STRATFIELD House, by Wellington Cottage Hospital, is to close at the end of January.
Four permanent and two part-time staff are affected along with 20 clients, some of whom visit the centre five days a week.
The centre provides specialist day care for the elderly with mental health problems and is a secure unit.
It is run by the NHS, which pays the staff’s wages, and Adult Social Care.
A ‘drop off information’ meeting was held at Wellington Library on Monday where clients were told their needs would be assessed.
One person who attended the meeting said: “We were not told anything, really. Staff were told they would be offered other jobs but this could be anything – cleaning, cooking etc.
“People depend on the centre – and it gives carers a break and some freedom, which is very much valued."
Two similar units in Taunton are also set to close.