I'm in this position right now, except my degree and training is all in mental health. I took a job at a residential treatment center as a behavior support/direct care, and lasted about 2 weeks. Could NOT do it. Turns out between my PTSD and my Autism, I can't deal with people at the therapeutic level for any real length of time. Maybe an hour or two a day, 2-3 days a week, but not full-time. Right now I'm working at a residential facility for IDD individuals, but I work in the barn, taking care of the therapeutic riding horses. MUCH easier on me mentally, enjoying the physical work, but the pay is total crap. Can't live on this. Have to find other work, which is damn near impossible for me.
So no, you aren't alone in this. Even having TONS in common with clients doesn't mean you can care for them as full-time work. Lots of us can't handle the stress of their issues on top of our own.