Are you still seeing this doctor? Are you in therapy at the moment?
If you're still seeing this doctor, I think it would be good to have a conversation about this. It's hard without knowing more of the situation, but I do know that sometimes, my therapist will be saying one thing, but I will hear something else. Not literally, but I might read more into her words than she is actually saying, or add meaning to things that she didn't put there. I interpret things the wrong way at times, but I'm aware that I do it, so I'm generally pretty careful at checking whether that's what I'm doing or not.
If you're not seeing that doctor any more, have the conversation with your current doctor. Unless you are in inpatient treatment at the moment, or highly supervised, I can't imagine they are considering you to be a danger to other people.
Did he say in what way he considered you to be dangerous?