I grew up with zero, and I mean ZERO physical contact and affection and security etc. Lots of violence, abandonment etc etc. Many people go through this, and don't develop this weird reaction to physical contact that I have, but somehow this is what happened for me.
I have a total intolerance of physical contact, at the same time I have an unbearable need for physical contact. At the same time I have huge attachment problems - the same push pull dynamic.
As I see it, which is probably an obvious viewpoint, is that you learned as a child, again and again, that physical contact is dangerous and will inflict pain. We do not want pain, so, especially as children, our brains quickly learn what to do to avoid it being inflicted upon us. A child touches the hot stove - it hurts - a reflex makes the child pull away - the child does not touch the stove while it is hot again, it checks before touch or just avoids it all together. You also learned that attachment is dangerous and bad, and that being touched by those you're attched to (yes?) is dangerous - brain does not want, brain wants to avoid.
At the same time, you are human, you do need both of these. You long for it, but as you say, freak out when attachment happens/seems to be happening. You never had the attachment you were supposed to, and you long for the feelings of bonding in itself and touch, but then you've been taught to avoid both. You never had the security, attachment hormones, love and stress release of a good, long hug, touch was never what it should've been to you.
I long for being held and feeling secure in that. I did not have physical abuse as a child, "only" the six months of being raped last year. When someone's hand appears to be going towards me I have to keep myself from running or punching them, even when it's the girl in class sitting next to me and just reaching for something. Your fear of touch probably goes a lot deeper, but I do relate. And I'm very, very glad I could provide you with some relief - there is (supposed to be) a simple connection between attachment and touch, and I believe there's connection in you fearing them. :hug: