I have similar, I had to put my foot down about something the other day, very new thing for me to do, and despite the people involved not being around in that moment, I did get scared that one would send the other to confront me, with the possibility of it becoming violent, even sexually violent.
Reasonable? No. Natural trauma response? Yeah.
In my experience you just have to practice when opportunities arise, and keep grounding techniques on you… and if at all possible a close friend who can give some encouragement, and more grounded perspective (that’s what helped me bail out of a situation where I felt very uncomfortable, which I almost gave up trying to get out of because of insisting.)
Most people ever will not become aggressive or violent for not getting their way. Vast majority of people can handle their emotions and not be cruel. The urge to be cruel doesn’t actually come to every angry person. Some people? May say something unkind, or be weird about it, but it’s more so down in petty territory, than threatening or flaming. I’ve had someone be irritated at me, but didn’t insult me or anything like that.
Violence and berating happen so much in private relationships such as abusive families and partners, especially for children, because as a child you have no power, really, and what’s happening won’t be seen or heard about by anyone. As an adult, people for the most part want to resolve xyz without putting the connections they have at risk, or their image. In my experience it’s a lot more civil. Even extended family VS direct is different in how free people feel to be unreasonably frusrated and direct it at you.
Also, work on any cognitive distortions you very likely have from growing up with your family will help. Do you view saying “no” as inherently mean? What needs do you automatically put below others? (I’m fine with sacrificing a measure of energy and time for someone, but am learning to take my stress cup more seriously).
Takes practice, but you won’t be attacked by random people for not doing whatever they want. The more you push through the fear, the more experience you get under your belt, which will prepare you for if/when you have to work a bit harder to put your foot down and be heard. People in normal life don’t use violence or aggression unless met with it, it’s most often trying to convince you away from your decision. Which is unpleasant, but not an attack.