Boundaries are not about changing the other person. They are about changing our behavior to manage what we let into our lives and not. It's not about consequences for another person. Consequences may happen as a result of boundaries, but it is rare that "consequences" should be the primary goal. The other person may change, but trying to change or penalize the other person actually should not be primary goal, or else both parties often end up frustrated and resentful.
The goal should be change in what is in and not in your life, and to send a clear message to the other person about what you will and will not let in your life. This lets them be an adult and decide if they can or are willing to do what it takes to be in your life.
In your example, if behavior you don't want to let into your life is yelling (which is reasonable), then think about what you can do when he yells to change what you can do to reduce or eliminate the amount that you allow this behavior in your life. This does not mean his behavior is your fault. This is simply focusing on changing the only person you have any control over: you.
Communicate to him the boundary, and then communicate to him what you will do if the behavior continues, if the boundary is cross. Then if he does the behavior, stick to it. He will either change or not, but that's out of your control. Take back what is under your control.
An example of how to apply this to yelling or hanging up on the phone: communicate to him in a relationship affirming way, that you want to be in a relationship with him (if you do). Avoid using the word "but" as it invites argument. Tell him something like, "I want to be in the relationship and talk with you, and I need to be able to talk with you without yelling or being hung up on." Explain that if he continues to hang up on you, then you will only speak to him at length in person where he can not just impulsively hang up on you, or etc. If he yells, you will ask him to lower his voice once and if he doesn't, you will walk away or end the conversation until he is calmer. If the pattern continues without change, then you will consider if you can stay in the relationship. Perhaps there will be less contact or you will need to know he is getting help for the behavior and making improvements... I dunno what your boundaries and limits are, and what is and isn't ok with you - these are just examples of what someone might do in this example. Pick what is right for you and what you need to stay in this, if you are able.
In this way, he will face a "consequence" for his behavior that is about you doing what you need to take care of you, and doing it by focusing on you will reduce how easy it is for him to try to escape blame for his behavior. The more you take responsibility for you, the more he will hopefully be more responsible for himself.