What
@Sweetpea76 said.
Really. Hard lesson learned...the courts really don't give a damn, and ANYTHING you do can be looked badly upon, (or looked well upon, but family courts are jaded, and just assume everyone sucks, and no matter how much your lawyer tries to paint you in a good light, their lawyer will paint it in a bad light).
I used to drive myself CRAZY with this stuff. 5 years later?
Denying visitation looks bad.
Giving visitation looks bad.
Really. They BOTH look bad. One makes you look controlling and abusive, the other makes you look neglectful and a liar. And that's true for absolutely everything.
So screw what something looks like. And go for precedent. AKA act as if you've already got the ruling granting exactly what you want to happen. Because anything you allow? The courts will tend to allow MORE of. And anything you don't allow? The courts will either tell you to stop that, or go ahead and keep the status quo. If you talk with him, his family, meet up with him, allow him over? The courts will see that as you're "willing" to do all those things, have been doing them by agreement, and there should be a lot more of it! 9 times out of 10. So just do not do ANYTHING you do not want to be doing. Do not set precedent. If he wants XYZ, make him work for it. Don't just give it to him. It won't look good. It won't help you keep him out of your child's life later. It will just drive you crazy now, and crazier later.
So since everything looks bad? Do what actually IS good.
They might take that away from you later, so revel in it now. For real. This is your chance to live how you want to live. WITHOUT this asshole having any rights to you, or your child.