Hello Casey,
I wouldn't beat yourself up for wanting to do what's "right" in allowing your son to have a relationship with his father, however - he doesn't sound like he's going to be a healthy contribution to either one of you. If anything, the situation will escalate with manipulative...