I feel for ya... it is a tough case.
I'm on the other side of that, having kids with ex's... and that was good, but hard also. My last one got to the point were she literally made life so hard for me, I just couldn't keep doing it as she made me physical sick by knowing what to do and say for my PTSD to worsen. Then she blamed me for walking away, saying she knew it was just a matter of time, but she was the one who did it, not the kids, not me. Me and kids were happy together x times a year. She made my life absolutely shit, so disgruntled and spiteful she is... and I just hope you don't punish him like mine did me, until he leaves totally from the kids lives.
My ex made life easier for herself with her actions... and the kids and me suffer as a result. My kids will grow-up thinking daddy doesn't love them and want to be with them, whilst every year it tears me apart a little more.
Separation is tough...