Raj - first of all, my heart goes out to you. You and your family have been through a lot.
It would be very reasonable to exclude him from your home. If you and your partner (if you have one) want to still stay in contact with him, maybe you could meet him for dinner, in a public place or do other things together, away from his daughter, with the condition that he is in treatment. If he is not in treatment, then no contact would be reasonable too.
Keeping him out of your home and away from your daughter will keep him from traumatizing your daughter and you. If you meet him in a public place, it will give you a safer place to still have a means of contact with him. If you require he is in treatment in order to meet with him in public, it will be the best way to push him to get help. Keep good boundaries to keep your other kids AND YOU as safe as possible.
Do you have any support for you and your other family members in sorting this out? Do your other kids have any support? (Other than the ODD son) No matter what you decide to do, this is really hard stuff to be going through. I commend you on your courage to deal with it instead of turn a blind eye like some families do.