NDKnight
Any witness to that sort of event is also a victim. One of the most difficult things many survivors have to overcome as adults is feelings they should have done more. Another big issue is guilty over having remembered "enjoying" it. The thing about people like your abuser is that they are extremely adept at exploiting vulnerabilities and using a system of rewards to continue to ensure loyalty and silence.
You were a child. You were not supposed to protect anyone, someone was supposed to protect YOU. Even if some of this continued into later years, parts of you were still a child.
The tactics used by serial pedophiles wouldn't necessarily work on an adult. Yet as adult survivors, often we attempt to "take responsibility" (which is a very adult thing to do), but when the abuse took place, there was simply no way to prevent it.
We would have stopped if we could have, if we had understood then what we know today, and that is the legacy. We are left to deal with all the scars, and often we choose unhealthy ways of doing this, often we experience revictimization by our own hand.
I would only add that your feelings as described above, are very normal. The fact that you even have them is healthy, even. I think you are on track and once you are able to grieve your own victim hood, squarely fix blame where it belongs, you can be free........