I've never been raped or sexually assaulted, and I'd say I've fantasized about "dirty stuff".
[I apologise if anything I say offends anybody - that's not my intent, I'm trying to answer as honestly as I can. If you have problems with this stuff, or could be triggered by discussion of it, you may want to consider skipping this post.]
Prefacing by saying that fantasy and reality are not the same thing...
I sometimes like things rough (occasionally very rough - this is not the same as being forced, or being raped). That said, on occasion I've had fantasies about being forced. I like loss of control (to an extent) both in fantasy and in actuality. But I believe there's a big BIG BIG difference between having a fantasy and actually wanting to act on it, or wanting something to happen in real life. Just as I believe that (so long as it's not harming you in any way, physically or mentally) there's a big BIG BIG difference between a fantasy being acted upon between loving partners playing trust games, even ones involving a limited amount of pain, (where things stop if either one is uncomfortable with it, without question or rebuke), and anything that's real. I also believe it can be healthy to have fantasies, even very dark ones, so long as you aren't hurt or disturbed by them.
I think, though, that if you have been assaulted or raped, it's probably all a lot more complicated than 'just' a fantasy. Then I'd guess (as a layman) that there's probably a lot more to it that may need to be explored, or at least considered. It may be a fantasy or preference (I'd imagine that's perfectly possible), but it may be an attempt at reliving something that's happened to you. That's something for a therapist to deal with, because it (again, I'd imagine) would be different for everyone, and t probably runs very deep indeed (I'd also imagine a lot depends on your background, and how you were brought up to view sex itself).
I'd suggest that if you feel disturbed by anything you fantasize about - or if you can't control that fantasy - that may be a pretty good warning sign that it's more than a 'normal' fantasy.
I think (from the perspective of never having been assaulted sexually, only physically), that what you say is very true - that "normal" women have dark fantasies too, but don't find them disturbing, because the trauma isn't there in the first place.