For me, I wouldn't have been able to accurately start piecing together my childhood without a T that I trusted. Memory is such a difficult and unknown area.
It could be that there's more to this thing with your grandad than you currently recall.
It could be that there's something else, with a completely different person, but your brain considers this memory with grandad to be safer to let you access than the other stuff.
It could be that the emotions attached to this memory are the only reason that this memory keeps coming back to you, and figuring out what those emotions are linked to, like completely different situations that your brain currently won't let you access where those same emotions came up, are why this memory has significance.
At the moment, unfortunately, allowing yourself to sit with "I just don't know yet", is safer and more helpful than pulling and pulling.
Or it could be that what you've always brushed off about your psychological abuse as a child is a much much bigger issue than you've allowed yourself to consider before.
All of those (and plenty more) are real possibilities. But trying to pull at memories can very easily make our brain "fill in the gaps" with stuff that we've heard about or read about, just to solve the questions, and not necessarily accurately. And our brain creating inaccurate memories only creates more problems, without getting to the real issues.
Working through these issues with a T experienced in trauma gives you a safe process to answer these questions in a way that is much more likely to give you accurate answers that you can recover from, at a pace, and with the skills, that you're going to need to cope with the information as it comes.
It actually sounds like what you already know about your childhood is more than enough to explain why you've had the struggles that you've had. The abuse you know you went through, is a much bigger deal than has been safe to acknowledge. Until we're separated from that situation, the human brain considers it unsafe to acknowledge "my primary caregivers were abusive". Until now, your brain probably hasn't allowed that to be an option.
But for some reason (and there are infinite possible reasons), you brain is throwing this memory at you with uncomfortable emotions that it sounds like you need to process. It also sounds like you could do with some support to acknowledge the abuse that you do remember from your childhood, and finally start feeling the shame, anger, grief, and all the other complex emotions that you haven't been allowed to feel and process before.
Stick around. This place is really supportive, and you'll find lots of great advice from people who genuinely relate.