Blaming the other person when they whisper, or forget that they didn't tell you something is GASLIGHTING in Narcisstic terms
Or it can be a human failing - I can have the "did you not listen to me" conversation 10 times a day with my husband - and then remember that I actually didn't tell him, or he didn't hear me talking to him or didn't pick up what I said or the importance of it.
"You are not listening" is just a phrase, a set of words in a particular order, the thing that makes it a trigger is the meaning you attach to those words based on your previous experience. That's the process
@joeylittle describes in explaining what that phrase brings up for her. The only person who can challenge or change that bit is you - your trigger, your interpretation, your reaction, yours to manage.
My sense of you is that you'd like the world to work around you, to avoid doing things you find difficult, to know what your triggers are and not press your buttons in any way and if - inevitably - something does happen that you consider a trigger, it's the other persons fault for triggering you. I know you'll say that's part of your APD diagnosis, and that may be the case, but while your diagnosis might make it harder to challenge your thinking, you still need to do it or you'll forever be in this loop.
One of the most valuable things for me about this site is the explicit understanding that we are all responsible for our own triggers and how we react to them. It's helped me take that understanding into my daily life and stop hoping that people would just not trigger stuff in me and has helped me find ways to cope with shame, blame, anger, fear, vulnerability and sense of exposure that arise when I feel triggered.