I have a great relationship with my psychologist but like you I still have times where I think that he can't stand me , that he is bored with me and even the smallest change in the tone of his voice will set me off ,I am a shy person and find it hard to open up but over a period of time I have learnt to discuss things with him and tell him how I am feeling.
I think that therapy is about all kinds relationships(including the one you have with yourself ) both in the therapy room and outside and if you have difficulty with relationships(which I do as well) then you start finding faults in people so you can push them away and maybe you are doing that with your T.
I agree with
@Rumors he may have got little details wrong but on the whole he has an idea about the whole story and seems to want to help you and also we all need to remember that therapists have to see quite a few clients and sometimes they gets little things wrong but they are only human.
You have two choices :You stay with this T and talk things through with him (you could send him a email first if that helps) and discuss your concerns/issues, how it makes you feel when he forgets things and I know it is hard but you also need to talk about your therapeutic relationship OR you look for a new therapist who you think you maybe more suited to you but be aware that you may find fault in them as well.
I know it is really hard but sometimes we have to take a leap of faith and start to learn to trust the other human being who sits in the same room as us and wants to help us.
Wishing you good luck in whatever choice you decide to go for and I really hope things work out for you.