today I was out walking and ran into him doing the same, and he was in a chatty mood and stopped me to talk, wanting to know everything from my opinions on religion to my favourite leisure activities, even inviting me out (I got out of that one). He's outwardly so charming, it's hard to disengage without feeling like I am being rude. This went on until I made up some excuse about something I had to do at home. But why do I even care about being rude to someone like that? I want to say "I know who you really are, and I want nothing to do with you, good riddance!"
I hate being polite but I hate being the bad guy even worse or having people think that I'm some horrible person or something.
It seems like this is a black and white way to view things. Your conclusion is that in order to be polite and nice, you must listen to him ramble on and on and talk to him much longer than you would like. If you don't do that, then you are rude, and thus your other option is to tell him good riddance!
I think that there is another way to look at this. Hmmm, how do I say this...?
Boundaries can be the most polite and nice and respectful thing to do for others, and ourselves. Listening to someone talk on and on can be more about our desire to avoid the very real discomfort (and even fear) of gently setting a very real boundary, than it really is about being polite to the other person.
If I was talking on and on to someone who didn't want to talk to me, or even have a connection with me, I would feel like it was actually rude and disrespectful to me for them to just keep talking with me.
Narcissists tend to be drawn towards people with crap boundaries. I have an uncle who is a very classic and strong narassict. When my boundaries are weak, he is drawn to me like a fly is to honey. Oh dear, it is so hard to handle. Eventually I toughen up and gently draw the line, both in words and inside myself. He tends to loose all interest in talking to me when I am respectfully clear that I don't want to talk to him.
For you, this might be a chance to learn stronger boundaries too - which is actually the most respectful way to treat another person. The most polite way to be. For both of you.
The next time you encounter him, and you feel scared to say "no, I can't talk now" - remember the politeness sometimes means saying "No." Or "I can't talk now" or "I need some space" or etc. It can be more polite that continuing to engage him and let him think that you do want to talk with him. It is so easy to say, and hard to do. I'm sorry that you have to live right next door to someone like him!