It sounds like maybe he does need to learn to compromise some. If he doesn't want to go out, he can at least be present when you visit. You are making all the concessions now, he can make one or two.
Like maybe tell his mother and aunt he is having a date tonight and see if they need anything BEFORE you get there. They would probably like him to have a nice evening. Perhaps the other upset elderly lady can make it for an hour or two on her own if she is in a nursing home.
Also, if you guys can't go out on dates, you can plan special "at home" dates... like a dvd film festival, or inside picnic. Or find an inside activity you both like... puzzles, board games, cards.