I'm pretty good at remembering directions. Probably in part because I delivered pizzas for a few years back before GPS was everywhere.
People comment on it. Fairly routinely, if I'm driving someone pretty much anywhere, they ask how I remember where I'm going. I often take back routes, or change routes through the day based on peak traffic periods, all just off the top of my head.
I have a GPs in the glovebox, but I can't remember the last time I used it. If I'm driving somewhere new, I pull out the old street map, get a general sense of where I'm going, and just take it from there.
Being able to do that I think is like any other skill. You learn how to do it, then you practice. If you keep using the skill it gets better. If you never really learned/practiced? Then it makes sense to me it doesn't come naturally to you.
There was a time when people needed to know which way north was in the same way we now need to know whether there's phone reception. This is no longer the norm in the first world. If you have your phone, you don't need to know where you are, or where you're going. So, there's no longer a reason for people to develop the skill.
So, it may feel like it means there's something wrong with you. But from personal experience? Having a good sense of direction, and rarely getting lost (unless I'm seriously dissociated, in which case I usually stop the car altogether) is at least as peculiar, based on what people reflect back to me.