It doesn't sound selfish to me.
I am just guessing but maybe his desire to go on this trip with just his friends is a way for all of them to escape responsibility temporarily and feel some carefree freedom. That doesn't mean he doesn't love you or the kid(s) or that he wants to be away from you.
I have twice spent weekends away from my husband on girls only shopping trips to other cities. Now granted it was only a 3 day trip not a full holiday. I did not come up with the idea. I didn't stop and think "hey I'd like to go on a trip and leave him behind." A friend suggested it to a few of us, several people were interested and I realised that it sounded like fun and that I would quiet enjoy some girls only time. Its entirely possible that something like this is occurring for him only with men and motorcycles.
I know several people who ride motorcycles, some more safe than others, but they all list the feeling of freedom as being the main reason they love it. I don't know you or him enough to really comment on this, but from the people I know I would say its not an easy hobby to just give up. And you might want to have this as a separate discussion.
I think the biggest issue here is really that you feel that you have a need for his time, his attention, his companionship that isn't being met. You want him to spend more time with you, and you would like to go on holiday with him. So maybe approach him calmly with this in mind. You could share with him how his motorcycle trip is making you feel upset, neglected, whatever word describes how you feel.
I definitely think you should talk to him about it though. Maybe just have one conversation about your needs and how you feel (as in "I am feeling xyz" not in a "you aren't meeting xyz").
And than a separate conversation about his hobby and your concern for his safety (which maybe results in getting him to be more protective or whatever), probably best to have this discussion before he goes on his boys trip.
That's just my 2c and how I would try to handle this. But I really don't think you are being selfish.