For me, I think I would have two issues with this.
The first is that, if I knew my siblings were seeing my therapist in my therapy room, it would feel like it's not really "my space". That would be regardless of whether I had started seeing the therapist first or whether I was the late comer. My partner used to see her therapist in the same building as mine. And a therapist friend of mine also worked in the building - for several months, her room was next to the room I had my sessions in and we would often bump into each other on the stairs before or after my session. I actually found both those things really challenging. Even though my partner saw a different therapist to me in a different room on a different floor on a different day, it somehow felt that somehow it wasn't "my space" anymore. I actually asked her in the end that if she and her therapist ever had to swap rooms temporarily and she ended up in my room, could she please not tell me as that would freak me out!
Second thing - I would see this as a conflict of interest from the therapist as you are not doing family therapy together. A therapist needs to be on a client's side, must remain objective, should not have their own agenda and has to stay out of judgement. I think that's an incredibly difficult thing to manage and keep "clean" when you're dealing with related people and hearing all sides of things and different perspectives on the same situations/people.
I'm not a therapist but I work on personal development stuff with clients and I wouldn't do individual work with individuals who are related, partners or possibly even who are very good friends. One client coming up in discussion during a session with another client just doesn't work. And you can't really listen to something objectively and with total, clean, open curiosity if you've already heard about it from someone else. It's just potentially really messy and very it's hard for the practitioner to totally be in best service to each client all the time if clients are in close relationships with each other.
So, for me, it would be an ethical no-no.
I do think it's worth a conversation with your therapist. He has said he won't see them you are uncomfortable with it so I would encourage you to be honest with him and explain that you're not comfortable with it. And to then also let him know that you feel some guilt about that and let him work through that with you.