People who do this are usually not nearly as talented or capable as they say they are. The only exception to that in the medical field IMO might be brain and spinal surgeons...they seem to be genuinely hyper-competitive and have an excess of confidence.This doctor has hyped himself up so much that he's convinced me that no one else I see will be as good as him. In fact, he's told me that exact thing multiple times.
So unless this doc is going to be cutting you open to do some extremely risky and complex surgery? I'd say you should feel fine about walking away.
And frankly, even if this is a high-risk surgical situation? It's a competitive field. Find someone you can feel comfortable with. Consider bringing your mom into your appointments. It sounds like you are comfortable talking with her, so don't hesitate to include her as an extra layer of protection.