I am genderqueer. I'm female-bodied but I don't specifically always feel female; I sometimes feel male. I sometimes feel like I'm neither. I sometimes feel like I'm both. I mostly present as female and I'm fine with female pronouns for the most part, but I sometimes feel like I'm living a lie.
I had a partner who is transgendered. She's in her late 20s and has just started transition. I have a friend who is in her 50s who is transgendered - she came out as transgendered a few years ago in her late 40s and has completed transition. I know a few other people who are transgendered and did not come out as trans* until in their 50s and 60s. It's never too late to transition. I also have another friend who is FtM and he is short for a guy - about 5'2" at most. He has never experienced prejudice regarding his height.
I hear what you're saying about your height, though. Patriarchy and the 'traditional' accepted concept of what a man is supposed to be is ridiculous - no man should be judged on his friggin' height, just like no woman (or man or any other gender, for that matter) should be judged on their body size or shape. Unfortunately, society is extremely heterosexist and extremely cissexist; society exists locked in binaries and heteronormativity. It makes me sick how narrow-minded and uninviting society is.
Ultimately, you need to do what is right for you. Being transgender isn't a choice, as you know - you didn't choose to be trans* but you do have to live with it and you need to work out what is the best option for you in dealing with it. I know that being told "just be yourself" is very reductionist and simplistic because it's nowhere near as easy as "just be yourself" for trans* people. Trans* people are incredibly marginalised people who are at huge risk of receiving hateful, even sometimes violent behaviour from others. The suicide and attempted suicide rate among trans* people is over 40%. The LGBTQIA movement is mostly about gay and lesbian people - the T for transgendered in the acronym is mostly invisible and hardly ever talked about.
Do you have a therapist to talk to about gender dysphoria and about trans* issues? Do you have a gender counsellor or psychiatrist to talk to? I feel that is something that would benefit you, to be able to talk openly about what you're going through, to help you work out what the best course of action to take is. It sounds to me like you need a counsellor or a psychologist/psychiatrist specifically qualified in gender dysphoria and transgender issues because you need to talk to someone who actually understands what your situation is about without demoralising you with cissexist platitudes and advice that erases your problems and issues.