Well, let's just say I waited a long time - too long - to buy my first pair of heels. I was well into middle age before I dared to give it a try - even just wearing them at home. I felt guilty about wanting to try wearing high heels; I mean, what if I liked it?
You mention job and social life as the things that are holding you back. But you don't need to wear them out and about if it makes you feel uncomfortable. Sure, quite a few of the guys on here do - and cut quite a dash. But it doesn't mean you have to. You can buy yourself a pair of heels tomorrow and wear them to your heart's content at home or in private or in whatever way or place works best for you. It's not a contest to see who can push the envelope the farthest.
As for me I own a couple pairs of stiletto boots, but I don't wear them out and about, but instead at home while I am working. I do this in part because I am not well practiced in wearing them, in part because of where and how I live they would only get ruined, and partly because my natural style runs to knee and OTK boots with chunky heels, rather than stilettos. I bought some stilettos because they are the Ultima Thule of heels and (for guys) taboo breaking and I like wearing them, but when I am getting dressed the chunky heeled boots call out to me more often (which is why I am not well practiced in stilettos) I sm comfortable wearing feminine knee of OTK boots (with and without heels) out and about, but that's me.
The point is it is not a competition. Wearing heels it doesn't need to be about towering stilettos or cutting a dash in public. It's about freedom of choice and your own personal style. The world doesn't make it easy for a guy to wear heels, but you don't have to wear them in public if you don't want to. Wear them at home. It's better to do that than never to try at all.