I can't say that it's all about heel height for me, but it's a major factor. Every once in a while, I see something that catches my eye and has a low heel, or even no heel. I have learned to resist the urge to buy, unless there's a very specific reason to do so. The reason is not that I've got anything against the lower heels, but I find that they just don't get worn.
A prime example of this is my so-called "farmer's market" sandals. I have several pair of low-heeled wedges, and several pair of effectively 3 inch chunky heeled sandals, which I used to wear to work my wife's vendor stand at farmer's markets. All of them are very attractive sandals, and I have gotten numerous compliments on them, but I find that I never wear them these days because I don't have to. The reason I don't have to is because I can wear 4 inch heels just fine without really thinking about it. Three years ago, possibly even two years ago, I would worry, "I wonder if these shoes are a mistake?" And yes, sometimes at the end of the night, my feet would really be feeling it. That just doesn't happen now.
Ironically, three out of five of my last shoe purchases were sub-4 inch.