I have definitely created a two-tiered system with my shoes. First class is for walking, and second class is for showing. Actually, that should probably be the other way around, as I wouldn't wear most of my walking shoes in any but the most casual situations. What this mean is that 80% of my collection is worn very little, and I should probably be more ruthless about getting rid of pairs I haven't worn for a couple of years.
Most of the time, but not all, the trouble is caused by a combination of leather upper with a man made lining. That combination does not seem to age well when exposed to oxygen. My all leather shoes do not seem to have this problem, and my plastic shoes seem to have it less. Sandals seem to suffer worse than fully enclosed shoes. Maybe this is a false perception on my part, as once this problem starts happening on a sandal, you might as well throw that pair away, because the flaking is usually quite visible as they're being worn, whereas with a fully enclosed shoe, such as a pump, you can't really tell until the shoes are removed. The latest example of this was when I wore my red Steve Madden open toed mules to church a few weeks ago, and by the time I came home, I noticed I had red flakes all over my toes. Not a good look, and there is nothing to be done about it. They're just gone.
I used to have a rule that if I couldn't walk at least two miles in reasonable comfort, I didn't keep the shoes. Nowadays, I do not maintain any such hard and fast rule. I think that I probably could walk two miles in a lot of my "dress" shoes, but I don't want to for a number of reasons.