Why isn't hi-end audio an aspiration for most? It certainly has a 'bling factor,' as much as exotic cars and fancy wristwatches, can be shared with other non-enthusiasts as much as home theatre and offers the same kind of 'marginal' benefits, given the cost, as owning a Ferrari or Patek Phillipe compared to more pedestrian forms of the same product. Because the pursuit is not just about spending money on equipment, and involves tons of time, effort and socialization around minutae, just like any other serious pursuit. Think about how many people not just buy a fast car (Porsche, Ferrari, etc.), but then spend the time to work it into a track-day toy, fidgiting with suspension changes, braking, weight loss and aerodynamics. The popularity of that kind of single-minded activity probably brings the population of people willing to commit the time, effort and money down to a small fraction....
And, add to that, the fact that music listening is, for cultural reasons, not something most people do as an "activity." (Rarely do people sit and 'listen' to reproduced music- it is there as background for another activity).