How can anyone afford this ?

I consider myself a dedicated audiophile. I am 36(which I am guessing is a little younger than the average here) and single. I have been interested in high-end audio since I saw my uncle's Mcintosh and Threshold equipment for the first time when I was 5.
Since joining the workforce and saving a little I have always been trying to put together a nice system on a budget. I do OK financially(I am a systems engineer) but I do live in NYC which may put some of this into perspective.
Over the last 6 months I have struggled to buy(all used on Agon) a pair of Dynaudio Audience 42s and a Threshold CAS2 amp, Chang CLS3200, and cables(I haven't gone out[I don't have a girfriend], purchased anything else and really haven't eaten too much to be able to afford these and it is still a real stretch). I am using the amp with a direct connection from my CD/DVD player(Cambridge Audo Azur 540D...slightly modified[op amps, PS caps, bypass caps] that I have had for almost 10 years). A fellow has a Threshold FET2 series II(to match with the CAS2) he is holding for me but that seems like a pipe dream at this point along with a turntable.
A few years ago the analogue bug hit me.
I had a setup consisting of an Audio Analogue Settanta integrated and a Nottingham Horizon SE turntable with a Rega RB300 tonearm with the Incognto rewire and structural mod. This was not an expensive kit by any stretch but for me it almost put me in the poor house. I had to sell the entire rig to pay my bills and it hurt.
It seems over the last 10 years or so I have not been able to keep a kit for more than 6 months before I had to sell it. Whenever I don't have a rig I am constantly scanning the online Ads lusting for the next bargain to set up a system and cannot even listen to music on a mass market rig(I have been spoiled).
Anyway, I guess my question is how can anyone normal afford this hobby? What type of money do you have to be making to be able to enjoy this hobby.....$100,000/year? $500,000/year? Do you need to be worth millions? $5,000 barely gets you in the door(some interconnects cost more) and you could possibly spend millions. I am not looking to put together a $10,000 system(not even close...and that is modest in this hobby) but if I wanted to I don't see it ever being financially possible. If I had a girlfriend or a family(hopefully someday) I would not event be able to think about this hobby with a good conscience. I guess I am wondering if all these people in this hobby are millionaires? I am close to selling my rig again to pay the bills(the amp needed repair/recap and that was $450). Any advice for an audiophile who lusts to put together a nice rig but can't afford it? Should I get out and save for 5 or 10 years and then try again? Maybe I am in the wrong hobby but it is more addicting that crack to me(and more expensive). Maybe I should be a crackhead instead...that might be the only thing to make me forget about it. Thoughts?