Starting a high-end audio shop... Am I crazy?

Your desire to 'throw money out the window' is equal to your desire to burn piles of cash on a useless fantasy.
Sorry, but nearly ALL the stores I know of that are 'bricks and mortar' are either going bankrupt, or are nearly doing so.
Only the really big private stores have a chance. you got $25,000,000.00 to invest on a 'midlife crisis'?
If you started a web business, you might have a starter business.
Locally in SE Wisconsin:
Flanners: just moved to a BIG new location, does TV and install big time. (they are the $25,000,000. store) great place!
Sound Investments: How they stay in business is a mystery?
Audio Emporium: nice place (no comment here, just too close to home.. I like them)
Sound Stage: turned into a small, barely surviving 'last stand'.
So!!! you too can go grey worrying about loosing everything you own.
Why not just blow $80,000. on a Corvette?
Or find a hot babe and run away to Mexico?

One thing I haven't seen mentioned is have you made any contacts with manufacturers yet? If there are exisiting dealers in your area for desirable lines you will likely not get those lines. You may be stuck with less desirable lines that are harder to move. Opening a brick and mortar store has a lot of up front costs that you stand a good chance of losing if you don't make it. Additionally, the business will become a time hole, you have to be there when the store is open then you have to take care of all the other things like installations, stroking the good customers, accounting, local politics, etc. I don't recall the number, but there is a very high percentage of small businesses that fail in the first year, and many of those are selling much more popular items. High end audio is such a small sliver of the overall electronics market that it will be an uphill battle. I would definitely go with the recommendation of sticking with your current job and doing this on the side. I think many of the small time folks run the business on the side and use it to feed their hobby. I think the bottom line is, how much can you stand to lose? That will determine how deep you should get into this!

I agree with Rec and Gregm. The overall customer base is relatively small, there is very little support infrastructure, and something not mentioned by either - a fair number of us (part of your customer base) are natural-born cheapskates. I, too, would highly recommend starting your efforts as a "paying hobby." As Gregm noted, the "value proposition" of high-end audio is "weak." The most successful dealers I have run into have something which makes them "sticky" as counterparts. They cater to their customers like personal servants, they specialize in one particular kind of item (only independent manufacturers, only analog, etc), or one particular kind of sales (low margin high turnover, low turnover but one-of-a-kind (or very rare) items), or they have a personality which drives them to be mavens. In each case, the "value" is in the person offering the goods, not the goods themselves.

As a personal aside... Every now and then I try to think of myself trying to make a business out of my hobbies. I have been able to imagine success in a few different hobbies. Audio ain't one. There is A LOT of competition out there.

That said, dreams are there for a reason... best of luck!

As someone that started a business when I was 36, although not a high end audio, this thread struck a cord. You have to have a few things to be able to make any startup work. First, you have to either be single or have a partner that completely believes in you. This is more important than anything else in my opinion. The amount of time you will spend on this will be more than your current job. Second, you need to have almost zero debt outside of your mortgage. If you have two car payments, credit cards, student loans and various other debts, you have almost no chance of making this work. A change of lifestyle will be in order. I went from going our to dinner three times a week to about three times a year. Third, you must like and be willing to work 14 hours a day, six or seven days a week for years and not become exausted. The amount of work to get things going is at times overwhelming so be prepared. And then fourth, you must have a passion for the work you are doing. There will be days that you just can't believe you left a great job for what you are doing, and there will be days you can't believe anyone would pay you for what you are doing. Good luck with your endevor.

Keep day job, start business on Internet first. Learn tricks of trade, establish contacts with manufacturers, test the Internet with good marketing to see what will be the reaction to your new business.
If the reaction is good, quit your job. After 2 years establishing income on the internet, take another look into opening a retail location. Better still, build a dedicated room and work from home to keep your overhead lower.