What to upgrade first for better sound quality?

Yes I have the MAC MA6600 retail $6,500. My first inclination was to get an external phono amp that could run both MM and MC cartridges. This way I could use my current cartridge later upgrading to a MC. I was leaning towards either the Herron VTPH-2, Chinook or an AR. Biggest issue here was matching up with input impedance of only 20k. Herron looked like it might be the way to go. I do think about going tube at some point and don't believe I really 200w per channel. Listening room is say 12x18. This hobby can be daunting at times but I love it.

You can always get more hung up with things like a cartridge, then oops the tone arm is not the best for this cart. and the whole set up doesn't work optimally with this table etc. etc.. It is kind of humorous though that an $80 mat made such a big impression.
You really haven't said what the sound was missing or described this poor quality sound. It might be useful to know what your hoping to hear that your current set up is failing to provide.
If the McIntosh amp is really mid fi as described by others then you know what you need. My own philosophy is that the speakers are the most important elements of a sound system. I see how they sell very expensive cartridges now.

I'd also strongly suggest you do a LOMC cartridge. I'm a big Dynavector fan, but there are several in the $2K range that are very good. Do some auditions. Consider a Sutherland 20/20 for a head unit - very nicely engineered and very reasonable second-hand.

Good luck & happy listening!

Very good advice so far!
How many watts you need depends on the sensitivity of your speakers. Why most tube users like highly sensitive speakers. But I feel, just me personally, that the amp controls the tone of your system, I started into this hobby years ago with an Adcom amp, 200wpc, very nice for the price! And I went through 3 pairs of speakers and the overall sound of the system really didn't change. Sure one speaker may have been slightly louder, or had cleaner treble, but the big bass sound of that amp was a constant. Switching to the Forte' amp I've got now changed the entire character of the stereo.
And of course the preamp feeds the amp, also very important to the basic sound of your system. This is critical, mess up here and nothing before it,
or after it, can make a difference.

But I am speaking in generalities, I've read articles that advise one buy a stereo from the speakers back, buying your source gear last. This may be good advice? And I agree with the others that your cartridge and phono pre are not up to the rest of your gear. A really good phono pre, which doesn't have to cost a lot, can really wake up that analog front end! It was a major step up for me, when I went to a separate phono pre it really woke the analog up and it sailed past my digital front end in one leap! Personally I went from the one built into my Parasound Preamp to a $1000 Phonomena, which isn't even a super choice but it still wowed me big time! More air, more soundstage! More bass!

This revelation prompted me to jump up a few notches to a much better cartridge. I was worried, will I hear a real difference between a $500 and a $1500 cartridge? I mean, enough to justify the cost? And wow did it ever! Jumping from a Sumiko Special III, to a Lyra Delos was some of the best money I ever spent in my system! On my vpi Classic. I'm not that familiar with your particular gear, but this was my experience with my system.

And look at it this way, if you do buy a cartridge and preamp, it will make a major upgrade to your sound that you will hear and appreciate! Even if it's not the tweak you are looking for, it will still be money well spent. And also leave a few hundred for a new phono cable! A good phono cable doesn't have to break the bank, great cables can be had for reasonable prices.

And I will say that you do have some nice gear, I'm sure it will be easy to coax more out of it.

09-08-15: Elunkenheimer
Did a little web surfing, but do you have a $6000 Mac MA6600 with a $3800 VPI Prime table all driven by a $700 Ortofon cartridge?

So if you vary the funds spent on cartridge you can vary the sound quality respectively by increasing or decreasing?