From just a distortion perspective, speakers do several orders of magnitude more damage to the signal that decent electronics. Factor in room interaction and it's a no- brainer where the emphasis needs to be.
I quite agree - it is a no brainer really. I would certainly NOT recommend to spend as much money on Interconnects/Cables as on the speakers/room acoustic treatments.
Most speakers have $100 wholesale mass produced drivers - you are paying almost entirely for veneer and cabinet work in most cases. Drivers are what actually makes the sound in many $50K+ systems...yes really... several $100 transducers powered by $1000's of wires and 10's of thousands of dollars of source and amplification.
Imagine if just a little more money went into the quality of the speaker drivers and into room design/acoustics instead...a scary thought.