I hear ya ..... at least in theory.
It's been a 40 year voyage into audio for me, and your right, when I stepped up to a high-end modified deck, and got some sot after NOS tubes for it, the differences in the entire system were profound. Transparency, timbre, deeper back stage, the music was cleaner. This was to a greater extent experienced when multiple instruments were playing simultaneous. The more complex the music, the noticeable differences were again heightened.
At the time I was using an M400 Carver cube amplifier, and it was fine, but I knew there were more esoteric amplifiers to be found. (I had owned a Classe' Audio DR-25 before the Carver). After living with the Carver for 20 years I went out and bought a pair of Classe' Audio DR-9's. Well blow me over with a feather, the sound stage was massive. Subsequently I sold off one of the DR-9's ... they sound better in stereo than in mono.
I toyed with various speakers, using a pair of Roger Sound Lab, Elan's (modified), as a reference. They're a passive radiator design with a beautiful silk dome tweeter. For the money (including mods), they were a great bang for the buck (& could be made to sound incrementally better), but I've been at this game awhile and know what a great speaker can sound like. So better speaker were the next step.
I had some Paradigm's, got a hold of a pair of Meadowlark Huron Hot Rods, and finally settled on a pair of 946 Focal Electra's. New, these speakers retailed for approx $8,500 dollars. Lets be clear, the Focals are worlds apart from the RSL Elan's, but are the differences as profound as it was with the Cd upgrade, or the amp upgrade? In a word, no. Taking into consideration that the price difference between the Elan's and the 946's is about $6000-$6500 dollars, or 71%.
Now the difference in the two decks was $800, to $2100 (including the mods and the tubes), or 62%.
Comparing the data .... I'd say your Front End theory is pretty safe.
Obviously there's more to a system than these simple examples, but IMHO the concept of building a system that you like to live with (or aspire to move up to), should be built around your source. That's the place to begin. Why?
People move, over the years the size and shape of your sound rooms will change. Speakers should suit your room, so when you change your room, chances are you'll need to change your speakers. Your source remains the same.