I would also say the preamp. The preamp establishes the character of the whole system. Good speakers accurately produce what they are given. So if your electronics are have a weakness, then the speakers will reproduce the faults. Better speakers will only make that worse. I think we may overestimate the source. There are many very musically involving CD players, tuners and turntables that are not the latest technology.
I picked up a Cambridge Audio Dacmagic DAC recently to just to play around with because it was being given away ($100 with a $50 digital cable for free included). The Dacmagic is basically the guts of the CD6 single player with better power supply, etc. This thing is very musical. You just need to ignore in your head it is old Phillips bitstream technology. My 24bit player has better resolution, but it no more musical.