I thought that I would throw in a dissenting opinion. While I agree that the speakers have a greater effect on the total sound, I would say that the distortions of loudspeakers are, basically, consonant with the sound of the music itself. On the other hand, the sound that electronics produce are electronic sounding distortions, which are much more irritable, even in small amounts, than the distortions of loudspeakers. So, cast my vote for the amp being much, much more important.
Oh, and without the amp, no sound comes out of the speakers; the amp is the chicken.