How much difference in sound?

Coffeey,

What's lost upstream (at the source) cannot be recovered downstream (amp/speaker end). That said, upgrade should occur at the weakest link of the chain--whatever it may be in a given system.

Kenobi

But once you have a decent cd player, at that point thwen is when you notice just how big a difference a nice amp will do.

Garbage in/garbage out .
I would suggest that you get the best CDP used , that you can afford , and a decent set of IC's . You will hear the difference and what the rest of your equipment is capable of sounding like . A solid/well built unit with digital out should take you down the road for quite a while and is able to be altered by the addition of a DAC if you care for change .

Speakers and amps can change a signal from the source but they can not do much to improve it . If you want to truly 'build' a system I think that you should start with supplying your system with the best signal that you can afford . Then at least you will hear what the rest of your components are capable of rather than limiting their ability .

Good luck .