It is definitely not the speakers. Other than the tweeks you are trying, I vote for the source CD player and the amplifier. I've run CDM-1's with amplifiers costing well over $2000 and they sounded great.
If you want to upgrade at some point, I would start with the amplifier, then the CD player.
If you want to upgrade at some point, I would start with the amplifier, then the CD player.