2 channel system?


in a 2 channel system (cd player or integrated amps) which component affects the sound the more? im looking to upgrade gradually and im wondering which component will have a more direct impact on the sound? i understand that matching of components is very important and all parts play an important role in the overall sound? but in your opinion which one affects it more? thanks in advance.
inspiral
I third or fourth the source recommendation, what's lost(or not read) can not be recovered down stream. Buy the best source you can this way there is no guessing about what the recording is suppose to sound like. For a while(recently) I was running a $9k cd player into a pair of monoblock amps that can be picked up used at agon for under a grand(they are no longer made) and it sounded much better then when I was using my $5k amp with a cheap CD player- and guess what deductive logic will tell us that combining the two(good cd player and good amp) yielded the best results. Avideo brings up a good point about not forgetting cables, once you've tried good cables you simply can not go back, it is often what makes or breaks a system. One piece at a time and it'll be done in no time :)
I'd suggest going with amp/preamp first. I only say this because it is good to see what the source you currently have can sound like. It's hard to judge sources with bad preamplification/amplification. Get amplification/preamplification that matches your speakers.

KF
I agree with amplification first, and if separates, the preamp before the power amp. What good is a great source sent through inferior amplification.

I also agree with the cable comments. Most people put a cheap cable on a lower cost source. You'd be surprised how good a lower cost source will sound connected with better cables.
You said that you wanted to upgrade the integrated amplifier and did not wish to go with preamp/amp.I would suggest you get one of the following: Rowland Concentra I or II,Gryphon Tabu Century,Gryphon 2100 or 2200,YBA Passion Integre.Do not stay in the middle,no source will help you achieve a good sound if you do.Then try different players and judge them accordingly.In the beginning this may sound like a waist of money but in the long term,I think,you will be pleased.CD-players come and go,good amplification stays.That's what I would do.
I've no quarrel with starting with the source. I would caution against starting with amplifiers, at least until you have selected a style of speaker. Will you choose cones, planars, electrostatics ... monitors, full-range .... a subwoofer ...? These choices will have a huge impact on what amplifiers will be necessary and suitable, let alone preferences among them. At least the source components are pretty much free of this critical interdependency.