Speaker first. Audition speakers at your house and commit lots of time evaluating them. Make sure you listen to various types of speakers (box sealed, box ported, OB, planar, high efficient, low efficient, one/two/three/etc. drivers, sat/sub, etc.) As you form your opinion make sure you like it at the volumes you'll typically play and can listen to it with most (if not all) of your recordings for long periods of time without fatigue. Then as time goes on you can improve your source if necessary or you just get itchy.