If you're In Toronto, check these places out:
Rotate This (Queen & Bathurst)- Mostly new and some used vinyl with an emphasis on ROCK and a selection of jazz, R&B, reggae, etc. Very cool and helpful staff.
Slinky Music (Queen & Spadina)- Mostly new and some used vinyl with an emphasis on dance floor friendly music and a choice selection of hip hop, R&B, jazz, reggae, etc. Very cool and helpful staff.
Play De Record (Yonge & Dundas)- Definitely a DJ shop, but with a choice selection of new jazz, soul, funk and rock with a dance floor friendly vibe. These guys source out some really cool, obscure stuff, that I will never see anywhere else. Very cool and helpful staff.
Criminal Records (Queen & Spadina- across from Slinky Music). These guys have the best prices in the city for rock and a small amount of jazz. Very enthusiastic staff.
BTW, these shops are not all catering to audiophile types, so vinyl gets opened and handled, so you can sample the wares before you purchase it. Just ask at the counter before you open any sealed vinyl as policies vary from place to place and you may find the staff not so cool and helpful after you open something you shouldn't have. If the copy's sealed, they may have it on CD for a listen.
For strictly used stock, check these out:
Vortex (Yonge & Eglinton)
She Said Boom
Babel (Queen & Ossington)
There's several shops between Rotate This and Slinky Music, that makes for a nice walk and digging session.
Enjoy.