This is not a recommendation for everyone but if your jazz tastes run to free jazz, improvisational jazz, and modern jazz, as well as modern classical, and the late 60s and early 70s progressive scene, there is one place that is absolutely the best place on the planet on a per square foot basis (because it is a small basement shop): The Downtown Music Gallery (now in Chinatown). Check it out:
http://www.dtmgallery.com/Main/index.htm
Think Andrew Hill, Cecil Taylor, John Zorn, Keith Tippett, Anthony Braxton, Soft Machine, Terry Riley, Phillip Glass, King Crimson, X-Legged Sally, Miles Davis, Coltrane, Ornette Coleman, Brotherhood of Breath, Robert Wyatt, Last Exit, Peter Brotzmann, Mathew Shipp, William Parker, Larval, Gunter Hampel, Misha Mengelberg,..."that" sort of thing.
You've never seen most of the stuff nor heard most of the artists, but they've got stuff playing all the time in there and I've picked up many things just because I was there and heard something playing. I've spent a couple of hours in there on more than one occasion. A real treasure.