What should I do

What sort of late night hours and dB levels are you talking about. Are there neighbors above, below and/or next to your listening room? Talk to your neighbors about what you're doing, even invite them over for a listen. It actually might be better for everyone if you did some sound transmission inhibiting reconstruction to your listening room. Different equipment might not be the optimal solution.

Headphones aside, I totally understand your predicament of apartment living and maybe I can offer a suggestion. If you listen to rock this may be out, but I've spent a lot of time collecting music I can listen to late at night at low volumes - solo classical guitar, solo piano, violin/guitar, string quartet, etc. There's a lot of good stuff out there without excessive bass and drums that you might be able to get away with. I personally find it preferable to headphones, even good ones.

08-01-13: Danlib1
Just one guy's opinion...Buy a Burson Soloist headphone amp and a pair of HiFiman HE-500 or Audeze LCD 2 or 3 (as your budget allows) and prepare to really enjoy the music.

+1 on the Audeze. Right now I am using Sennheiser HD-600 with a Grommes PHI-26. The HD-600 sound great, but the Audeze sound really GREAT.

Seeking your advise again had Vandeersteen 3's then bought a used pair of 5's. Best speakers I ever had. Had to sell when I sold home and downsized to condo. Boy are you LUCKY. Tried the Quatro's ,but not in the same league as 5's. Like the Hardeths at this stage of listening.and size of room . Gamut equipment to good to give up. Based on your comments ,I've zeroed in on WOO WA-22 headphone amp , Woo sells Grado's looking at GS-1000 i based on those owning Harbeths .What do you think? At this stage of my life 72 . Listening to classical,opera and Beatles,Pink Floyd and 70's light sometimes. Advise KEEP the 5's as long as you can and Don't get old . Thank you pointed me in the correct direction .

That will undoubtedly be a killer combo, but just keep one thing in mind. The Grados are really, really open, meaning you can hear every noise in the room, almost as if you were wearing nothing. You'll have to listen to them as if you were listening to the speakers - in a quiet environment. If that works for you, I would vote you proceed. Best of luck.