Are Headphone amps worth the expense?

Are you into DIY?

Try building a CMOY headphone amp. Strictly low cost and fun. The good news is you can than 'roll opamps' and test the different ones available.

I use OPA2134s as a good compromise.

Very good amp when properly built. Not as good as the 'best' but no more than 5% of the cost.

http://tangentsoft.net/audio/cmoy-tutorial/

Aren't there special recordings made for headphone listening?

I think they are called binaural and are made with microphones in the ear positions of some kind of artificial head.

Am I mixing this up with something else?
Didn't the Carver Sonic Holocaust try to eliminate the out of phase stuff coming from the other speaker? Didn't Polk make speakers with double mids and tweets which had a wire pair between speakers to remove this out of phase information mechanically, if you will? The double speakers were located about 6" apart, the distance between the ears so the time cues remained.
This is not a new idea.