Listening Fatique- componet or speaker issue.

Hello everyone. I've been posting various questions in other topics trying to solve a problem that couldn't discribe until now. I jumped into the audiophile hobby these last few months full head-on. I got the best that I could for about 1200 and finally ended up with as my first set-up.

Totem Rainmakers (straightwire octava II)
Nad c320bee
Source (pc m-audio,foobar) will be (modded squeezebox2)

In the begining (sounds like the bible I know) after I hooked it all up I was estatic. I've never listen or owned any quality audio eq and this was a huge leap for me. The sound was or so detailed, huge, realistic and plain great. This only brief sessions. Now that all the newness factor is gone and I actually live with these I seem to be in a delima.
Normally I play my music at low levels simply because during the week days I come home late and I live in a tenant building (rules are rules). At lower levels I can listen to the music for a fair amount of time no problem. No real hint of fatigue.
Now my fatigue seems to occur at normal or slightly higher than normal listening level. I don't get seem to get headache as most other people. My ear mostly feel "tired" and occasionally I do get a ring. The sound coming out also sounds compressed and boxy.
I don't know if it's a speaker componet (source, amp, or room) but it's really getting me bumbed out. I brought my new setup so I can listen to more music and make it a more consistant part of my life as a musician (gotta study the greats) not so my ears ache and feel tired.
When I changed the source from my dvd to my computer (m-audio foobar) it became better but it's still there. I posted a thread about my amp changing but know now that I didn't ask the right question.
Now my Rainmakers are rated at min 50w at 4 ohms and maximum is 100w I think. I decided on the nad mainly on suggestion on this and other threads and read that a couple rainmaker owners match them. I like the amp quite a bit its warm and quite detailed but can this fatigue I am experiencing also be occuring because of the amp. Could this fatigue be just the normal distortion created by my speakers not being driven well at higher levels?
Or are the rainmakers just naturally fatiguing and harsh when pushed? I need advice from the higher ups.