Here is some additional information on lister fatigue that I have copied from the Teck Talk forum:
"Listener fatigue is nearly always a consequence of distortion.
Two distortion sources are IM distortion and odd-ordered harmonic distortion. Both are known to be irritating to the human ear- and this has been known since the early 1950s.
What is less understood is some of the studies of the human ear that have occurred since then that relates directly to listener fatigue.
One of those things is that the ear/brain system uses odd ordered harmonics to sort out how loud a sound is. To do this it is very sensitive to the presence of odd ordered harmonics. Audiophiles have terms for the presence of trace amounts (less than 0.01%): hard, harsh, bright, etc.
This is why two stereo pieces can measure flat on the test bench but one might sound bright while the other does not.
In addition it is useful to know that the ear is tuned to birdsong frequencies. Knowing that makes its easier to understand how the ear can be that sensitive to the presence of the 5th, 7th or 9th harmonics.
The ear translates distortion as tonality. A 2nd harmonic is interpreted as warmth, a 7th as brightness.
IM (Intermodulation Distortion) is a form of harmonic distortion that the ear usually translates as brightness. This is where two more more tones can interact in the stereo system to produce other frequencies, the sums and differences of the tones involved. It is caused by non-linearities in the system, any place where the tones can interact, such as a feedback node in an amplifier, poor power supply design or a breakup in a speaker cone.
There is a special form of IM distortion called inharmonic distortion that only occurs in digital products (good luck looking for the specs on that- most manufacturers don't publish the numbers). These are intermodulations between a fundamental frequency and the scan frequency of the recording rather than harmonics based on the fundamental tone as is encountered in analog recordings. The ear treats these as brightness as well.
There is a tipping point in the human hearing perceptual system where the ear will favor distortion as tonality over actual frequency response errors. As a result is is often more important to have low distortion rather than flat frequency response. For this reason its usually best to reduce the types of distortion that the ear finds most objectionable and noticeable before going after frequency response errors."