Where does fatique come from?

Distortion is very fatiguing.

I believe it varies with the person involved.

Think about other situations that are fatiguing, such as driving at night in the rain. You have to concentrate at a very high level to correctly interpret what you see. The paint lines between lanes are hard to see, there is a lot of glare from oncoming headlights and visibility is just downright poor. Your brain can expend a lot of processing power to "correct" the information it is receiving. Most people feel absolutely drained after a long drive in those conditions.

Same thing with concentrating on a difficult test where a minor misreading of a test question could lead to a wrong answer.

Our brain has an amazing ability to sort through the information it receives (think of concentrating on one conversation in a noisy room), but that effort comes with a price tag attached. The harder it has to work the faster we feel worn down by the task at hand.

It is no different with listening to music. However, different people listen for different things. A particular overemphasis or deficiency may bother one person (and thus require more mental correction) more than it bothers another.

As an example, one listener may value midrange tonal accuracy more than good bass. Good bass with a inaccurate midrange will annoy that listener while another listener may be blase about a slightly off midrange yet strain mightily to supply the missing info to a bass line that is important to him. There are obviously a wide variety of parameters for each listener to prioritize in a fashion that is important to them.

That said, some errors are more likely to bug a high percentage of listeners. Edgy distortions probably top the list of near universal annoyances, but even then we all know people who are perfectly happy with their table radio with the blown speaker.

Usually from distortions occurring in the mid to upper frequency range.

One of the most common sources of fatiguing distortion comes from individual component's inability to accurately reproduce sharp transients and dynamics accurately. When these occur, the system cannot respond fast enough and fatiguing distortions occurs. Some may find sharp transients and extreme dynamics fatiguing as well even when accurately reproduced because these create the high pressure level variations in the air as sound is transmitted that can stress or fatigue the ear itself.

The problem is much more common with digital sources compared to analog/vinyl sources which is why I believe more find digital to be fatiguing than vinyl.

Planar and electrostatic speaker systems in particular ten to not move as much air and typically produce transients well and do not deliver in genreall deliver the extreme dynamics compared to conventional speakers using dynamic drivers. This makes them often an attractive option for those that may be fatigued by more dynamic designs.

As covered by folks above, fatigue is mainly caused by brightness especially in the midrange to high frequency range. Sometimes there are simple solutions that can ameliorate on this "fault" other than swapping equipment or speakers. Two effective way out is in speaker placement and room treatments. Room treatments will greatly enhance the sound system while eliminating any deficiencies such as sharp piercing treble(which results in fatigue) or bass boom in the low frequencies etc. Also, you might be surprised that a simple speaker positionining may eliminate fatigue altogether by a minimum or no toe-in speaker placement.

Maggies are well known for being "easy on the ear". They are not fatiguing at all, sound better the longer you listen, and you are seldom inclined to stop as a result of fatigue. These are planar designs that have very good transient response (low distortion there) and they do not move a lot of air compared to conventional full range dynamic designs, so dynamics are generally not as fatiguing either.

Omnidirectional speakers, like OHM Walsh designs, can also be comparatively easier on the ears as well because they do not blast all the sound and the associated high pressure levels in the air directly at you. They tend to fill the room with music instead, more like what occurs at a live, un-amplified musical performance, which enables realistic listening levels with less fatigue as well.