Where does fatique come from?

As covered by folks above, fatigue is mainly caused by brightness especially in the midrange to high frequency range. Sometimes there are simple solutions that can ameliorate on this "fault" other than swapping equipment or speakers. Two effective way out is in speaker placement and room treatments. Room treatments will greatly enhance the sound system while eliminating any deficiencies such as sharp piercing treble(which results in fatigue) or bass boom in the low frequencies etc. Also, you might be surprised that a simple speaker positionining may eliminate fatigue altogether by a minimum or no toe-in speaker placement.

Maggies are well known for being "easy on the ear". They are not fatiguing at all, sound better the longer you listen, and you are seldom inclined to stop as a result of fatigue. These are planar designs that have very good transient response (low distortion there) and they do not move a lot of air compared to conventional full range dynamic designs, so dynamics are generally not as fatiguing either.

Omnidirectional speakers, like OHM Walsh designs, can also be comparatively easier on the ears as well because they do not blast all the sound and the associated high pressure levels in the air directly at you. They tend to fill the room with music instead, more like what occurs at a live, un-amplified musical performance, which enables realistic listening levels with less fatigue as well.

Yes, direct sound to the listening position is certainly not too good. With the drivers directed away from the listening position especially the tweeters, the ears will not be subjected to direct sound that may cause fatigue. One may also be able to crank up the volume higher without hurting the ear-drums with some proper speaker placement especially with minimum toe-in. In this sense, I suppose that omni-directional speakers may hold an advantage although I have no experience with Ohm Walsh designs.

BTW, this is an excellent question. The best way to a pleasant listening experience is to eliminate or minimize factors that cause fatigue. It is something that is not discussed as often as are things like tubes versus SS, vinyl versus digital, detail, imaging, etc.

I am with Mapman and Ryder's rationale. Simply excessive energy in upper mids and lower highs in general. But it could be the entire midrange, if excessive. You would want as flat as possible in the mids and lower highs, then roll off is preferable in highs. Find a component in your system that is the culprit. It could be speakers, source or preamp or even cables. Look to the room in the end unless it obviously has overtly exposed reflective areas. In any case make your evaluation in near field listening to rule the room out. For now.