Do distortion 's affect enjoyment of speaker?

No distortion does not affect the enjoyment of loudspeakers, and no there will be no consensus on this, so stop hoping.

Speakers can have many types of distortions, divergence from linear frequency response on-axis, divergence from linear freqency response off-axis, time based distortions, IM distortion, dynamic distortions such as compression, distortions of rise time and settling, and many more. If you think that focusing on very low harmonic distorion is the answer, by all means take that route.