I am in a good position to offer an opinion as I started out with Infinity Reference series speakers which have a basic wood cabinet, and comparing those speakers to my current Magico S5's and other extremely inert speakers like Crystal Arabesque which I have experience with, my view is yes speaker distortion does negatively impact your enjoyment of music.
Firstly, cheaper wood cabinets can 'sing', in effect adding harmonics to the sound which distorts the original signal. In terms of driver distortion, ported designs like Wilson in my experience suffer from bass overhang which creates a bloated sound as the cone is unable to follow the signal as accurately as Magico's sealed design. This smears the upper frequencies which reduces the speaker's coherency and can lead to listener fatigue.
The Magico S5's for example use extremely rigid cabinets made from 1/2" thick aluminium extrusions. The S5's, like all Magico speakers also benefit from a sealed design as I mentioned which allows the bass drivers to operate in perfect piston motion and more accurately follow the signal. But of course there is no such thing as a free lunch, and the trade off is usually lower efficiency.
With Magico speakers (perhaps more than any other dynamic speaker i've heard) your ears need time to adjust due to the absolute lack of any perceptible distortion products, the accuracy of the drivers & exceptional resolution. You hear more of the information in the recording, including subtle background information and layering in the music which adds a whole new dimension to the music in terms of realism. A wordy response, but hope that helps.
