The kind of distortion is extremely important. 1% being composed of a nice, clean second harmonic isn't good in my opinion, not because it doesn't sound good, but rather because you're basically using the amp as a tone control. That's significant color to add to a signal. That's not to say it sounds bad. A lot of people want that kind and level of coloration. Nobody likes complicated distortion. Some fuggly mix of 5th, 6th, and 7th order distortion at .1% is going to be a lot more offensive than a big, pretty 2nd order at 1%. The kind of distortion is more important than the amount.
The problems you see in class D are the same you have with class B. The lower the power output, the more the distortion dominates. Class D just tries to minimize the distortion by reducing the time the transistors transition from off to on by pulsing them very rapidly. But no matter what, transistors switching off and on will always make huge amounts of high order distortion. It's unavoidable. It's just imppossible to fully filter it out. If it were possible, it would have been applied to class B amps by now.