Quality inversely related to power output?

Not true. As Gregm says - it is simply more difficult and more expensive to build more powerful amplfiers with similar performance to lower powered amps.

If your only measure of quality is distortion then stick to headphones.

As far as I am concerned, dynamics and ultimate SPL output is also quality. If an unamplified piano or drum set can produce north of 110 db SPL on transients then the low distortion qualities of an underpowered system is rather a mute point - as it can never realistically reproduce the range of natural sound of musical instruments in the first place...

I am of the "first watt" camp.If you don't get the first one "correct",then the rest doesn't matter.
It is much harder to get multiples to act as one/two.That being said,you'll never drive 86/88db efficient speakers with 3W to a desireable level.

only when they're grown from magic beans

There are few cases where larger amplifiers will sound better, all other things being equal. The disclaimer is important, since there are so many variables in amplifier design. The 'first watt' concept has a lot of credibility, since in many cases the amplifier will be making less than a watt most of the time. Most of the power is used on peaks.

In tube amplifiers the size of the output transformer is paramount. The issue is getting full bass performance vs HF performance. This is challenging and you can assume that any high powered transformer-coupled tube amplifier will have limited bandwidth.

With SETs this issue is even more pronounced and is why the ones that are 7 watts or less are also considered the best sounding. Time was that the 300b was King, 5-7 years ago the 2A3 took ascendancy (about 3.5 watts) and nowadays you hear an awful lot about 45-based amplifiers (maybe about 1 watt).

OTLs do not have this issue; IMO/IME the bigger you make them they better they sound. As far as I can tell this is the only amplifier technology for which it is true that bigger is better.

With transistor amplifiers the issue is all the paralleled devices, which will 'hog' current to a much greater degree than paralleled tubes will. Thus its pretty much mandatory that in a larger power amplifier, all the output devices are matched. Another issue you get into with larger power amplifiers is the extreme levels of negative feedback that are employed- such designs rarely bring home the bacon insofar as the rules of human hearing are concerned.

All this points to the efficiency of the loudspeaker as being quite important overall- since truly musical high powered amplifiers are nearly oxymoronic in general, one will always be challenged to get low efficiency speakers to sound like real music.

All this points to the efficiency of the loudspeaker as being quite important overall

However, this now requires "all other things being equal" in speaker design. It is generally not as easy to find a full range speaker of 100+ db SPL with acoustical performance that is competitive with the many speakers designed between 85 to 91 db sensitivity. (Generally designers equalize drivers downwards to achieve a desirable acoustical performance (crossovers etc.) because the raw driver output is rarely flat and smooth on and off axis acros the frequency range.