Is it possible that high powered amps delivering 1 watt will sound worse than a lower powered amp delivering that watt? I would hazard to guess that the answer is yes. I've heard a few demonstrations of high powered amps vs. low powered amps from the same manufacturer, and often, the lower powered version sounded better where high power was not needed. Sure, a whole lot of other factors are not controlled and the design and components, etc. could be radically different between the two amps, so I am not drawing any hard conclusions. I know some solid state designers have gone to almost heroic extremes to minimize the number of output devices in their design because they claim that paralleling multiple devices degrade the sound. I have no idea if this is true, but, at least this is a plausible explanation of why some don't like the sound of high powered amps on high efficiency speakers.
As for the issue of high efficiency speakers that are difficult loads, I have found that compatibility with tube gear IS very much related to the difficulty of the load. A speaker, like some of the Wilson speakers, are surprisingly efficient (mid 90s), but none sound very good with most tube gear because they are difficult to drive. Other much less efficient speakers, e.g., Spendors, are MUCH more compatible with low-powered tube amps. I recently heard an old BBC monitor speaker (mid 80s efficiency rating, a 15 ohm nominal impedance) used with some quite low-powered pushpull amps (5 watts or so) and the combination sounded fantastic.