Csontos, you put your finger on the leading edge of what is possible with transistors.
Here is one of the difficulties you are dealing with. Transistors have a non-linear capacitive aspect inherent in the junctions of the devices. In fact some semiconductors take advantage of this capacitance, as in the example of varactor diodes that are used in the tuning of modern FM radios.
This non-linear capacitance is magnified by the amount of current through the junction. It results in non-linearities in the amplifier that employs such devices. Now if you were reading between the lines, one way to reduce this problem is have the amp drive a higher load impedance. This will reduce the current in the driver transistors and output devices.
There are other advantages to a higher load impedance besides this one, but that is one effective means of reducing harshness in transistor amps. Of course you will get less power, but in high end audio usually we are more concerned about sound quality rather than raw sound pressure.
Now in tubes there is also an inter-element capacitance; the difference is that it is constant and unchanging regardless of signal level rather than non-linear and changing. This makes it a lot easier to build a circuit that is low in odd ordered harmonics.
This seems to me to be one of the things that has to be dealt with in a transistor design before overcoming the apparent advantages of tubes insofar as listener fatigue is concerned. One patented method to get around this problems involves heating the power transistors to a high (+100c) temperature- I have heard one of those amps and its quite impressive- but also very expensive- over $100,000 for a 100-watt/channel amplifier. Plus it made all the heat of a class A tube amp of the same power.
So it seems to me that in more practical terms tubes do have the upper hand in this regard, as its fairly easy to build a zero-feedback tube amplifier. That is hard to do with transistors; but Ayre and the Pass First Watt amps are examples, and IMO these are some of the best transistor amps made today.