What matters most in speaker design?

What is great sounding speaker? Sound depends on many factors and people have different preferences. Do you prefer speaker with underhung motors (less distortion and less dynamic) or overhung motors(more distortion and more dynamics) . Do you listen soft or loud. Do you listen to Jazz or Organ music? Do you crave for pinpoint imaging or heavy metal punch. Do you care how large sweet spot is. How does it perform in small or large room etc. How easy it is to drive.

Frequency response test is pretty much useless. Bad speakers with horrible phase errors might show with excellent flat frequency response.

I would agree with Shadorne on transducers. If pair of speaker costs for instance $1k then manufacturer sells it to stores for about $500. Components cannot cost more than 1/3 of it (to cover labor, expenses etc). Now we are left with $170. Both cabinets will cost $50 both and we're left with 120$ for let say 6 transducers and crossover components. That's why my speaker had originally $25 Vifa tweeter and Mylar + electrolytic caps in crossover. Good tweeter is 10x this and good cap is $100 (I just replaced $15 caps wit $100 caps of the same value and sound + imaging improved a lot).

There is no universal answer but you won't get very far with cheap transducers (especially tweeters). I don't design speakers but after concept design I would start with the best tweeter I can afford and worked down from it.

Uniform polar response with a slow monotonic decrease in total power above a few kilohertz preferably with less output towards nearby objects.

Getting there is a function of driver diameters, driver bandwidths, spacing, baffle shape/dimensions, and the cross-over.

There appears to be a point at which drivers are "good enough" in terms of distortion and stored energy.

Provided output levels are limited, you sit closer to accomodate the reduced directivity, and get farther away from the side walls my Linkwitz Plutos ($140 driver cost) are surprisingly close to my Orions ($1400).

"There appears to be a point at which drivers are "good enough" in terms of distortion and stored energy."

Drew - are you saying that Beryllium tweeters in new Revel Saloon 2 or diamond tweeters in B&W are mistake (unnecessary expense)?

I originally thought flat frequency response was a primary goal. And it is, IF damage done to achieve this is minimal.

BUT having listened to single driver speakers with no x-over and then adding different tone controls, [DEQ24/96, Parasound Z-pre, 3-band tone control] to flatten the response, I can say any manipulation of the signal makes the sound worse in more important ways. Assuming the driver F-R is fairly flat to begin with. For example, NOT metal with some crazy break up mode or peaky high efficiency Fostex/Lowther.

I find I can mentally compensate for uneven F-R BETTER than the distortion and phase shifts that come with ANY tone control that I have found. A classic case in point is the Eggelston Andra that switched from an uneven mid-driver with no x-over to a flat F-R but with x-over. Surprise was many listeners preferred the older design.

Also, I am adjusting the signal the "right" way, at line level, not AFTER the power amp where passive x-over parts will color the sound even more. And for those who HATE tone controls, you should be aware that designers build tone controls INTO passive x-overs to flatten bass or add the famous 5dB boost at 100hz.

Expressed here better than I can:
Read Johnnyb53 re Flat F-R

So to answer the OP's question in terms of the driver itself, I will respond by repeating what Seas says:

1) Low non-linear and modulation distortion.
2) Excellent transient response.
3) Good coil excursion.
4) Pistonic response.
5) Large windows in the basket to reduce sound reflections, air flow noise, and cavity resonances.

It's been reported that John Dunlavy thought step response was what mattered most in speaker design.