What is really happening is that we are able to hear what negative feedback does to sound
negative feedback just ensures linearity - it makes sure that the output matches the input. An electrical circuit operates at close to the speed of light...I doubt anyone can hear negative feedback in modern circuitry. Slew rates of good amps are typically 50 volts per micro second or enough to accurately reproduce a signal of well over 100 KHZ without distorting the signal. Since it is accepted that people rarely are able to hear anything above 20 KHZ then it is extremely doubtful that any slight anomalies of this kind of order are actually audible.
Speaker cones will try to keep moving because of inertia, the suspension pulls them back and as they move they induce current (EMF) in the coil which the amp will sense and will dampen by driving the output to match with the input. Provided the drivers have low mass, a high Xmax and a strong magnet then a powerful low output impedance amp should be able to control the driver well. If the driver is low quality, with a weak magnet and a low Xmax then it doesn't matter as much if the high powered amp has a high damping factor as the cones become harder to control as they travel well outside the linear range of the magnetic field...in this case, ultimately, the suspension pulls them back in but unfortunately at this point you have large amounts of audible distortion).