Reviews with all double blind testing?


In the July, 2005 issue of Stereophile, John Atkinson discusses his debate with Arnold Krueger, who Atkinson suggest fundamentally wants only double blind testing of all products in the name of science. Atkinson goes on to discuss his early advocacy of such methodology and his realization that the conclusion that all amps sound the same, as the result of such testing, proved incorrect in the long run. Atkinson’s double blind test involved listening to three amps, so it apparently was not the typical different or the same comparison advocated by those advocating blind testing.

I have been party to three blind testings and several “shootouts,” which were not blind tests and thus resulted in each component having advocates as everyone knew which was playing. None of these ever resulted in a consensus. Two of the three db tests were same or different comparisons. Neither of these resulted in a conclusion that people could consistently hear a difference. One was a comparison of about six preamps. Here there was a substantial consensus that the Bozak preamp surpassed more expensive preamps with many designers of those preamps involved in the listening. In both cases there were individuals that were at odds with the overall conclusion, and in no case were those involved a random sample. In all cases there were no more than 25 people involved.

I have never heard of an instance where “same versus different” methodology ever concluded that there was a difference, but apparently comparisons of multiple amps and preamps, etc. can result in one being generally preferred. I suspect, however, that those advocating db, mean only “same versus different” methodology. Do the advocates of db really expect that the outcome will always be that people can hear no difference? If so, is it the conclusion that underlies their advocacy rather than the supposedly scientific basis for db? Some advocates claim that were there a db test that found people capable of hearing a difference that they would no longer be critical, but is this sincere?

Atkinson puts it in terms of the double blind test advocates want to be right rather than happy, while their opponents would rather be happy than right.

Tests of statistical significance also get involved here as some people can hear a difference, but if they are insufficient in number to achieve statistical significance, then proponents say we must accept the null hypothesis that there is no audible difference. This is all invalid as the samples are never random samples and seldom, if ever, of a substantial size. Since the tests only apply to random samples and statistical significance is greatly enhanced with large samples, nothing in the typical db test works to yield the result that people can hear a difference. This would suggest that the conclusion and not the methodology or a commitment to “science” is the real purpose.

Without db testing, the advocates suggest those who hear a difference are deluding themselves, the placebo effect. But were we to use db but other than the same/different technique and people consistently choose the same component, would we not conclude that they are not delusional? This would test another hypothesis that some can hear better.

I am probably like most subjectivists, as I really do not care what the outcomes of db testing might be. I buy components that I can afford and that satisfy my ears as realistic. Certainly some products satisfy the ears of more people, and sometimes these are not the positively reviewed or heavily advertised products. Again it strikes me, at least, that this should not happen in the world that the objectivists see. They see the world as full of greedy charlatans who use advertising to sell expensive items which are no better than much cheaper ones.

Since my occupation is as a professor and scientist, some among the advocates of double blind might question my commitment to science. My experience with same/different double blind experiments suggest to me a flawed methodology. A double blind multiple component design, especially with a hypothesis that some people are better able to hear a difference, would be more pleasing to me, but even here, I do not think anyone would buy on the basis of such experiments.

To use Atkinson’s phrase, I am generally happy and don’t care if the objectivists think I am right. I suspect they have to have all of us say they are right before they can be happy. Well tough luck, guys. I cannot imagine anything more boring than consistent findings of no difference among wires and components, when I know that to be untrue. Oh, and I have ordered additional Intelligent Chips. My, I am a delusional fool!
tbg
TBG: Two amps that reproduce music differently enough to be heard will NOT sound identical in a DBT. But how do we know that two amps reproduce music differently? You say they do, but how do we know you are right?

Let me pose the question a bit differently. Here we have two amps that are not distinguishable in a level-matched, quick-switching ABX test, generally regarded in scientific circles as the gold standard for determining audible differences. A subjectivist claim that these two amps reproduce music differently. How would he prove that they do? Whatever he does, he has to use a blind test, because a sighted test can prove nothing about audibility. That's settled science. So what kind of test should he use?
The standoff between Pabelson and Tbg reminds me of the stalemate between the external-world skeptic and the dogmatist.

Skeptic: You don't know that you're not a brain in a vat of nutrients, being stimulated by a computer simulation, carefully monitored by a team of scientists, to think you're in a real, concrete world... the world you *think* you're in. Since you don't know you're not a brain-in-a-vat, you don't know anything mundane about the external world, e.g., that you have two hands.

Dogmatist: I know I have two hands! If I know I have two hands, then I know I am not a handless brain-in-a-vat. Therefore, I know I am not a handless brain-in-a-vat.

One man's modus ponens is another man's modus tollens, as the saying goes.

(For non-logicians, modus ponens is: If P then Q. P. Therefore Q. Modus tollens is If P then Q. Not-Q. Therefore, not-P.)

Pabelson: DBT shows no audible difference between cables, therefore there is no audible difference.

Tbg: There is an audible difference between cables, therefore, DBT is flawed.

Logic alone (formal logic) cannot settle the dispute, any more than logic can settle the skeptic/dogmatist dispute.

But in this case, it's odd to think of Tbg's favored cables being a/b'ed with cheapos, without his being able to tell the difference, and then, only when told the true identity of the cables, his insistence that there *is* a perceivable difference. Very odd.

Here's a question for the doubters of DBT-ing. Given that there are perceptual biases at work (expectation, confirmation, endowment effect, etc.) how would one test for such biases? That is, what *would* count as two components sounding the same?

Suppose you have two amps that are identical except one of them has a beetle put inside and the beetle runs around, I don't know, defecating in there. And then reviewers praise the beetle effect: "Widened the soundstage by meters! You don't need golden ears to hear this one!" How would you go about evaluating the beetle effect?
As I have said at least 5 times, your statement that, "ABX test, generally regarded in scientific circles as the gold standard for determining audible differences" is not true. But neither of us will ever convince the other, so why don't we just drop it. I can accept your statement that those advocating it would not be numerous enough to justify a magazine so it is a moot point.
Interesting discussion but I think the main point or lesson of DBT tests is being missed through the discussion of semantics and detail.

DBT tests are significant for audiophiles because they show that, for the handful of those people and equipment tested, differences cannot be audibly detected between some types of equipment, at least not easily, even when people are trying. DBT's show that it is typically hard to discern differences with good quality SS amps, CD players and a variety of quality cables.

This raises questions about reviewers abilities to hear strong differences between certain types of equipment, but it does not conclusively prove that they cannot hear these differences (unless they were DBT tested with the very equipment under review).

It also suggests that some types of equipment are either less crticial (cables) or of universally engineered high quality (many amps and CD players); in these cases, different equipment choices are unlikely to make a significant difference to the sound that is heard. DBT tests also tell audiophiles that some things are a big factor...speakers, speaker placement and tube vs SS amps, for example. In essence, DBT's confirm what is probably a gut feel for any serious listener who has played around with a variety of equipment over the years.

However, it remains possible that some people out there might be able to hear a difference....for differences there surely are...however minute and undetectable to those tested so far. So those with "golden ears" can keep on searching for nirvana, everywhere and anywhere, including but not limited to fancy power line conditioners and other tweaks.

If DBT test result reports do not conclusively prove that a cheap amp is just as good as a very expensive one to your own ears...then exactly what use are they????

I suggest they simply offer some guidance as to where an audophile might spend a higher proportion of their effort/money in improving the sound of their system. They also suggest that the "emperor may have no clothes" in some cases....so, for certain types of equipment, be a little wary of exhorbitant prices and rave claims by audio salesman/reviewers!
Shadorne, I understand your moderate position. Please take no offense when I simply say that I strongly suspect that DBT is an invalid testing of what people hear. I am not really concerned either that some, myself included, cannot hear differences in the typical same versus different format so commonly used in DBT. People do hear differences when double blind testing is just a which do you prefer of amps A, B, and C. I don't really have much trust in many reviewers and don't need their inability to hear difference in same/different tests to be convinced.

I absolutely concur that we need to be wary of exorbitantly high priced equipment and rave reviews and claims by salesmen and reviewers. But we should equally be ready to hear true quality in some more expensive equipment. Quality parts cost money and research and design work has to be paid for. Too often I have heard some expensive gear that truly is excellent in my opinion and which I remain thrilled with. My Reimyo PAT777 amp and Shindo Labs turntable are but two examples. I also have a relatively inexpensive line stage, phono stage, and universal player that are at least the equals of much more expensive equipment, again in my opinion.

I once heard a $350,000 amp at CES. I listened with no intention of ever buying one. It was the best sounding amp I ever heard. The Stereophile reviewer also loved it, but it measurements looked bad and so they dismissed it. The objectivists ranted that Stereophile should not have even reviewed it. I ranted that they should have heeded their ears rather than their inadequate instruments. I still would not consider buying it largely because I just cannot afford it.