Why is Double Blind Testing Controversial?

Blind and double-blind is a way, as we all know, to attempt to remove the subjectivity from what is hoped to be an objective evaluation. It's great for testing some things in which there is a clear hierarchy of "poor" to "excellent." Eyes, ears, and taste buds, however, don't conform to objectivity. Try a blind wine-tasting to see how many diverse opinions are being swished around the tongue and spat out into the bucket.

I tried a blind (but not deaf!) test last night, as it so happens, of some Vampire vs. Nordost vs. van den Hul interconnects.

It's REALLY hard to change cables with one's eyes shut! I knocked my turntable onto the floor, stepped onto the open try of my CD player, connected the phono stage to the tape deck, smashed my 180gm Dylan LP, and shoved my index finger through the cone of my Cabasse.

After all that, all the cables sounded like s**t. In truth, the van den Hul bested them all, followed (at some distance) by the other two. The $15 Vampire sounded no different than the $150 Nordost.

Blind testing is fun, as Twl notes, but very difficult to implement in real life.

What an interesting thread. I've othen wondered if there would be value in SB/DB testing and why it isn't done more often. It seems to me that the reason for such tests is twofold, firstly to identify on an objective basis specific differences between components - eg transparency, ambiance, soundstage, resolution, focus, brightness, transient attack, clarity, etc; and secondly which component gives greater musicality over a period of time eg which gives the best pleasure, which is nearest to the original recording, which is nearest to live sound. The problem is that everyone hears sound differently. That's why some like the sound of the Festival Hall, and can't bear the sound of the same orchestra playing the same piece under the same conductor, in the Albert Hall (I live in the UK). I suggest the success or failiure of SB/DB testing would depend on its its ability to allow the listener to consistently pick out the component which gave them the greatest pleasure, or produced the specific type of ambiance, transparency, etc. that they wanted.

Most people can tell the difference between listening to components placed on a bog standard shelf in the living room, or on specifically designed audio stands such as the Sistrum, or Townshend: whether they prefer one over the others is dependent on how they hear sound. It's the same with cable burn-in: I defy anyone with normal hearing not to agree there is a difference between a fully burnt-in cable and a virgen cable. But that difference may be totally unimportant to them.

So maybe BS/DB testing at an individual level would have value, if one had the time and the money to spend on it. Otherwise, perhaps the knowledge - gained by quick A/B comparision, reviews, and most importantly people's opinions on forms such as this one - that component A has slightly more of what we want in terms of amibiance, transparency, etc; is about as far as we can reasonably expect to go, and very probably good enough for all but the most "golden-eared" of us.

TWL, I'm not following your logic or maybe I just didn't read it carefully enough.

First, you make a point that DBT proved to be virtually the same as non-DBT for you. (BTW, congrats on the ~ 100% results. I wish more "reviewers" would do the same exercise.)

Then, you basically claim that DBT is primarily for people that want to save money (or claim that their lower priced system is just as good as any other.) This seems to be a stretch to me.

First, I doubt that most DBT proponents advocate it to justify their inferior system. You're taking a cynical view of an opposing viewpoint and making a generalization. Regardless, even for those who use DBT to solely justify a lower priced product, what other method would you recommend that is better? You, yourself, said the results were identical.

Bottom line -- DBT is either an accurate test or not. If it isn't, then your 100% non-DBT concurrence with DBT tests would be a bad thing wouldn't it???

I advocate this testing as a way of attempting to control variables in subjective evaluations, not as means of disproving the merits of high-end, high- priced components.

Labtec, just because I have used blind testing in the past, does not mean that I consider it terribly useful. I can much more easily get the answer I need regarding the performance of a product, by just doing a simple listening test, and dispensing with all the blindfolds, and mystique of blind testing. It is simply not necessary for me. If I find that something sounds so close to what I have, that I can't really tell, I don't really need to be considering an upgrade to that product. Very simple really.

As far a blind testing being used by others, I don't think that it is used for testing the sound of equipment by very many people at all. I think that it is used for the purposes I stated, which is to "refer" to blind-testing, as a diversion from the real issue.

I am making a distinction here, between actual blind testing, and the way it is "referred to" in the context of this discussion. It is referred to as "scientific" reasoning that blind testing will show that audiophiles cannot hear what they claim to hear, and that is what I take issue with. It has been my experience that blind testing actually bears out the statements that audiophiles make about hearing differences in equipment. Not in all cases, because some equipment is not different enough for many, or any, to hear. In most cases that I have seen, it is different enough to hear.

I have never claimed that there are differences in ALL cases, and this is borne out in regular informal listening tests as well as it may be in blind testing.

So, I feel that blind testing is not yielding any more information than we already can obtain by simple listening testing. But I also feel that blind testing is being used for an entirely different purpose, in the context of these discussions. In these discussions, it is often claimed that no difference in sound exists between low and high priced cables, or amplifiers, which are the 2 items most often cited in this context. It is then generally stated that if we were to be placed in a blind testing situation, that we would not be able to tell the difference, and thus are wasting money by buying higher priced "audiophile" cables and amplifiers. I feel that since it is obvious, not only by my experience, but by simple empirical inspection, that items made with different designs, and different components will have a different sound, that this line of argument is not related to equipment at all, but must have another purpose. And I have stated what I feel that purpose is.

But it also has another effect. The haranguing of audiophiles that they are imagining things, and "scientific" testing will prove that they are incapable of making sonic decisions, serves only to shake the confidence of a listener, and make him feel that he is imagining things that he is actually hearing. This is a destructive effect, that may in fact lead him to make a bad decision regarding the quality of his audio gear, if he is sufficiently swayed by this line of thinking. If there is truly anyone here who thinks that lamp cord is the best speaker cable obtainable, then they need to take up another hobby.

This is really an extension of a 25 year old battle, which started out in the 70s with the "measurement" people. They read a couple of ads, and thought they knew everything. Then they went around proclaiming that a Technics reciever would be as good as anything you could buy, because it had .0000001% distortion, which we now know was provided by gross amounts of negative feedback, and had hugely destructive effects on the sound quality. But, did this stop the measurement people from running around telling everyone that they were wasting their money on "audiophile" amplifiers? No. In fact they continued to rant about how no high-end amp could be worth the money, when it had higher "measured" distortion than the Technics reciever. And that anything different we heard in these high-end amps was either our imaginations, or it was "euphonic" types of distortion, which is some unknown type of distortion which somehow makes us like the sound more, while still actually being bad. They are still at it with this one. I hear references to "euphonic" distortion all the time on these pages, whereby the implication is made that a tube amp cannot be accurate, because of distortion measurments. After the world caught on to the gag, these measurement people had to "hole-up" for a while. Now they are out again in full force with this again with the cable thing, and they are throwing the amps in again too, in case they can get any more "mileage" out of that one.

While certain measurements may be a decent indicator of whether a component will perform in certain ways, the final arbiter is the ear. This equipment is made to be listened to by ears, and the ears are the judge, regardless of whether the measurements line up or not. The most perfectly measuring amp in the world is not worth a thing, if it sounds like crap. Conversely, an amp that sounds fantastic is worth plenty, regardless of what the measurements are.

This is the same thing as we are seeing here. And the same techniques are being used. Point at a measurement or "scientific" test, and proclaim that anything that you hear that doesn't line up with this data, is imaginary. And to make matters worse, proclaim that if there is no measureable data, that it cannot exist. This is not scientific at all. In the case of blind testing, it is used as a "scare tactic" to make audiophiles think that if they were actually to be placed in a blind testing position, that they would fail miserably, and their entire system is nothing more than an imaginary delusion.

Why would people do this? It serves no purpose but one. To justify their decisions to not make high end purchases, by concocting this story. It truly amazes me that this gets any mileage at all. It is so patently absurd. It flies in the face of what is obvious. They want you to disregard what you experience, and replace it with a reliance on some number, or worse yet, replace it with another school of thought, based upon the "fear" that they try to instill in you that you could not "hold up" under the scrutiny of this blind testing. I find this very objectionable. For someone to imply to me, that I am incapable of making my own decisions based on my own experience, is not going to fly too far with me.

You can all make up your own minds on this matter. And anyone who has the schools of thought that I negatively referred to has every right to maintain them. They have every right to express them. As do I. It is up to you to separate the wheat from the chaff. And the decisions you make will ultimately define what your system sounds like.