Why is Double Blind Testing Controversial?

We've covered this before. As such, all i'll add to this thread is that Craig and Albert cracked me up. If DBT threads are going to be good for laughs like that, bring them on : ) Sean
>

The use of blind and double-blind procedures presumes one is employing the logic of hypothesis testing. That is, that there is a null hypothesis (i.e., that there are no differences between two treatments—in this case, two sets of interconnects) and an alternate hypothesis (there is indeed a difference). Experimenters are more than experimental custodians. Their biases and expectations can profoundly influence a study. To the extent that all people (including experimenters) have biases, one would double-blind the treatments to reduce among other things "experimenter effects." It’s surprisingly easy for an experimenter to influence a study (e.g., Stanley Milgram’s famous obedience studies). It is also easy for other participants (formerly known as “subjects”) to influence each other (e.g., Ash’s line judgment experiments where participants tended to agree with Ash’s confederates that clearly dissimilar lines were the same).

There is a famous researcher/psychologist/statistician by the name of Robert Rosenthal who once told his students that he had obtained two breeds of rats from another famous researcher. One type of rat was called “maze smart” and the other was “maze dull.” Dr. Rosenthal asked the students to teach these rats to run though mazes (ah, the power of cheese). After a few weeks or so the students were asked to show off their rats’ maze prowess (as it were). The “maze smart” rats performed significantly better than their “dumb” counterparts. The kicker here is that the rats were OF THE SAME SPECIES. One cannot infer that the students intentionally influenced the training, but it most certainly was measurable. Moreover, when the experimenter bias was measured it turned out that the “smart" rats owners had "imparted" a greater positive measurement bias than did the “dumb” rats owners negative measurement bias.

There are probably much better examples than these, but I’m in a hurry to go downstairs for dinner :-) so I’ll wrap this up soon.

Something else to consider is that “different” does not mean “better.” People’s ability to remember sounds and colors varies greatly but rarely is the memory accurate after a short decay period. With audio equipment evaluation, it tends to result in a bias for a certain “sound” regardless of whether or not that sound is authentic. When it comes to making a decision as to whether one component is better than another, it probably makes the most sense to have a reference. In the case of audio, I’d say that reference should be THE REAL THING. It’s not practical to have live orchestra tag along on equipment tests but it doesn’t hurt to keep that in mind. Some people go on and on about how they prefer one cable to another because their favorite is “warm” or whatever. Real sounds from an orchestra or a band are not necessarily “warm.”

All that said, if one believes that a $6,000 set of interconnects sounds better (they just might sound *different*) than a $70 pair then let ‘em. The more expensive cable might even sound closer to reality. One would hope that the more expensive cables aren’t just mostly cosmetics and markup.

--Paul

p.s. and yes, spending time with a set of cables or anything else in the system is a great way to know if one really likes the sound. On a marginally related note, a friend of mine once said “I’ve never owed a handheld device that I liked after having it for a week.”

Well said Bomarc. I'll go a little further along that road. Most people who argue in favor of DBT's in these threads imply that audiophiles don't want to discover that they can't tell one thing from another. I think audiophiles believe that DBT's are not good because that's what they've been told by the buff mag writers (aka "reviewers," but really just story tellers), who don't want to be exposed. You see, Sean, in a DBT, Mikey can't tell those preamps apart either.

There are reviewers with real scientific credentials and experience who advocate and use DBT's, but not many.

They don't have to be short. You can listen to whole pieces of music. And you can start by familiarizing yourself with the components under test until you are sure of what it is that makes them sound different, then try to tell them apart blind, not trying to prove there is no difference, but to confirm your hypothesis that they are different. In home, one obvious difficulty is exact matching of spls.

I have done blind testing myself, on many occasions, as I have mentioned before, in an earlier thread on this subject. When I worked in a high-end audio shop, when there were slow times, when nobody was in the store. Myself and the other employees would do it for fun. It was like a contest. We would set up equipment when 1 person was out of the room, blindfold him, and bring him in to listen. Then we would see how many components in the system he could identify. Everyone was surprisingly good at this, but nobody was 100%. Of course, we were going for multiple components at the same time. With only 1 component in a known system, we all could indentify it virtually every time. Now, we were very familiar with the sound of all of the gear we tested this way, so that made it easier. But, it wasn't that tough to do. It was in a fairly relaxed environment with no real pressure. Just fun.

So, I am not afraid of what would be the result of these tests, in my case. I cannot say I could be 100% in any test, but I am sure that I can be accurate enough to satisfy any tester that I am not guessing. Even with unknown equipment, I can identify differences accurately, in a short time span. I think many people can. I also think that some people cannot. This would not be the lack of difference in equipment, but a difference in people. It is not scientific to label the results as "no difference" in equipment, when it is an inability on the part of the listener to percieve the difference. Many people think a car radio sounds like a good audio system. Clearly they are not listening to the entire presentation, but just the superficial aspects of the sound. They are not aware of how to listen for "differences" in the sounds of items. They are simply "superficial listeners". This type of listener cannot be relied upon to discern differences in equipment. They listen as "background music". A person who knows what to listen for, will easily tell the differences in components.

And again, I will state that I think that this whole issue is a "red herring", that is brought up by those who have convinced themselves that there is no difference in equipment, or cables, or whatever, in order to satisfy their own minds that they do not have to spend money on such things, and still have the best. In my opinion, the desire to save money is stronger, even in the most ardent audiophile, than the desire to spend a lot of money fruitlessly. I would say that I, and everyone else here, would rather spend less, to get the same performance, if we could do it. And there are ways to do it. The most expensive item is not always the best sounding item. But sometimes it is. And making claims that "in a real scientific environment we couldn't tell the difference" is simply a diversion.

In virtually every case where this is mentioned, it is in a context of "wasting money on snake oil". This is the crux of the matter. It is a matter of justifying expenditures.

So now, we have a whole different scenario. Now we have a subject brought up, which is the real center of the matter, which is,"Is it worth it to me to spend alot of money to get a certain level of performance increase, when I am not sure of the outcome?"

There's such a plethora of cables and components, and claims of grandeur, that some people cannot cope with it, and punt. Instead they divert their attentions to claiming that there is no difference between these things, and stick to it. They use a "scientific" argument that they know nobody is going to use, to back-up their idea.

In some cases, they will be right, and there will be no noticeable differences in some items. This only adds to the confusion, because it lends credence to the extrapolation that there is no difference in anything.

If these folks want to believe that, then that is their prerogative, and they are entitled to believe that. But to tell the rest of us that we are "deluded" by our unquenchable desires to spend money, that we would manufacture these differences in our heads, just so we can spend more money, is not passing the "smell" test.

I'll use myself as an example. I "claim" that I can hear differences. But I don't want to spend any more than I have to, in order to get the sound quality I want. I can honestly say that I may hear very small, or even no, sonic difference between certain items that have significantly different prices. When I arrive at a situation like that, I call the lower priced item a "bargain", and I buy that one. Or I may say,"I like that one better, but the small difference is not worth the extra money to me." Isn't that a more "measured" approach than what we are seeing here? That is what everyone else does. When my girlfriend goes shopping for a dress, and she sees one she likes for alot less than a similar one, she remarks on what a "good deal" she got. She doesn't come home proclaiming that "there is no difference between dresses" so she bought the cheapest one in the store. After all, they all perform the same function of a covering, right?

All of this is much ado about nothing. People will buy what they want. Then they will justify it to themselves, or others. That is life.

Twl, thanks for taking the time to clearly articulate what is obvious to many of us, some products are superior to others and the decision is usually, whether the improvement is worth the cost. Also, thank God, there ARE superior products that cost less than the competion. Then there is also the issue of synergy and the proper matching of components. Judging from your system profile, it's obvious you are doing your homework. God bless.

TRUTH is controversial, to be sure - follow this link Audio Asylum post concerning hyperbole