In defense of ABX testing


We Audiophiles need to get ourselves out of the stoneage, reject mythology, and say goodbye to superstition. Especially the reviewers, who do us a disservice by endlessly writing articles claiming the latest tweak or gadget revolutionized the sound of their system. Likewise, any reviewer who claims that ABX testing is not applicable to high end audio needs to find a new career path. Like anything, there is a right way and many wrong ways. Hail Science!

Here's an interesting thread on the hydrogenaudio website:

http://www.hydrogenaud.io/forums/index.php?showtopic=108062

This caught my eye in particular:

"The problem with sighted evaluations is very visible in consumer high end audio, where all sorts of very poorly trained listeners claim that they have heard differences that, in technical terms are impossibly small or non existent.

The corresponding problem is that blind tests deal with this problem of false positives very effectively, but can easily produce false negatives."
psag
"LOL, but doctors in that time period of history didn't know any better."

That was my point. The only thing they knew for sure about blood was that if you loose too much of it, you die. Armed with only that info, I stand by my statement. lol. Maybe there was some logic to it, but I don't see. I guess its possible that every time a doctor saw a bleeding, injured person, he thought that was the body getting rid of some excess blood. Why blame it on the arrow stuck in the arm.

About the rest of your post, my comments on all this, in context, is in reference to the thread the OP mentioned on Hydrogen Audio. The complaint was that reviewers were listening to audio components and then basing their review on what they heard, and not doing any type of scientific listening tests. They've complaining about this very thing for years. My comment was that if these types of test are so damn important, then just do them already. Even if its just a few tests just to show us all how to do it. Instead, its just year after year complaining, and they never do anything. That said, if they can come up with some kind of useful tool to help better evaluate audio equipment, I know I would be interested in seeing it. Why not? My personal opinion is that they don't have the guts to do anything they talk about. There's always the chance they would be wrong. They have too much invested in the argument.

"In ABX testing, the definition of the cumulative response is trivial: can reliable hear a difference or not. But because the stimulus is likely to be imprecisely defined, absence of reliably making the distinction does not mean there is no difference."

I agree with that. My view is that you would have to tell the test subjects what they're listening for. There's really no way around it, they have to know. To offset results that are not accurate, you can increase the number of tests, or try's, each subject takes. So, for example, if you were trying to test to see if a difference can be heard between a silver cable and a copper cable, all other things being equal, maybe have them listen to 50 or 100 samples. Maybe they can get lucky and guess correctly for 5 or 10, but 100 is highly unlikely. Not only that, under the same scenario, you can tell them exactly what they should be listening for. If there is really no difference to be heard, over time/individual tries, the test subjects will have to trend towards a 50/50 split. It won't matter what they think, or know they can hear.

To offset results that are not accurate, you can increase the number of tests, or try's, each subject takes. So, for example, if you were trying to test to see if a difference can be heard between a silver cable and a copper cable, all other things being equal, maybe have them listen to 50 or 100 samples. Maybe they can get lucky and guess correctly for 5 or 10, but 100 is highly unlikely.
01-21-15: Zd542
50 to 100 samples you say? Why not make it 100 to 200? Over how many years do you expect your ABX listening test experiment to take?

Just curious have you ever A/B compared 2 or 3 cables to one another? More than 3 or 4 at a time? Could you hear audible differences between the cables?

50 to 100 samples.... Do you believe there are people in the world that can tell which key of a piano is struck on a tuned grand piano in a blind test? Do you think their brain learned the sound of each key in the span of a week or so, or even a few months or so? How about in a year?
.

This bleeding thing is a bit off topic, but since it keeps coming up, I thought I'd clarify that issue. (because I have no good opinion on the ABX thing)

During ancient and medieval times doctors believed in "the humor theory". It's pretty complicated and a bit funny by modern standards, but the short explanation is that the blood carries liquids called "humors". A sick person has bad humors in their blood and you have to let it out. Thus "blood letting". A person in good humors is healthy.

This theory died out in the 1700's and early 1800's, when the new and wonderful "germ theory" of disease became more popular.

For medical testing, doctors would normally draw a sample of the blood and check the color and taste. Good humors taste good, I imagine. =-}

http://en.wikipedia.org/wiki/Humorism

I read a lot of ancient writings...
"50 to 100 samples you say? Why not make it 100 to 200? Over how many years do you expect your ABX listening test experiment to take?

Just curious have you ever A/B compared 2 or 3 cables to one another? More than 3 or 4 at a time? Could you hear audible differences between the cables?"

Yes, I have. I did an experiment a few years ago and compared AQ Cheetah IC's to a pair of AQ Panther IC's. Both cables are identical except for the conductors themselves. One silver, one copper. The goal was to see of a difference could be heard between the 2 metals, and nothing else. It wasn't about what one sounded better, just if there was a difference. There was 4 of us took the test and we listened to 100 samples of a 10 second audio clip that took around 30-40 minutes for each of us.

"50 to 100 samples.... Do you believe there are people in the world that can tell which key of a piano is struck on a tuned grand piano in a blind test? Do you think their brain learned the sound of each key in the span of a week or so, or even a few months or so? How about in a year? "

Actually yes, and I can prove it. My brother has something called perfect pitch. He can tell with 100% accuracy what any note or cord played on any instrument is, and if its in tune or not. I don't have it myself, but if you have ever played an instrument, you can develop something called relative pitch. Its not as good as perfect pitch, but its a skill that can be learned. For me, I needed to develop the skill somewhat when I played drums in school. If you have ever seen kettle drums or tympani, they have to be tuned to a certain note when you play them. That is what the food pedal is for, it sets tension on the drum head. Anyway, you have to be able to set the drums to different notes while the band is playing. To do this, you tap it very lightly (because the band is playing), and hopefully tune it to the right note before you need to play it. Its not an easy thing to do, but its a skill that can be learned.
Yes, I have. I did an experiment a few years ago and compared AQ Cheetah IC's to a pair of AQ Panther IC's. Both cables are identical except for the conductors themselves. One silver, one copper. The goal was to see of a difference could be heard between the 2 metals, and nothing else. It wasn't about what one sounded better, just if there was a difference. There was 4 of us took the test and we listened to 100 samples of a 10 second audio clip that took around 30-40 minutes for each of us.
01-22-15: Zd542

There was 4 of us took the test and we listened to 100 samples of a 10 second audio clip that took around 30-40 minutes for each of us.
01-22-15: Zd542
And the findings, results, of the listening test?

Just curious what is behind your thinking of needing so many samples for your listening test?
.