Tbg: If these tests didn't yield positive results, they'd be useless for research. Just because they don't yield positive results when you want them to doesn't make them invalid. A good example of a mix of positive and negative tests is the ABX cable tests that Stereo Review did more than 20 years ago. Of the 6 comparisons they did, 5 had positive results; only 1 was negative. (The one negative, however, used similar cables and had subjects listen to music rather than noise. In most of the other 5 cases, the measured differences were much greater; in one, they listened to noise rather than music--it's easier to hear level and frequency differences with full-spectrum noise than with music.)
I presumed you knew statistics. 15 out of 20 is the 95% confidence level, which means that we can be 95% sure that the listener really heard a difference, and wasn't just guessing lucky. The 95% threshold is a reasonable one in this case.
I suspect the tests you did involved multiple listeners listening at the same time. It's better to use one subject at a time, and to let the subject control the switching. But the Stereo Review tests used multiple listeners at once, and got plenty of positive results. Subjectivists often object that ABX tests use quick switching between components, but there's solid research showing that this approach actually works better--it's easier to hear differences when you can switch immediately between the two. I know subjectivist audiophiles consider that heresy, but the research is pretty clear.
Some manufacturers use DBTs, others don't. It makes no sense for components where differences are undeniable (microphones, turntables, cartridges, and speakers are good examples). As for "voicing" of amps and cables, people who claim to do that without DBTs are either fooling themselves or trying to fool you.
Almost nobody has a preconceived notion that things sound the same. Many objectivists used to be subjectivists till they started looking into things, and perhaps did some testing of their own.
As for reviews, a high-end magazine that used DBTs couldn't survive. Advertisers would pull out, and readers would revolt. Better to give the people what they want.
I presumed you knew statistics. 15 out of 20 is the 95% confidence level, which means that we can be 95% sure that the listener really heard a difference, and wasn't just guessing lucky. The 95% threshold is a reasonable one in this case.
I suspect the tests you did involved multiple listeners listening at the same time. It's better to use one subject at a time, and to let the subject control the switching. But the Stereo Review tests used multiple listeners at once, and got plenty of positive results. Subjectivists often object that ABX tests use quick switching between components, but there's solid research showing that this approach actually works better--it's easier to hear differences when you can switch immediately between the two. I know subjectivist audiophiles consider that heresy, but the research is pretty clear.
Some manufacturers use DBTs, others don't. It makes no sense for components where differences are undeniable (microphones, turntables, cartridges, and speakers are good examples). As for "voicing" of amps and cables, people who claim to do that without DBTs are either fooling themselves or trying to fool you.
Almost nobody has a preconceived notion that things sound the same. Many objectivists used to be subjectivists till they started looking into things, and perhaps did some testing of their own.
As for reviews, a high-end magazine that used DBTs couldn't survive. Advertisers would pull out, and readers would revolt. Better to give the people what they want.