Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera


Folks-

Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.

In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.

As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:

1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;

2.  Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars; 

3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?

I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?

Thanks for listening to my ranting...Gerry 
128x128Ag insider logo xs@2xgerryah930

Showing 2 responses by teo_audio

Reviewers, and audio magazines in general, are built around the idea of only reviewing gear they know to sound good from their own listening at shows and at various other locations and locales.

It would be economic suicide for any published magazine to review stinkers, and trash them.

It’s a very simple bit of economics as tied to the human condition.

Stereophile has said this flat out, and said it fairly often.

No negative reviews as negative reviews are never allowed to have a chance to form in the first place. Everything reviewed is filtered on multiple levels, before the given review is published.

The best you’ll get (toward the idea of a negative review) is to have the given item ’damned with faint praise’, if the selected item they thought would sound great and review well, does not quite meet that high quality minimum.

It does not mean that smaller publications or non paying (no advertisers and no paying to read the reviews) scenarios where equipment trashing is done...are somehow more honest, it’s just that there is ZERO, I repeat ZERO economic position available for anyone trying to run a company (of a publishing nature) that produces only (or high levels of) negative reviews of commercial products that are in this area of the economy.

Importantly, Stereophile additionally states -far more often than they speak on only publishing good reviews-..that if they don’t review something...that does not necessarily mean it is bad or not good. There’s lots of gear out there and they can only review a very very small percentage of it. They state this openly, regularly in the magazine, about 2-3 times per year, one in every instance of ’recommended components’, and in other places and ways.

There is a film called the ’American ruling class’, where the ’fictional’ main character in the ’pseudo-documentary’ interviews the actual and real ownership and publisher of the NYT (at the time). (everyone interviewed in the film is for real) He asks a real question and gets a real answer. The editor says they only publish editorial that is favorable and in line with their advertisers and that’s the way it is done. Period. Any other way and the newspaper would simply not exist. The end.

Now, let’s talk about the US medical research INDUSTRY, that + trillion dollar medical/pharmaceutical juggernaut’s ’factualization’ arm.

Additionally, if the audio world was about 2-3-4 times bigger than it is, then publications could get into the idea of a more ’biased toward negative’ review standard. Then a world with audio magazines with some negative reviews could stand and survive.

Also audio is specifically NOT a ’first past the physical post’ type endeavor. Where everything is in full black and white factual norms. We still don’t have a full and correct handle on how people hear and understand audio quality, nor how distortions and micro-distortions interfere and integrate with our not yet understood hearing functions.

We can send out the exact same audio cable to four different people and get back four very different assessments of how the cable sounds.

Eg, the post right above this one illustrates the complexity of the audio world and marketplace, quite well. It’s a mess that has no zeroing point or norms that can be discerned beyond a basic direction in desires. Sometimes.

We’re dealing with mental wiring and associated system which are all INDIVIDUAL in end points. There is NO capacity to perfect anything on paper or in the so called real world of audio.

If you want to normalize all individualism (hearing and associated wiring, intelligence, etc) down into a black and white repeating standard and give away all the things that make you--you ..and kill all of humanity down to a repeating widget that comes off an assembly line of repeating identical items..and do the same for everything you know in the human world ---basically kill off humanity and the world into a dead, non-living nothingness..THEN.. you can have your black and white perfection in high end audio.
It's an advertising/awareness thing. That which is reviewed garners sales, pure and simple.