Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera
Folks-
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
not sure you’ve been doing decent research then, the Rossini and Vivaldi have been reviewed at length on every mag and the network box is now getting a bunch of reviews
I would argue that in fact DCS is more reviewed than the others you mention
DCS do however try to manage the second hand market in a Ferrari like manner and hence you find factory approved used items from them which come with after sales service and software upgrades as opposed to the private sale non supported ones, obviously the latter are cheaper but the buyer has the choice of which way to go
As to the reviewers they all have biases which are not hard to spot once you spend time reading and listening. With so many on line opinions it’s not too hard these days to find someone who’s ears you trust
I think your conclusions are a bit flawed. I can agree that TAS and 6 Moons are a pay to play, Stereophile is not in that category. John Atkinson, the Editor of Stereophile is a man of high integrity. I have met him, talked personally with him and he really adheres to the values that Gordon Holt put down for Stereophile in its infancy. I know of one reviewer, Myles Astor who is honest as the day is long and would never be biased towards a certain manufacturer.
The SA10 only came out a year ago, so it is possible that the mainstream magazines will review them. There really haven't been too many cd/sacd players reviewed over the past couple of years as most people are going for music servers and streamers instead.
I guess I am asking about the degree of bias in these reviews, to what
extent are products reviews influenced by the manufacturers and dealers,
and where is the objectivity in this domain?
In a nutshell, yes, most product reviews aren't worth the paper they are printed on. I let all of my glossy magazine subscriptions lapse many years ago. The only information to be found in these "reviews", or "ads" whatever you want to call them are photos and specifications.
These magazines/websites are part of the marketing branch of high end audio. So yes, they would rather write no review at all than a negative review. They might bash one of the smaller, non-paying players once in a while just for appearances.
Make no mistake, all reviewers are HEAVILY biased towards the manufacturers. Their job is to push product, an extension of the marketing department if you will
Some companies frown on reviews. Some companies regularly try to get magazines and reviewers to do their products. No site just BUYS what they want to review (though some reviews are certainly done by owners of said equipment) So some companies do not get reviewed.. They just will not loan out equipment for reviews. Other companies flood the system with equipment to be reviewed.
There are two factors here. The speed with which products get reviewed. (Time of product launch to time of review publication.) And whether a brand/product gets reviewed at all.
The frequency with which Musical Fidelity appeared in the pages of Stereophile a few years ago was a running joke in the audiophile community.
Meanwhile, try searching for reviews of PMC speakers in those same pages.
Partly it has to do with company policy, partly how well a foreign company is represented in the US and what kind of PR campaign they sponsor, partly on reviewers' individual preferences, partly on kismet, and probably 3 or 4 other factors as well.
In short, it's hard to read anything into the patchy coverage other than it's undeniably patchy.
Along with biased reviews in this forum and others, you will often find the warning that to determine whether a piece of audio gear is right for you, you must listen to it in your own system, preferably for an extended period of time. That is the bottom line in this pursuit.
I know of no way around it. Audio is an extremely subjective pursuit and there are no two combinations of components, rooms, hearing and tastes that are the same. So the words of others can be taken as no more than suggestions as to what gear you might want to research. Listening at the dealer or an audio show cannot assure you of what a component will sound like in your system.
The only alternative that I know of is getting to know a good audio dealer and trusting him or her. This does work for some but again it’s not a certainty.
1. there are symbiotic relationships b/w the manufacturers and the rags 2. there are symbiotic relationships b/w the manufacturer’s and the readers of the rag, and 3. there are symbiotic relationships b/c the rags and the readers. = clusterf*ck.
The truth lies somewhere in the middle of this relationship triad, which is why we still speculate about it ..... lol.
Reading reviews in the audiophile magazines should be considered information gathering, or general entertainment, and nothing like reading a scientific peer-reviewed journal. I have read lots of reviews; I haven't read any that concluded with something like, "Save your money, it's a bust." And surely, out of hundreds of reviews, based on statistical probabilities, it seems that, if reviews were being objective and transparent, that should have been the analysis on some piece of equipment. I have found the best source for reviews is Audiogon, when actual owners or former owners describe their experiences with different manufacturers and equipment models. The dealers on Audiogon need to clearly identify themselves since they clearly have a sales objective in mind, which by the way, I am all for provided they tell the truth about what they know and don't know concerning their equipment.
My thanks to all of you for the responses, since I am naive in this domain, and expected audiophile magazines might actually provide unbiased product reviews.
One last tidbit of information, since we routinely perform high-throughput data analysis on billions of sequence reads - the name "Diana Krall" is significantly over-represented in the discussions of musical content between 2015-2017, both on Audiogon and in TA and Stereophile magazines. I guess she is a popular artist and was also once a student at Berklee in Boston (like I was, a long time ago!) - Rip Van Winkle, aka Gerry.
I've noticed a major change in the quality of reviews in TAS since HP departed for the Audio Valhalla in the Sky. Seemingly, every item in that publication is "superior" in some way or another. Perhaps that's true, but the wry comments, jabs and jibes I so appreciated from HP and his then peers are all gone.
To talk about "objectivity" with respect to audiophile opinions is far fetched ,but the uniformly adulatory tenor of TAS reviews these days calls motives into question. The overwhelming amount of advertising between the covers may indicate a problem in that regard.
Stereophile seems a lot better than TAS. I prefer it anyway.
Owner reviews may be tainted: who wants to readily admit to having purchased a stinker? I've bought a couple, but was hesitant to admit the mistake when I actually had the bothersome component in my own system. This forum is always interesting and informative, despite potential ownership conflicts.
i can only speak from personal experience. My products have received good reviews in Stereophile without advertising with them. Conversely, I am unable to get my products reviewed by The Absolute Sound.
The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base).
roberjerman The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base).
There is a fine line between “honest and forthright assessment” and closed mindedness and self aggrandizement.
“What I have learned after six decades in audio.” - Peter Aczel
@gerryah930 - have you considered that products that are reviewed are actually pretty darn good?
And perhaps - that's why the review is (almost) always favorable?
After reading lots of reviews, I have found the I can generally tell when a product may be falling a little short of the reviewers expectation - they "generally' start to nit-pick.
And the opposite occurs also - if the product is fantastic the reviewer goes out of their war to extol the virtues of a component.
Sometimes "numbers" are not enough - you just have to learn to "read between the lines".
As for Diana Krall... - a great many of her albums are a product of exemplary sound engineering/recording and final mixing and more often than not lacks that "wall of sound", which allows a reviewer to focus on the details in the recording. - As opposed to another delightful singer, Norah Jones, who's albums are also very well engineered but are quite varied in their production and can sometimes lack the more natural tones of a Diana Krall album.
These observations are result of hundreds of hours of listening I my self have conducted whilst reviewing (as an amateur) a small number of products.
Unlike many, my reviews were not paid for, but I was enthused by the outstanding abilities of those products.
I'm not defending professional reviewers - but there is always another side to the coin
Having experienced just how difficult it is to write a review that is engaging, informative, concise, insightful and more importantly useful to the reader, I think it is important for people to consider some of the "intangibles" at play.
BTW- do you collect stats on how many products are returned without a review?
Another reason why reviews are almost always favorable is because the magazine and or reviewer doesn’t really want to kill a small company, especially in view of the fact that sometimes things happen, you know, such as damage during shipment to the reviewer, operator error, failure to follow instructions, that sort of thing. Also, sometimes there are trade-offs for the device under review which would be spelled out in the review. For example, cost, size, complexity, peculiarities, etc. Finally, reviewers and magazines are not all the same. Reviewers and magazines can certainly have unique or different approaches to conducting reviews. It certainly wouldn’t be unusual to see products reviewed that are trending or that the reviewer and or magazine would suspect of having interest for their readers.
I am not trying to insight a so-called flame war here, but I have to agree with willemj.
If audiophile magazines are "hobbyist" journals, then I would stop the product reviewers from including what appear to be plots of measurements of impedance versus phase, spectral decay, and amplitude versus frequency. I enjoy looking at these graphs but in almost all cases, the reviewers will only hint at issues (e.g., speaker impedance spikes or drops), but the review will end with statements such as "best speakers I have heard, so I bought the review pair" and so forth. I can only believe that there is some economic incentive that is not disclosed.
Comparisons to a hobby and a profession:
1) I used to be involved in Porsche club racing, so I would read porschephile and generic car magazines. This would certainly be considered a wealthy person's hobby, but the car magazines would provide head-to-head comparisons of performance between brands and would make disparaging statements about cars that were poor performers. I have never read head-to-head comparisons in a given category of hardware in any high-end audiophile magazine (e.g., different speaker brands in the same room at the same time, driven by the same audio hardware);
2) In biomedical research, there are many problems and challenges, but to suggest that our collective intent is to do anything but try to arrive at so-called "ground truth" is not accurate. That does not mean there is no publication bias (see [1]), but at least there is recognition that this issue exists, and many attempts to deal with the problem.
You missed the elephant in the room. Ordinary bits of wire. Not long ago, bits of wire were correctly viewed as bits of wire. Today gargantuan sums are paid for ordinary wires in the misguided belief that ordinary bits of wire do more than anything else in the audiophile’s setup. The fact is they don’t. What they do is lead to massive profits and mark ups for something cheap with no substantive benefit to the user. Audio reviewers and press have jumped on the band wagon (incentivized by advertising) and stores have jumped on the opportunity to sell additional cable trinkets and jewelry at extremely high mark ups to buyers.
The advertising lie is that “everything affects the sound” and that “exceptional ears and exceptional systems will require these trinkets to sound their best”. A story just like the emperor’s new clothes which audiofools eat up readily. Smugly the audiofools look down on anyone with less than $1000 bits of wire and snear at those that don’t hear a difference as being deaf or not having a high end system. The whole domain is like high end skin care - packed with excruciatingly expensive products that do nothing more than appease the ego of the vain. The bits of wire are made of fairy dust meme also undermines real audio innovation.
Are you at Johns Hopkins? My daughter is studying there currently.
Reviewers, and audio magazines in general, are built around the idea of only reviewing gear they know to sound good from their own listening at shows and at various other locations and locales.
It would be economic suicide for any published magazine to review stinkers, and trash them.
It’s a very simple bit of economics as tied to the human condition.
Stereophile has said this flat out, and said it fairly often.
No negative reviews as negative reviews are never allowed to have a chance to form in the first place. Everything reviewed is filtered on multiple levels, before the given review is published.
The best you’ll get (toward the idea of a negative review) is to have the given item ’damned with faint praise’, if the selected item they thought would sound great and review well, does not quite meet that high quality minimum.
It does not mean that smaller publications or non paying (no advertisers and no paying to read the reviews) scenarios where equipment trashing is done...are somehow more honest, it’s just that there is ZERO, I repeat ZERO economic position available for anyone trying to run a company (of a publishing nature) that produces only (or high levels of) negative reviews of commercial products that are in this area of the economy.
Importantly, Stereophile additionally states -far more often than they speak on only publishing good reviews-..that if they don’t review something...that does not necessarily mean it is bad or not good. There’s lots of gear out there and they can only review a very very small percentage of it. They state this openly, regularly in the magazine, about 2-3 times per year, one in every instance of ’recommended components’, and in other places and ways.
There is a film called the ’American ruling class’, where the ’fictional’ main character in the ’pseudo-documentary’ interviews the actual and real ownership and publisher of the NYT (at the time). (everyone interviewed in the film is for real) He asks a real question and gets a real answer. The editor says they only publish editorial that is favorable and in line with their advertisers and that’s the way it is done. Period. Any other way and the newspaper would simply not exist. The end.
Now, let’s talk about the US medical research INDUSTRY, that + trillion dollar medical/pharmaceutical juggernaut’s ’factualization’ arm.
Additionally, if the audio world was about 2-3-4 times bigger than it is, then publications could get into the idea of a more ’biased toward negative’ review standard. Then a world with audio magazines with some negative reviews could stand and survive.
Also audio is specifically NOT a ’first past the physical post’ type endeavor. Where everything is in full black and white factual norms. We still don’t have a full and correct handle on how people hear and understand audio quality, nor how distortions and micro-distortions interfere and integrate with our not yet understood hearing functions.
We can send out the exact same audio cable to four different people and get back four very different assessments of how the cable sounds.
Eg, the post right above this one illustrates the complexity of the audio world and marketplace, quite well. It’s a mess that has no zeroing point or norms that can be discerned beyond a basic direction in desires. Sometimes.
We’re dealing with mental wiring and associated system which are all INDIVIDUAL in end points. There is NO capacity to perfect anything on paper or in the so called real world of audio.
If you want to normalize all individualism (hearing and associated wiring, intelligence, etc) down into a black and white repeating standard and give away all the things that make you--you ..and kill all of humanity down to a repeating widget that comes off an assembly line of repeating identical items..and do the same for everything you know in the human world ---basically kill off humanity and the world into a dead, non-living nothingness..THEN.. you can have your black and white perfection in high end audio.
It’s not really an elephant in the room. 🐘 It’s a nothing burger. 🍔 And there’s nothing ordinary at all about most high end cables, from the purity and crystal structure of the metal, to the controlled directionality of the cable or power cord, to the dielectric material, to the method of welding and particulars of the connectors, geometry of the conductors, etc. The longer this cable debate goes on the weirder it gets. 😳
“The listening panel were all well known and experienced listeners.”
I’m sure. As I’ve said repeatedly you don’t have to look too far to find a test that’s inconclusive or even negative. I suggest throwing the whole test out.
The point is that frequency of coverage, or indeed coverage at all, in the mainstream mags is no indication of anything really. Certainly not of a component's quality or popularity.
No negative reviews, in my view, means no other review can be trusted. I know collusion when I see it and that is exactly what stereo magazines and manufactures have done for years. Any objective scientific review will have negatives. I should know I have edited scientific publications for years, where lots of critique can be found.
So, caveat emptor when reading about and buying stereo gear. Actually I prefer to hear form actual users on Audiogon about specific products. They are ofter much more critical.
Finally there are a few really good companies who tell the truth. Example, Heron Audio. How many reviews have you seen on their products. My guess, they are outside the "Club".
browndt No negative reviews, in my view, means no other review can be trusted. I know collusion when I see it and that is exactly what stereo magazines and manufactures have done for years. Any objective scientific review will have negatives. I should know I have edited scientific publications for years, where lots of critique can be found.
>>>>Actually, many if not most reviews have always included negative aspects, if any, as well as positive ones of the device under test. For example frequency response curves and radiation patterns, distortions of various types. The idea they would Gild the Lilly 🌷doesn’t really hold water IMHO. In fact, one reason they provide negative aspects is too prevent or dispel the impression that the reviewer or magazine is biased or that there is any sort of collusion. They aren’t stupid.
Good read, so far....and that despite the Cable Drifting : )
I'll add to @williewonka 's point:
...have you considered that products that are reviewed are actually pretty darn good...
In my view, most gear that makes it to market is very good, especially when considered on it's own. That in itself eliminates highly 'negative' reviews.
I'm searching my memory for a component that I've had in system that I would consider a turd on arrival and which remained so.....
This includes speakers which are quite variable (as you know) and some tube components (early in my journey) that were overly euphonic. For both component classes it was not that they were poor performing but rather that I didn't care for their sound.
For me, reviewers offer a great service that I am grateful for regardless of bias or vested interests. It's the way of the world, even the medical and science ones. Reviewers I have personally interacted with have been no different than my audio friends and acquaintances. A pleasure.
I can also understand, if that's how one chooses to roll, ignoring their reviews. I, however, find the toxic mudslinging unseemly.
Reviews for audio anything should be taken with a grain of salt. It is up to the individual to realize that a certain amount of product bias exists in all reviewers skewed toward their benefactors or personal biases. Their publication can be useful for a buyer to determine which direction he may want to try but should never be used as the gospel truth. Audio is a very personal matter and a 100000 dollar system is no better suited to propel the individual human into audio nirvana than a 1000 dollar system. The human condition determines when nirvana has been obtained, not reviews. All of us in these forums have reached nirvana at some time of our continuing search for a better nirvana without reviews simply by trial and error. We still do it. Just look at all the equipment ads. We are our own very best reviewers. All different and yet the same. One persons failure to enter their nirvana opens the gates to nirvana for someone else. Isn't it wonderful?
@browndt Absolutely! There are numerous manufacturers that for one reason or another are not members of "the club". They good ones soldier on, on the strength of their product, word of mouth, etc., without a boost from S-phile or AS.
I think this sounds ridiculous. To the best of my knowledge, editors request products. A good deal of the time those requests are made to distributors and dealers, not manufacturers. A lot of these guys have explained their selection process.
Many times, a reviewer asks to review a component that he (or an editor) heard at a show and with which he was favorably impressed. Although reviewers at the 2 big magazines rarely trash a product, a good review usually informs the reader about the qualities of the component and the characteristics of its sound to the reviewer's ears. It's up to us to listen to the components which appear from a review to meet our preferences and then go listen for ourselves.
I enjoyed your opening query. I do not view your illustrations as ranting in the least. I, too, am interested in the Marantz SA-10 spinner and it has been a very slow roll-out into the Audio press. This is odd indeed, as Marantz, tends to receive favorable coverage on its Reference products. Still, being hopeful that Stereophile, TAS or one of the other top publications will test and review this spinner in 2018. I am in the process of finding and auditioning. Based on your locale maybe you can do the same.
In general, the audio press is placing more emphasis on streamed audio, which they see as the future, and less attention is being paid to disc spinners of all types. Those of us with considerable digital disc libraries will likely find it increasingly difficult to access reviews of disc players.😟
Good question. There's little to none. My conclusion is if it's not blind, they didn't hear it. The only time I would pay any attention whatsoever to a subjective review is if that person continually came to the same results as I.
Regarding #2, you'll usually see an influx of items for sale when the next "hot" thing comes out. The herd is always chasing the newest rabbit. Doesn't necessarily mean people are dumping them for cause.
Regarding #3, there are plenty of technical articles showing why MQA is just a money-making scheme.
BTW, there have been blind tests on SACD, including an AES paper. I think you can guess what the results were. However, there was an audible difference in the noise floor when there was no signal and high system gain at a level where music would have been "unpleasantly loud".
After reading both regularly for about three years now, my feeling is that Stereophile is more objective and professional in reviewing equipment than The Absolute Sound. Although the latter magazine occasionally publishes well-written reviews, sometimes it is impossible to tell whether the price quoted for a speaker is for one speaker or a pair. Stereophile is not uniformly positive, and points out when the item being reviewed doesn't work, or stops working (or blows up), and when it provides no discernible improvement in sound quality. They also give the manufacturer a forum to respond to reviews.
I have found that both magazines have helped me get educated on new stuff in the hobby after a 20-year hiatus. And in my opinion the best writers in the audiophile universe (Art Dudley, Herb Reichert and Michael Fremer) are all at Stereophile.
No one would pay for, or bother to return to read, about bad equipment that cannot cut it. The choices of what to review start from components which seem ’promising’ and which already have some plus vibes. So there IS a bias before reviewers even select equipment to review, to choose items which (from word of mouth) stand out. Or from companies which have consistently made fine products.And IMO those reasons are why there are very very few bad reviews.
I subscribe to both TAS and Stereophile, and see now and then some negative comments in reviews. How the item compares not so favorably.. etc, or how the item has flaws.
As far as stuff that breaks. It all really is about would the manufacturer replace it? and how fast? Stereophile definitely states if the item failed for any reason. Some companies stonewall. So for things that break, I find the actual company service attitude is way more important than if it arrived broken or broke during the review..
The main purpose of most reviewers is self preservation. They pander to certain manufactures and favour certain flavour of the month products. Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard.
Magazine reviewers are almost entirely misinformed and misguided idiots to a man. Its a bit like selling soap, every year new improved products are launched... Yet somehow the reference products of yore remain unsurpassed.
For decades they have lied about digital sources, they didn't have a clue about speaker isolation, recommending spiking ad infinitum, and most of the hacks can't write for toffee. Memorable reviews? Ha! You'd be lucky to find a dozen.
Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality. Trust it at your peril. Now if only the manufacturers weren't such liars. Now if only we didn't like the lies so much....
cd318 The main purpose of most reviewers is self preservation. They pander ... Honesty is so far down the list of priorities ... integrity ... is the first thing that gets thrown overboard.
Magazine reviewers are ... misguided idiots to a man ... For decades they have lied ... Hi-Fi journalism is akin to mainstream pornography ... if only the manufacturers weren’t such liars ...
Why so angry? What makes you such an objective observer of all things audio?
It is gradually becoming increasingly apparent that user "cd318" has his own agenda that he is enforcing at the same time that he reserves the right to himself to attack others who disagree with him however a major difference is that we don't know what users' "cd318" interest is in the matter which makes it difficult/impossible for readers of this forum to meaningfully assess his "opinions" on matters that he states here with such forcefullness. But I admit comparing Hi-Fi news to pornography is a little bit weird and makes me wonder what he is doing with his Hi-Fi News magazines but of course that is his solitary business and not for us to question challenge criticize or condemn so long as no one is being hurt it is fine.
You must have a verified phone number and physical address in order to post in the Audiogon Forums. Please return to Audiogon.com and complete this step. If you have any questions please contact Support.
