“Don’t believe online reviews of health products, they’re ‘skewed’,” the Mail Online reports.
A psychologist compared online reviews of three medical products with results from clinical trials, and found the reviews are skewed towards the positive.
The author of the study, Dr Micheál de Barra, wanted to look into whether people who have had good outcomes from treatments are more likely to go online and give positive reviews than people who have had average or poor outcomes. As such, the product reviews provided by online retailers may be distorted.
The author looked at Amazon.com – the US version of the site – and analysed two cholesterol-reducing products and one weight loss treatment.
In general, he found the extent of cholesterol reduction or weight loss reported by online reviewers was substantially greater than that demonstrated in randomised controlled trials, a more reliable source of evidence on effectiveness.
The research highlights an underlying issue with online reviews, whether they are for health products, films or books. Online reviews are arguably subject to a type of reporting bias in that they are written by people who take the time to write them.
This means it’s far more likely that these reviews are written by people who have very strong views, either positive or negative, about a product than people who would score it three out of five stars.
Where did the story come from?
The study was authored by a single researcher from the University of Aberdeen and published in the peer-reviewed journal, Social Science and Medicine.
No sources of financial support are mentioned. The author reports support and advice from four named individuals, but otherwise declares no conflict of interest.
The UK media’s coverage of this study is generally representative, but it would be unfair to single out Amazon or the products discussed as having particularly misleading reviews.
The site and the products just happen to be the ones the author looked at. It’s likely other products on other sites have equally skewed reviews.
What kind of research was this?
This study aimed to look into the potential issue of biased reporting by online retailers around the effects of medical treatments.
The author reports how people “often hold unduly positive expectations about the outcomes of medicines and other healthcare products”.
He suggests the reason for this could be that people who have had a positive outcome from a treatment are more likely to be vocal about this, compared with people who’ve had poor to average outcomes.
This means the information available to others through customer reviews on retailers’ websites is likely to be distorted in favour of the product.
The author focused on several questions:
What did the researcher do?
The author aimed to see whether the reviews of medical products published by the international online retailer Amazon.com are consistent with outcomes of the same products reported in clinical trials.
If there is no reporting bias, the average outcomes should be more or less the same.
He found three medical products available that met the following criteria:
The three products that met these criteria were two cholesterol treatments – Benecol Smart Chews Caramels (not available in the UK) and Nature Made CholestOff (a herbal supplement available over the counter) – and a weight loss drug designed to reduce the amount of fat the body takes from food, called Orlistat (also available over the counter under the supervision of a pharmacist).
As the author says, it’s likely that many other treatments would also have met these criteria, but these were the first three identified.
He included a total of 908 reviews of the two cholesterol products where the reviewers also provided information on changes in blood cholesterol levels.
These were compared with the size of the effects reported in a systematic review of clinical trials.
There were 767 reviews of Orlistat included – a specific weight change was reported in about a third of these.
A recent systematic review was identified for comparison, which included two clinical trials of this drug.
What did he find?
For the cholesterol products, the author found the blood cholesterol reductions reported by online reviewers were significantly greater than those reported in clinical trials.
For example, Benecol users reported a 45mg/dl reduction in cholesterol, compared with a range of 9.28-24mg/dl reduction reported in trials.
Similarly, the weight loss reported by Orlistat reviewers was significantly greater than in the clinical trial participants (specific kg loss not reported in the study).
The author calculated that the “reputational distortion” (the perceived benefit) for the three products was three to six times greater than the actual size of the benefit gained from the treatment.
When looking at the one- to five-star grading system, on average, reviews that were shared had 0.55 more positive stars than those that weren’t shared.
What did the researcher conclude?
The author concluded that, ”People with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others.
“People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations.”
This unique study suggests that, in general, online medical product reviews may give a distorted and enhanced perception of the effectiveness of the product compared with that actually demonstrated in randomised controlled trials.
The author discusses potential theories around this. For example, it may reflect the fact people are more likely to post a review if they found something good than if the benefit they found was not that remarkable or there was no benefit at all.
He also suggests people may not wish to dwell on prior periods of ill health, whereas a positive recovery is something they may want to share with others.
People who remain in poorer health could also have lower mood and be less inclined to engage in sharing information about their health.
But, as the author acknowledges, in the case of raised cholesterol this theory doesn’t provide the whole answer because the condition doesn’t cause as many obvious health problems.
However, it’s important to note a few points:
Links To The Headlines
Don’t believe online reviews of health products, they’re ‘skewed’ and ‘misleading’, study concludes. Mail Online, February 10 2017
Amazon weight loss product reviews ‘positively misleading’ customers, say researchers. The Independent, February 10 2017
Links To Science
De Barra M. Reporting bias inflates the reputation of medical treatments: A comparison of outcomes in clinical trials and online product reviews. Social Science & Medicine. Published online February 10 2017