User Reviews: Useful, but know what you’re using

Type in the words user reviews on Google, and you'll get more than 160 million results. As many as 80 percent of shoppers nationwide now base at least part of their buying decision on these digitized experiences, including the ones on model pages, available to ConsumerReports.org subscribers. You'll also find user reviews on retailer sites like Amazon, Home Depot, Lowe's, Sears, and thousands of others.

User reviews can provide the feeling of reassurance you'd get from a friend or neighbor. Some cover models Consumer Reports hasn't tested. And many offer unique insights about specific models, including breakage and other durability complaints that may not show up in our tests or brand-repair surveys. Even months of rigorous testing may not uncover flaws that could take a year or more of everyday use to surface, which is why we monitor user reviews closely. What's more, an unreliable model may be too new or too limited in sales to affect our survey results for an otherwise-reliable brand.

That's why we were concerned when roughly half of the more than 100 user reviews for the Hoover WindTunnel Bagless S3765-040 and the Kenmore Progressive 27514 canister vacuums cited durability complaints. Both are CR Best Buys. Both also come from brands that have been reasonably reliable in our surveys. We noted those issues in our most recent report on upright and canister vacuums and promised to follow up. Our research confirms that user reviews, while often helpful, may not tell the whole story.

Our experts at the Consumer Reports National Research Center began by recontacting more than 1,200 Hoover and Kenmore canister owners who responded to our online Annual Questionnaire on brand reliability. The number who owned the Hoover WindTunnel and Kenmore Progressive was similar to the user-review total for those models. But our scientific sampling provided a more-representative look at their experiences—good, bad, and indifferent. It also yielded different results:

According to our respondents, both models broke less than the average in our surveys for those canister brands, even when we included broken belts—an inexpensive do-it-yourself repair we don't score in our repair data. Most who owned the Hoover and nearly all who owned the Kenmore said they would definitely or probably buy that model again.

Our research into user reviews for vacuums highlights several factors that can skew reviews for other products or services, wherever you find them:

• People are likelier to review a product they strongly like or dislike than something they find simply okay. Our analysis of more than 1,800 user reviews for uprights and nearly 500 for canisters found that most were either very positive or very negative, with few in between. Good surveys, including our brand-repair surveys, are designed to eliminate those biases. They also eliminate biases related to a product's age and other factors, which user reviews don't do.

• Even the most thorough reviewer may lack the context that comes from extensive side-by-side tests. For example, few could buy, test, and comprehensively compare the roughly 30 cars or trucks, 60 vacuums, and 80 refrigerators and TVs in a typical Consumer Reports Ratings. A model someone loves or hates could be far less lovable or loathsome if the same person compared it with several dozen other models.

• Some reviewers may be especially hard on a product or use it incorrectly. And some may inadvertently praise or pan the wrong model, since vacuums and other appliances can include 20 or more letters and numbers a reviewer may—or may not—get right.

• Users and manufacturers may also try to skew the results by intentionally posting biased reviews themselves or through others. Some reviewers could actually be shills.

All of those issues explain why we don't base our Ratings and model recommendations on user reviews. That doesn't mean they aren't a useful adjunct to our unbiased, scientific tests and brand-repair surveys. User reviews can also tell you about unique features that meet special needs. Just don't assume they're all unbiased or scientific.

The bottom line: Consumer Reports buys and tests more than 3,000 products and analyzes more than 1 million survey responses each year to create our Ratings and brand-repair histories. We're taking a close look at all of our model-specific user reviews in an ongoing effort to learn from them and make them as helpful as possible. Meanwhile, use our Ratings and repair data first for something you'd buy for yourself or a loved one. Then use our user reviews to help inform that decision.

We invite you to share your experiences in our user reviews and forums.

Comments

[Michael Mercier]

I recently bought a copy of your March edition on 100 television reviews. Though your test methods seemed adequate I found one catagory missing and that was user feedback on reliability and repair history for all brands. One of the striking things I did notice was how Samsung consistently made the top of the list for quality in your reviews.. As a consumer who has had bad experience with Samsung products I feel I need to clue you in on the facts. If you go to the "Consumeraffairs.com" website and check out the home entertainment category and select televisions or plasma tv you will find numerous complaints listed by angry consumers who bought not only samsung products but zenith and others. Every one of them has experienced product defects that were never resolved by samsung. And these are plasma and lcd tv's that were purchased from 2005 right up to the present. I myself have filed a complaint against Samsung for the total indiverence they have displayed when trying to contact them about their products. With regards to my own problems it goes like this. I purchased a Samsung SP-R4232 plasma tv in December of 2005. It has been well cared for and the only thing I used it for was for viewing dvd movies. On February 14, 2009 the tv abruptly stop working for no apparent reason. When I tried to turn it on that day the picture came on momentarily and then went blank. I could hear a lot of clicking noises coming from the back of the unit like it was cycling thru the turning on phase. I was able to still shut it of with the blue switch in the front panel. I got on the phone and called Samsung directly to talk with a technician and they asked me the usuall questions, model, serial numbers, etc. which I provided. They took a few minutes to look it up and informed me that my warrantee had expired a couple of months previous and that there wasn't much they could do for me except refer me to a repair service in my area. I proceeded to explain to them the technical problem I was experiencing and the technician seem to know immediately what the problems was. He informed me that it was a power supply issue. It seems that this is a common problem with this model. Anyway I now have a $2200 tv sitting in my living room that is totally useless. They have $2200 of my money and I have a tv that should have lasted more than 3 years and 2 months. It is going to cost a little over $500 to repair for parts and labor. That is 25% of the purchase price to repair. I will not buy another Samsung product ever again and you at consumer report need to let people know what is truely going on with their products. I am not alone in this opinion. And I understand that it is difficult for a company like yours to give bad ratings on products because of liability issues. But people need to be able to make an informed decision when it comes to products that have short term and long term defects/problems and when you omit these facts that does not provide them a complete set of facts to make that informed decision.

[susan monroe]

i bought a dirt devil ultra vision turbo, i am very disapointed in it, very poorly useable, i canot use the hose to clean up dust in corners etc, no suction on it at all, dust flies out it all over, i have went over it to make sure that it is properly put together, it is, but it is not a user friendly for allergies with the dust flying all over, i have wiped off the sweeper then used it again, you couldnt tell i had wiped it off,i have put my hand over the hose to feel the suction for picking up dust, nothing then i looked at the hose where it was connected, and where the force air for suction was to come from, it sure felt like it was just attached to the side of a piece of plastic, i used some money to get a sweeper instead of paying a bill, on a sweeper that didnt work, what a waste, im not recommending a dirt devil to any one, now the sweeper is in the trash.

[chris hansen]

I've had a strong response to the Consumer Reports Article on how to use consumer reviews. In particular, there seem to be many (i.e., hundreds) of individuals who disagree with recent findings on your vacuum cleaners. Initially, I was pleasantly surprised to see that Consumer Reports had addressed the disparity between product reviews and consumer reviews. But your assessment is filled with self righteous spaghetti logic. You seem to be contrasting the weakness of qualitative study to quantitative study, but you do a poor job of relaying the meaning to your readers. I want to know if if the band on my vacuum breaks every time I use it. Further, from consumer reviews it does not sound as if this is an easy fix. You explain that you reexamined these issues and submitted another sample. But you do not address any of the complaints your readers have posted. It seems to me, as you have taken a strong stance on the qualitative complaints of your readers, this was a wonderful opportunity to test them. An opportunity you not only have failed at but seem to blame on your readers (i.e, you seem to suggest those readers who submitted reviews on just these machines were biased or they have misused the machines in question?). I'm sure there is a demographic that takes great solace in knowing their purchases are statistically/quantitatively good buys, on the measures you choose to implement. But at the end of the day, I want my purchases to make my life easier - and you seem completely unable to grasp this. Your technicians and editors should get more sunlight. For the first time in 15 years I have serious misgivings into the quality of your publication. I'm very disappointed.

[Thomas E. Bow]

I'm glad you addressed the topic of user-posted product reviews since these are increasingly becoming commonplace. I have been a long-time CR subscriber because I find the information you publish to be invaluable. I have also found user reviews on sites like Amazon and e-Pinions to be helpful, especially when I'm interested in a product CR has not reviewed.

I agree with you though that one has to consider the origin of the data and the methodology employed in order to make appropriate use of it (though this is true in evaluating any data, including data from CR). I further agree that in reading any review it is important to consider who the reviewer is and their motivations. User-posted reviews are NOT qualitative research; They are subjective opinions. And our opinions are a function of who we are as people and our motivations. Of course it is highly unlikely that we'll know any particular reviewer outside of what he or she posts in the review. But the review itself can provide clues as to who the person is and their motivations.

My approach to evaluating reviews is to read all available reviews on the product and look for specific, recurring themes. If multiple users are highlighting a specific circumscribed problem with a product, I tend to think there is something to it. Reviews that are exceedingly glowing or critical I find suspect, especially when they lack specific detail, since I believe that it's rare to find a product that's either perfect or without any redeeming qualities. Finally, sometimes a reviewer will inadvertantly reveal more about him/herself than about the product in question. I once read a review of an automobile in which the reviewer went on many tangents that portrayed a person with questionable judgment. In other cases I've found critiques that just seem suspect on the surface, e.g., a claim that auto sound system knobs "fell apart" after 2 months. Was the reviwer abusing the sound system knobs? Is tise reviwer rough or careless with their things in general and that's why they break in 2 months?

Thinking about user-posted reviews in the ways I described above helps me evaluate the quality of the information provided and assists me in my purchase decisions.