Inside CR Test Labs: How Consumer Reports tests products

Readers of Consumer Reports and ConsumerReports.org have plenty to say—last year, 127,887 of our readers called, wrote, or e-mailed our customer-relations department with comments and questions about the thousands of products we test each year. In the home-and-garden area, they wondered about refrigerators that don't keep food cold, dishwashers that drown out conversations, and funny-looking twisted lightbulbs that claim to save energy.

The occasional rhetorical question arrives in our inboxes, as in "What the heck were you thinking?" (Actually, we get that genre of query pretty often, some of them not suitable to print here.)

Here we address some of the common inquiries we get about how we test products for the home. If you've got a question about a home-related product, send it to us at home@cro.consumer.org.

How do you pick the models you test?
We try to test models that represent the spectrum of products in a given market. Our analysts seek out products with new features and technological advances and a wide range of prices. After they analyze market share, marketing strategy, and advertising and promotional materials, they contact manufacturers to determine whether items will be available for at least three months after a report is published. The analysts then recommend a list of models that managers in our technical and editorial divisions review.

During the next step, staff shoppers buy the products at retail outlets in the throughout the Northeast—our offices are in the New York City suburbs—or online, never revealing that the purchases are for Consumer Reports. (We want to ensure that we test the same products you'll buy.) When we need to buy best-selling regional brands, we use shoppers around the country. Most significant, and unlike most other publications, we buy everything we test.

In rare instances, when a product isn't in stores yet, we buy it from the manufacturer, revealing this in our report; we'll subsequently test a version that we buy at retail and report on those findings.

How long does it take from when you decide to test a product to when you publish the results in a report?
Typically it takes about 9 to 12 months from the time we contact manufacturers for product information until a report appears in the magazine and/or online, a time frame that includes testing.

Note that for some fast-changing products like GPS units we can buy and test units and publish testing information in as few as two weeks. For other products that don't change as fast or for seasonal items (say chain saws), testing can take three to four months.

There are exceptions. Full testing of interior paints, for example, takes six months, but full testing of exterior paint requires up to three years.

How do you test?
Our experts develop tests that re-create the experience you'll have with the product. They also consider industry standards for testing a particular product; note that those tests usually gauge only a minimum level of performance while our tests aim to find the highest-performing products. We develop tests for those products that lack industry standards for ease of use.

In some cases, an industry models its tests on ours. For example, we developed an emissions test for vacuums that determines how much dirt and dust blows into the air when a model is running. The industry then devised its own test that’s based on ours, and now that test is the industry standard.

What do you do if a product malfunctions or breaks during the test?
When either happens we buy two more of the same product. If the new versions do not exhibit the same problem and we suspect the original problem was a quality-control issue, we base the results on the models that performed correctly; we chalk up the problem to an isolated issue. If either (or both) of the new samples exhibits the same problems, we make a judgment on whether it is a flaw in quality control or design and factor this into our Ratings.

A product-design flaw means that most consumers will experience problems with this item, while quality-control issues—materials, assembly, packaging, shipping—should not affect all of the products.

Why are some of the products you recommend in an issue no longer available in stores?
Again, our testing cycle is key here. The August 2008 issue of Consumer Reports arrived at your home and appeared on newsstands and on ConsumerReports.org in early July. Our market analysts recommended and purchased models throughout the fall of 2007 and contacted the manufacturers again before publication to confirm availability. But sometimes after we publish and despite our efforts to confirm that a product is still available, it sells out or is discontinued, or distribution changes.

How do you determine the prices you list in the magazine and online?
We collect data from a number of sources to arrive at approximate retail cost that will give you an idea of the price you'll find at stores, taking into consideration the manufacturer's suggested retail price, prices at stores across the country, and the prices we paid. Your actual cost might vary because of special promotions and coupon offers.

What do the bracketed letters and numbers at the end of the model name mean, as in the KitchenAid KBFC42F[SS] refrigerator or LG WM0642H[W] washing machine?
Manufacturers use the letters or numbers to denote the product's color or finish. The [SS] above indicates the refrigerator has a stainless-steel finish, and the [W] tells you the color of the product is white. Sears uses numbers to indicate the color of its Kenmore appliances. The Kenmore 7425[2] top-freezer is white—the bracketed 2 tells you so. If you see [9] at the end of a Kenmore model number, for example, you can be sure this appliance is black. When you see a [D] in the ratings for any product, it indicates a product has been discontinued; a similar product might be available.—Kimberly Janeway

Essential information: Find out where to get the best deals on appliances and who makes the appliances you own by doing a little appliance sleuthing. And check out our brand-new Home Improvement Guide interactive, which offers buying advice on appliances, tools, and building materials for every area of your home.

Comments

[K P]

I am looking for information on carpet shampooers for home use.

[JB]

How then does pricing fit into the ranking system? There have been a number of times where a product ranks superior to others based on the objective criteria, but receives an overall lower score due to higher pricing?

[Dana White]

I am also looking for information on carpet shampooers and am surprised this is not something Consumer Reports has tested yet. Can you please let us know if/when you plan on testing these?

[mariepontes]

i would love to be a product tester call me or send information to marie pontes 2900 rio grande dr antioch ca 94509

[Al Luehmann]

I am looking for any test results you might have for carpet shampooers or other consumer/professional ways of cleaning residential carpets - beyond the regular vacuuming?

[Bonnie K. Eichelberger]

I was really surprised when I came to this web site to look for your commendations on Carpet shampooers. I am going to buy one as an anniversary present and to my surprise NO RECOMMENDATIONS!!!

[DEBBIE MARVEL]

I NEED INFORM. ON CARPET SHAMPOOERS. THE BEST TO BUY FOR THE PRICE. WHERE TO BUY IT. THANKS DEBBIE

[Margaret]

I need a carpet shampooer that is the most powerful. I have pets. Please advise.

[marytclark]

I am so surprised that you have not rated carpet shampooers. I need one now. Please add this item to your list of needed ratings. Thank you.

[Kathy C.]

I agree with Bonnie (10/15/08). I subscribed to your site recently thinking it was a plethora of information. A friend asked me to research carpet shampooers this a.m. and I am astonished that there aren't any recommendations / test results. Specifically looking for the shampooer that best cleans stains from a variety of dog "accidents". When will this info be available? Soon I hope considering the number of requests...

[Lindsey B]

I also would like to see testing results from carpet cleaners. They are becoming a more common household tool and need to be rated.

[susanf]

Seems like no one is going to answer our queries regarding carpet shampooers. So, since they're not going to give any indication, I'm going to offer at least one opinion on the Hoover Steam Vac Dual V Widepath w/ Auto Rinse. I bought this in March of 2006, tried to use it 2x/year until the Summer of '07 when we started a major renovation project. The project is now complete and I was looking forward to cleaning my carpets, which badly needed it, before the holidays. Bad news!! The clean water tank had a crack on it, spraying water everywhere. I searched and found the replacement part and it costs over $45 (incl. shipping). I've decided not to invest in the replacement part since after receiving it, I may discover that other parts may need to be replaced and I'll end up spending as much as if I just bought a new one. So, that's where I'm at... looking to buy a new one.

Although this model did an excellent job of cleaning, and wasn't hard on my old back, it had some annoyances, such as the dirty water reservoir was rather small, needing frequent emptying. I won't be replacing this model with a similar upright model. Instead, I'm going to look for the more conventional "steamer" model that looks more like a wet/dry vac. It doesn't have scrubbers, and is MUCH harder on the back, but the basic design of the model doesn't have any expensive plastic parts that would break easily.

Anyway, that's my 2¢.

[Linda 54901]

I have been looking for ratings on carpet shampooers for over a year.
What a disappointment to find nothing. This is NOT an inexpensive purchase, so I was sorry to see no ratings at all. Please check into this. Thanks.

[Nancy]

We too are looking to purchase a carpet shampooer. We have read that some people believe that a dry chemical should be used, rather than a steam shampooer. Please include a recommendation regarding this option when you publish your review, which I see is promised for 2009. Hopefully we will find the review early in the new year!

[Candace]

Where can I find information on Carpet Shamphooers? So far I have not been having good luck. Hoover has lost my vote - I own the F7458900 and it's been a money pit. Fortunately the first repairs were covered under warranty but after that it's quite an expense. I really like the machine when it works but the company went really cheap with using "Brittle Plastic".

Thanks, look forward to your reviews.

[Raymond DuBois]

Looks like Consumer Reports, reports on only some things! I am now rethinking my membership as it is up shortly anyway. So far this site has not paid for itself, and probably won't!

[Judy Nelson]

I am also looking to buy a carpet shampooer and can't find any information. Hate to spend all that money and get a turkey. Any good feedback would be appreciated.

[Donna Armond]

I am in need of information for shampooing my carpet.I am not sure which to choose.Please hurry with those 2009 ratings so you can help me decide which one to buy.Thank you.

[Carol Charley]

Well, I see I am not the only one disappointed in CR with regard to no information on Carpet Shampooers. I have subscribed to the magaziine for years and years and once to CR-Com. I guess I am spoiled by CR covering many of the products I am looking to buy. However, it has been over 2 years since I last looked for info on shampooers thru CR to no avail. Come on guys, those of us who depend on you count on your opinion especially for costly items. Please get something in print on this subject soon! Love, love Consumer Reports Mag. and read it cover to cover. Thanks for the great work.

[stephanie kilburn]

I too, am disappointed in Consumer Reports. The few times I have used my subscription, it was to look up items that you DO NOT RATE! Please add carpet cleaners and sewing machines to your list of appliances to rate. I will probably not renew my membership, as it has been a waste of money.

[Steph]

I am looking for a carpet shampooer that works best for cleaning up after our dogs and cat and kids.

[Jim Fishback]

Very disappointed not to find ratings on carpet shampooers

[Kim]

Okay, I just paid to subscribe to Consumer Reports on-line because I am looking for a carpet shampooer... and you don't have any reviews! I'm definitely going to cancel my membership.

[Michelle]

December 2,2008
Come on guys I can't agree more with all the others who posted their concerns. I'm in the market as well for a shampooer for my carpets. I have spent so much time on CR website and nothing.

I just renewed my web subscription for this reason. I'm so disappointed. Will there be any light at the end of the tunnel. It sounds like we are all holding off on making this purchase.

Please CR help us out here. Thank you waiting for your reviews.

[Hattie Mitchell]

I am looking for a recommendation for carpet shampooers. Surprisingly you don't have any. When can I expect this information, if at all?

[rbervin]

I too am looking for a shampooer. When will you answer our question? Richard.

[Joe]

It seems like CR won't respond to the outcry for a carpet shampooer review so I’ll share my personal experience with you. I have two Chocolate Labs who are capable of making quite a mess on the carpet be it by shedding or “accidents”. Over the past three years I’ve gone through two Bissell’s and one Hoover. I took all of these to a local guy for repair and this is what he said. All of the machines that you get at your local department store are junk. Until recently Bissell had a slight advantage over Hoover not because of quality but because the parts were made in America. Now, like Hoover, they have to be obtained from China through a long and often uncertain process. My Hoover only lasted two month with no replacement parts available. Each Bissell averaged 5-6 months.

The repair man told me that the only reliable option was a commercial model which averaged $1,000. Sounds bad right? Well I’m up to $600 over the past two years with my Chinese junk. Decide for yourself since CR won’t!

[Joe]

It seems like CR won't respond to the outcry for a carpet shampooer review so I’ll share my personal experience with you. I have two Chocolate Labs who are capable of making quite a mess on the carpet be it by shedding or “accidents”. Over the past three years I’ve gone through two Bissell’s and one Hoover. I took all of these to a local guy for repair and this is what he said. All of the machines that you get at your local department store are junk. Until recently Bissell had a slight advantage over Hoover not because of quality but because the parts were made in America. Now, like Hoover, they have to be obtained from China through a long and often uncertain process. My Hoover only lasted two month with no replacement parts available. Each Bissell averaged 5-6 months.

The repair man told me that the only reliable option was a commercial model which averaged $1,000. Sounds bad right? Well I’m up to $600 over the past two years with my Chinese junk. Decide for yourself since CR won’t!

[r houchin]

I too am disappointed that I subscribed (again) only to find that you have no information on carpet cleaners or shampoo machines. Guess I wasted that $5.95. Won't let that happen again! Live and learn I guess.

[Laura]

I too signed on looking for ratings on carpet shampooers/steam vacs. I'm very surprised they aren't here. Please please please start rating these. I'd like to purchase one soon, and I thought my yearly membership to CR would really help with the decision. I guess not.

[Mike]

I too am looking for Steam Vac recommendations. Come on consumer reports! Get with the program.

[Theresa]

I am also in need for another carpet shampooer. I was hoping for some insight, since I have several pets. I use to have the Hoover Steam Vac Dual V, but the water tank is having problems. The valves at the bottom of the tank that regulate the water, have stopped working. The machine soaked my carpet last time I tried to use it. I have had it for a couple of years, but I only have a couple bedrooms with carpet. I need to buy one soon.

[Jim]

Some time ago consumer reports had a review that stated buying a carpet shampooer was not worth it and rather renting or hiring a cleaning service was often the best way to go, but that has been some time ago.

[Marian]

Add me to the list of folks looking for a recommendation on home carpet cleaning machines, either the kind a consumer can purchase, or rent at a store....or even ratings on the national services who offer cleaning. It shouldn't be that hard to compare and contrast steam cleaning to chemical cleaning, and it would certainly be more valuable to a consumer to hear it from CR than from the salesperson with the carpet cleaning company.

[Hernan]

Where can I find information/recommendation/ratings/prices for a carpet steam cleaner? This is not an appliance to buy without advice. Thank you.

[kmj]

I have to add my two cents on the general carpet cleaner/floor steamer lament. I finally decided to sign up for CR online, feeling sure it would hold the key to the best steamers on the market, before I invest $400. Now I find there is nothing useful in terms of a review, but several long blogs about how CR has not reviewed any floor steamers or carpet cleaners other than vacuums. How disappointing! These items have become increasingly popular these days; please, CR, listen to your audience and do a review so that people don't get taken on a shoddy piece of equipment.

[joy wrinkle]

I subscribed to Consumer Reports because I couldn't find any information until I did. Now YOU DON'T HAVE WHAT I WAS LOOKING FOR UNLESS YOU MOVED IT. I NEED A CARPET SHAMPOORER & PAID YOU $26.00 TO HELP ME & I FEEL LET DOWN.

[Amy]

I am also looking for information on carpet shampooers and cannot believe that CR has not rated them. I have Berber style carpet and understand that most carpet shampooers do a poor job of cleaning Berber style carpeting. I ordered a rug shampooer off the Internet and only after I received it did I realize that it does not extract the dirty water after scrubbing the solution into the carpet! And the return policy prohibited returning an item if it had been removed from the box! I was hoping that CR could help me find the right shampooer to buy next time. So far I have had very little luck in finding information on ANYTHING I have searched for since subscribing to this service!

[C. Batton]

I, too, was looking for some information/recommendations on carpet shampooers and found this long list of requests for the same. Carpet shampooers have been around a long time, have a valuable use in many homes but, sadly, find all sorts of reviews/recommendations on various technological items. Please help the simple folks who still do their own work and give us some recommendations. Thanks.

[Vicki Sternick]

We have an older Bissel Carpet Cleaner with Cannister on wheels with one hose that attaches to your faucet and a long hose with different attachments. We used one for about 10 years and then gave it to a neighbor when we moved and bought another one. I like it because it cleans well, but is hard on the back.

Last fall we bought a Bissel Proheat 2X. I used it to clean one
bedroom carpet and then went back to using the older Bissel with
Cannister. The Bissel 2X is good, but I got tired of emptying
the dirty water resevoir so frequently and switching to the hose
to clean in some areas.

[Joan]

I just renewed my online subscription because I need to purchase a new carpet shampooer. I have an Oreck now but it's not working great any more. When it was new, I loved it. I saw an infomercial on Hoover dual V and it nearly had me convinced. But I really hoped you all would have rated them by now. Big disappointment. I am going to cancel and get a refund. This disappointment isn't worth my $26.00 annual fee.

[William Bardelmeier]

Google "carpet cleaners" and you can locate reviews of some carpet shampooers. The Royal "Procision 7940 carpet extractor" is rated considerably better than Bissel and Hoover. Cost is around $299.00.

[D. Williams]

I signed up for Consumer Reports On-Line because I was looking for a carpet shampooer. As I look at all of the comments that have been posted, it appears that I'm not the only CR subscriber that is surprised and disappointed to see that there are no CR reviews on CARPET SHAMPOO MACHINES. After all of these inquiries, I would have thought, by now, there would be a review posted. ????

[Carol Kimmitt]

WHEN are you finally going to do a review of carpet shampooer/steam cleaners? I have been waiting nearly 2 years for a review on these and I see others have been waiting too! I hesitate to purchase one until I read your review on testing them. Can you just give us a estimated date on when you expect to publish information on the most requested item? Thanks!
ck

[Mike and Steph]

I have been trying to use a Bissel carpet shampooer and it is letting me down and not getting up animal messes, even with the specific animal cleaner being used in the machine. I was hoping to find information on a carpet shampooer that actually does a good job sucking the water/cleaner that captures the now dirty liquid back into machine from the carpet, as this is where mine is failing miserably.

Customer Service,

Can you post an entry to let all concerned parties know whether or not testing is being pursued for this product?

Thanks!

[jeanie]

Could someone please help with a carpet steamer? Don't want to invest a lot of money into something and not get good qualitiy. When will you test these?

[Jene Johnson]

Does CR read the comments that people write in? Look at dishwashers, Cr states a kenmore is the pick of all dishwashers, Then I read the comments by people that own that particular model. Most of them are very negative. So who do I beleive? The people or CR? I lean very hard towards the the people who have owned and hated this machine

[Scott Rafferty]

I have been a member for years and have always wanted a review of carpet shampoo/steam machines. I have had a Kirby, Bissell and Hoover and all have their problems but I would truly like this subject researched.

Your subscribers are the type of individuals who value the research you provide and would like to see this subject covered. Like the SI swimsuit issue, this could be your biggest seller..........

[The Moose]

CR has posted here:
http://blogs.consumerreports.org/home/2008/03/appliances.html
that they are testing carpet cleaners/shampooers. Scroll down to the paragraph in red.

[Marie Konat]

Come on already. How many people have to write in for you guys to provide some information on carpet scrubbers?

[Kathy]

The reason I subscribed to CR today was to get info on carpet shampooers, or extractors, as was mentioned in the paragraph I read that enticed me to purchase an online membership. What a rip off and disappointment. CR you are not what you used to be.

[Dianne Parker]

I subscribed when I needed advice on purchasing a DVD recorder. I was disappointed to find so few reviewed - none of them the latest models or otherwise available locally. Still, I did get some good tips on features to consider when choosing.

Now... add another request for a long-awaited review of floor steamers to the list! There are dozens of models available now for hard floor surfaces in response to customer demand. Where are corresponding reviews?

I heartily agree with Scott R: "Like the SI swimsuit issue, this could be your biggest seller.........."

[Karan]

I can't understand why you won't rate carpet steamers. I have purchased 2 Bissell cleaners in the past three years.They run great at first, but then fail miserably. The Hoovers I have had in the past lasted longer, and although they were heavier, noisier, and larger, they lasted longer. What's up with the lack of interest in a study? Seems like there is clearly an interest.

[Charles Lammers]

I've been dealing with cats and dogs for years. I've had a lot of luck with the Bissel upgright carpet shampooers. I haven't tried anything else. I would love it if CR would start rating these things. I bout the small hand-held re-chargeable Bissel, big mistake. The battery on that thing is useless. Would be nice if CR would help me avoid bad products in this category.

[T]

WOW is all I can say..Just signed up with the hope of finding a good review for a shampoo carpet cleaner. Now signed up and very disapointed that at CR doesn't have one.

[Sue]

I can't believe there aren't reports/recommendations on carpet shampooers! I have pets too and the carpets are a mess. I have a Kirby vacuum/shampooer, but it just doesn't do the job well anymore; guess it's tired. I even had it checked out. Was going to buy something today, and now don't know which one to get! Come on CR!!

[Kimberly Janeway, associate editor]

Thanks for reading our blogs. We're testing carpet shampooers now and we're scheduled to publish the results in the March 2010 issue of Consumer Reports and online in early February.