Inside CR Test Labs: How Consumer Reports tests products

Readers of Consumer Reports and ConsumerReports.org have plenty to say—last year, 127,887 of our readers called, wrote, or e-mailed our customer-relations department with comments and questions about the thousands of products we test each year. In the home-and-garden area, they wondered about refrigerators that don't keep food cold, dishwashers that drown out conversations, and funny-looking twisted lightbulbs that claim to save energy.

The occasional rhetorical question arrives in our inboxes, as in "What the heck were you thinking?" (Actually, we get that genre of query pretty often, some of them not suitable to print here.)

Here we address some of the common inquiries we get about how we test products for the home. If you've got a question about a home-related product, send it to us at home@cro.consumer.org.

How do you pick the models you test?
We try to test models that represent the spectrum of products in a given market. Our analysts seek out products with new features and technological advances and a wide range of prices. After they analyze market share, marketing strategy, and advertising and promotional materials, they contact manufacturers to determine whether items will be available for at least three months after a report is published. The analysts then recommend a list of models that managers in our technical and editorial divisions review.

During the next step, staff shoppers buy the products at retail outlets in the throughout the Northeast—our offices are in the New York City suburbs—or online, never revealing that the purchases are for Consumer Reports. (We want to ensure that we test the same products you'll buy.) When we need to buy best-selling regional brands, we use shoppers around the country. Most significant, and unlike most other publications, we buy everything we test.

In rare instances, when a product isn't in stores yet, we buy it from the manufacturer, revealing this in our report; we'll subsequently test a version that we buy at retail and report on those findings.

How long does it take from when you decide to test a product to when you publish the results in a report?
Typically it takes about 9 to 12 months from the time we contact manufacturers for product information until a report appears in the magazine and/or online, a time frame that includes testing.

Note that for some fast-changing products like GPS units we can buy and test units and publish testing information in as few as two weeks. For other products that don't change as fast or for seasonal items (say chain saws), testing can take three to four months.

There are exceptions. Full testing of interior paints, for example, takes six months, but full testing of exterior paint requires up to three years.

How do you test?
Our experts develop tests that re-create the experience you'll have with the product. They also consider industry standards for testing a particular product; note that those tests usually gauge only a minimum level of performance while our tests aim to find the highest-performing products. We develop tests for those products that lack industry standards for ease of use.

In some cases, an industry models its tests on ours. For example, we developed an emissions test for vacuums that determines how much dirt and dust blows into the air when a model is running. The industry then devised its own test that’s based on ours, and now that test is the industry standard.

What do you do if a product malfunctions or breaks during the test?
When either happens we buy two more of the same product. If the new versions do not exhibit the same problem and we suspect the original problem was a quality-control issue, we base the results on the models that performed correctly; we chalk up the problem to an isolated issue. If either (or both) of the new samples exhibits the same problems, we make a judgment on whether it is a flaw in quality control or design and factor this into our Ratings.

A product-design flaw means that most consumers will experience problems with this item, while quality-control issues—materials, assembly, packaging, shipping—should not affect all of the products.

Why are some of the products you recommend in an issue no longer available in stores?
Again, our testing cycle is key here. The August 2008 issue of Consumer Reports arrived at your home and appeared on newsstands and on ConsumerReports.org in early July. Our market analysts recommended and purchased models throughout the fall of 2007 and contacted the manufacturers again before publication to confirm availability. But sometimes after we publish and despite our efforts to confirm that a product is still available, it sells out or is discontinued, or distribution changes.

How do you determine the prices you list in the magazine and online?
We collect data from a number of sources to arrive at approximate retail cost that will give you an idea of the price you'll find at stores, taking into consideration the manufacturer's suggested retail price, prices at stores across the country, and the prices we paid. Your actual cost might vary because of special promotions and coupon offers.

What do the bracketed letters and numbers at the end of the model name mean, as in the KitchenAid KBFC42F[SS] refrigerator or LG WM0642H[W] washing machine?
Manufacturers use the letters or numbers to denote the product's color or finish. The [SS] above indicates the refrigerator has a stainless-steel finish, and the [W] tells you the color of the product is white. Sears uses numbers to indicate the color of its Kenmore appliances. The Kenmore 7425[2] top-freezer is white—the bracketed 2 tells you so. If you see [9] at the end of a Kenmore model number, for example, you can be sure this appliance is black. When you see a [D] in the ratings for any product, it indicates a product has been discontinued; a similar product might be available.—Kimberly Janeway

Essential information: Find out where to get the best deals on appliances and who makes the appliances you own by doing a little appliance sleuthing. And check out our brand-new Home Improvement Guide interactive, which offers buying advice on appliances, tools, and building materials for every area of your home.

Comments

[K P]

I am looking for information on carpet shampooers for home use.

[JB]

How then does pricing fit into the ranking system? There have been a number of times where a product ranks superior to others based on the objective criteria, but receives an overall lower score due to higher pricing?

[Dana White]

I am also looking for information on carpet shampooers and am surprised this is not something Consumer Reports has tested yet. Can you please let us know if/when you plan on testing these?

[mariepontes]

i would love to be a product tester call me or send information to marie pontes 2900 rio grande dr antioch ca 94509

[Al Luehmann]

I am looking for any test results you might have for carpet shampooers or other consumer/professional ways of cleaning residential carpets - beyond the regular vacuuming?

[Bonnie K. Eichelberger]

I was really surprised when I came to this web site to look for your commendations on Carpet shampooers. I am going to buy one as an anniversary present and to my surprise NO RECOMMENDATIONS!!!

[DEBBIE MARVEL]

I NEED INFORM. ON CARPET SHAMPOOERS. THE BEST TO BUY FOR THE PRICE. WHERE TO BUY IT. THANKS DEBBIE

[Margaret]

I need a carpet shampooer that is the most powerful. I have pets. Please advise.

[marytclark]

I am so surprised that you have not rated carpet shampooers. I need one now. Please add this item to your list of needed ratings. Thank you.

[Kathy C.]

I agree with Bonnie (10/15/08). I subscribed to your site recently thinking it was a plethora of information. A friend asked me to research carpet shampooers this a.m. and I am astonished that there aren't any recommendations / test results. Specifically looking for the shampooer that best cleans stains from a variety of dog "accidents". When will this info be available? Soon I hope considering the number of requests...

[Lindsey B]

I also would like to see testing results from carpet cleaners. They are becoming a more common household tool and need to be rated.

[susanf]

Seems like no one is going to answer our queries regarding carpet shampooers. So, since they're not going to give any indication, I'm going to offer at least one opinion on the Hoover Steam Vac Dual V Widepath w/ Auto Rinse. I bought this in March of 2006, tried to use it 2x/year until the Summer of '07 when we started a major renovation project. The project is now complete and I was looking forward to cleaning my carpets, which badly needed it, before the holidays. Bad news!! The clean water tank had a crack on it, spraying water everywhere. I searched and found the replacement part and it costs over $45 (incl. shipping). I've decided not to invest in the replacement part since after receiving it, I may discover that other parts may need to be replaced and I'll end up spending as much as if I just bought a new one. So, that's where I'm at... looking to buy a new one.

Although this model did an excellent job of cleaning, and wasn't hard on my old back, it had some annoyances, such as the dirty water reservoir was rather small, needing frequent emptying. I won't be replacing this model with a similar upright model. Instead, I'm going to look for the more conventional "steamer" model that looks more like a wet/dry vac. It doesn't have scrubbers, and is MUCH harder on the back, but the basic design of the model doesn't have any expensive plastic parts that would break easily.

Anyway, that's my 2¢.

[Linda 54901]

I have been looking for ratings on carpet shampooers for over a year.
What a disappointment to find nothing. This is NOT an inexpensive purchase, so I was sorry to see no ratings at all. Please check into this. Thanks.

[Nancy]

We too are looking to purchase a carpet shampooer. We have read that some people believe that a dry chemical should be used, rather than a steam shampooer. Please include a recommendation regarding this option when you publish your review, which I see is promised for 2009. Hopefully we will find the review early in the new year!

[Candace]

Where can I find information on Carpet Shamphooers? So far I have not been having good luck. Hoover has lost my vote - I own the F7458900 and it's been a money pit. Fortunately the first repairs were covered under warranty but after that it's quite an expense. I really like the machine when it works but the company went really cheap with using "Brittle Plastic".

Thanks, look forward to your reviews.

[Raymond DuBois]

Looks like Consumer Reports, reports on only some things! I am now rethinking my membership as it is up shortly anyway. So far this site has not paid for itself, and probably won't!

[Judy Nelson]

I am also looking to buy a carpet shampooer and can't find any information. Hate to spend all that money and get a turkey. Any good feedback would be appreciated.

[Donna Armond]

I am in need of information for shampooing my carpet.I am not sure which to choose.Please hurry with those 2009 ratings so you can help me decide which one to buy.Thank you.

[Carol Charley]

Well, I see I am not the only one disappointed in CR with regard to no information on Carpet Shampooers. I have subscribed to the magaziine for years and years and once to CR-Com. I guess I am spoiled by CR covering many of the products I am looking to buy. However, it has been over 2 years since I last looked for info on shampooers thru CR to no avail. Come on guys, those of us who depend on you count on your opinion especially for costly items. Please get something in print on this subject soon! Love, love Consumer Reports Mag. and read it cover to cover. Thanks for the great work.

[stephanie kilburn]

I too, am disappointed in Consumer Reports. The few times I have used my subscription, it was to look up items that you DO NOT RATE! Please add carpet cleaners and sewing machines to your list of appliances to rate. I will probably not renew my membership, as it has been a waste of money.

[Steph]

I am looking for a carpet shampooer that works best for cleaning up after our dogs and cat and kids.

[Jim Fishback]

Very disappointed not to find ratings on carpet shampooers

[Kim]

Okay, I just paid to subscribe to Consumer Reports on-line because I am looking for a carpet shampooer... and you don't have any reviews! I'm definitely going to cancel my membership.

[Michelle]

December 2,2008
Come on guys I can't agree more with all the others who posted their concerns. I'm in the market as well for a shampooer for my carpets. I have spent so much time on CR website and nothing.

I just renewed my web subscription for this reason. I'm so disappointed. Will there be any light at the end of the tunnel. It sounds like we are all holding off on making this purchase.

Please CR help us out here. Thank you waiting for your reviews.

[Hattie Mitchell]

I am looking for a recommendation for carpet shampooers. Surprisingly you don't have any. When can I expect this information, if at all?