• U.S.

EVALUATING THE BUYER’S BIBLE

10 minute read
Paul Gray

GINNY LUI SITS UNDER A 100-WATT light bulb, studying spoons. The spoons have gone through eight full cycles in one or another of a battery of 20 dishwashers; Lui’s task is to examine each one for telltale signs of dirt. If she detects a speck, she must then decide just what sort of speck she has spotted. Is it gritty? Is it “medium soil” or “heavy soil”? When she runs through all the spoons, Lui turns to plates and cups and knives and glasses that have been emptied from the dishwashers, making the same inspections and rendering the same hairsplitting verdicts. She guesses she has made 30,000 such judgments in the past eight weeks.

Appearances to the contrary, Lui is not some hapless soul condemned to a lower circle of Dante’s Inferno. She has been happily doing such work for eight years as one of the 95 product testers for Consumer Reports, the monthly magazine that has served as an indispensable guide for millions of shoppers since 1936. And a major reason for CR’s success is the rigorous, painstaking, maybe obsessive and some might even say nutty striving for objectivity that Ginny Lui and her colleagues bring to the testing of products.

By most measures, CR is thriving as never before. Between 1990 and 1993, its monthly circulation ballooned from 3.8 million to 5 million. Although that figure has remained flat since then, CR estimates its total readership, including pass-along and library copies, at 19 million. Consumers Union, the magazine’s parent organization, operates on an annual budget of $133 million, most of which is raised through subscriptions and book sales. And in 1990, aware that its devoted audience was getting older (median age: 42), CR revitalized another magazine devoted to young people into Zillions: Consumer Reports for Kids, which comes out six times a year and reaches 280,000 home subscribers plus numerous school programs. Its topics (“Which Ice Cream?” and “Fast Food: What’s Good?”) are keyed to young consumers who may, somewhere down the line, want guidance in buying a new car. CR has even gone electronic: it is now available on several commercial online services and produces its own syndicated television programming, which is currently carried by about 50 local stations.

As an independent ombudsman for the American buying public, CR has always attracted some ire and resentment-and not only from manufacturers. In 1939 a House committee branded Consumers Union subversive, thanks to the magazine’s perceived antibusiness bias. Though such witch-hunting may seem laughable now, executives at CR and Consumers Union are not smiling at what some detractors and publishing analysts are saying about the magazine today: 1) its methods of testing and evaluating products are sometimes flawed and less objective than they appear; 2) the magazine has evolved from a neutral reporter into an advocate of politically correct but scientifically dubious environmental causes; and 3) as a general-interest publication, CR can no longer keep up with the flood of new products and hence cannot compete with the growing number of specialty magazines that rate, say, computers and the tidal wave of new software.

Critics aside, few signs of stress can be detected at the magazine’s headquarters, a 180,000 sq. ft. complex built in 1991 at a cost of $40 million in Yonkers, New York, half an hour north of Manhattan. An eerie serenity and strange things abound here. There is, for example, a 30-ft. by 30-ft. by 30-ft. chamber, its beams sunk deep into Westchester County bedrock, that stands free of the rest of the building. The purpose of this noiseless, echoless room, immune to all outside vibrations short of the nuclear or the heavily seismic, is to test under absolutely pristine aural conditions the performance range of stereo speakers. “Nobody else could afford this room,” says Alan Lefkow, CR’s director of electronics testing. “Other magazines don’t use them.”

Elsewhere, Ruth Greenberg, a veteran CR assistant project leader on appliances, is grilling 40 microwave ovens on their ability to pop popcorn. She introduces a bag of Paul Newman’s popcorn to a Whirlpool microwave oven. “First, I hit the Popcorn button,” she says, “but then I will compare it to the bag’s instructions to see if the popcorn feature works. I’ll measure how much popcorn popped and how many kernels will be left over.” Guided by its Popcorn button, the Whirlpool shuts off after 2 min. 20 sec. The yield: four cups of popcorn and, Greenberg estimates, 200 unpopped kernels. “I’ll go back and count the kernels by hand.”

Such meticulousness has obviously been designed to take the guesswork out of testing. But some critics charge that impressionistic, subjective judgments nevertheless find their way into some CR evaluations. In its January issue the magazine ran a chart listing its overall test results for eight mid-priced 1995 sedans. In second place in overall excellence, just behind the Toyota Camry, was the Dodge Intrepid. But in a separate story headlined WOES BESET CHRYSLER, the magazine said, “Through the first quarter of 1994, fully 24 percent of owners of 1994 Intrepids reported at least one problem they considered serious, an especially disturbing result for such new cars.” As a result, CR announced, it was dropping the Intrepid from its list of recommended models.

Consumers who were confused (the second-best sedan is also a turkey?) could link elbows with the ranks of Chrysler executives who were aggrieved. Bob Moser, who bears the formidable title of director of customer problem identification and resolution, argues that the CR methods of evaluating the Intrepid compared fresh apples and a few outdated lemons. CR’s citing of drivers’ complaints about 1994 models, he says, “carries the assumption that the problems continued through the 1995 models.” Not so, Moser insists: “We know where our cars are versus any time in the past, and in fact these 1995 models are the best vehicles we have ever manufactured.” Counters CR director of automobile testing Robert Knoll: “Automobiles and auto companies don’t change that rapidly. We get a good enough sample of a model to see if they follow a pattern. When we predict a pattern, it usually does develop.”

This disagreement between CR and Chrysler underscores the difference between objective tests and predictions that are based on a sample of subjective responses (how serious, for example, is a “serious” problem?). And what about products that do not perform tasks, like automobiles or dishwashers, but are rather made to be ingested, e la orange juice, which is featured in the February 1995 issue of CR? Isn’t one person’s ambrosia another person’s pig swill? “We have a standard of what is good orange juice,” says CR spokeswoman Rana Arons. Tasters attempt to quantify attributes such as “sweet” and “astringent.” “But you’ll never see in Consumer Reports anywhere, ‘We like this best, this tastes good.’ We never say to our testers ‘what tastes best.’ ” Which is a confusing assertion, given the headline at the top of the February cover: WHICH ORANGE JUICES TASTE BEST?

Charges are also growing that CR regularly crosses the line between impartial objectivity and committed advocacy. In one sense, this complaint is nothing new. In its very first issue, May 1936, the magazine ran a story on the dangers of lead in children’s toys. In the 1950s, CR warned-correctly-that radioactive fallout from nuclear tests in the atmosphere was contaminating milk products; in the ’60s and ’70s it repeatedly urged auto manufacturers to install seat belts.

But in those campaigns, critics argue, CR had hard science or, in the case of seat belts, exhaustive testing and research on its side. The magazine’s more recent warnings, they add, have not been similarly grounded in facts or else have made minimal risks seem major ones. In May 1989, CR did a story on the health risks posed by the continued use of the chemical Alar to ripen apples. The piece made news in other magazines and on TV and helped spread alarm about the chemical; eventually, stores took Alar-treated apples off the shelf, costing U.S. apple growers an estimated $100 million in discarded produce. While some, including CR, still maintain that Alar could pose a hazard, the National Cancer Institute, the American Medical Association and the Surgeon General concluded that the chemical put apple eaters in little if any danger.

A similar controversy has blown up over CR’s condemnation of bovine somatotropin (bst), a genetically engineered hormone to increase milk production in cows. Again, the magazine’s original warning in the May 1992 issue (udder insanity) was widely reprinted, prompting consumer resistance to the use of bst on dairy cows. And again, a small battalion of other environmental watchdogs refuted the claims of danger, including the A.M.A. and the Food and Drug Administration.

Dale Bauman, a Cornell University professor of nutritional biochemistry, claims that Consumer Reports erred in its bst story by relying solely on its in-house critic in the field, Michael Hansen, rather than a panel review. “Nobody has the credibility to handle a wide range of issues,” Bauman says. He adds that he found a Hansen report on bst “replete with mistakes” and that the fda sent Hansen a letter listing all the errors. “I don’t think this helps the magazine’s credibility,” Bauman observes.

CR’s technical policy and public-service director Edward Groth III calls the fda letter to Hansen “hogwash, a propaganda document put out to discredit his report.” Groth defends Hansen’s expertise and explains CR’s position on bst: “The literature shows there is possibly a problem but no conclusive proof. Scientist A says we should be cautious. Scientist B says let’s go ahead. Science sometimes carries more weight than it should. Science is good, but in policy you need value judgments too.”

But is setting policy, rather than rating products, CR’s purpose? Richard Greenhaus, 65, was a testing engineer for the magazine for 18 years; he left in 1990 because, he says, CR had “changed from an interested consumers’ group to a bureaucracy behaving like a branch of government.” Greenhaus charges that Consumers Union president Rhoda Karpatkin “is a politically active Democrat” and that CR has changed from conducting disinterested research to promoting a consciously liberal agenda. Says Karpatkin, citing a 1986 letter from Ronald Reagan praising Consumers Union’s “competence and impartiality”: “I do not engage in partisan politics at all.”

This back-and-forth sniping suggests that the world has grown more complicated than it was when Consumer Reports made its debut almost 59 years ago. Subscribers still swear by it. Cheryll and Tommy Schechtman, both in their early 40s, recently built a new house in Palm Beach Gardens, Florida, and wanted to upgrade all the major appliances on a $10,000 budget. Says Cheryll: “We’re pretty trusting about what Consumer Reports recommends.” But the exponential growth of new products threatens to swamp the magazine that was established to rank them for the buying public. Hundreds of new computer-software programs and games appeared in the months preceding the last Christmas buying season. “We’re not going to test software in the foreseeable future,” admits CR’s Alan Lefkow. “It is beyond this department.”

In the beginning, Consumer Reports and Consumers Union were virtually alone in calling for a fair shake for buyers; now the field is jammed with such advocates-some of whom regularly snipe at the magazine that started the trend in the first place. And with the g.o.p. controlling both houses of Congress, and a move from government-as-watchdog to a deregulated economy, CR’s brand of activism may draw increasingly thunderous criticisms. If so, will the noise matter inside a 30-ft. by 30-ft. by 30-ft. echoless chamber?

–Reported by Tom Curry and Stacy Perman/Yonkers and Jane Van Tassel/New York

More Must-Reads from TIME

Contact us at letters@time.com