Google's War Against Rotten Search Results

The search giant fights to defend itself against spammy intruders.

By Harry Mccracken Thursday, Mar. 03, 2011
    • Illustration by Topos Graphics for TIME

    Google's algorithm—the legendary, largely mysterious technology that determines which sites get shown in search results and in what order—is good. It's good for the people who quickly find useful stuff on the Web. It's good for the sites they find. And it's very, very good for Google, which handles two-thirds of all searches in the U.S. and reaps billions from the ads it displays.

    Not all media coverage of Google-the-company is a lovefest, however. Products such as its Twitter clone, Buzz, have been pummeled by the press ; incidents such as its accidental collection of data from Wi-Fi networks during Google Maps Street View photography excursions have been deeply controversial . But Google-the-search-engine has worked so well for so long that it's rarely been subject to serious criticism.

    Lately, though, some professional Googlewatchers have wondered if the algorithm—which is supposed to measure both a site's popularity and relevance by tallying links, keywords, and other factors—is broken. In November, for instance, the New York Times' David Segal reported on a seller of designer eyeglasses whose marketing strategy was to rise in results—by getting the Web talking about how lousy its customer service was. The same reporter wrote last month about JCPenney.com's #1 position in results for searches such as "dresses," "bedding," and "area rugs," which stemmed at least in part from bogus links at irrelevant sites such as BulgarianPropertyPortal.com.

    Much of the recent negativity has involved pages with a smattering of questionable content and a ton of related advertising showing up at or near the top of search results—such as a few shoddy paragraphs about dishwashers wrapped in paid-links to dishwasher merchants . In a perverse twist, the ads that dominate the page are often sold by Google and placed on the site; when visitors click, the site proprietors get paid a finder's fee.

    All along, Google has been responding in fits and starts. After the Times story on the rogue eyewear merchant appeared, the company announced that it had revised the algorithm to detect negative sentiment about online sellers and ensure that it didn't give them a boost. It also reacted to JCPenney's unnaturally high positioning by manually demoting the site in results.

    Then, on February 24th, the company made its highest-profile moves so far to buttress the quality of its results. U.S. users started getting a major update of the search engine with aims to push low-grade pages that unfairly game the system down in results. According to Google, 12 percent of all searches will now get different results than if it hadn't made change.

    "Scientific measurement shows our search quality [was already] the highest it's ever been," says Google Fellow Amit Singhal, one of the keepers of the algorithm. But he acknowledges that the company started noticing more sketchy items slipping into results, months before the blogosphere went ballistic. "We've been at it a while," he says of the effort the company has invested in the new revisions. "Making knee-jerk changes is not in our DNA."

    Search-engine upstart Blekko recently gave some major content farms the death penalty: Twenty sites that its users flagged most often as spam no longer appear in results, period. Google isn't going that far, and won't talk much about its precise tactics—it doesn't want to teach sneaky sites how to work around them. Singhal did tell me that the changes don't target specific companies and sites. And rather than eliminating pages altogether, it's simply making it tougher for them to wriggle their way to the top of the first of the first page of links, where most users will see them.

    Some of the sites that may suffer offer nothing but material swiped from elsewhere or gibberish stuffed with keywords. Others, such as EzineArticles, are overrun with contributors who are bigger on vested interests than writing skills. "Driving in a limousine is just like a fantasy," explains one "article" there. "There will be thousands of people in the world who'll be longing to get a ride in limousine." If you burrow through to the end, you learn that the piece was written by, yup, a limo driver.

    The trickiest matter is material produced by major "content farms" such as Demand Media (parent of eHow) and Yahoo's Associated Content. These companies mass-produce articles tailored to match Google's top queries, and they're not consistently terrible—they're just inconsistent, period. Small wonder: They crank out items at such high volume, for so little cost, that they're at the mercy of the armies of freelancers who write for them. Associated Content, for instance, recently sought a 300-word report on what it's like to undergo "Tommy John" elbow surgery. It offered up-front pay of eight bucks.

    Only a few days have passed since Google's big upgrade, thus it's premature to gauge the lasting impact—in part because the content producers it hurts will surely regroup. "There's always been this cat-and-mouse game," says Greg Sterling, an editor at Search Engine Land. "This is another episode."

    According to one independent analysis , the overall rankings for EzineArticles and Associated Content had tumbled, but eHow's had actually improved. Some sites that are neither content farms nor shady spammers, such as Apple-news specialist Cult of Mac, seem to have gotten caught in the crossfire and have seen traffic plummet; Google says it's working to undo any unintentional damage .

    All Google search changes, including the new ones, are subject to further adjustment: The company experiments with 6,000 modifications a year and rolls out 500 of them. It has every reason to do whatever it takes to preserve its algorithm's long-standing reputation for excellence. If consumers start to regard it as anything less than good, it won't be good for anybody—except other search engines.

    McCracken blogs about personal technology at Technologizer , which he founded in 2008 after nearly two decades as a tech journalist. His column for TIME.com, also called Technologizer, appears every Thursday.