Seek on the Net, and Ye May Not Find

  • Share
  • Read Later
Looking on the Internet for that proverbial needle in a haystack? Don’t count on any of the much-touted web search engines to find it for you. That’s the conclusion of two New Jersey computer scientists whose search engine study was published in Thursday’s issue of the journal Nature. The researchers found that the most comprehensive engine, the relatively unknown Northern Light, was able to cover only about a sixth of the estimated 800 million pages in cyberspace. The coverage of the better-known engines was even less impressive (Altavista, 15.5 percent; HotBot, 11.3 percent; Infoseek, 8 percent; Yahoo, 7.4 percent; and Excite, 5.6 percent). Worse, the researchers concluded that the engines are being overwhelmed by the burgeoning web, covering a diminishing fraction of pages with the passage of time and taking longer to list new sites.

"This is yet another confirmation that the Internet is a wild jungle growing out of anyone’s control," says TIME writer-reporter Chris Taylor. "Not even the greatest search engine can cover more than 16 percent of it." So what is a surfer to do? As a short-term solution, says Taylor, "use more than one search engine." For the longer term, the cyberspace industry is going to have to develop new approaches. A promising one, says Taylor, "is the attempt by some companies to develop a system for archiving the whole Internet." Such a system will not only have to be swift enough to keep pace with the thousands of sites born each day, but smart enough to get around those that die each day too. Not to mention transparent enough to be consumer-friendly.