Data Miners

Quick: What do Hamas terrorists have in common with Martha Stewart? No, we're not talking about their public-approval ratings. Rather, both may have drawn unwanted scrutiny in part because of the same piece of software.

The data-mining algorithms of ClearForest, based in New York City, are at work within both Israeli security agencies and NASDAQ. Israel uses them to drill for hidden connections among suspected terrorists: say, a pattern of phone calls shortly before each of several suicide bombings. NASDAQ uses the same software to detect block trades of stock quietly placed just before the release of company news including sales by relatives of ImClone's founder, Sam Waksal, who this fall pleaded guilty to insider-trading charges, and his friend Martha Stewart, who remains under investigation (and has denied any wrongdoing).

Both NASDAQ and Israel's security services are sprawling organizations, bombarded daily with terabytes of information, any bit of which may prevent a catastrophe, whether measured in lives or in retirement savings lost to fraud. And these days, data-mining software, combined with technologies that connect disparate computer systems and databases, is making it possible for everyone from police departments to clothing merchants to global manufacturers to search through ever expanding data warehouses and draw valuable connections that would otherwise be lost to human eyes.

Consider the British aerospace firm BAE Systems. Software developed by Autonomy, based in Cambridge, England, connected BAE's research databases and alerted civilian aircraft engineers to the fact that the wing-construction problem they were working on was also being addressed by the company's military division. Ending this duplication helped the company save millions of dollars.

Even as the application of data mining has spread throughout the retail, manufacturing, pharmaceutical and financial sectors, the lingering tech slump has slowed the technology's growth. This year will go down as the worst in history for information technology, with sales falling 2.3% after averaging a 12% annual growth rate over the past 20 years. Data-mining companies have been among the hardest hit in recent years: MicroStrategy, based in McLean, Va., rode the near messianic predictions of founder and CEO Michael Saylor from a market capitalization of $1 billion to more than $20 billion and back again in less than 18 months.

Few companies attained the heights of MicroStrategy or the ignominy of its fall after it confessed to overstating earnings. But the combination of shrinking corporate IT budgets and a lack of interest among small and midsize companies laid waste a host of firms that once promised to be the brains of the information revolution.

Today, however, companies that excel in connecting the data dots are finding a lifeline in a customer whose IT ineptitude is matched only by its means: the U.S. government, which will spend $53 billion on information technology this year. The Federal Government's inability to share and analyze information became clear in the months after the 9/11 attacks. An FBI agent in Phoenix, Ariz., who was suspicious of Arab flight-school students was not aware that he had a colleague in Minneapolis, Minn., with the same concerns. Immigration officials didn't know that a Saudi suspected by the CIA of terrorist ties had applied for a visa. "If you look at the lessons learned from Sept. 11," says Ed Sketch, director of Ford's Learning Network, which uses Autonomy software to categorize and distribute information to all its salaried employees, "this technology is slap-bang in the middle because it sorts the relevant from the irrelevant to keep an organization from drowning in data."

It didn't take long for data-management companies to realize that if their software could find links in customer buying patterns and improve retailers' inventory decisions, perhaps it could find links among the government's vast terrorism-related intelligence warehouses and enhance the government's ability to prevent the next attack. After 9/11, many tech companies saw opportunities for both patriotism and profit. Oracle offered to donate the software to create a federal identity database. Siebel Systems CEO Thomas Siebel boasted to a House subcommittee that had his company's software been used by law-enforcement and intelligence organizations before 9/11, "there may have been a different outcome."

Promises of quick fixes have faded as the scale of the government's challenge has become clear. But the early setbacks have not deterred companies from setting up shop in or near Washington to get closer to the action. In the past year, Raytheon and EMC launched a joint government IT unit, and SAP relocated its global public-sector headquarters to the Washington area. And PeopleSoft launched two new products designed for the Homeland Security market.