Quick: What do Hamas terrorists have in common with Martha Stewart? No, we're not talking about their public-approval ratings. Rather, both may have drawn unwanted scrutiny in part because of the same piece of software.
The data-mining algorithms of ClearForest, based in New York City, are at work within both Israeli security agencies and NASDAQ. Israel uses them to drill for hidden connections among suspected terrorists: say, a pattern of phone calls shortly before each of several suicide bombings. NASDAQ uses the same software to detect block trades of stock quietly placed just before the release of company news--including sales by relatives of ImClone's founder, Sam Waksal, who this fall pleaded guilty to insider-trading charges, and his friend Martha Stewart, who remains under investigation (and has denied any wrongdoing).
Both NASDAQ and Israel's security services are sprawling organizations, bombarded daily with terabytes of information, any bit of which may prevent a catastrophe, whether measured in lives or in retirement savings lost to fraud. And these days, data-mining software, combined with technologies that connect disparate computer systems and databases, is making it possible for everyone from police departments to clothing merchants to global manufacturers to search through ever expanding data warehouses and draw valuable connections that would otherwise be lost to human eyes.
Consider the British aerospace firm BAE Systems. Software developed by Autonomy, based in Cambridge, England, connected BAE's research databases and alerted civilian aircraft engineers to the fact that the wing-construction problem they were working on was also being addressed by the company's military division. Ending this duplication helped the company save millions of dollars.
Even as the application of data mining has spread throughout the retail, manufacturing, pharmaceutical and financial sectors, the lingering tech slump has slowed the technology's growth. This year will go down as the worst in history for information technology, with sales falling 2.3% after averaging a 12% annual growth rate over the past 20 years. Data-mining companies have been among the hardest hit in recent years: MicroStrategy, based in McLean, Va., rode the near messianic predictions of founder and CEO Michael Saylor from a market capitalization of $1 billion to more than $20 billion and back again in less than 18 months.
Few companies attained the heights of MicroStrategy or the ignominy of its fall after it confessed to overstating earnings. But the combination of shrinking corporate IT budgets and a lack of interest among small and midsize companies laid waste a host of firms that once promised to be the brains of the information revolution.