Data-mining initiatives. (Up front: news, trends & analysis).

To harness the vast information flow generated each day, computer scientists are developing sophisticated software that can instantly mine streaming data without ever needing to archive it.

According to The Christian Science Monitor, specialists worldwide are working to enable users quickly to navigate and retrieve information nuggets from vast storehouses of data that are growing at unprecedented rates by developing new tools to "mine" digital databases and the streams of information that feed them.

Data storage is growing by leaps and bounds, experts say. For example, each day the U.S. intelligence community collects information equal to all the printed pages in the U.S. Library of Congress. The World Wide Web is growing by more than 1.5 million Web pages daily, taxing the ability of the current generation of search engines to track down the answer to a user's query quickly and accurately. Overall, the average size of a database and the software needed to use it are growing faster than computer-processing speeds, which double about once every 18 months.

Meanwhile, storage capacity has grown even as the cost has plummeted. Four years ago, a credit-card-size hard-disk storage device in a typical consumer laptop computer might have held 6 gigabytes of data--enough space to store six conventional movies. Today, laptops have hard drives capable of holding from 40 to 60 gigabytes and more.

Such trends are prompting researchers to explore more sophisticated data-mining technologies. In addition, recent U.S. government initiatives such as the Homeland Security Act may require an ability to search existing financial, criminal, immigration, or other fixed databases, as well as to monitor streaming video and audio sources for evidence of potential terrorist activity.

The federal government's Total Information Awareness (TIA) project, spearheaded by the Defense Advanced Research Projects Agency (DARPA), has been called "the mother of all data-mining projects" by some. The research and development program, headed by John Poindexter, aims to identify, track, and prevent individuals from planning and organizing terrorist activities. Much of the effort, according to Poindexter, will focus on unifying and probing databases that carry information on financial transactions. The program will also create large databases that sift through the purchases, travel, immigration status, income, and other data of millions of Americans. There are three parts to the TIA project:

* Voice recognition--sifting through electronically recorded transmissions and providing rapid translations of foreign languages

* A tool to find connections between transactions such as passports, airline tickets, rental cars, gun or chemical purchases, arrests, and other suspicious activities

* Collaboration--a mechanism to enable information and analysis sharing among agencies

Experts say the government may have this technology up and running within a year. But the key will be facilitating information sharing among departments. The technology to mine these data sources is there today, but developing systems to talk to each other may be a challenge for some time to come.