Pigeonholing text.

The enormous task of categorizing and retrieving information from the vast quantities of text stored in digital form has spurred the development of a variety of strategies for finding the textual needle in the database haystack. Most of these automated techniques rely on the identification of specific words and phrases after sentences and paragraphs are stripped of extraneous material (SN: 9/7/91, p.155). However, such methods often require some degree of expert human participation in their development and setup. They have trouble with misspellings and garbled text, and they are usually suitable only for specific topics or languages.

Now, Marc Damashek of the Department of Defense's National Computer Security Center at Fort George G. Meade, Md., has developed a text categorization and retrieval technique that works equally well in any language and requires practically no human preparation. His method, known as Acquaintance, is purely statistical. "No prior information about document content or language is required," Damashek says.

His software divides text samples into sequences made up of a given number of consecutive characters, then computes how often each distinct sequence appears in the document. To gauge similarity, Damashek assumes that two documents showing comparable patterns are likely to deal with related subjects.

Tests of the technique show that it performs well for grouping documents by language, topic, and subtopic, Damashek says. He describes the method in the Feb. 10 Science.