End Email Chaos: An Introduction To Email Data Management.

Editor's note: The following is excerpted from a Special Report published by OTG Software.

Email is now the information lifeblood of business. It is so reliable, fast, inexpensive, and widely accessible that it has displaced traditional corporate communication systems--including postal services, couriers, faxes, and to a lesser extent, telephones--and become an integral part of daily business operations. In fact, it has become a treasure trove of information critical to the ongoing success of the enterprise--now holding an estimated 60% of the average company's most vital business data (up from 33% in 1999).

Unfortunately, business email is quickly becoming a victim of its own success. While email volume continues to grow at exponential rates, its value as a business tool and data center is being seriously compromised by a lack of centralized administration and record management inherent to traditional communications systems.

With no central control or administrative support for classifying, indexing, filing, storing, and retrieving the 3.5 billion email messages that U.S. businesses are transmitting daily, it's no surprise that everyone is experiencing some sort of fallout.

A Recipe For Email Chaos

The tidal wave of email is affecting everybody, from users and IT administrators, to management and record keeping professionals. Typical users find themselves swamped with an average of 70 email messages per day. They can barely find time to respond, never mind devise a method for managing their personal email archives.

Thus, it becomes increasingly difficult to locate information when it's needed, and users spend countless hours searching for or reproducing data that is often inaccessible or lost. Over 80% of end users cannot retrieve archived emails without the help of an IT administrator. Even with IT assistance, 26% of companies still cannot retrieve emails from backup.

As a result, the burden on IT administrators continues to grow. The average IT administrator spends 5-6 hours each week recovering old messages for end users. (For messages that are a year old, it can take an email administrator more than 11 hours to recover a year-old message from an archive.) Limited processing and storage capacity of workstations further hinders the ability of users to accomplish their business objectives. Research shows that a typical 3,000-user email system handles more than one terabyte of message traffic annually.

In addition, IT administrators spend another eight hours each week backing up and archiving emails so that burgeoning message stores do not bog down the email servers. Because end users don't want to discard messages, there is a constant struggle to provide adequate storage space without compromising system reliability. Not only are IT managers responsible for ensuring continuous availability of the server, they must protect email communications during and following virus attacks.

Meanwhile, management needs to protect confidential information contained in email from loss, theft, or inappropriate disclosure. Just as important is an email policy that prevents the waste of employee time and computer resources due to junk mail, spam, chain mail, and other frivolous or nonproductive mail. One multinational company placed the cost of junk mail at one dollar per employee per day.

Finally, despite the fact that email is an integral part of business communication, the lack of administrative control over email jeopardizes record management procedures and the ability to comply with regulatory and legal requirements. Without a formal filing and retention policy, past emails become a maze of unrelated communications that make responding to legal discovery and FOIA requests time consuming and costly. So time consuming and costly, in fact, that many organizations opt to risk noncompliance or settle disputes rather than incur the expense of retrieving archived email.

Current Practices In Email Data Management

In an average office, each user is currently responsible for managing and archiving the estimated 300MB of email received on his or her desktop annually. Even when desktop archive utilities and training are provided, the results vary according to individual user needs and work habits, especially given the lack of formal policies on message categorization or retention. The lack of formal retention policies also contributes to swollen message stores, as users dislike discarding messages, which can lead to service shutdown. Therefore, an unmanaged collection of personal archives and over-full mailboxes is not in the best interests of the organization.

With responsibility for email system performance and availability, the IT department is concerned with preventing over-full message stores--a serious and routine threat to email server performance. Because one email server houses the mailboxes of multiple users in an organization, the server can easily fill up, resulting in a system shutdown, and consequently cutting off email service to those users. To safeguard against overfull message stores and ensure removal of old messages, current IT practices include sending "delete your messages" notices to all users, or blind purging of the message stores. In the case of Microsoft Exchange, which accounts for approximately 50% of the email market, the send and receive services are shut down when the message store exceeds its limit (after administrative notices have been sent). In addition to reducing the volume of messages on the server, such practices also encourage IT organizations to implement guidelines that treat old email messages as potential liabilities and recommend actively removing them from server message stores.

Current IT practices to minimize message storage also include encouraging personal desktop archives, routinely making backup tapes of all email servers, and restricting the size of messages or attachments. Since personal archives will be spotty in that not every employee will properly archive email, and in that they might be discarded should the employee leave the organization, backup tapes become the only centrally controlled and available archive of message stores. When it is necessary to recover messages--either for enterprise users in need of information or for discovery requests--IT administrators can do so only by accessing the data on backup tapes-an expensive and time consuming task. For example, recovering all archived messages that concern a specific subject requires initiating a server with the appropriate email program, loading an archive tape, and searching the tape for relevant email messages--over and over, until all the tapes have been loaded and searched. With the cost of searching an indivi dual tape running as high as thousands of dollars, this method is extremely costly.

Limits on message or attachment size can impact an organization's effectiveness. For example, if a sales person is unable to receive a large RFP as an attachment to email, the organization cannot respond, and loses the opportunity. Similarly, if an employee is unable to send a large file in response to a customer request for information, it reflects poorly on the organization.

Conflicting with the need to maintain system availability are two factors: (1) the onus on the IT department to ensure disaster recovery, and (2) the need to retain email for the necessary length of time to enable the organization to satisfy external requests for information--such as SEC audits, due diligence requests, FOIA requests, and legal discovery requests.

Information contained in old email messages may also be required for any number of internal business purposes. For example, the sales department may need to access a draft of a contract; customer service may need to review communications that promise certain rewards to customers; human resources may have a question about an employment offer. And because each individual user decides which email messages to retain or delete, much of this information is hidden from the rest of the organization. Thus, when an employee leaves, or is suddenly unavailable or even on vacation, the treasure trove of mission critical information in his archived email could be lost.

Meanwhile, discovery requests may arise as a result of FOIA requests in state or federal organizations or as a routine part of enterprise litigation. Regardless, responding to these requests places a significant burden on IT resources. Backup tapes must be restored to a replica of the production email server. A search may span several email servers and, in some cases, the personal archives on hundreds of desktop computers. The search may also need to cover a series of backup tapes made through a target calendar period. Thus, the estimated costs for fulfilling a single discovery request run from tens to hundreds of thousands of dollars. For example, the cost of recovering 246,000 emails from approximately 4,900 backup tapes will cost the White House an estimated $10 million. No wonder many companies choose to settle cases, even when they are not at fault, so as to avoid the high cost of discovery.

Guideline For Effective Email Data Management

Government and industry organizations have worked for several years to develop record management guidelines for electronic documents, communications, and email. Several publications can serve as references for various types of organizations.

One such reference is the DoD 5015.2 specification, titled "Design Criteria Standard For Electronic Records Management Software Applications," which is designed to meet federal (NARA) guidelines and existing regulations.

For corporations, ARMA International released the draft version of the "Guideline for Managing Email" in April of 1999. ARMA's International Standards Advisory and Development Committee and the members of its Email Guideline Task Force received input from a wide range of state, federal, educational, and commercial representatives to develop the guidelines.

For financial institutions regulated by the SEC, requirements for managing electronic communications including email were published as amendments to SEC Rule 17 in February of 1998.

While these various guidelines apply to a diverse range of organizations, it's clear that any effective solution for managing email must include the following capabilities:

1. Archival of email documents and attachments--and associated address and routing information--in original electronic form;

2. Creation of an email policy that addresses message retention and filing requirements;

3. Automatic and content-based classification of email to folders/categories within the system;

4. Creation and execution of disposition instructions for each email folder/category;

5. Protection of the archive against unauthorized access;

6. Inviolate auditing of administrator access to archived documents (i.e., administrators cannot tamper with audit records);

7. Powerful search and retrieval tools for end users and administrators, based on a full-text index and user-defined metadata; and

8. Use of random-access, lowcost, and non-volatile media for long-term storage.

Goals Of Enterprise Email Data Management

Ideally, a message store management solution should both address the basic requirements stemming from published guidelines and add value by bringing the power of data management to email systems--in essence, channeling the daily stream of email messages and attachments into a tool that provides competitive advantage.

Any such solution should address the requirements of users, IT administrators, management, and record keeping professionals by providing the following benefits:

* Reduced IT Costs--Superior Email Server Management. A product that combines automated capture, integrated support for low-cost mass storage, and content-based classification rules can transform an organization's temporary cache of messages on the server into a tool for enterprise document management. Email servers are freed of message overload, improving performance and availability. Additionally, without the requirement to restrict message size, larger and more complex documents can be transmitted. As a result, email can serve as the basis for strategic planning and business development efforts, without impacting IT.

* Reduced Costs for Routine Data Retrieval. Research shows a typical user will need to recover messages or documents from an email archive about 15 times a year. Subsequent analysis shows that large companies spend an average of $193 per user, per year to retrieve messages from their email archives using current methods. Reducing the per-user, per-year costs of retrieval should be an inherent factor in any message store management solution.

* Quicker, Lower-Cost Discovery Actions. A product with full-text indexing and cataloging features enables email discovery actions to be completed in hours instead of weeks, greatly reducing expenses. A 'freeze' feature that may be used at the onset of any investigation to protect message categories from automatic destruction is also desirable.

* Increased Productivity -- Fast and Efficient Access to the historical body of email messages and attachments. Full-text indexing combined with a powerful search engine would allow end users to access their own messages, while only authorized managers or administrators could search across multiple mail-boxes to meet audit/regulatory requirements, discovery requests, or other business needs.

* Record Management. An effective solution ensures adherence to formal email policies with enterprise-level data management tools that categorize and manage email through a useful life cycle. By using a tool that integrates record management functionality into email systems, organizations can easily comply with SEC. federal, and state/local requirements.

* Virus Recovery and Data Protection. A message store management solution that ensures email records are tamper-proof throughout their life cycle and guards against abuse by monitoring compliance with corporate policy is critical. Additionally, it should facilitate clean up of message stores following virus attacks, enabling recovery within hours rather than days and guarding against loss of information.

Chris Gray is the director of product management at OTG Software (Bethesda, MD).