Changing Technology Requires A New Look At Enterprise Email Management.

This article is the second in a two-part series. The first part appeared in the June issue of CTR.

Critical information is routinely destroyed or lost among the mountains of deleted or lost messages. As a result, organizations now face a difficult choice: Either accept the high cost of system administrators and user productivity losses when deleted information must be restored from backup tapes or re-created by the users, or accept the mail system performance and reliability degradation that occurs when the message store is allowed to expand unrestrained. A study by Creative Networks (Palo Alto, CA) found that restoring and re-creating deleted messages represents a significant cost for organizations, averaging $193 per user in large Exchange installations, which is more than the cost of all technical support and help desk combined.

In addition to the administrative and productivity costs, information lost from the email system poses other problems. Companies are beginning to realize that email messages and attachments represent business records that must be retained and managed securely in order to support regulatory compliance, avoid legal fines or litigation costs, and satisfy auditing requirements. Increasingly, corporate legal departments are trying to impose corporate standards on email archiving and retention procedures. In addition, email represents an increasingly large part of the corporate "knowledge asset". Organizations want to leverage this asset through knowledge management tools, sometimes referred to as data mining, to maintain a competitive advantage.

According to David Ferris of Ferris Research (San Francisco), there is a growing market for archiving and retrieval products in the email management space. Companies that are legally required to implement some form of records retention rule or schedule to hold their data are likely to be early adopters of these archiving and retrieval products.

The two most likely types of early adopters, according to Ferris, will be:

1. Organizations that are subject to government regulations, such as utilities, federal and state government, communications, education, and defense.

2. Organizations that are subject to product liability lawsuits, such as industrial, retail, and high technology companies.

Ferris further states that the best prospects for the service are businesses that meet both criteria, including such sectors as banking and brokerage, pharmaceutical and medical, chemicals, insurance, energy, automotive, aerospace, consumer/food, and engineering.

Doculabs' Richard Medina comments that in order for this type of application to be truly useful it needs to make life for the IT people easier. According to Mr. Medina, any email archiving application must address three different problems:

1. IT/Operational Issues--can't impact performance or increase volume of storage.

2. Records Management--demonstrate regulatory compliance, audit ability, and legal compliance.

3. Strategic Benefit--handling and enhancing e-business concerns.

Messaging Bandwidth Demand To Grow 3 - 5 Times

According to another David Ferris study, in only 12 months companies can expect their messaging bandwidth requirements to be three to five times that of today. Most of this growth will be driven by a rapid increase in average message size, as more messages include attached word processing, spreadsheet, and presentation software files. Today's business users send about 15 email messages a day and receive about 20 messages a day, on average. This volume is expected to grow by some 60% and 80% respectively over the next year.

Most users keep hundreds of old messages. As the number of daily messages grows, and those messages increase in size, organizations may see a growth in storage overhead of 100% to 150%.

According to Ferris, "One of the most surprising findings of the study was that despite the increase in message traffic and the increase in average message size, IT managers only forecast a growth of 50% in their message storage requirements over the next 12 months." He believes they are grossly underestimating the impact of growth on their infrastructure.

Ferris points out that these growing personal libraries of email messages are of growing interest to top management concerned about the company's exposure to costly legal discovery processes.

"We recommend that clients start looking for tools to manage, archive, and delete old messages," said Ferris.

Storage analysts agree with this assessment. According to Jack Scott of the Evaluator Group, a consulting firm focused on aiding clients with mass storage issues, "Email is the leading consumer of storage." It is estimated that 5.1 billion messages worldwide are generated per day.

As a counterpoint to this, Scott points out that storage companies in general do not recognize document management, records management, or email management as an opportunity, mainly because they don't receive any significant demand for it from their users. Email administrators tend to understand this problem but historically have little clout within their organizations.

Storage management is no longer just about archiving data. The quantity of data being archived has become an unnecessary burden to IT. Some organizations predict that just thirty percent of their email data actually needs to be archived, and out of that thirty percent, all of it requires proper categorization and retention management. This has created an unnecessary burden on organizations in the following areas:

* Storage costs

* Access to the right information at the right time

* Litigation risk and protection

* Compliance with corporate policy and regulations

Most analytical reports today cite exorbitant factors when projecting enterprise data growth over the next five years. For example, a META Group analyst who spoke at a May 2000 eCRM conference projected data increases of a hundredfold within five years through the year 2004. Enterprises that are having difficulty coping with three terabytes (TB) of data today need to quickly find solutions for dealing with 300 terabytes of data tomorrow. Since the May 2000 conference, other META representatives have validated this growth factor as well as the urgent need for in-depth strategic data management planning.

A recent Deutsche Banc Alex Brown data study found that e-business data would grow from 30% of the total data in the first year of activity (1999 in most cases) to 75% of the total data in the fourth year. This growth represents a commanding data swell of 400% per year.

Understanding and acceptance of these predictions comes only after considering the scope of new business initiatives and the technological capabilities that both enable and support them. New e-business applications like web-front management (clickstream analysis), one-to-one customer relationship management (CRM), personalization and encounter management, supply chain management, call event detail analysis, and digital certification significantly add to an enterprise's existing IT-supported agenda. In addition, new nonscalar data types (objects) including images (drawings, X-rays, etc.), streaming audio, and video dramatically expand the data inventory.

Surging TCO Costs

It's easy to suppose that somehow more disk storage and technological advances will ease the cost of managing this tidal wave of new data. However, even with the expected continuation in RAID price/TB decline, the total expenditure necessary to accommodate the projected data growth will escalate more than tenfold in the next five years.

This message is startling. Over the next five years, given a six-fold decrease in price/TB and a hundredfold increase in data, a thirteen-fold increase can be expected in total data management costs.

Email Management: A Complex Problem

Once email has been recognized as a significant issue for organizations to manage, why can't the problem be solved through awareness and appropriate business practices? Unfortunately, the answer leads back to the original intention for the usage of email systems as just a very simple and easy means of communication. The mindset has been that it was much less formal than a letter and considered insignificant once it had accomplished its original intent of simple communication. This has been done with practically no thought of how long the email was to be retained.

This entrenched email usage pattern has become so highly recognized that many companies are attempting to provide solutions to this problem. However, addressing the issue goes much deeper than it appears on the surface. Even if organizations are able to convince themselves and their email users to begin managing their email information properly, the problem that remains is, what does this mean? If a company creates locations or adds tools to provide email management functionality, how does everyone know what emails need to be managed?

Central policy and proper business practices at a corporate level are required to establish these rules. Many organizations already recognize that email created in the course of business is corporate information, not personal, and must be treated as such. But just establishing policy is insufficient; everyone must be trained not just on how, but on what to file as a corporate email and what to discard. This requires extensive training and leaves categorization open to many sources of interpretation and error.

The following methods have shown little or no success:

Manual email policies on top of MS Exchange/Outlook usage. Some organizations are placing policies on email usage that require email users to sort through their email data and keep only what they need. Unfortunately not everyone knows what data is required to be kept for the benefit of organization as a whole. End users also tend to be burdened with so many other tasks they either do not have the time or the incentive to participate in supporting labor-intensive email policies. Training costs become high, interpretation of what is pertinent to the organization becomes varied and disjointed, digital corporate assets slip through the cracks and redundant data is being archived; in all a high cost to the organization.

Traditional Records Management (RM) packages/ engines. Traditional RM packages that comply with the DoD 5015.2 standard for electronic records management include basic hooks for managing email. These hooks however, do nothing more than add additional functionality for the end user to perform the archiving; once again a manual intensive solution as described in the previous point.

Email archiving with manual and simple automated business rules. Currently available email archiving packages depend on unsophisticated rules that already come with MS Exchange/Outlook. These systems do not manage a full-scale retention plan and do not have intelligent learning software to actually understand the content/context of an email to automatically determine if it is important and where to store it.

What Has Effectively Addressed This Problem?

With the above examples, it is clear what is not working, but what can be done and what will provide an effective solution? TrueArc (formerly Provenance Systems, Inc) has eleven years experience in solving this specific problem. As a result, TrueArc's solution for Microsoft Exchange using ArcIQ is an extremely sophisticated and accurate software intelligence engine capable of providing completely automated identification and categorization of electronic email records.

Solving The Problem

Most automated systems tend to use simple rules and/or keywords as a means of automating the capture and categorization of email messages. Unfortunately, unless the email content has the specific keywords or structure to satisfy the rule, the email will be overlooked and the solution will be ineffective.

ArcIQ is trained on a sample set of email messages to learn and understand the patterns (terms) that are most statistically significant across messages already archived in each category. This enables email messages to be evaluated in realtime with automatic determinations on whether the email should be archived or eliminated. ArcIQ can be combined with an organization's business rules to provide a very sophisticated method of automated capture that is more effective than training all end users. Studies have shown that end users are already overloaded and are therefore not willing to participate in archival training sessions. End users are very inconsistent when it comes to identifying what email messages should be archived and where they should be stored. ArcIQ is unbiased, consistent, and tireless, with the ability to be retrained at any time for constant improvement and accuracy.

It has been shown that the normal business email user will attempt to retain or store an average of 500 megabytes of email per year. An organization of 3,000 users would require over 1.5 terabytes of additional storage every year. However, it is possible to reduce the amount of storage by as much as 70% by applying the intelligence provided by ArcIQ.

This represents not only a considerable savings in storage hardware costs, but also through more effective use of the employees' time. Employees are not required to evaluate each and every email to determine not only should it be archived, but also where it should be stored. Additionally, because ArcIQ consistently categorizes the retained email messages, retrieval can be accomplished in a real-time, interactive environment.

Tim Shinkle is CTO at TrueArc (Arlington, VA).