Reducing business risk through cost-effective e-mail management: "can legacy systems effectively manage today's data volumes and varying data types?".

Sybase

Organisations are facing increasing risk because they cannot control what their employees do with e-mails. When asked to provide e-mail records by a regulator they incur considerable cost and often fail, sometimes facing fines and litigation. currently e-mail systems are based on legacy technology and, according to analysts, can cost over $2000 per user, per year to maintain. yet, despite these high costs, accurate records are not kept, backups take days to run, and information is lost.

The Challenge Of Unstructured Data 

Organisations are under more pressure than ever before to keep accurate records of their business activities. In many cases this is easy, as the databases that run applications like payroll or trading systems have very efficient record keeping mechanisms built-in.

Unfortunately, around 70% to 80% of the average organisation's records are held as unstructured data. this includes word documents, web pages, voice recordings, instant messages and e-mail.

For most people, e-mail is the tool of choice for all manner of interactions--everything from personal arrangements to legal contracts are communicated in this way. unfortunately, e-mail systems have no way of differentiating between the content of messages, which means that an e-mail about a birthday party is treated with the same "one size fits all" policy as an e-mail about an employee appraisal, or the legal details of a major contract.

Increasing regulation and litigation over recent years means that organisations now have a legal obligation to keep certain types of information for a specific period of time, while other information can be legally deleted much sooner. Several companies have had to pay multi-million dollar fines for not being able to find a specific e-mail, or in some cases because a regulator has found an incriminating e-mail, which could legitimately, have been deleted.

Organisations try very hard to keep as much e-mail data as possible, usually by storing backup tapes of each e-mail server for many years, and sometimes even keeping backups of users' local e-mail files, resulting in the data being kept forever.

Relying on such inefficient backup strategies is both difficult and expensive, as organisations often have an e-mail server in each office to provide users with the best performance. In big organisations there could be as many as 3000 individual servers to maintain and backup.

Distributed architecture is one of the main contributors to the backup problem. unlike Internet Service Providers (ISPs) that run large, efficient, single data-centres, other types of organisation have to run many local servers to provide the performance and features that users require, such as shared calendars and fast local messaging. a big disadvantage of having many local servers is that e-mails are duplicated almost every time they are sent, and organisations end up paying for each message to be stored many times in many locations. When the backup tapes are run, they pick up the duplicated e-mails and duplicate them further.

Regulated organisations often have to reproduce e-mail data for investigations by their own legal teams or for industry regulators such as the Securities and Exchange Commission (SEC) or the Financial Services Authority (FSA).

At the moment this is a very expensive process, as the It department has to send a team of people to painstakingly search through thousands of backup tapes. this may take weeks and may cost a significant amount, as each tape has to be loaded into an e-mail server to be accessed. an old tape may need the specific version of software that created it in order to retrieve the e-mails. Sybase has evidence of a company spending $2M looking for a single e-mail, and believes this is not an unusual sum. In many cases the search team do not find what they are looking for, but do find information that should have been deleted, which can have expensive consequences.

Many organisations have tried to implement local e-mail archiving solutions, but have run into difficulties as these solutions are locally managed and do not address the massive duplication of content and excessive use of storage. Because the local archives do not communicate with each other, there is no way of removing excess content and no way of being sure to eliminate a single e-mail when its retention period has expired. also, the e-mails are loaded into the archive without any useful form of indexing or compression, using the storage space as a large, unstructured receptacle, resulting in excessive storage requirements.

A further problem with existing archiving solutions is that they often fail to capture important inbound and outbound e-mails at the SMtP gateways where all an organisation's external e-mail is processed. capturing everything is crucial, particularly if a company uses mobile devices.

Why are e-mail systems so expensive to run?

E-mail is notoriously expensive for organisations to operate. according to analysts it is common to find companies spending more than $2000 per user, per year. this sounds unreasonable considering that an ISP account costs around $20 per month, and there are many free webmail services.

One reason for the high cost is the way that organisations run e-mail servers close to the users (in the same office). If a centralised architecture was deployed, similar to that used by ISPs, the costs would be much lower as there would be fewer servers to manage and backup. this is unlikely to happen, however, as the features users require depend on the provision of local servers.

A second reason is that historically, backup systems were designed to enable an organisation to re-build a server in the event of it failing. now, backup systems are being relied upon as the primary source of archive data even though this is not what they were designed for. as a result they are highly inefficient, volatile, and extremely expensive to run. In addition, e-mails deleted between backups are not archived.

Probably the biggest contributor to the high cost of running e-mail systems however is the fact that in a distributed architecture, duplicate records consume the majority of storage.

To summarise, the main factors contributing to the high total cost of ownership of e-mail are:

* Duplication resulting from the way e-mail protocols work

* A large number of servers to manage

* A large number of servers to backup

* Backup tapes are not designed for archive data

* A vast archive of backup tapes to manage

* High storage costs

* Data-centre overheads

* Operational costs

* Search costs for regulatory investigations.

Surprisingly, software and network costs only account for a small proportion of the total.

Reducing e-mail costs with Sybase

Sybase retention Server provides organizations with a completely different way of managing the information contained in e-mails. by using a data warehouse to act as the global e-mail archive, a significant amount of the cost associated with running e-mail systems is removed. Implementing a data warehouse means that duplication of stored documents is reduced and backup tapes can be used as they were designed to be used, purely for backup purposes. they do not need to be kept for more than 30 days.

E-mail administrators are given the tools they need to manage e-mails sensibly by choosing to restrict the amount of data that an individual is allowed to keep for example, or allowing users to search their own historical records in the data warehouse.

Administrators also no longer need to worry about maintaining local desktop (.pst) files for users, as all the data will be stored centrally. Keeping down the storage on users desktops makes the desktop environment more efficient, and reduces the risk of leaked data in the event of a lost laptop or other mobile device.

Once the e-mail data warehouse is in place and the mountain of backup data has stopped growing, a project can commence to systematically load data from old backup tapes into the e-mail data warehouse. the old data can be cleaned up, and any e-mails past their deletion date can be disposed of.

In the event of a regulatory investigation that requires retrieval of information contained in an e-mail, search parties no longer need to be sent to the tape warehouse to wade through a mountain of backup tapes. Implementing Sybase Retention Server enables any e-mail to be recovered in a few minutes, by a single person, dramatically reducing the cost involved.

A summary of the business benefits

Deploying Sybase Retention Server:

* Reduces business risk by providing a comprehensive archive

* Ensures compliance with regulatory demands

* Cuts the cost of completing an investigation

* Enables implementation of an efficient retention policy and deletion of expired e-mail

* Reduces operational cost through correct use of backup and storage

* Lowers the cost of storage by reducing content kept on e-mail servers

* Decreases backup and restore times.

A solution designed with customers in mind

The e-mail module has been specifically designed to address the business requirements of a number of large organisations.

In order to mitigate risk and reduce exposure, it provides organizations with an infrastructure that keeps records of all e-mails for a defined period of time. It is supported by technology that is tried and tested, and can be run as a mission-critical application with 100% uptime. The system is completely open-standards based and supports a service-oriented architecture (SOA), the basic functionality:

* Captures 100% of all e-mails--internal, external and from all devices

* Stores messages in a global repository

* Can find any message sent, on demand

* Has reporting facilities that enable high quality forensic searches

* Makes efficient use of storage by reducing duplication

* Tags e-mails with a policy code to enable bulk deletion

* Reproduces a users' mailbox at any point in time for an investigation

* Is transparent to users

* Offers high availability 

* Is standards based.

User requirements are reviewed and updated on a regular basis and, along with feedback from customers, are used for continual product development.

An overview of the technology

Figure 1 shows the logical architecture of the Sybase retention Server. e-mails are collected from the SMtP gateways and e-mail servers at local points of presence, where they are converted into a more efficient relational format.

Batches of e-mails are then sent to a global sorting database where various processes are run to clean the data. at regular intervals, batches of e-mails are loaded into a global data warehouse where processes are run daily to remove duplicate e-mails and to ensure the continuing quality of the data.

E-mail attachments are also stored in the data warehouse, but they are hashed using an Md5 or SHa-1 algorithm before they are put in the database. this creates a small digital "fingerprint" of the information and protects againsttampering, as the original can always be compared to the hash to check authenticity. It also enables a process to look for duplicate copies of information and delete them where appropriate.

To help manage the retention of e-mails, they can be tagged with meta-data, either when they are loaded into the data warehouse or retrospectively. This enables a policy code to be tagged onto each message to signal when it is eligible for deletion. Retention periods can be set by location (specific country, office, or e-mail server), by date created, by sender, by recipient, or even by scanning content for key words. The data warehouse is based around Sybase Iq, which is designed to cope with large quantities of data and is highly optimised for e-mail. Information is made available to a number of popular business analytics engines such as cognos and business objects and customised reports and data-sets can be generated as part of the implementation process.

It also can be multiplexed across several servers for scalability and resilience, and if necessary replicated with a mirror in a different geographic location. the entire architecture is designed around high availability and scalability, with redundancy built into each layer. Sybase supports a number of different operating systems including Sun Solaris, HP UX, Red Hat Linux and IBM AIX.

www.sybase.com