The emergence of e-vaulting: electronic vaulting is a compelling improvement on traditional in-house data backup and recovery functions.

If small- and medium-sized businesses thought that data availability was not crucial to their operational success before September 11, 2001, they do now. While these companies realized that their data was the lifeblood of their businesses, many believed that they just could not afford the costly staff, services, and capital to properly protect that data offsite. More often than not, they backed up to tape and prayed they never had to use those tapes to recover lost data.

Even large enterprises have not implemented bulletproof backup and recovery procedures on a consistent basis. But in the wake of 9/11, businesses large and small want to ensure they can recover from a disaster. Larger enterprises are revisiting their backup procedures and expanding their infrastrures, smaller businesses are developing recovery plans first time.

According to "Standard IT Processes: Getting to Yes at Intel," by Bill Whittington, "a modern states that those who conduct backups are only two restores away from a pink slip."

Events such as 9/11 have increased awareness that disaster are necessary, but the reality is that the primary loss within an organization--people--has nothing with headline-grabbing disasters. According to a recent published by CBL Technologies, approximately 15 percent unplanned downtime occurs because of human error. Human error is perhaps the most common cause of data loss, but certainly not the only one that impacts a company's valuable data.

How many times have employees deleted a file that they should not have? How many companies have filled a disk drive to 100 percent capacity, causing the server to crash and corrupting large chunks of the data on that disk? These are daily occurrences that data protection methods and schemes must take into account.

Essentially, a company cannot afford not to protect its data. Hardware can be replaced but data is, at best, difficult to recover and, at worst, impossible to recover.

Traditional Data Protection

The traditional method for protecting computer data is to transfer a copy of the data from a hard disk to removable media such as a floppy disk or magnetic tape. Until recently, the low cost of tape-based storage relative to the cost of online hard disk storage made magnetic tape the most practical method for backing up data. Nonetheless, there have always been inherent problems with the use of magnetic tape for backup. Despite the high cost of available alternatives, those problems have been increasingly unacceptable as part of the risk of doing business. The inherent problems with the traditional magnetic tape backup method include:

* Slow speed of backup--Because tape uses a linear recording format, it takes more time to write backup data to it vs. disk. Traditional tape backup strategies including full, incremental, and differential styles require significant bandwidth to meet short backup windows.

* Inability to audit backup--With distributed systems, it is difficult to verify that tapes are being properly written to with each daily backup.

* Data is not immediately offsite--Without manual intervention, backup tapes remain in the tape drive, leaving the data vulnerable to physical events.

* Infrequent verification of backup data--Most people do not activate the option to "verify after write" on their tape drives because this adds 30 to 50 percent to the time required to complete the backup process.

* Lack of data encryption--Most tape backup hardware or software does not encrypt backup data, leaving it exposed to unauthorized access.

* No 24x7 access to data for recovery--If tapes are removed from the drives to be sent offsite (often the recommended method), there can be a significant delay in those tapes returning for recovery purposes.

* Changing tape formats--New generations of tape technology are introduced on a regular basis. When IT departments upgrade their tape technologies, few worry about recovering data from previous backups. Upgrades can make it difficult to satisfy corporate retention policies for longer periods of time.

Tape is still a good option for long-term retention of data. It has excellent price performance characteristics and likely will continue to have a price advantage over disk for the foreseeble future. Therefore, for those applications requiring infrequent access and those that can afford to wait a longer period of time for "restores" tape will be there medium of choice.

Electronic Vaulting Emerges

Electronic vaulting predicates the appearance of storage area networks (SANs) and network-attached storage (NAS) devices on the storage landscape. In fact, forms of electronic vaulting have been in use since the early days of computer networking. Wide-area network architectures, as well as the TCP/I set of protocols that form the basis for the internet, have long provided a method to move data electronically from a local system to a remote one.

Electronic vaulting is commonly defined as the movement of electronic data over private or public communication lines from a local (primary) computer storage device to a remote (secondary) storage device for the purpose of restoring the data in the event the primary copy is lost. Electronic vaulting also is referred to as "e-vaulting" and "online backup." These terms imply the same process of moving electronic data from a local organized to manage its own internal data protection service.

Market Size for Electronic Vaulting

According to author Fred Moore's 2001 findings on the market penetration of e-vaulting:

   The combined disk, and optical storage industry is approaching $50
   billion in revenue, while the storage management software industry
   now exceeds $5 billion in annual revenue. The powerful dynamics of
   the storage industry, coupled with the abundance of bandwidth, are
   creating new and disruptive events that are reshaping the
   established methods for storing, retrieving, communicating, and
   managing data.

E-vaulting represents a major transition away from traditional in-house data backup and recovery functions. The outsourced solution transitions the backup and recovery function of a company--non-core but critical functions--to an automated third-party service provider. As a result, the total market value for e-vaulting is the union of

* traditional tape backup and archiving software products

* traditional offsite tape storage or "tape vaulting" services

Each time a company converts its data protection methodology to an e-vaulting solution, it is displacing the value propositions of both tape backup software and offsite storage with a compelling improvement on each function. As a result, analysts believe that both markets are taking a fundamental evolutionary step.

The market size for BSP services is currently less than $100 million annually. Though the served market is small today, the addressed markets are large. For example, backup/recovery and digital archiving software was a $1.8 billion market in 2001, according to Gartner Inc., and offsite tape vaulting services comprised a $1.5 billion market in 2001, according to PRISM International. In addition, the migration toward outsourced data protection solutions represents a transition away from direct-attached tape drives and their associated tape media because e-vaulting can be a direct substitution. Each of these markets represents a multi-billion-dollar IT spending practice.

Into the Future

The simple fact that information is increasingly "born digital" and, once rendered digitally, remains in that form, means that more and more information will be backed up, recovered, and stored on digital media. As a result, the enormous $8.5 billion global outsourcing market for paper records storage also will transition over time toward digital storage services. This transition won't happen overnight and it will be many years before any significant impact is felt in the legacy paper records storage business. Nonetheless, legislation such as the Health Insurance Portability and Accountability Act (HIPAA), the Electronic Signatures Act, and the Gramm-Leach-Bliley Act drive organizations to protect, backup, and archive data in ways that increasingly perpetuate digital storage services.

The transition to e-vaulting from traditional tape-based backup and offsite vaulting services has been predicted for some time, according to Gartner's Enterprise Storage Group and other respected storage analysts. However, the "primordial mix" for supporting this transition has only existed for a relatively short period of time. That mix of elements includes

* cheap, reliable, widely adopted high-bandwidth network connections (DSL or faster)

* affordable, reliable, and massively scalable Redundant Array of Independent Disks (RAID) storage systems

* new software technology engineered "from scratch" specifically to minimize the amount of data required to transmit a full backup across a network and to maintain the history of those backups in a very small storage footprint at the remote location

The software technology necessary to offer e-vaulting services through BSPs is fundamentally different than traditional tape-based backup and recovery solutions. To ensure the feasibility of automated data protection services, the following engineering challenges had to be addressed:

Challenge 1: The development of communication techniques to handle large-volume data movements over public networks, with a high level of reliability needed for outsourcing the data backup and recovery function. Additional communication-related challenges included simultaneous multi-party processing and enhanced digital security.

Challenge 2: Creation of a storage architecture with efficiencies that greatly exceed the limitations of historical data compression techniques. It was important to minimize the amount of data transferred across the network to effectively recreate a full backup and equally important to minimize the amount of data required to represent a comprehensive history of backup data.

Challenge 3: Aggregation of all operating system (OS) platforms to a common back-end platform. This change reduces the cost of configuration on the back-end. It also accommodates the recovery of data from one OS platform (OS of origin) to any supported OS platform. Another benefit of this common format is to mitigate the problems associated with tape media obsolescence.

The result of the e-vaulting development effort over the past 10 years is a comprehensive solution that addresses the challenges described above. This solution fits between mirroring and long-term archiving on the data lifecycle spectrum. As shown in the data life cycle table, data protection, whether for a period of hours or up to one year, is the target market for the e-vaulting solution set.

Mirror, Mirror on the Wall ...

Some confuse mirroring with e-vaulting. This is understandable given that marketing literature and Web sites promoting these technologies claim similar features and benefits. Perhaps the most concise clarification is provided in "Mirroring Isn't Backup" by Rick Cook:

   Although data mirroring, especially remote mirroring, is an important
   tool in business continuity and disaster recovery, it is not a
   substitute for conventional backups. It offers no protection against
   several important causes of data loss, such as accidental deletion or
   virus infection. Anything that happens to the primary data happens to
   the secondary [mirrored] data.

Mirroring technology provides different features and benefits for the user compared to e-vaulting. As shown in the accompanying table, mirroring solutions are complementary to, and not mutually exclusive of, e-vaulting solutions. It is common to see companies using a local mirroring solution to provide real-time data protection for certain mission-critical applications that cannot afford more than a few minutes of downtime. And, that same company will use an e-vaulting solution to back up the mirrored copy of the data to a remote location for disaster recovery purposes. In addition, that company also may use evaulting as the principle backup method for protecting the data on the balance of its application servers.

Business Case Assumptions

Depending on the soft cost-savings that an organization can attribute to an e-vaulting solution, it is typical for a company to save significant dollars using e-vaulting while improving the level of service for the data protection function. The return on investment (ROI) is illustrated in the accompanying table.

The retention schedule for backup data is as follows:

* Daily backups are saved for one cycle (one week--7 days).

* Weekly backups are saved for one cycle (one month--5 weeks).

* Monthly backups are saved for one cycle (one year--12 months).

* Total number of backup sets/generations to support this schedule is 24.

* Tapes used are DLT4000 type with a 20GB capacity and an estimated drive cost of $3,000; individual tapes cost $50 apiece.

Backup administration involves

* offsite administration

* tape handling and labeling

* backup monitoring and troubleshooting

* backup set up and scheduling

* installation of software upgrades

* labor rates estimated conservatively at $25 per hour including benefits

In addition:

* Offsite costs are calculated by having 20 of the 24 tapes offsite at any point in time. Daily pickup and delivery is estimated by taking 20 business days and multiplying by a per-visit charge of $15. (Storage charges are calculated based on a charge of $1 per tape per month.)

* Emergency services are required to bring a tape back from offsite to restore an entire disk drive or directory to the server. (It is estimated here that one emergency delivery is required per month.)

* Restore time is required to carry out 10 "ad-hoc" restores per month to satisfy user requirements; eight hours are allocated (one day) per month to support this function.

* Tape drives are continually being improved to enable the latest technology for speed and reliability, but they are mechanical devices and will fail over time. It is estimated that the DLT4000 will be replaced with new technology every two years.

* Software license and maintenance costs are a constant per year for tape backup software.

E-Vaulting Architecture

There are normally three main components of an e-vaulting solution: agents, administration, and backend server. The software agent is installed on each system to be protected. The administration utility connects to the agents and allows for the scheduling of backups, defining backup parameters, monitoring backup activity and, when required, handling restore requests. The agents "wake up" at the scheduled backup time and transmit the "delta" changes to the remote server. The remote server accepts the changes, manages the retention life cycle, and serves up the data when a restore request is received.

When getting started, the next step after the e-vaulting agent software is loaded and the backup parameters are defined is to perform a full backup. The client can choose one of the following backup methods to initialize or "seed" the e-vaulting process:

* Do the first backup over the network (depending upon network speed and impact on production activity).

* Send a full backup to a local tape device, which is then transported to the remote e-vaulting server for initialization.

* Run a full backup across the local network to a portable e-vaulting server, which is then transported to the remote site for initialization.

Once the remote e-vaulting server is initialized, a full backup is normally never required again. Notifications are sent via e-mail--depending on agent settings--upon successful completion and/or exceptions. The process is completely automated from that point forward and virtually eliminates human intervention.

The recovery procedure is easy to follow. A wizard within the administration utility guides the user through the retrieval process, prompting for date of backup, file/directory name, and other pertinent information. If the recovery is for a relatively small volume of data, it is generally performed immediately over the network connection. A complete recovery, on the other hand, is typically done from either tape or a portable e-vaulting server. In a complete recovery scenario, the customer restore request is handled at the remote e-vaulting server by competent staff who will initiate an export to tape or to a portable e-vaulting server. Either way, the restored data is delivered to the customer's site and recovered to the original or an alternate server. The portable server approach is much faster than using tape for larger volumes of data (greater than 40GB).

The e-vaulting data flow illustrations on pages 29 and 30 describe the detailed flow of information from the client's server across the network and into the recovery-ready state at the remote e-vaulting server system. The delta processing technique results in minimal data travelling across the network to effectively recreate a full backup of the protected data. The empirical evidence from millions of backups performed since the mid-1990s is that 1 to 2 percent of the total uncompressed data is transmitted across the communication line each night. Thus, if the client's server has 100GB of storage designated for backup each night, then the amount of data transferred is typically 2GB or less. If that data were transmitted over a network with available bandwidth of 1.5MB/sec (offered by a T1--or DSL-type connection) then the backup window would be less than two hours on a typical night.

The fact that most organizations have implemented highspeed Internet connections to support e-commerce and Web hosting applications has been a boon for the e-vaulting business. Most of these Internet connections are under-utilized after prime office hours and are usually fully available to carry backup data during the 6:00 p.m.--6:00 a.m. window.

Companies that have implemented e-vaulting solutions experience the following benefits:

* Shorter backup times--E-vaulting typically reduces the backup window from hours to minutes.

* Central control--Through the administration utility, hundreds of remote systems can be managed from one or more locations. Log files can be reviewed, users receive exception notifications, and spot checks can be performed on any system to verify the backup process is operating with integrity.

* Data is immediately offsite--One of the most important things learned from the 9/11 disaster is that data has to be remote from the primary site to ensure full protection. With e-vaulting, the data is offsite as soon as the backup is completed.

* Security--E-vaulting solutions encrypt data at the agent before departure and the data remains encrypted throughout its life cycle at the remote server.

* 24x7 access to data for recovery--Most recoveries tend to be of the small ad-hoc variety. E-vaulting solutions provide instant access to data and that significantly improves the level of service an IT group can deliver to its users.

* Labor productivity--The IT staff managing the traditional backup process now have more time to work on other tasks to support the business.

* System independent format--IT departments avoid the cost and hassles of continuously upgrading tape equipment. They also can eliminate worries about legacy data when migrating from one operating system to another, making it easier to satisfy corporate retention policies.

At the Core

This article:

* Discusses the need for effective data protection methods

* Examines electronic vaulting technology and uses

Key Questions to Ask Potential Vendors

* What are the vendor's experience/qualifications specific to the industry?

* How long it offered sound is the company?

* How many clients does it have?

* How many remote backup locations are there?

* How much data can it handle?

* What platforms does it support?

* What are the backup procedures in place for the remove storage facility?

* Can it service all your locations?

In addition, customer should

* make sure the vendor can handle all the data in the customer's backup window

* ensure that a restore can be performed either online or via removable media

* verify that the solution backs up all system states, security, and other operating system attributes for the files

* make sure the vendor can send the older backup data to removable media for long-term retention

* ensure that the solution allows for back up of open files and databases

* verify that the solution can work over their existing network and/or internet connection and does require extensive network re-design

* ensure that both backups and restores can be performed from a central management console

The Next Steps

* Set up a simulation or test using an e-vaulting service. Or pick the distributed location causing the most trouble and give that site to the vendor on a paid trial basis.

* Extrapolate corporate communication requirements from the test.

* Get a qoute for the company's full requirements and create the business case to move forward with a full solution to backup problems.

* Assign a project manager in the company to organize the rollout of the solution on all distributed servers. Perform semi-annual or annual tests on a subset of company servers.

Buyer Checklist

What to Look for in an e-Vaulting Solution

* automated and unattended backups

* centralized management of the backup and restore process from one or more locations

* open file and open database backups

* backups that take less time than a traditional full-tape backup

* bundled software licensing fees that do not incur additional fees

* secure, dedicated computer facility that will house data

* complete control of files and directories to be backed up, with file filtering capabilities

* ad-hoc file restores around the clock via end-user or central administrator control

* no special hardware requirements or changes to the network

* ability to restore data either over the network or via the media of the customer's choice

* ability to back up more than one operating system platform with the solution and an option to restore data from one platform to another

* user-managed data retention schedule

* several levels of data encryption

* automatic restart and resume capabilities for handling a variety of network conditions

* exceptions and problems encountered reported to an administrator in electronic form automatically

* detailed usage reporting available

* ability to archive data for long-term retention without requiring a local tape drive

Cost Considerations

A detailed cost analysis will often reveal hidden expenses. When adding up costs, consider the following:

* a declaration fee for full-system restores

* ad-hoc restore fees

* set-up and install costs

* communication costs (if using an existing Internet connection, this should be free)

* archive tape fees

* usage fees (based on time, quantity, frequency, and other parameters)

* cancellation fees for contract termination

Data Protection Lingo

Compression Technology: Virtually all backup software uses compression technology to reduce storage requirements. This technology can be software- or hardware-based and reduces the amount of data to be transmitted or stored by an average ratio of 2:1. Compression ratios depend on the type of file being compressed; therefore, a company's actual experience could vary. On average, text files, word processing documents, and most databases experience the highest compression ratios.

Full Backup Method: A full backup copies all files and directories that have been selected for backup. A user can select an entire disk volume or choose specific files and directories to back up. On the downside, a full backup takes the longest time, and many files may be backed up unnecessarily because they may not have changed since the last backup.

Incremental Backup Method: An incremental backup copies only those files that have been modified or changed since the last backup. The obvious advantage with this method is that files that have not changed do not require back up, thus reducing the time required to do the backup. Using this option is just as simple as a full backup; however, there is a definite downside. The restoration process requires you to first restore from the first full backup and then to apply all incremental backups up to the required date of restore. This can be a tedious process. Typically, an incremental backup method requires a weekly full backup to help reduce the time to restore and reduce the potential for restore errors.

Differential Backup Method: A differential backup copies all those files that have been modified or changed since the last full backup. This differs from the incremental that just copies the files changed since the last incremental backup. While it takes less time to back up than the full backup method, it requires more time than the incremental. This is because it must back up all the files that have changed since the last full backup and not just since the last incremental. On the positive side, the restoration process is simpler and quicker than the incremental. With this method, one needs only to apply the last full backup and the latest differential.

Delta Backup Method: In general, it is not practical to transfer full, differential, or incremental backups across a network. Network bandwidth is normally a bottleneck that renders traditional backup methods obsolete. Most e-vaulting systems are designed to minimize the amount of data transferred across the network by using an enhancement on the incremental method called delta processing. Delta processing makes it possible to transfer just the changes that occurred since the previous backup. This procedure makes it practical to back up data to a remote locale over relatively low-speed communication lines.

Delta processing uses an algorithm that compares the image of each data file with the image of the same file from the last backup. This comparison generates a list of blocks within the data file that have changed. These blocks are compressed and encrypted and then transmitted across the network. To illustrate the difference between the methods, consider an example of a 100 MB data file containing customer information. If changes made to the customer data over the course of a day total 1MB, then approximately 1MB of data would be transmitted to the remote site to complete the backup. With the incremental or differential methods, the entire file of 100MB would be backed up.

The "effective compression ratio" obtained by the delta method can range from 50:1 to 1000:1. These large ratios make it feasible to back up large quantities of data over common network connections.

Backup Timing: All backup methods require some form of "time stamp"to enable a proper recovery. This is important in database scenarios where updates may occur 24 hours a day, seven days a week. Some companies who run this type of operation actually take the system down for a small period of time to perform a backup. This ensures a clean copy of the database with no partial transactions. If the system cannot be taken down for a period of time then traditional backup methods do not provide a stable backup solution. For example, if a backup is taken off an open database then there is no guarantee which transactions made it into the backup and which did not. In such cases, special backup functions have been written into the database system that allow for backups to take place while updates are held in a queue. This backup is usually written to a temporary file, which is an image of the database. This image normally would be written to tape but also could be sent electronically to a remote location. Also, online databases may create journal files (lists of transactions) continuously, which allow for restoration using the last full backup plus all transactions up to the end of the journal file.

There are other solutions to this problem that function at the operating system level. These solutions intercept any write updates to open files during a backup and they store the previous data in a cache area. This allows the backup to read the data as of the point in time the backup started. Once the backup is finished the cache is released. The big benefit of these solutions is that mission-critical applications continue running 24 hours a day, seven days a week, and complete backups are obtained with integrity.

Be Prepared

In order to bid, vendors will want to know

* recovery goals and backup window

* detailed information on server hardware and operating system (Compile a list of servers to be backed up, including brand name, model number, nomenclature, and estimated volume of data storage.)

* existing communication facilities connecting locations--Internet, frame relay, ATM, etc.

* firewall management (How are special ports on a firewall opened to allow backup data to travel across the network?)

* corporate retention schedule for computer data

* how many people will be administering the backup/restore process and where they are located

* location of alternate processing facility to be used in the event of a disaster

Mirroring Is Not Backup

Consideration                   Synchronous   Asynchronous   E-Vaulting

Protection Against Corruption       No             No           Yes
Protection Against Deletion         No             No           Yes
Cost of Solution                   High        Medium-High      Low
Distance Limitations               Yes             No           No
Link Speed Required                High          Medium         Low
Performance Impact                 High          Medium         Low
Failover                         Automatic       Manual       Requires
                                                              Restore
Multi-platform                      No             No           Yes

Return on Investment with E-Vaulting Solution

Traditional Tape

Current Cost Item                  Soft Cost   Hard Cost   E-Vaulting

Total tape inventory--30
  tapes @ $50 per tape                           $1,500        $0
Backup administration--8
  hr/wk/server @ $25 per hour       $10,000                    $0
Offsite storage--20 tapes
  offsite x 12 months @ $320 per                 $3,840        $0
Estimated annual emergency
  retrieval services @ $50
  per month                                        $600        $0
Restore activity--1 day per
  month per server                   $2,400                   $240
Replacement of tape drives
  every 2 years                                  $1,500        $0
Current backup software, updates
  and maintenance per server                     $1,000        $0
Daily backup with 24 generations
  of data for 20GB                                           $8,400
(cost per GB/month = ($35x20GB)
  x 12 months)
Total cost (soft and hard)          $12,400      $8,440      $8,640

Grand Total (annual)                       $20,840           $8,640

References

Brooks, Jeremy. "Protecting Your Data With an IT Continuity Plan." Itsecurity.com. Available at www.itsecurity.com/papers/cbl12.htm (accessed 25 November 2002).

Cook, Rick. "Mirroring Isn't Backup." Available at http://searchstorage.techtarget.com/tip/1,289483,sid5_gci805340,00.html (accessed 25 November 2002).

Moore, Fred. "Storage--Infinite Disruption" Horizon Information Strategies. 2001.

Whittington, Bill. "Standard IT Processes: Getting to Yes at Intel." Storage Magazine. July 2002.

Ray Ganong is Chief Technology Officer at EVault, a data backup and recovery provider. Ganong has worked in the data storage industry since 1989. He may be reached at [e-mail].