Methods of Linux bare metal restore and disaster recovery.

Most companies think they are prepared for disaster by backing up critical data, which they can quickly restore if a loss occurs. Few companies, however, are prepared for a catastrophe-total system failure-in which they lose all their data as well as the entire operating system, applications, settings, patches and more, reducing their system to "bare metal" and experiencing costly downtime.

The Cost of Downtime

Rebuilding even a single system from scratch can take a few hours or several days-time in which management, employees and customers have no data access. As a result there are no sales, no customer service, no product shipments, no supply chain management, no call centers ... no revenue.

How Much Does a Single Hour of Downtime Cost?

The Meta Group stated that, in 2002, system downtime cost American businesses an average of $1 million an hour. A survey done by Contingency Planning Research and published in the April 3, 2000 issue of InternetWeek details hourly downtime losses for several industries:

Industry                    Cost per hour

Brokerage operations        $6,450,000
Credit card authorization   $2,600,000
Package shipping services     $150,000
Catalog sales center           $90,000
Airline reservation center     $89,000

In today's 24x7 world, even if downtime is only $1,000 an hour, a three day shutdown would cost a company $72,000. At $10,000 an hour it would be $720,000. Few companies can survive such a hit. In fact, a chilling study from Contingency Planning and Management magazine discovered that 40 percent of companies that were shut down for three days failed within the following 36 months.

Starting Over

Recovering from a bare metal disaster is a highly complex operation that can take hours, days or even weeks to return the system to its prior state.

Common tasks include:

* Repartition hard drives

* Recreate volume groups and logical volumes

* Remake software RAID devices

* Remake file systems

* Reinstall the Linux OS from the original distribution media

* Recreate users and groups

* Reinstall hardware drivers

* Reinstall additional packages and third party software " Reinstall all software updates

* Reconfigure the network and firewall

* Redefine and reconfigure printers and other peripherals

* Reinstall and reconfigure web, FTP and email servers

* Reinstall backup software

* Reinstall database software

Linux System Recovery Tools?

Some UNIX administrators, who routinely use recovery tools like AIX "mksysb" and HP/UX "make_recovery," are reluctant to move to Linux because that OS has no similar standard tools for full-system recovery. In addition, most Linux distributions are fairly unsophisticated about how they install themselves. In most instances they use a single hard disk that is partitioned with a single root filesystem containing all system files as well as other applications and user data.

In more sophisticated configurations, an administrator might take advantage of performance, flexibility and availability benefits by spending considerable time reorganizing the system to use LVM, software RAID, enhanced filesystem types and other advanced techniques. Unfortunately, all this time-consuming and costly hard work, which may evolve over several years, must be repeated if the system crashes or new hardware arrives.

Fortunately, third party vendors are making tools available for Linux systems to ease this pain. Often referred to as "system recovery," "disaster recovery" or "bare-metal restore" tools, they are capable of restoring a typical Linux system to its previous hardware configuration. Currently there are two methods of accomplishing this: disk image recovery and file level recovery. Each has its own benefits and pitfalls.

Disk Image Recovery

A disk image backup program views the hard drive simply as a group of sectors. It backs up all sectors (not individual files) on the hard drive containing the data and creates an exact image. By operating on the disk sector level, the program backs up everything-system files, master boot record (MBA), partition tables and all user data-with no regard to their content or structure.

Some disk image backup tools, which are a bit more sophisticated, first back up the partition table information, then each partition independently as if it were a disk image. This adds a bit more flexibility by allowing IT to restore a single partition instead of the entire disk.

A disk image restore begins by booting the system from media (typically a CD-ROM) that includes a simple OS, device drivers for accessing the disks and backup media, and a utility for rewriting the data. If the entire disk was backed up, then the entire disk's data is restored. If individual partitions were backed up, then the disk partition table is first re-created and data is restored to each partition.

While in some instances this can provide the fastest means of backing up and restoring data, the method's inflexibility may cause considerable grief during a system restore. A disk image backup is like a block of ice that will only fit properly into its original container. If the "container" has changed (such as a different disk size, type or location), the data will no longer fit, and the entire backup is rendered unusable.

File-level system recovery

In contrast to a disk image backup, a few backup tools and products are available that better understand the operating system they are running on. These tools generally record the system, disk and filesystem configuration, then use this information to rebuild the system from the ground up. The backup usually consists of a file-by-file backup of either the entire system or of each individual filesystem.

Borrowing the previous analogy, if a disk image backup is like a block of ice, a file-level backup is like water; it can be shaped to fit into any container. As a result, disks, partitions, filesystems and other storage tools can be reconfigured for the new hardware, and data files can be simply restored onto the new configuration. It's like pouring water into a different glass; the data doesn't care about the storage configuration it's restored onto.

This process of altering, then building a new storage configuration can be a sophisticated task in which most DR products show their limitations. Though most will allow alterations of disk partition tables, they fail to understand more advanced and increasingly common disk configuration tools such as the Logical Volume Manager (LVM) or Software RAID (meta-disks). These "logical disks" must also be altered to match the new hard disk configuration before the data can be restored. If the DR software is not capable of understanding and guiding the user through the process, it forces the user to manually alter and build the configuration, usually through command-line utilities and manual editing of configuration files.

If all is well, the data files are restored and the system disk is made bootable once again. The new system, even with a new disk configuration, should operate the same (and often better) than the original.

Regardless of the product selected, users must be absolutely sure it will recognize, support and rebuild all of the storage options that have been implemented. Whenever possible, a disaster recovery test should be fully performed. Businesses shouldn't wait until disaster strikes to find out whether or not they made the right choice.

Anthony Johnson is president and CEO of Storix, Inc. (San Diego, CA).

www.storix.com

Pros                              Cons

* Fast backup if most of the      * Backup will take longer backing
  sectors on the disk contain       up sectors of the disk not used by
  real data                         real data
* Simple, non-OS-specific backup  * Backup cannot be restored to
  and restore process               different disk types, sizes or
                                    locations
                                  * Cannot restore partial disk
                                    backup-no selective file
                                    restore
                                  * Restored data files retain the
                                    same fragmentation as before
                                    the backup

Table 2. Disk Image Backup/Recovery: Pros and Cons

Pros                               Cons

* Faster backup when disk or       * Can take longer to read individual
  filesystems contain free space     files from a full filesystem than
* Backup can be restored to          image backup
  different disk types, sizes and  * Depending on the utility,
  locations                          reconfiguration to new hardware
* Faster system I/O after a          may be complex
  recovery since all files are     * Takes longer to restore files
  de-fragmented                      and directories
* Selective restore of individual
  files and directories without
  complete recovery

Table 3. File-Level System Backup/Recovery