Backup & recovery using revolutionary MAID architecture: part 1.

We'd all be doing disk-to-disk backup if we could justify the cost. The main reason that most of us have stayed with tape for so long (despite its limitations) is that there simply wasn't any other medium that could reach the scale and cost of tape--until now. This article will talk about the issues with current backup and recovery strategies and introduce you to a revolutionary new architecture--Massive Array of Idle Disks (MAID)--being adopted by today's cutting-edge storage vendors (such as COPAN Systems) to deliver all features of both disk and tape--at the price of tape.

Tapes Have a Good Side

In all the excitement about disk-to-disk backup, some have forgotten that while it has limitations, tape is actually well suited to the applications that it was designed for. Tape is a serial medium, and backup and recovery (the most common use for tape) is a serial application. Most backup environments perform their backups as a batch process sometime during the night. Each server to be backed up queues up the files that changed the previous day, and sends them in a series of batches to the backup server. Full backups are even more serial in nature, as they are a bulk transfer of every file from the server to a backup device. After being backed up, the data is infrequently accessed for recovery. Even environments that do many restores still perform many more backups than restores. This makes backup and recovery ideal for a serial medium such as tape.

Tape is also very fast, and is often faster than the file systems that it is backing up--due to its streaming nature. While an individual disk is going to be faster than an individual tape, you usually don't back up that way. Most people are backing up to a large, somewhat fragmented, file system, and its data must traverse two servers, two I/O systems, two memory banks, one IP network with multiple hops, and sometimes two Fibre Channel networks to get to its final destination. The result is that tape drives are usually starving for data, and thus do not reach their optimum transfer rate.

Tapes also hold a lot of data. We've come a long way from the 1GB and 2GB tapes that dominated the industry ten years ago. Tapes now have anywhere from 50GB to 500GB of native capacity--sometimes storing over 1TB of compressed data on a single tape.

Perhaps the second greatest advantage that tapes have over disk is that they are very easy to take off site. While disks are small enough to be carried off site, it has not been logistically feasible to do so. While 3.5-inch and 2.5-inch ATA disks are designed to handle a 300G and 900G shock, respectively, that is not the issue. RAID arrays were simply not designed to support this kind of usage. A removed disk in a RAID array is seen as a failure that needs to be repaired, not something that was sent somewhere on purpose. Tapes are not like that. If you want a tape off site, simply remove it from the library and send it there.

Issues with Tape

Anyone who has used tape for any length of time knows that it is an imperfect medium. While tape drives can store vast amounts of information for very low cost, they are not the most reliable machines in the world. They are also difficult to stream at their maximum speed, and the management costs associated with these two limitations can be quite high.

The reliability issues that you have will depend on the particular brand of tape you are using, but one limitation that all tapes have is that they are an open system. That is, the media is inside the tape cartridge or cassette, and the read/write mechanism is in the tape drive. This means that the media must be partially removed from the cartridge or cassette to be used by the tape drive, which means that there is an opening through which the media can be accessed.

Unfortunately, this opening allows for the introduction of various contaminants that may damage either the tape or the drive. Of course, the mechanisms that handle the tape operate at very high speeds and therefore have a very low tolerance for such contaminants--thus the reliability issues. Capstans get gummed up, springs don't spring, and read/write heads get corroded. While some parts of the tape path can be cleaned using a cleaning tape, not all of the tape drive is in the tape path, and many people don't clean their drives as often as they should. At some point, one of the many mechanisms in the tape stops operating within its assigned tolerances, and the tape fails to perform its assigned task. This can be anything from a low signal-to-noise ratio and a high level of errors to a tape drive that eats a tape beyond repair. Anyone who has used a significant amount of tape has had a tape eaten by a drive.

Some tapes are also physically fragile, where dropping them from a height of just a few feet can make them unreadable without repairing the tape. Cassette doors can break off, spring mechanisms inside cartridges can be dislodged, or the entire tape cabinet may break apart. How often these things happen to you will be based on the type of tape you are using, and how carefully you handle your media.

The storage industry has designed ways to deal with the unreliable nature of tape. Tape drives perform constant read-after-write tests to make sure that what you think you just wrote to tape actually got written to tape. Some tape drives automatically notify backup software products when they need cleaning via the tape alert function. Backup software companies encourage making many copies of each backup image so that when one fails, you've got another copy. Despite all of these attempts, and many more like them, the amount of time that the average environment spends managing tape can be staggering. One activity involves identifying bad media and bad drives, which can be a rather laborious process and can only be done properly by using trend analysis. Once bad media or bad drives have been identified, they must be swapped out. This almost always involves some sort of downtime and a significant amount of hassle.

Another issue with modern tape drives is that if you want them to operate anywhere near their published transfer rate, you have to send them a stream of data that's close to that transfer rate. While this seems obvious, what's not that obvious is what happens when you don't send a drive a stream of data close to its published transfer rate. What happens is that it actually writes slower than the stream of data you're sending. This is all based on how the drive is designed. To increase the signal-to-noise ratio, the recording head moves very quickly across the media and only records at high speed (some tapes move as quickly as 150 inches per second). So a 30MB/s drive can actually only record data at 30MB/s. If you send it a stream of data slower than that, the drive will wait until the buffer is full, spin up the drive, write that data at 30MB/s, and then look to see if the buffer is full again. If it's not, it stops the tape, rewinds it slightly, and then waits before the buffer is full to begin writing again. This is referred to as repositioning; and if a drive does it a lot, we call it shoe shining, because the tape moves back and forth across the head like someone shining your shoes. The more your drive is shoe shining, the less time it has to actually write data. For example, if you send a 30MB/s tape drive a 30MB/s stream, it will write at 30 MB/s. However, if you send that same tape drive a 10MB/s stream, it may write at 5MB/s and waste 25MB/s of its potential bandwidth.

Therefore, most major backup products support multiplexing, which is the process of sending multiple streams of data simultaneously to a tape drive in hopes of sending it enough to reach its published throughput numbers. Unfortunately, multiplexing speeds up the backup, but slows down the restore. To restore one data stream, the backup software must actually tell the tape drive to read all data streams that were multiplexed with it, and then it disregards the unwanted data streams. The result is that the higher your multiplexing settings, the more your recovery speed will be negatively impacted. The difficulty is in finding a multiplexing setting that gives you enough speed on your backups, without decreasing your restore speed to an unacceptable level. The result is that storage administrators do either one of two things: ignore it or constantly tweak it. The ones that ignore it usually have way too many tape drives, and the ones that constantly tweak it spend a lot of time doing so. Either way, you lose money.

Enter D2D2T

One popular solution currently is disk-to-disk-to-tape backups, where backups first go to disk and then to tape. This is commonly referred to as D2D2T, along with two other acronyms: D2d2T and D2D2t.

D2d2T is a traditional tape-based system with a little disk typically for a few nights of backup data in front of a large tape library. This is also referred to as disk staging, because the backups are only staged to disk for a short time prior to being sent to tape. D2D2t refers to a backup system with a lot of disk in front and a little tape library in the back. The disks hold all on-site backups, and the tape is only used to create off-site backups.

Some customers have been doing disk staging for years. They purchased enough disk for at least one night's worth of backups, and all backups first go to the disk staging area and are then copied to tape. The tape copy is created locally and can stream the tape drive more efficiently than a network backup, resulting in more reliable use of the tape drive. Some backup software products have done this for a long time, and others have recently added this functionality.

Disk staging (i.e. D2d2T) offers the benefits of disk without having to purchase enough disks for all of your backups. But disk staging has limitations; since every backup is sent to disk and then to tape, you're actually making two backups for every backup you make. If you also want to have an off-site copy, you must create yet another copy. Therefore, a disk staging system might require a more powerful backup server to support the additional I/O requirements. Also understand that once a given backup is moved to tape, restores from that backup will come from tape, not disk. Restores from recent backups may come from disk if they're still in the staging area, and the chances of that happening will be based on your average daily change rate compared to the size of your disk staging area. In addition, since a full restore will almost always be pulling backups from more than a few days ago, full restores will almost always come from tape. Therefore, disk staging offers more reliable backups, but not necessarily more reliable restores.

If you want the benefits of disk for both backups and restores, you should buy enough disk to hold all on-site backups. The tape library would hold only enough tapes to make copies of a few days' backups, which would promptly be ejected and sent off site. Therefore, the size of the tape library could be significantly smaller than before, thus the acronym D2D2t, since the tape portion of the system is now a lot smaller. Such a system would give you the reliability, protection and performance of disk for both backups and recoveries, while still using tape for off site.

Both D2d2T and D2D2t solutions use RAID underneath, and backing up to RAID-protected disks is a very good thing. With tape backups, a single media failure can cause a whole restore to fail, and you never know if a piece of media is good until you use it. With RAID, a single bad disk can be replaced without data loss, and the system is constantly being monitored, so you don't have to wait until a restore to see if your media is any good.

Of course, with a disk staging system (D2d2T), backups are only RAID-protected until they're moved to tape. Therefore, many people would probably like to do all their on-site backups to disk (D2D2t). The problem is cost. With disk systems typically being 3-5 times the cost of tape, this is just not an option for most environments.

Disk to Disk to Tape Configurations

   * D2d2T
1  Backups reside on disk until moved to tape
   Most restores from tape

   * D2D2t
2  All onsite backups on disk
   All restores from disk, except DR

Part 2 of this article will appear in the September issue of CTR and discuss D2D2T options and the benefits of MAID technology.

www.glasshouse.com

RELATED ARTICLE: Why Are We Using Disk?

* Doesn't require constant streaming

* Disk is more reliable than tape

* Instantaneous loads and rewinds

* Don't need to multiplex

* Even if you multiplex, restores are probably faster

* Protected by RAID when on disk

* Disk to tape copies are easier than tape to tape

* Incremental backups with little data will not hurt other backups

W. Curtis Preston is vice president of service development at GlassHouse Technologies, Inc. (Framingham, MA)