Plan for the worst, hope for the best: backup & disaster recovery, part 3.

Part 1 of this column stressed the importance of performing regular periodic backups with disaster prevention in mind. Part 2 discussed hardware and software options.

Media

Many organizations scrimp on tapes. I recommend at least a 30-day rotation with a complete replacement after a couple of years of use. Tapes are cheap. Take a 4MM DDS-5 tape for example: if a quality tape costs $20, and you use it 40 times over a couple of years, that's 50 cents a use. Attempting to get more use on an older tape can prove to be a very false economy. Imagine having a disk drive crash and the old tape you used for the last backup turns out to be unrecoverable. I don't think anyone would want to try to justify "cost savings" to the Boss after such a failure. Practice preventative replacement. While other tape technologies might advocate thousands of passes, I've found from experience that less is better when relying on recovering data from such fragile media. I've actually had a few customers who bought new media for every nightly backup.

Clean your drive at the manufacturer's recommended intervals. A cleaning tape is simply non-burnished tape--it's rougher than the data-grade backup tape and must be used regularly to keep the heads clean. A cleaning cycle simply rubs a bit of tape across the read/write heads. While every tape drive is going to fail some number of hours after being placed into service, it will definitely cause problems before-hand unless you take care of it properly.

Operations

Operations is comprised of a wide variety of items where organizations can seriously break down operator training, regular scheduled device cleaning, simple tape rotation, offsite storage, or an adequate budget for backup and disaster prevention.

Buying hardware and software is a straightforward expense. While many organizations may scrimp here too much, a far greater expense is operating and managing backups. I've heard many a complaint of how expensive backup hardware and software is, but unfortunately many learn the lesson that not backing up, or doing it poorly, is a lot more costly.

Here is a typical scenario: Nightly unattended backup fails several nights in a row, and the person responsible for backups isn't sure whether the problem is hardware, software or operational. Without adequate experience to make a determination of the problem, another night's backup is scheduled to see if the problem simply goes away. This can go on for some days with backup failing every night. With each failure, the risk of a serious loss mounts, should a disk crash or a database corruption occur. Sooner or later an outside resource has to be pressed into service to figure out what is causing the problem. In defense, someone who rarely has to deal with backup problems can't be expected to diagnose problems that require a wide range of knowledge and experience to accurately determine the nature of the fault. An experienced technical support person can usually determine the cause of the problem by requesting a number of tests over the telephone. An experienced tech can often guess the cause of a problem before the caller has even finished detailing the symptoms--it's easy when a person deals with the same problems over and over again.

From an operational stand-point, there is no substitute for an observant operator who is charged with the responsibility of verifying that each backup is successful. When the backup program does report errors, it is important to have someone who researches and determines the probable cause, e.g. the file was skipped due to an open file, or there was a tape I/O error. The former is a benign error, probably caused by a user leaving his or her workstation logged on with an application open, while the latter is a fatal hardware error reported by the operating system after the program had a Dr. Watson (abnormally terminated).

Checking a backup log for successful completion does not automatically guarantee anything on the backup media can be retrieved. The only way to ensure that a tape can be read for retrieval is to perform a second pass on the tape after backup, using a verification routine, or to actually perform a sampling restore. Tape drives have different heads for writing and reading so you can have a problem with one and not the other. There are two ways to verify data after backup to minimize restore issues.

The first is calculating a check sum during the backup process, then reading the tape, recalculating the check sum and doing a comparison--they should be equal. The second method to verify data is to compare each block on tape to its equivalent block on disk. The problem with the latter is you can't allow anyone to update files or the verification process will fail. (As a matter of note, UltraBac Software was the first backup company to implement tape-based verification in the Windows market back in 1995.)

In closing, management must share in the responsibility of ensuring that an effective backup plan is implemented and maintained. The best method of protecting company information is by providing an adequate budget to buy a good hardware and software solution with technical support that provides the necessary help required to maintain day-to-day operations. Management should also validate that backup operators have sufficient training to perform their duties by requesting periodic "disaster recovery" drills. Backup and disaster recovery is too important for management to either ignore or delegate to inadequately trained employees. A good backup plan should have a contingency for every type of problem that can occur, so plan for the worst and hope for the best!

Morgan Edwards is the CEO of UltraBac Software (Bellevue, WA)

www.ultrabac.com