Disaster recovery planning.

It could never happen to us.

Disasters strike with impunity. Whole regions can be wiped out by a cataclysmic earthquake or hurricane; but disasters more commonly strike on a smaller scale and in many different forms. A company may lose its key decision-maker, a computer system may be wiped out by an insidious virus, or a key plant may be struck by lightning.

Regardless of the size or shape of the disaster, the results can be devastating. Too many individuals, departments, and organizations make the recovery process even more difficult -- or even impossible -- by not planning for disaster recovery. Internal auditors, whose detailed knowledge of the organization and how departments operate and interact, must play a key role in ensuring that a viable, up-to-date contingency plan is in place.

The Planning Process

The disaster planning process is an iterative one. Initially, many organizations with limited resources of time and money will develop recovery plans only for the functions that they believe represent the core of their business. Though limited in scope, such an approach can still be crucial to the organization's survival, and it provides a base for expanding the plan in the future.

Internal auditors bring a unique vantage point to the planning process. In many organizations, it is the internal auditor who will be able to provide the most lucid and objective overview of operations. In fact, the internal auditor may already be reporting critical vulnerabilities to management as part of regular operations.

Although disaster plans must obviously be tailored to the organization, several sequential steps will be fundamental to every planning process:

1. Obtain top management guidelines and prioritize the operations.

Input from top management is required to keep the planning process in perspective and to ensure participation by all groups. Top management must make key decisions regarding the length of time during which it will accept disruptions in each functional area, for example, and the amount of money it is willing to invest in standby equipment, back-up sites, testing, and other measures.

2. Identify unique vulnerabilities and other serious risks.

This is one of the points in the planning process where the unique perspective of the internal auditor can be invaluable. Internal auditors are any organization's resident experts in identifying risks and vulnerabilities and are well qualified to pinpoint critical interrelationships between functions.

A brainstorming process involving the employees themselves during departmental or group meetings will also be useful in identifying potential risks of which management may not have been aware. Brainstorming sessions will also help to build employee awareness of disaster planning.

3. Develop restoration strategies.

After a serious disaster, all or most functions are initially suspended and only gradually resumed. To facilitate restoration of operations, management should assign a "rank" to various functions, based on how long the organization can survive without each one. Typically, there will be three groups:

* Those that need to resume within 24 hours.

* Those that can wait up to a week.

* Those that are not under time pressure.

Once top management determines which functions it wants resumed and when, operations management can then decide on the best technical means to meet the recovery goals.

4. Assign the disaster team.

Management should appoint one person to be in charge of disaster recovery and one person to be second-in-command. Recovery responsibility for each functional area and as many specific tasks as possible within the area should also be pre-assigned.

Having assigned someone to be responsible for each recovery task and each functional area and having obtained management guidelines regarding the allowable time for restoring each function, detailed plans can then be created by the people who will actually be responsible for the recovery. The primary objective should be to ensure that, in the event disaster strikes, employees will know who is in charge of each area and what their own specific responsibilities are.

5. Take a complete inventory.

Disaster recovery planning includes comprehensive inventories of both basic and specialized equipment, in addition to forms, files, and other items. Special attention must be given to any items that are unique and would be difficult to replace. While most organizations have records covering the make and model numbers of their equipment at the time of purchase, they are usually not updated and almost never kept off site. Taking inventory is aimed at correcting both situations and should also include emergency vendor contacts for all specialty equipment and computer software, as well as listing sources for generic items.

6. Document the plan.

The planning process will ultimately result in a written plan. This document should specify various assignments, recovery procedures for each function, an updated inventory, and all key phone numbers. These phone numbers should include not only all employees, vendors, key customers/clients, and board members, but also after-hours contacts for vendors, insurance agents, and appropriate others. All key recovery personnel should have a copy of this documentation off site.

7. Review the plan.

After completion, all employees need to understand how the plan pertains to them. In addition to preparing employees, regular reviews strengthen the involvement of individual employees and provide a means of verifying that the plan is actually workable and that nothing has been overlooked. It will also help to surface the need for any additional cross-training required as part of disaster operations.

8. Test the plan, review the results, and modify the plan as appropriate.

Often much of the benefit of a test is lost because it is not properly planned. The following steps can help to ensure that the maximum benefits are derived from the testing experience:

* Establish test objectives beforehand.

* Conduct the test, document it, and evaluate results.

* Document conclusions and recommendations.

* Review with top management.

* Make corrections and changes.

If testing is too disruptive, it can become counter-productive. Each organization needs to make testing decisions based on its own dynamics. The following range of possible tests are listed in increasing levels of cost and disruption:

* Blink Test: Occurs when employees see their names listed as responsible parties for specific recovery tasks.

* Checklist Comparisons: Confirm that sufficient backup supplies and sources actually exist.

* Simulation: Ensures that "restored" backup data files have not been corrupted and can be read.

* Structured Walk-Through: Allows organization to rehearse the steps involved in relocating to a backup site.

* Parallel Operations: Facilitate tests of backup operations at a separate site.

* "Pull the Plug" Exercise: Involves full-blown relocation and recovery exercise. Can sometimes be combined with a planned departmental move or office relocation that requires a shutdown of operations anyway.

Testing doesn't have to be either expensive or disruptive. Those charged with disaster recovery planning should use common sense strategies with regard to testing. Often department moves and location changes can be used as testing opportunities.

Internal Auditing's Review

Regardless of the internal auditor's involvement in the process, the final recovery plan document needs to be reviewed and commented upon by the audit staff. While there is no standard plan format, the following elements should be covered:

* Management guidelines regarding downtime.

* Appropriate risk analysis, including any special conditions that may limit the organization's responsiveness, such as union limitations or dependence on individuals with special skills.

* Advance decision-making for all operations.

* Names of the disaster recovery team, along with their assignments.

* Procedures to be followed in relocating off site and operating on a temporary basis, including, for example, time frames, transition controls, and security and confidentiality issues.

* Plans for employee training and education with regard to disaster recovery.

* Attachments, including relevant telephone numbers.

* The process for testing, reviewing, and updating the plan to keep it current and accurate.

In reviewing the disaster plan, the internal auditor should evaluate:

* The inclusiveness of the planning process.

* The appropriateness of the plans and procedures, given the available resources.

* The completeness and comprehensiveness of the plan.

* Security and privacy implications.

* Future steps in the recovery planning process.

* Testing issues.

* Ease of updating and keeping the plan current.

* Organization-specific issues such as expansion plans and layoffs.

Obviously, the most important aspect of the auditor's review will be the overall assessment of the viability of the plan in terms of keeping the organization in business following a disaster.

Summary

The impact of any disaster can be mitigated by a realistic, comprehensive, and viable disaster recovery plan. Internal audit involvement will do much to reassure management that, should disaster strike, prudent and economical steps have been taken to safeguard the organization's assets.

Steven Lewis, PhD, CISA, is Editor-in-Chief of The Disaster Recovery Yellow Pages at The Systems Audit Group, Inc., in Newton, Massachusetts.