Disaster recovery planning for the distributed environment.

Mainframe-focused disaster recovery plans may be out-of-sync in the new era of distributed networks. It's the internal auditor's job to evaluate, assess, and recommend updated approaches and strategies.

The inexorable migration of critical applications from computer mainframes to distributed networks has created control challenges on several fronts. For internal auditors, ensuring that applications, networks, and data are covered by a disaster recovery plan (DRP) tailored to the distributed environment is an area of major importance. Companies that rely heavily on computer networks for routine business operations may be especially hard-hit by lapses in rethinking, planning, and implementing disaster recovery initiatives.

Simply defined, DRPS are comprehensive sets of policies, procedures, and resources employed before, during, and after a catastrophe. All scenarios must be considered in a DRP, including natural disasters such as fires, floods, storms, and earthquakes; hardware and software failures; power outages; and man-made threats ranging from equipment theft and vandalism, computer viruses, hacking, sabotage, and terrorism to non-intentional errors and mistakes. Effective plans enable an organization to minimize its losses; conduct business while resuming normal operations as quickly as possible; and maximize its chances of survival as a going concern.

The strategy for developing a DRP can be organized into six basic areas: business impact analysis; risk assessment analysis; risk mitigation strategy; recovery plans and procedures; alternate site considerations; and routine training, simulation, and testing. Each of these areas must be thoroughly considered, developed, and tested to ensure the overall success of the plan.

BUSINESS IMPACT ANALYSIS

The business impact analysis identifies and inventories mission-critical resources; quantifies the costs associated with failure to transact business due to the loss of those resources; and estimates the amount of downtime the organization can bear while those resources are being restored. The initial step in the creation and upkeep of any DRP is to identify and inventory mission-critical resources residing within the distributed environment.

Applications, systems, data, and network infrastructures that (1) are directly linked to the organization's livelihood and survival, and (2) would be significantly impacted should they be disrupted, are considered mission-critical. Examples include on-line reservation systems; customer accounts receivable; payroll; and other mainstream processes. Unlike the centralized data center, distributed systems can sometimes foster an environment where, unknown to management, unofficial, pseudo mission-critical systems can be developed and installed on networks. Such systems, which might include ad hoc reports and spreadsheets utilized by executive management in strategic decision-making, must also be identified so that a complete analysis can be performed.

Although the process of conducting an inventory may seem simple in theory, the reality may prove much more difficult to accomplish in a distributed environment. In this instance, the looming Y2K challenge may actually have a beneficial impact. Until recently, management could often justify decisions against performing a distributed network DRP inventory on the grounds of insufficient resources, other priorities, and the low probability of disaster. The millennium is now forcing organizations to take stock of their distributed network infrastructures.

Companies should take advantage of their organizations' Y2K remediation efforts, which typically include itemizing distributed applications and hardware. Such an inventory can readily be incorporated into the organization's distributed network DRP. Auditors should encourage these tabulation efforts and ensure that disaster and business recovery personnel play an active role in the

Y2K process.

Once all mission-critical resources have been cataloged, both recovery costs and lost revenue opportunities must be identified and quantified for each resource to determine the aggregate financial impact to the business. Organizations should be prepared for some hefty loss estimates. According to the Winter 1997 issue of Disaster Recovery Journal, organizations suffered significant disaster-related costs last year:

* Each on-line outage averaged four hours and cost companies an average of $329,000 in lost revenues and productivity.

* For each hour of unscheduled downtime, 355 worker hours were lost.

* Major businesses lost 38.1 million work hours, or $444 million in wages annually.

Although the story will be different in every organization, these sobering statistics provide a useful benchmark.

The final element to be considered in a business impact analysis is an estimate of the amount of time the company can afford to be without its mission-critical resources before suffering detrimental effects or, in a worst-case scenario, risking going out of business.

Once a business impact analysis has been completed, the probabilities of risks that are specific to the organization's mission-critical resources must be analyzed and weighed. This risk assessment analysis shows where company resources should be deployed for the development of a

The process of weighing risk probabilities must take into account the nature and design of distributed systems, which vary among organizations. Meaningful statistics unique to each type of business may prove difficult to acquire. However, general data loss statistics, such as those shown on page 47, are available, and they may provide some basis for correlating risks pertaining to network exposures. Such statistics can hardly be taken at face value across the board, since each organization's unique environmental aspects must be considered. Nonetheless, they may provide a starting point for establishing a more accurate assessment of risks.

RISK MITIGATION STRATEGY

Every recovery plan should emphasize preventative strategies that minimize the risk of a disaster happening in the first place. The following defensive measures may also help minimize damage and facilitate recovery efforts should a disaster actually occur:

* Enhance senior management awareness, and gain its commitment and support of the DRP plan. Examples of management support might include setting the proper tone for the organization and providing sufficient resources to make the plan viable. To increase the potential for success, senior management should participate in and be an integral part of the disaster recovery planning process and promote DRP awareness to all personnel.

* Ensure that competently trained certified professionals are employed to develop and maintain the company's contingency plans. Training is available from several sources. The Disaster Recovery Institute, for example, offers seminars and certification programs. The Institute's Certified Business Continuity Professional (CBCP) designation is awarded to individuals who pass the CBCP exam and meet specific academic and work experience requirements.

* Implement and enforce standardized distributed environment practices and procedures. If consistency is maintained throughout the organization, restoration and recovery efforts may prove less cumbersome. Distributed environment standards and procedures should be considered in areas such as LAN administration; security; dial-up and Internet access; naming conventions for programs, files, and network nodes; software change control; and backup and recovery procedures.

* Standardize approved hardware, software, and the voice/data communications infrastructure to prevent a proliferation of components from multiple vendors. Keeping the number of vendors to a minimum should reduce the complexity and resources required for PeP salvage, restoration, and reconstruction efforts.

* Conduct a thorough insurance policy evaluation to determine whether or not the organization has adequate coverage for the distributed processing infrastructure. Costs of mission-critical resources, electronic data restoration, business interruption and recovery costs, and other items of coverage may have been overlooked in insurance policies. Even an omission that seems minor could have catastrophic implications if it is not identified and resolved prior to a disaster.

* Secure routine and specialized vendor support, including predetermined plans and procedures for disaster recovery efforts. Routine vendors are classified as those suppliers who provide regular, ongoing services, such as hardware and software support; electronic commerce, including Internet service and value added networks (VANS); and telephone service. Specialized vendors include companies that furnish specific disaster recovery services, such as data salvation and restoration; alternate backup sites and office accommodations; and emergency leasing of hardware and equipment. Agreements should be negotiated in advance to ensure that vendors can deliver services and products on short notice.

A COMPARISON OF DISTRIBUTED AND MAINFRAME ENVIRONMENTS

Distributed Network Environments   Mainframe Environments

Multiple office locations that     A localized facility physically
lack sufficient physical and       secured and protected by
environmental safeguards.          environmental safeguards.

Critical software applications     Critical software applications
and data residing at numerous      and data residing within a
sites.                             centralized environment.

Hardware and software inventory    Readily identifiable hardware
that is difficult to identify.     and software inventory.

Ambiguous ownership of             Easily recognized ownership
applications and data.             of applications and data.

Greater likelihood of pseudo-      Few pseudo-production
production applications.           applications.

Under-utilization of system        Utilization of official system
development life-cycle             development life-cycle
methodologies.                     methodologies.

Inadequate program change and      Established program change
version controls.                  and version controls.

Multiple vendors that support a    Hardware and operating system
variety of hardware and            typically supported
software.                          by a primary vendor.

Mission-critical systems and       Systems supported and
operations supported and           operated by trained data
utilized by multiple end-users.    processing professionals.

Potential haphazard backup         Documented standardized
and recovery procedures.           backup and recovery procedures.

Insufficient and/or sporadic       Routine off-site rotation of
off-site rotation of critical      backup files.
backup files.

Inadequate testing of recovery     Periodic testing of the disaster
procedures.                        recovery plan.

Significant vulnerability to       Weak virus threat.
viruses.

Poor or non-existent programming   Mature programming standards
standards and procedures.          and procedures.

Lack of empowered centralized
administration.

Insufficient end-user training and
awareness of recovery procedures.

Overall lack of direction from
executive management.

Fragmentation of management
control resulting from networks
that cross department boundaries.

* Perform routine physical backup and off-site rotation of application programs, operating system software, data files, and documentation. Measures should be taken to ensure all mission-critical items are stored on client/server hard drives as opposed to user workstations and floppy disks, which often are not subject to recovery processes. If this transfer to the server does not occur, backup efforts could be compromised.

* Identify and address interdependent relationships in data and applications that are housed on both client/servers and the mainframe to ensure they will remain synchronized during backup and recovery. Interface applications that enable the exchange of data between the mainframe and networks will also need to be identified and inventoried since they are an integral part of this process. Failure to do so could complicate recovery efforts and compromise data and system integrity.

* Utilize Redundant Array of Independent Disk (RAID) technology to capture on-line, real-time transaction activity. RAID employs autonomous disk drives working in tandem with client/servers to create up-to-date copies of data files stored elsewhere within the distributed environment. Another benefit of RAID is its enhanced fault tolerance, or its ability to enable the system to remain accessible in the event of a partial disk drive failure. The placement of RAID capabilities at an alternate location would provide an additional benefit of enabling automatic backup and data storage.

* Work with specialized vendors that provide electronic vaulting, which enables companies to automatically download encrypted copies of their backup files to off-site secured data vaults. The utilization of these services, along with RAID, could significantly reduce the risk of human error or negligence associated with the conventional physical backup and off-site rotation of data files.

* Limit Internet access to those individuals who require it to perform their jobs. In addition, the existence of security safeguards such as firewalls help limit access between the outside world and the organization's intranet.

* Monitor and enforce computer virus policies and procedures to minimize the threat of business disruption. Preventative measures could include routine automated virus sweeps of the network, as well as educational programs that would explain to employees what they should do in the event that computer viruses are encountered in the workplace.

* Implement environmental damage prevention measures. Surge protectors and anti-static mats can help protect data and hardware. Regular testing of smoke and heat detectors can reduce fire risks; plastic sheeting should be available to protect computer equipment from water damage; and physical security measures should be in place to deter hardware theft and vandalism. Ecological safeguards, such as quake-proofing and shoring up vulnerable areas like windows against storms, should also be implemented.

* Promote employee safety awareness through training and education.

Appropriate strategies might include periodic emergency drills, which would educate personnel on first aid/CPR; the handling of hazardous materials; and the location of fire alarms, extinguishers, and emergency exits.

* Install alternative power sources, including batteries and uninterruptible power supplies. Alternative power sources are needed to minimize work disruptions, facilitate the controlled shut-down of networks, and enable utilization of emergency generators for prolonged power failures.

The old axiom, "it's better to be safe than sorry," is nowhere more appropriate than in this area. Minimizing the potential impact of a disaster is much simpler before, rather than after, the fact.

RECOVERY PLANS AND PROCEDURES

The organization's distributed network DRP should be a living, up-to-date document, thoroughly familiar to those who would need to rely on it during a recovery effort. Copies of the DRP should obviously be kept in a readily accessible off-site location.

Logistically, the implementation and execution of the DRP should be deployed along areas of responsibility and expertise. Specialized teams should be organized, including the following:

* INITIAL RESPONSE TEAM

Identifies and classifies the nature of a disaster and determines the extent of damage. Depending on the severity and magnitude of the situation, this team's decision will determine whether operations can continue on site or whether a transition to an alternate backup facility is warranted. If the DRP needs to be activated, other teams will be notified and called into action.

* RESTORATION TEAM

Coordinates the salvage, restoration, and reconstruction of distributed networks, data files, software, and the network infrastructure, including voice and data communication lines.

* RECOVERY OPERATIONS TEAM

Sets up and runs operations at an alternate location through the reestablishment of distributed network infrastructures, retrieval of backup files, transfer and reactivation of hardware and communication lines, and other relevant activities.

* BUSINESS RECOVERY TEAM

Ensures business operations can continue to function throughout a crisis until recovery efforts have been completed. The team oversees the ability of employees to gain access to alternate office and storage facilities that house supplies and specialized forms, such as customer invoices and check stock.

* LOGISTICAL SUPPORT TEAM

Provides logistical support to employees by ensuring personnel access to alternate offices and facilities and providing such services as travel and relocation support, on-site monetary advancements for unforeseen expenses, crisis management counseling, and family assistance.

A final note of caution for organizations that have already documented and implemented a DRP for the centralized mainframe - a tandem process should be established for linking the distributed network DRP counterpart. This step is especially critical in environments where the mainframe and distributed networks interact with each other.

ALTERNATE SITE CONSIDERATIONS

Hot sites, cold sites, and in-house backup systems can facilitate resumption of data processing operations off-site during a disaster. Each choice offers both advantages and disadvantages that should be carefully scrutinized before a decision is made.

A hot site furnished by disaster recovery vendors offers ready access to a facility fully furnished with data processing hardware, telephones, fax machines, telecommunications hook-ups, supplies, and other items. This alternative is likely to be the most prudent choice if the organization has determined it can survive for only a short time without its own systems. Although hot site vendors have traditionally offered services catering to mainframe clients, many now provide facilities equipped with fully operational LANS to accommodate changes in the marketplace. Another advantage offered by hot sites is that clients can conduct routine testing of their DRPS to train personnel and identify and correct problems in a non-emergency setting.

A cold site, also furnished by disaster recovery vendors, offers ready access to an empty facility or shell that must be furnished by the client during an actual emergency. Although cold sites cost less than hot sites, this alternative should be considered only if the organization can continue to conduct business as usual and can afford the luxury of time in setting up backup systems. An obvious disadvantage with cold sites is that no one will know if the DRP, will actually work until a crisis occurs. In addition, ensuring that vendors can quickly deliver leased computer hardware on demand may be a critical issue.

In-house backup systems placed at different physical locations may offer a viable alternative to relying on outside vendors. Advantages include reduced vendor costs and greater access and control over ancillary backup systems. The primary disadvantage with this choice is the tie-up of capital invested in equipment. In most cases, the equipment would not be utilized except in an emergency situation; yet it can rapidly become obsolete as a result of technological advances. If backup distributed systems are utilized for other purposes, logistical procedures should be in place either to share these platforms or to subrogate resident non-mission-critical systems with designated mission-critical applications in emergency situations.

ROUTINE TRAINING, SIMULATION, AND TESTING

Ongoing training at all staff levels, from top management down to the clerical staff, is essential to employees' adequacy and proficiency in DRP matters. Although investment in time and resources is required, the risks associated with employees who have not been sufficiently educated in minimizing risks and who are unprepared to participate in recovery efforts could prove disastrous. No organization wants to face situations where employees are forced to receive their initial DRP education as on-the-job training.

The routine simulation and testing of the DRP cannot be overemphasized. If there are problems or shortcomings, the best time to find out about them is during DRP simulations.

AUDIT'S RESPONSIBILITY

As noted in the IIA's Statement of Responsibilities of Internal Auditing, the internal auditor is responsible for reviewing the organization's methods for safeguarding assets and, as appropriate, verifying the existence of those assets. In a growing number of companies, the distributed network infrastructure itself has become an integral part of this equation.

Accordingly, it falls upon the internal auditor in these organizations to perform due diligence and sufficient auditing procedures to verify the existence of a viable DRP for the distributed environment. Failure to do so could violate professional responsibility and leave the company at risk for catastrophe. Unfortunately, just one minor problem with the DRP could affect the company's survival.

DISTRIBUTED ENVIRONMENTS STILL AT RISK

A 1996 Comdisco Disaster Recovery study of 300 businesses found that traditional data processing centers still get the lion's share of all disaster recovery dollars, while local area network (LAN) areas are left with little. Despite the fact that two-thirds of all companies in the U.S. have mission-critical operations running on local networks, the trend appears to be continuing.

In Comdisco's 1997 follow-up survey of more than 200 of the largest computer users in the u.s., Canada, and the United Kingdom, results showed that most organizations' distributed environments are still at significant risk in the event of a disaster. Overall, only 12 percent of companies surveyed had an effective DRP in place for their enterprise-wide computing systems. Only one-third of organizations with distributed systems possessed a written set of programs that designated alternative sites or outlined procedures for testing and evaluating their recovery plan for these systems. In addition, only two-thirds indicated that they use automated electronic backup. Such findings represent a significant concern to any organization employing distributed networks in a decentralized environment.

THE HIGH COST OF DISASTER

Organizations that fail to prepare for disaster face significant risks.

* Increased business disruption costs due to a lack of sufficient preparation.

* Decreased customer income resulting from missed opportunities.

* Reduction of market share due to a loss of customers to the competition.

* Potential lawsuits by customer and vendors due to the organization's inability to perform.

* Penalties and fines rendered by legal and regulatory agencies.

* Negative affect on employee safety, productivity, and morale.

* Increased insurance premiums or a loss of coverage.

* Damage to the organization's goodwill and reputation.

* Detrimental impact to stock value and bond ratings.

DRP Contacts

* Association of Contingency Planners 7040 South 13th Street Oak Creek, Wisconsin 53154 [phone], ext. 116 www.acp-international.com

* Business Continuity Planners Association P.O. Box 75930 St. Paul, Minnesota 55175-0930 [phone] www.bcpa.org

* Contingency Planning & Management Magazine 84 Park Avenue Flemington, New Jersey 08822 Phone [phone] www.contingencyplanning.com www.witterpublishing.com E-mail: [e-mail]

* Disaster Recovery Institute 1810 Craig Road, Suite 125 St. Louis, Missouri 63146 Phone [phone] Fax [phone] www.dr.org

* Disaster Recovery Journal P.O. Box 510110 St. Louis, Missouri 63151 Phone [phone] Fax [phone] www.drj.com E-mail: drj@.com

LANCE J. SEMER, CIA, CISA, CDP, is MIS Audit Supervisor for Avco Financial Services, Inc., a subsidiary of Textron, Inc. in Irvine, California. You may reach him via e-mail at [e-mail].