Printer Friendly

Disaster management: hot news in hot site selection.

THE CORRUPT FOREIGN PRACTICES Act makes corporate officers legally liable for ensuring the recoverability of business services following a disastrous event, but even without this federal mandate, disaster recovery planning is just good business. Disaster recovery planning has many names--business contingency planning, emergency response planning, disaster response planning, business continuity planning, and disaster preparedness planning. Regardless of the name, it is business insurance--nothing more and nothing less. That insurance can take the form of redundant capabilities within a company, mutual agreements with other companies, or agreements with third parties called hot site vendors.

In companies that rely heavily on automation and data processing departments, recovery planning is sometimes assumed to require the development of internal data processing recovery capabilities at remote corporate centers. Several drawbacks accompany this approach, however. For instance, a corporation must invest a large sum of money in redundant computer hardware systems and network interconnections, with sufficient excess capacities to accommodate the additional processing load from the center experiencing the outage.

Also, a company must maintain separate facilities to house these computer systems and networks and separate staffs to operate and maintain them. It must establish and maintain absolute standards between the various data processing sites to ensure compatibility of operating systems, applications, and communications.

A company must fund and conduct semiannual live testing of recovery capabilities at alternating sites to provide constant assurance that all systems and functions are current and compatible.

Recovery planning is sometimes based on binding contractual agreements with external companies at remote sites that have the same data processing capabilities as the subject company. This kind of reciprocal disaster recovery agreement also has a number of pitfalls. A mutual support agreement forces a perfectly healthy company to share a disaster or interruption with the contractual partner. Few companies are willing to jeopardize their own business viability by obligating their data processing resources to an external partner.

Under these arrangements, participant companies must agree to maintain all of their systems and networks in complete compatibility with each other. This restricts equipment upgrades, software enhancements, network protocols, hardware modifications, peripheral equipment upgrades, and overall processing capabilities. Any significant change by one partner forces an equivalent change by the other.

Because of the degradation and interference involved, live testing in these agreements is costly to both partners and difficult to schedule. Problems resolved during one test may be completely replaced by different problems in a later test due to some modification in equipment and connectivity or software versions. The host company's capabilities are greatly reduced during the test period. A blown test could create a real disaster at the host site by crashing the system or otherwise causing a loss of availability.

The budgetary expenses for redundant equipment and staffing and the investments in real property and upkeep required by the two previous options have led to a third option--hot site vendors. A hot site is a fully operational data processing facility configured to the user's specifications and available to the company within twenty-four hours. By paying a monthly subscriber fee, companies can contract with a commercial hot site vendor to use the vendor's hardware, networks, expertise, and staff to facilitate the recovery of data processing capabilities.

Commercial hot site vendors have multiple remote data centers, with work space, air conditioning, security, storage space, and an uninterruptible power supply (UPS). These facilities are filled with a variety of computing hardware with different capacities. In most cases, if they do not have what is needed they will agree to provide it as a part of their contractual obligation.

These vendors have complex and wide-ranging communications networks from the latest in fiber optics to every variation of copper wire technology available. They also have working arrangements with long-distance carriers and other communications providers.

Hot site vendors usually have extensive warehouse inventories of computer hardware that can be shipped overnight and installed at a customer-provided site. Some vendors even maintain large regional recovery sites for clients who require office space during the disaster recovery period. Many hot site facilities also contain large prepped areas for use as cold sites by any subscriber whose disaster recovery exceeds the period of time it is allowed to occupy the hot site. A cold site is a computer-ready facility held in reserve for the user's systems.

Vendors have staff members who are trained in all facets of hardware, software, and communications. During an emergency they do whatever is needed to keep a company in business. They also offer extensive assistance in the development and testing of recovery plans for an organization.

IF MANAGEMENT DECIDES TO SEEK A HOT site vendor to meet its recovery needs, it faces the task of determining specific requirements and choosing a vendor.

The project team. A project team should be assembled to gather and analyze information, compile recommendations, and if so directed, conduct contract negotiations with the selected vendor. The project manager should hold a senior position within the organization to be able to function effectively with all levels of management and across department boundaries. It is also important that the project manager be appointed by an officer of the company. His or her appointment and the project itself should be announced in a corporate memorandum that explains the purpose. The memo should affirm senior management backing for the project and solicit full support from all managers and employees.

The composition of the project team is crucial to the success of the project. It should be no larger than necessary to complete the associated work load; however, it is vital that the people chosen be highly competent and knowledgeable in project management of computer hardware performance and capacity requirements, network hardware performance and capacity requirements, network protocols, operating systems requirements, application software requirements, facilities and environmental requirements, and technical staffing requirements for the hot site team.

Once the project team has been assembled, it needs to formalize a plan of attack, establish milestones, and set realistic deadlines. One of the early steps in the project is to collect data on commercial hot site vendors from which the user company wishes to solicit bids. The user is not legally required to include every hot site vendor in the industry; however, the list should be as complete as possible. Vendors on the list should be given a brief explanation of the size and complexity of the recovery requirements and asked if they would like to participate in the vendor selection process.

Request for proposals. One of the most labor-intensive yet important steps in the project is the request for proposals (RFPs) to the hot site vendors. The responses received from the vendors will be based entirely on the information provided to them in the RFPs, therefore, this document should be as thorough and error free as possible.

If the user's communications network is critical, detailed diagrams showing the hardware components, line speeds, and protocols should be attached. The same goes for interconnectivity between systems, local area networks (LANs), and wide area networks (WANs).

If the company is concerned about the sensitivity of the information provided to the vendor, documentation should be clearly marked as confidential and the company's protection and handling requirements in the body of the RFP itself should be clearly specified. Vendors want the business, so it is unlikely they would intentionally misuse or abuse the information. Nevertheless, no question should be left in their minds as to its propriety.

Some hot site vendors will offer a formal presentation for management. It is neither beneficial nor appropriate at this early stage of the selection process to get bogged down in long, market-oriented, hard-sell presentations. The marketing representatives should be notified that the time for formal presentations has been scheduled later in the project and that they will be notified when the company is ready to begin.

While no right or wrong RFP format exists, an orderly and complete presentation of information will naturally yield the best results. The following format has been used successfully in the past:

Cover page. This identifies the document and the originating company.

Table of contents. This page identifies all major sections and attachments and shows where they can be found in the RFP.

General. A narrative description should state the purpose of the RFP; the general automation environment to be addressed; rules for responding and a description of how responses and vendor communications will be handled; and all pertinent dates and times, such as date prepared, cut-off date for vendor inquiries regarding RFP specifics, cut-off date for formal vendor RFP responses, proposed date for final vendor selection and opening of contract negotiations; and rules for protection and handling of the document.

Facilities. This section should contain clearly defined minimum requirements for housing computer and network hardware and peripherals. It should also discuss employee work areas, storage areas for magnetic media and computer room supplies, and security and access control systems and procedures. Additionally, the section can detail fire detection and suppression; power requirements, such as normal and UPS; air conditioning and chiller requirements; and other specific facility concerns.

Hardware. This section includes a complete and detailed inventory of all items of computer (non-network) hardware in use by the company. If the company has multiple computer sites, a separate RFP should be generated for each location. For each item of hardware, the brand name, model number, nomenclature, and number of units in use should be indicated, along with anything else that will help the vendor prepare a valid cost estimate.

This section should have space for the vendor to provide an overall hardware cost estimate, along with a line-item cost estimate next to each separate hardware item. At the end of this section, a questionnaire should request information pertaining to the non-network hardware requirements and how the vendor proposes to satisfy them. Any questions regarding vendor capabilities, redundancies, and responsiveness to change should be provided here as well.

Software. A complete and detailed inventory of all software in use on the computer systems that the vendor would be expected to provide at the hot site should be listed here. This inventory should be limited to computer operating systems, subsystems, communications, and other software installed and maintained by the corporate systems programming function. The list should include the manufacturer's name, the software application name, the current version date, and any other information that will assist the vendor in understanding the company's system software environment.

A space should be included in this section for the vendor to provide an overall software cost estimate, and next to each separate software item, a line-item cost estimate. At the end of this section, a questionnaire should solicit information on how the vendor proposes to satisfy the company's software requirements. Any questions the company might have about vendor software support capabilities, currency of releases and versions, and responsiveness to change should be included.

Network. This is a complete and detailed inventory of all network hardware relating to the computer processing environment described earlier. This inventory should include all devices, ports, connectors, cables, interfaces, and switches that make up the computer's communications network. The list should include manufacturer's name, model numbers, nomenclature, number of units in use, and any other information that helps the vendor prepare a valid cost estimate.

Space should be provided for the vendor to note an overall network cost estimate, and next to each separate network item, a line-item cost estimate. At the end of this section, a questionnaire should ask the vendor how it proposes to satisfy the company's network requirements. Any questions the company has about vendor capabilities, redundancies, and responsiveness to change should be noted.

Miscellaneous. A complete and detailed description of all miscellaneous services the company is interested in should be provided in this section. That could encompass arranging for travel for recovery teams, providing telephone communications for recovery team personnel, arranging for lodging for recovery team personnel, arranging for overnight mail services, purchasing operations supplies, and any additional costs relating to the provision of such services. The list should include a description of the need, such as green bar computer paper, magnetic tapes, and magnetic cartridges, and the number of units needed.

Space should be included for the vendor to provide an overall miscellaneous cost estimate, and, next to each separate service item, a line-item cost estimate. Questions concerning miscellaneous requirements and how the vendor proposes to satisfy them should be asked at the end of this section. Again, any questions the company has about vendor capabilities, redundancies, and responsiveness to change should be noted.

Testing. A description of the company's anticipated needs for hot site testing is addressed here. A part in this section should be included for the vendor to provide line-item cost estimates for each separate service, and an overall hot site testing cost estimate.

At the end of this section a questionnaire should be included requesting all of the information the vendor should provide pertaining to the hot site testing requirements and how the vendor proposes to satisfy them. Also included should be any questions regarding vendor capabilities, additional training services, and costs for additional testing time.

Attachments. Attachments to the RFP for any additional or supplementary information that might assist the vendor in a greater understanding of the hot site requirements should be included. These attachments can include network diagrams and excerpts from technical documents concerning needs.

WHEN THE RFP DOCUMENT HAS BEEN PREpared it should be checked by all project team members to ensure that the information is accurate and complete. Once the document has been verified and management gives the approval to release the RFP to the vendor list, copies should be mailed to all vendors simultaneously. The company should consider overnight mail if it wants to be sure that all vendors receive the package on the same day.

From release of the RFP package to the notification of final vendor selection, all participating vendors must be treated equally. Responses to individual vendor questions should be provided to all vendors. Any corrections issued or explanations given to one vendor should be provided to them all. This unbiased handling of vendors prevents the appearance of any one vendor being given an unfair advantage.

Great care must be taken in the event the company solicits an RFP response from a hot site vendor that is already contracting with the company for computer hardware or other services. A vendor that is already doing business with the firm may attempt to take advantage of its insider status with the company. If the appearance of favoritism is allowed to exist, the company may be setting itself up for a damaging lawsuit from a nonselected vendor that thinks it is being discriminated against.

Cut-off dates must also be applied across-the-board. If the company extends a deadline for one vendor to submit an additional inquiry or its formal proposal, then the company must extend the deadline for all vendors.

A formal file should be maintained for every participating vendor. These files should contain a chronological record of all communication, regardless of the medium, that takes places between a vendor and the hot site project team members. A diligently maintained record of all RFP activities could be vital if the conduct of the process is ever challenged by a vendor that was not selected.

When the cut-off date for the RFP submissions has expired, the company should analyze the responses received. If any vendor fails to meet the documented cut-off date and if the process has been thoroughly documented and fairly administered, the vendor should be notified in writing that due to its failure to meet the established deadline for RFP submissions, it has been disqualified from the process.

While going item by item through a vendor's formal RFP submission, if a response is unclear or incomplete, it is permissible to contact the vendor and request clarification for the item in question as long as no new information is given to the vendor and the communication is documented and filed. The vendor may not, as the result of such a query, take advantage of the opportunity to provide any additional information over and above that which was requested. Any unsolicited information received from a vendor as the result of a simple request for clarification should be disposed of and a file entry made describing the incident.

Once all of the submissions have been examined and the company is satisfied with the level of detail provided, a matrix should be created listing all participating vendors across the top of the matrix and the key RFP response items to be compared down the left margin. Word processing tools are particularly useful for automating this process and speeding up the activity. By entering each vendor's responses to the key RFP items, this document shows how the various vendors compare with one another on any of the RFP subjects. The company should not go to the extreme of charting every RFP question, since the effort would be more labor-intensive than beneficial. However, by carefully choosing the response items that are critical considerations for the company, a matrix can be produced that can significantly aid in the decision-making process.

As with other equal treatment activities with the hot site vendors, any tool that combines or evaluates the merits of the various participants must be protected. Such information in the hands of any one vendor could prove devastating to the competitors and to the company, as lawsuits would almost certainly follow the compromise of such a document.

The short list. Once the evaluations and comparisons of the competing vendor RFP submissions have been completed, the hot site project team should be ready to take the next step. If one of the participating vendors is in a position of undisputed desirability, the team may choose to go directly to the final vendor selection. If several of the participating vendors are similar in their proposed disaster recovery solutions and costs, the team needs to create a short list of those vendors that provided similar proposals. This process involves reviewing the vendor comparison tool and selecting those vendors that most nearly meet all of the documented requirements and capabilities that fall into the most preferred price range, if price is a factor.

In the first case, the team needs to notify the nonselected vendors in writing that they have been eliminated from the selection process. The team is not legally obligated to tell them why they were not selected. It may wish to tell them that if problems arise in the contract negotiations the team will reopen the RFP process.

In the second case, the team needs to notify the short list vendors in writing that they have made it through the first selection process and that they are invited to continue their participation in the remainder of the process. They should be informed that the team will be contacting them by a certain date to schedule formal proposal presentations at the company site.

Vendor presentations. Prior to the formal vendor presentations, the project team should draw up a meeting agenda. Vendors tend to spend hours touting the merits of their company and the shortcomings of their competitors. The project team must develop a presentation agenda that addresses the needs and interests of the project, concentrating more heavily on technical issues and capabilities.

Once the selection team has agreed on a presentation agenda, the group should document the major issues of concern and key topics of interest, and develop a list of managers and technicians who need to attend at various points during the presentations. The selection team should schedule attendees for the specific times when their specialty topics are discussed to streamline the process.

This is the opportunity to put the vendor through its paces. The vendor's presentation team includes the vendor's most knowledgeable personnel. The purchasing company's team members should confront the presentation team with any questions and concerns that impact the hot site selection project. Any questions that are left unanswered should be documented for a formal follow-up. At a cut-off date, following the completion of any planned hot site visits, issues that are still unresolved should be documented as negative factors for the vendor involved and should help in the final vendor selection process.

Hot site visits. The second part of the formal presentation process should include a visit by key members of the project team to each vendor site proposed as the primary recovery site for the company. The vendors generally welcome the opportunity to show potential users around their facilities. The team must express its needs regarding what it wishes to see and who it would like to discuss technical issues. This agenda should be provided to the vendor well in advance of the scheduled visit. By the time the hot site visits are completed, no unanswered questions or unresolved issues should remain. If answers cannot be provided during the hot site visit, a firm deadline should be set for the vendor to follow up with a response.

One valuable contributor the team should take along on its hot site visits is the company's disaster recovery or security specialist. The disaster recovery or security specialist should perform a high-level survey of the disaster recovery and security systems installed at the site. While many of the findings of such a survey may not result in significant changes by the vendor, they can be used by the company as measurement factors to help in the final vendor selection decision.

Final selection. When the disaster recovery project team has returned from the round of hot site visits, each member of the group should prepare two documents. The first document should be a narrative trip report in which each person relates his or her perceptions and opinions about the vendor hot sites visited. These reports should address each vendor's strengths and weaknesses relating to each team member's area of expertise in the project.

Perceptions of individual experts on the team are important. Although they are inexact factors of evaluation, they can be useful in the event of a tie in the comparison of two vendors. These individuals are often the ones who would have to go to the hot site and actually perform the recovery. As a result of their discussions and inquiries with the technical staffs at the vendor hot sites, they may have a higher comfort level with one vendor's capabilities, flexibilities, expertise, or experience. If all other considerations in the evaluation process have resulted in a tie, the deciding factor may be the project team's comfort level with the vendor it ultimately selects.

The second document that should be prepared following the hot site visits is an extension of the vendor comparison matrix described earlier. By listing key hot site considerations down the left margin and the vendors across the top margin, a comparison chart is produced that indicates how each vendor rated on the evaluation items. This document should be kept highly confidential and should be properly classified and distributed to only those persons with a need to know.

When all documentation has been completed and all team members have had adequate time to digest the information, the final selection meeting can be scheduled to vote on which vendor is the most preferred. This decision should be based on the consensus of the group. Dissenting views or concerns should be discussed and resolved.

After all issues have been addressed, if there is still disagreement among the group members, the project manager must make the final decision based on a majority opinion. In writing up the recommendation to management, the project manager should indicate that the decision was based on a majority rather than a unanimous vote and provide a synopsis of what caused the disagreement and why the majority decided the disagreement was not significant enough to delay the project.

Regardless of whether the selection of the hot site vendor was unanimous or by majority vote, the project manager should be prepared to defend the group's decision to senior management. Hot site contracts are an expensive form of insurance and most corporate executives are not going to obligate the company to such large expenditures unless they are convinced that the choice made is the best for the corporation. Once the responsible officers are persuaded, they will authorize the project manager to proceed to the next step of the project.

Contract negotiations. On receipt of senior management's approval regarding the project team's final vendor selection, the project manager must ensure that the other vendors are notified in writing that they were not selected. The company is under no obligation to provide these vendors with details about the selection process or the factors that contributed to their not being chosen.

Next, the selected vendor should be notified. Most hot site vendors try to persuade companies to accept their standard contract. However, all standard corporate contracts are intentionally written to favor the corporation they are developed by, particularly in issues involving liability or litigation.

Hot site vendor standard contracts often establish limitations on their liability for problems that might occur at any point during the life of the contract. Many times these limitations, if unchanged, could result in the subscriber suffering significant losses in the event the vendor failed to meet its contractual obligations to recover the business processing environment and communications networks.

But what happens if the vendor suffers a data center disaster? For example, company A suffers a disaster and calls its hot site vendor. It implements its recovery plan and ships all tapes and recovery people off to the hot site to recover its systems.

Now, suppose while company A's recovery team is feverishly loading all of the company's back-up tapes onto the vendor's computer system, the vendor's air-conditioning chiller ruptures a pipe at a faulty solder joint. Before the water can be shut off, the hot site subflooring has several inches of water in it, requiring that all power be shut down until the water can be pumped out and the subfloor area sufficiently dried to allow equipment to be powered up again.

The vendor offers to move company A's recovery team to another hot site in the same facility, however, another subscriber has been conducting recovery testing on the system, and it will take twelve hours for the vendor to remove all of the other subscriber's data and people, sanitize the system of the other subscriber's information, re-initialize the disk farm, reroute network connections, reconfigure the system to the company's specifications, load up the new operating system, test the changes made, resolve any glitches, and restart the company's recovery process. In the meantime, company A is losing $50,000 an hour in lost purchase orders.

The result is that the recovery takes twelve hours longer than the maximum allowable downtime contracted for, and the losses over this extra twelve-hour period amount to $600,000 over and above the losses the company has already accrued from the initial disaster. Because this additional loss resulted from the vendor's disaster, the company considers the vendor liable. At this point, the vendor's lawyer points out to company A's lawyers the clause in the vendor's standard contract that clearly states that under such circumstances the vendor's liability is limited to a maximum of $100,000 per disaster. This is one of the many little contractual hitches that can haunt a company if it has entered willingly into a standard contract.

Contracts for hot site services are complex, legally binding instruments. Ideally, a company's lawyers should work with the disaster recovery project team to ensure that all of the concerns of the corporation are addressed in the contract. These issues might include provisions for nonpunitive upgrades or downgrades of system capacities and capabilities during the life of the contract, maximum allowable downtimes after which the vendor pays escalating penalties for delays caused by the vendor, favorable terms for early contract termination under various conditions, and nonpunitive contract termination if the vendor instigates hardware or other changes that are incompatible with the subscriber's requirements.

The vendor's legal staff evaluates what the company proposes and responds with the vendor's counterproposals. The process involves patience and compromise. There may be issues where both parties do not want to accept what is demanded. In such circumstances, the company should not be bullied or intimidated into giving up vital business protections. The secret to success on both sides is compromise, compromise, and more compromise.

The hot site industry is a lucrative and highly competitive business. The vendors that were not selected in the original selection process will still be out there, and most of them will jump at the chance to re-enter the competition for business. Some vendors may be more willing to negotiate than others. Potential buyers should find one that is eager and willing to negotiate a mutually beneficial contract.

Al Foster, CPP, is an asset protection specialist with US West Risk Management in Englewood, Colorado. He is also a Certified Disaster Recovery Planner (CDRP). Foster is a member of ASIS and the Association of Contingency Planners.
COPYRIGHT 1993 American Society for Industrial Security
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1993 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:includes related article
Author:Foster, Al
Publication:Security Management
Date:Apr 1, 1993
Previous Article:Has your store had a check-up?
Next Article:Are we shortchanging ourselves?

Related Articles
Anatomy of a recovery.
Taking the disaster out of recovery.
Protecting million dollar memories.
Vaulting provides disaster relief.
Disaster recovery planning important topic since Sept. 11.
Keep a safe distance.

Terms of use | Copyright © 2017 Farlex, Inc. | Feedback | For webmasters