Planning for resiliency: why regular testing of your business continuity and availability plan is more critical than ever.

Following September, 11, 2001, more companies have started investing in business continuity and disaster recovery planning and have continued to invest heavily in business continuity and availability strategies in the wake of the recent hurricanes. In the event of a site outage or disaster, these companies discover that business continuity requires more than access to key enterprise applications or individual servers. Keeping a business running also requires access to a whole range of vital information and the ability to maintain connections with customers and partners.

These same organizations now face the daunting task of validating the plans drafted by their organizations, and of training their recovery personnel to use these plans effectively. As many companies are discovering, validating a plan is both absolutely necessary and can be more complicated than originally anticipated.

Rehearsals are critical for validating any business continuity and disaster recovery plan because they typically provide the most realistic training environment possible. However, many businesses run the risk of rendering such plans obsolete if not irrelevant by failing to test them on a regular basis, especially as the business grows and changes.

Results of a recent HP-commissioned survey of 200 companies in the United Kingdom found that while 84 percent of these organizations claimed to have a business continuity strategy, only one third test their strategies on a regular basis. About 29 percent of respondents reported they perform tests at six-month intervals, and 37 percent undertake testing annually.

Despite these alarming statistics, there is evidence that those who are testing their business continuity plans are more engaged than ever, in part because they have to be. Companies have found that testing their plans in a piecemeal rather than a unified way has proven to be a flawed strategy.

For example, they may have found in a disaster that the interconnections between databases and an application server did not work, or that the workstation under the receptionist's desk with CRM data on it needs to be tested along with servers that run financial reporting to ensure that a holistic view of the customer remains intact. Alternatively, they may have found that systems administration teams that were assumed to be available during a disaster could not be, because they may be taking care of injured colleagues or family, or they were simply unable to leave the geographic area. In HP's experience, these companies are demonstrating a greater sense of urgency to conduct such tests since Hurricanes Katrina and Rita.

As regulatory bodies levy business continuity requirements in the financial services, insurance, energy and healthcare industries, testing has gained importance. In the end, it is how one performs in an actual event that will determine how the governance bodies will regard the business. One of the most important and valuable aspect of business continuity testing is the validation of documentation and processes. Recovery procedures, server failover processes, call trees and resource listings cannot be proven accurate and useful until tested.

An accurate, updated plan is critical when a business's very existence is on the line. Experts reportedly have estimated that a company loses an average of $84,000 to $90,000 for every hour of downtime, according to published reports citing IDC Research and Strategic Research. More than ninety 90 percent of companies that experience one week of data center downtime will be out of business within 12 months, according to reports citing a study by the National Archives and Records Administration in Washington D.C.

As companies increasingly recognize the importance of testing, several trends are emerging. While disasters such as hurricane, fires, earthquakes and floods were the focus of most testing in the past, companies increasingly tailor their tests to develop improved strategies for handling a broader array of specific threats, ranging from minor incidents like power outages to more serious threats such as chemical spills and civil unrest to terrorist attacks and pandemics.

Organizations also are exhibiting a greater sense of urgency to expand the boundaries of testing beyond typical line-of-business items such as information recovery, or applications to handle accounts receivable and payable. Realizing that addressing operational risk with point solutions is an ineffective strategy for complex IT environments; these organizations are taking a more holistic approach to business continuity testing.

This is especially necessary for businesses that automate critical business processes, for businesses that increasingly operate in a 24x365 world, for businesses that engage with their customers on-line and on the phone. In all of these environments, business continuity, availability and security are highly interdependent. Recognizing this interdependency, organizations now understand that testing needs to examine vulnerabilities in an integrated, systemic way to ensure delivery of the service levels their business needs following a disaster.

Piecemeal testing of parts of an environment also has given way to rehearsals in which organizations can determine whether they can run their entire business elsewhere. More companies are testing how well they can run their business at a remote site, on a server shipped to them in the wake of a disaster, or at a recovery center run by an external contractor, such as HP.

Holistic testing has become viable in part thanks to new technologies that make testing programs for disaster recovery both more efficient and reliable. While in the past, each server would have to be recovered individually; virtualization technologies now allow recovery of numerous virtual servers on one computer, dramatically cutting the time and personnel needed to recover multiple servers.

The move away from storing data on tape to disk-based recovery or active data replication technologies is also is proving both more efficient and reliable. Tapes often physically break, and the media stored inside the cartridge degrades over time. In addition, disk-based recovery enables an IT department to move away from the complexity that accompanies tape backups, including problems with drivers for backup software and patch levels to manage. With a disk replication strategy, the data is checked as often as it is written and validated, facilitating early detection of a hiccup or failure in backup or recovery processes.

Beyond testing business-critical IT operations, organizations are recognizing that testing staff skills and best practices in a broader operational context plays a vital role in successful business continuity, too. For example, testing scripts today might explore how well a disaster recovery team coordinates with an executive board to respond to employee and media inquiries. The scripts may include detailed procedures for handling employee communications, ranging from employee phone trees, to the development of inter-company Web sites designed to provide employees with recovery status updates, where to go for human resource and operational questions, next steps, and work arrangements.

A hard look at alternative work arrangements following a disaster has led to increased testing of remote work options. Many organizations are exploring the use of better tools to facilitate work from home--ranging from providing laptops for home use to terminal servers that allow employees or customers to run a client application on a PC anywhere while the processing is done on a home server.

To ensure this alternative work environment remains secure, companies are expanding the use of virtual private networks which provides secure tunnels over the Internet between remote sites and the home offices or an alternative recovery sites.

At the same time, companies also are exploring the secure use of wireless communications devices ranging from laptops and PDAs to cell and satellite phones to ensure continuity of communications, given the collapse of hardwire and some wireless connections following Hurricane Katrina.

As testing expands to encompass remote work at home or at alternative sites nationally or even internationally, disaster recovery centers are now rarely confined to the four walls of a data center.

A viable recovery plan in fact envisions that the four walls of a specific data center may no longer exist. It must acknowledge that neither an organization's people nor its equipment may be available following a disaster, and that employees may not be able to access applications or business and customer information from home. That's why it is critical that companies test business continuity plans to ensure that their mission critical data fails over to a remote site where authorized users can access it, much as an individual might stow a copy of a will with a lawyer in the event of a personal disaster.

A business continuity and availability plan is not relevant until it is tested. Testing is critical to ensure that any business continuity plan is accurate and functional under the most unfavorable conditions. An effective test can result in many mishaps, but organizations should not shrink from making mistakes. It is best to make, identify and correct mistakes before disaster strikes. Only regular and ongoing testing will do this for you.

John Bennett is worldwide director, Business Continuity & Availability Solutions, HP.

www.hp.com