Protecting records in the face of chaos, calamity, and cataclysm: even organizations that do not think they are prime targets for terrorists do not have the luxury of considering themselves exempt from disaster planning.

The impact of the events of September 11, 2001 (9/11), will be felt tot decades in many areas of national and international affairs. One business impact has been on efforts to prevent and mitigate the loss of records and information. In terms of records and information, 9/11 was, by far, the largest disaster in American history. What conclusions can be drawn from the tragedy, and what lessons can be learned by those who manage records and information for organizations in government or business?

First and foremost, no amount of planning could have anticipated the incredible level of destruction; no contingency planner could have imagined the devastating events. The loss of life was astounding. In fact, the loss of people and their technical talents set the attacks apart from any disaster that preceded them. Some companies lost most, if not all, of their disaster recovery team, as well as large numbers of their information technology (IT) staffs. This made the recovery efforts enormously more difficult than they otherwise would have been.

The World Trade Center (WTC) Towers housed about 1,200 businesses that employed some 40,000 people. The largest tenants included some of the world's prominent financial institutions. Among them

* Morgan Stanley Dean Witter 

* Bank of America 

* Deutsche Bank 

* Oppenheimer Funds

* Credit Suisse First Boston 

Many other large financial institutions maintained offices in the WTC's immediate vicinity and were extensively damaged if not destroyed. According to "Terrorist Attacks Have Far-Reaching Effects on Businesses," a recent Disaster Recovery Journal article, more than 15 million square feet of office space were destroyed or damaged--an amount comparable to the entire downtowns of Atlanta or Miami. The Emergency Operations Center for the city government, located at 7 World Trade Center, caught fire and eventually collapsed, leaving New York City without a centralized facility to manage its emergency operations.

Disaster Recovery Planning

The terrorist attacks changed the face of disaster preparedness. In their wake, traditional disaster recovery practices have been shown to be inadequate. Businesses must now prepare for the kinds of threats and a scale of potential damage never before imagined.

According to a survey conducted at the Disaster Recovery Conference, prior to September 11, only a small percentage of disaster recovery plans addressed terrorist acts, and still fewer addressed acts of war. In light of the attacks, 97 percent of respondents indicated that their plans required at least some revision. A Disaster Recovery Journal article, "What Disaster Recovery Experts Were Thinking Just After the Attacks," states that clearly the respondents were under-prepared for threats of terrorism, war, biological hazards, and explosions.

Even organizations that do not think of themselves as prime terrorist targets do not have the luxury of considering themselves exempt from the need for disaster planning. Terrorism and threats of war must be taken seriously in all disaster planning scenarios.

The 1993 terrorist bombing of the WTC taught tenants to be ready. Charles Phillips of Morgan Stanley Dean Witter told Information Week that the WTC "was probably one of the best prepared office facilities from a systems and data-recovery perspective." In an interview with Storage Management Solutions, Merrill Lynch's Director of Global Contingency Planning, Paul Honey, stated that the Y2K experience greatly improved his company's ability to respond to the 9/11 disaster. The huge expenditures of Y2K yielded small dividends at the dawn of the new millennium, but were priceless on September 11. Without such prior efforts, the information losses of 9/11 have been much worse.

Smaller Organizations Most Vulnerable

The ability to respond to a disaster successfully is not an optional management initiative but an essential component of the cost of doing business. For the most part, the critical systems of major financial institutions and other large businesses seem to have recovered quickly and according to plan.

"In the most high-risk, high-exposure environments, we had great success," Joseph Walton of EMC Corp. told Information Week. In general, however, small and mid-size businesses were more vulnerable because few had the budget for real-time data backup or offsite recovery facilities. Walton also said that some smaller companies "were just put out of business." Their ability to get back to business depended on whether they recently saved essential data to disk, tape, or other media for storage offsite, and many had not done so.

New Government Initiatives

The 9/11 disaster revealed a major problem in the ability to fight terrorism--the inability to share intelligence data among various agencies of government. The detection and prevention of acts of terrorism poses truly unprecedented information challenges. Success in the war on terrorism will depend on extensive sharing of computer data and records among intelligence and law enforcement agencies at all levels of the U.S. government. The FBI, the CIA, the National Security Agency, the Immigration and Naturalization Service, and many other government agencies are now trying to share intelligence and investigative data on an unparalleled scale. Part of the challenge is that dozens or even hundreds of databases are involved, running on a wide variety of computer and software platforms. Only a few of the systems are currently interconnected. The resolution of this problem is a major challenge for the U.S. government.

The notion that the value of information is directly proportionate to its accessibility has been an article of faith in information management circles for decades. The 9/11 nightmare underscored this cardinal principle. The seamless transmission and sharing of information, both inside and outside the firewall, is needed now more than ever. Organizations do not have the luxury of designing information infrastructures solely for their own use. The value of information can be optimized only through greater accessibility.

President Bush recently signed the Homeland Security Act into law, authorizing the formation of the Department of Homeland Security, which will have the authority to develop a plan to inventory and protect the nation's critical infrastructures--telecommunications, financial and banking, energy, and transportation. Safeguarding the IT infrastructure in both the public and private sectors is expected to be a key part of the new department's work. Moreover, the U.S. Department of Justice has proposed new legislation that will give it the power to prosecute computer crimes as acts of terrorism.

A Success Story: Recovery at Merrill Lynch 

When Wall Street went back to work on the Monday following the attacks, most of the world saw what Wall Street and Washington wanted it to see: that financial markets had reopened, trading systems were up and running, and investors were buying and selling millions of shares of stock, seemingly without a glitch. "America's economic system is intact," Richard Grasso, chairman of the New York Stock Exchange (NYSE), was quoted as saying in a Fortune article, "Telco on the Frontline." However, things weren't that simple.

No story illustrates an impressive response to disaster as well as the recovery at Merrill Lynch, one of the world's preeminent securities and investment companies. Merrill Lynch's Honey has said in numerous media interviews that the company activated its disaster recovery plan within minutes of the attacks and immediately began transferring business-critical functions to its command center in New Jersey. Because the 60,000-square-foot facility had been pre-designed as a corporate-wide disaster response facility, all personnel knew immediately where to dial in to transfer information. This allowed transactions throughout the company's global offices to continue basically without interruption.

As was widely reported, trading operations were transferred to London, Tokyo, and Hong Kong. Moreover, Merrill Lynch devised a plan to use a telemarketing service and the company's public Internet site to communicate with displaced workers. In 1999, the company had installed a global trading platform that meant all its business was on the same system--and that the system would operate even if part of it went down. Merrill Lynch deployed 5,000 technical employees, many working 36-hour shifts, to re-establish communications to the stock exchange.

This foresight did not mean that everything went perfectly, however. When the stock market opened on Monday morning, Merrill Lynch's traders lacked the electronic data feeds normally used to send orders to the exchange floor. They had to make do with the telephone. Instead of transmitting buy and sell orders electronically, clerks rushed to large whiteboards where they scribbled ticker symbols with felt-tipped pens. Research analysts, who had escaped the WTC with none of their old research reports, worked from home. They began assembling and transmitting research reports directly in the e-mail messaging environment. Still, Merrill Lynch was never unable to execute buy and sell orders for securities--a very impressive achievement.

How "Remote" Must Backup Be?

As mentioned, Merrill Lynch transferred certain business-critical functions to its command center in New Jersey. Although its remote processing capability was successful, 9/11 has caused serious consideration of just how remote an organization's backup facilities should be. Sixteen months later, according to "U.S. May Require Backup Wall Street," a Wall Street Journal article, federal regulators are considering a plan that would require the nation's largest banks and securities firms, whose operations are critical to the integrity of the nation's financial systems, to establish backup facilities hundreds of miles outside of Wall Street and New York City.

Largely for reasons of convenience, many firms had established their backup facilities within 20 to 30 miles of the city. In some cases, these facilities used the same power and telecommunications grids as were used for the primary sites, which meant that full operations could not be restored for days or even weeks. To cite one example, the contingency floor of the NYSE is located in Brooklyn, where the offices of the Securities Industry Association, the technical backbone of the NYSE, are also located. NYSE Chairman Grasso has stated that these facilities are too close to Wall Street to act as an effective backup site, and alternate sites are currently being evaluated.

The new regulations, which have been prepared by the U.S. Department of the Treasury, the Federal Reserve Board, and the Securities and Exchange Commission, would require regulated companies to ensure that critical payment and clearance functions could be restored in full on the day of a catastrophic event.

An important lesson is that backup facilities supporting mission-critical business functions should not be located where they may be subject to the same disaster that affects the primary site.

The Technology Infrastructure

One of the biggest lessons learned from 9/11 was that, although it sustained severe damage, the IT and telecommunications infrastructure supporting the WTC did not collapse.

The area around the WTC in lower Manhattan is one of the most telecommunications-intensive sites in the world. The attacks knocked out telecommunications in a huge swath of lower Manhattan, including the data lines that served the NYSE and a custom software system that helped the "Big Board" communicate with its trading partners.

Under great pressure and horrific circumstances, Verizon, the major provider of local phone service in the area of the WTC, was able to pull off a feat few thought possible: It rebuilt its network in lower Manhattan during the course of a single weekend.

"On Thursday there was literally no hope," said Lawrence Babbio, head of Verizon's disaster recovery team, in an interview with Computerworld. In the same article, Paul Lacouture, president of Verizon's network services division, said, "I've gone into our buildings after fires. I've restored our networks after floods and earthquakes. This was a combination of all those things, times a factor of three or four."

To cite just one aspect of the restoration effort: The major cellular carriers (AT&T Wireless, Verizon Wireless, and Cingular) all deployed temporary Cellular on Wheels Systems (COWS) to replace downed towers and restore wireless phone communications at both the WTC and Pentagon sites.

Despite the success in restoring data and phone communications, 9/11 underscored the vulnerability of the existing telecom network in the United States, a vast jumble of copper and fiber lines, wireless transmitters, and computers that is much more fragmented than it was a decade ago.

Despite the WTC experience, disaster recovery plans must assume that phone service and Internet connections may not be restored for up to several weeks following a cataclysmic disaster.

Centralized vs. Decentralized Computing 

After 9/11, only a few companies required help with mainframe recovery; the biggest challenges were restoring mid-range systems and servers, restoring connectivity to computer networks and desktops, and obtaining access to stored backup information. Whether organizations should implement centralized or decentralized computing strategies is an issue that has been debated for decades. With the rise of client/server computing during the 1990s, following the end of IBM's dominance in highly centralized mainframe computing, decentralized computing became very popular. During and after 9/11, many organizations were crippled because they could not access the servers where their data was stored. The terrorist attacks have thus underscored the need to deploy digital assets in a manner that is designed to reduce the risk of attacks on any one location where these assets are deployed, according to InfoWorld article "Enterprise Storage."

In the wake of the attacks, organizations must consider strategies for dispersing computer data over multiple servers and multiple locations. One such approach to decentralized computing that is receiving increased attention is the use of storage area networks (SANs). SANs are dedicated storage networks in which servers and storage devices are connected by hubs and switches. The network's software permits the centralized management of data regardless of platforms or media. The data itself can be easily and quickly dispersed to secure, offsite locations. In a SAN data storage environment, if one site is taken out of commission, the whole enterprise is not affected by the event. The SAN architecture simplifies the process of creating mirror data images at remote locations, thus ensuring that remote, up-to-date copies of databases are available in case of disaster at the main location.

The major lesson is that organizations must now take steps to make their businesses less dependent on a single office or data infrastructure.

A Safety Net

Despite massive breakdowns on the telecommunications and computing front, the Internet never skipped a beat. As originally planned by the U.S. Department of Defense, the Internet was designed to remain invulnerable to acts of war. If one part were to be disabled, the entire system would not be put out of commission. This is exactly what occurred 9/11. The packet-based, asynchronous Internet enables messages to travel by a variety of routes to reach recipients, thereby reducing the risk of overall system failure. When phone service failed, people turned to the Internet. If Internet connectivity was lost and organizations couldn't get e-mail in and out, they established temporary access via other Internet service providers. Finally, it was widely reported that many organizations relied heavily on instant messaging to communicate with employees because it was the only method by which to do so.

The major conclusion and lesson to be learned: The 9/11 tragedy is very likely to spur use of the Internet and Internet protocol (IP)-based networking as the next generation of disaster recovery plans evolve. Far from discouraging organizations from doing backup of the Web, the opposite is much more likely to occur.

The Internet is, however, extremely vulnerable to the risks associated with cyberterrorism. In the Information Week article "Terror Attack Brings Renewed Emphasis On Security" Dennis Treece of Internet Security Systems Inc. states: "The way I look at vulnerability is to see how dependent companies are on Internet connectivity." Companies whose entire business is based on the Internet are completely vulnerable.

The Vulnerability of Paper

The 9/11 tragedy has had a huge impact on the way in which organizations view paper records. This disaster was unique in that almost no paper records survived the terrorist attacks. Except in cases where people took some paper documents with them in their briefcases when they escaped the collapsing buildings, virtually all paper records were destroyed.

What lessons can be learned about paper records from what happened on 9/11? The following points are most significant:

* Organizations that rely heavily on paper documents are most vulnerable to significant losses. Law firms and insurance companies are examples of businesses that remain very paper-intensive and, therefore, most vulnerable. As noted earlier, smaller organizations are generally more at risk than larger ones.

* Because paper records have been declining in importance relative to computer-based records for many years, organizations that have aggressively applied new technologies to automate their business processes are much less vulnerable than those that have not.

* For most large organizations today, to lose all paper records would be extremely inconvenient, but it would not actually put the organization out of business. To cite just one example, the headquarter offices of the Port Authority of New York and New Jersey were located in the WTC. Nearly all paper records were destroyed but the Port Authority remained in business and says it is now recovering.

* On the other hand, for an organization to lose all its computer records would be truly cataclysmic and would very likely result in the demise of the organization.

* To cite another example, Marsh & McLennan, a WTC tenant, had been engaged in a five-year program to convert paper documents to scanned images. The effort paid off. More than 25 million business documents that had been converted from paper to scanned images were backed up offsite and, therefore, were saved from destruction. According to InfoWorld, only one business day's worth of records was lost.

Offsite protection has always been a better and more reliable means of protecting vital data, but 9/11 highlighted the utter futility of onsite protection strategies. For many years, records management specialists have relied on fire-resistive filing cabinets and fireproof vaults as the primary methods for protecting paper records onsite in cases where it is not feasible to protect them by sending either the originals or duplicate copies to a secure offsite location. However, if an organization is vulnerable to a terrorist attack of the severity of those that occurred on 9/11, the only feasible strategy to protect the records is to get them offsite.

Where paper records are concerned, the major lesson to be learned from 9/11 is that organizations should adopt the long-term, goal of converting to digital format every paper-based record-keeping system of mission-critical importance as soon as resources and priorities allow.

Most experts usually recommend that organizations give themselves five years to get out of paper, at least for their most important, business-critical applications. Generally, paper should be reserved for small quantities of records of low value--personal working papers kept for convenience or reference at or near the workstations of employees. Most, if not all, records of official character and high strategic importance should be converted to digital format as soon as possible. Every disaster planning initiative should incorporate a document digitization strategy to make this happen.

Digital Preservation Best Practices

September 11 resulted in a flood of retrievals for data residing on backup media. These attempts to retrieve old data uncovered significant problems and the need for better long-term data retention practices. In many cases, the data on the backup tapes could not be accessed. In some cases, this was due to data errors or defective media. In other cases, when information several years old needed to be reconstructed, conflicting formats sometimes stymied the efforts.

Best practices for long-term data retention include

* selection of storage media specifically intended to support extended-term data retention

* standardization of file formats

* proper management of metadata

* proper maintenance of systems documentation

* proper housing of stored media

* regular inspection and maintenance of stored media

It is important to inspect archived data regularly to ensure that it can be processed and that the medium hasn't become outdated, given current backup systems. Organizations should implement formal practices to ensure the integrity and retrievability of backup data, no matter how long it may reside on storage media.

Lessons Learned

* Disaster recovery plans should be summarized in a few pages and always carried in a briefcase and/or a PDA.

* These plans should clearly delineate the responsibilities of all employees and include critical phone numbers, communications inventories, hardware and software inventories, master call lists, master vendor lists, and inventories of offsite storage facilities.

* Vital records and data should never be stored solely on local PC hard drives. Organizational policies should require that all mission-critical documents and data be saved to network servers so they can be routinely backed up.

* Keep copies of all software and passwords offsite, including client and server installation disks.

* According to the Disaster Recovery Journal article "Real Life After September 11," data about new computer systems under development is an element in disaster planning often overlooked by busy IT departments in the rush to bring new applications on stream. Many systems in a stage of development need to be backed up, particularly if they are deemed business-critical.

If American governmental and business organizations apply these lessons, they will be in a high state of readiness to meet whatever emergency may arise in the future. Where the integrity of their recordkeeping systems is concerned, they will be better and stronger than ever.

At the Core

This article:

* Discusses the impact of 9/11 on disaster planning

* Discusses centralized vs. decentralized computing

* Provides best practices for long-term data retention

References

Andolsen, Alan A. "On the Horizon." The Information Management Journal. March/April 2002.

Ballman, Janette. "Merrill Lynch Resumes Critical Business Functions Within Minutes of Attack." Disaster Recovery Journal. Fall 2001.

--."Terrorist Attacks Have Far-Reaching Effects on Businesses." Disaster Recovery Journal. Fall 2001.

Behar, Richard. "Fear Along the Firewall." Fortune. 15 October 2002.

Brewin, Bob. "Nation's Networks See Sharp Volume Spikes After Attacks." Computerworld. 17 September 2001.

Chandler, Robert C. and J.D. Wallace. "What Disaster Recovery Experts Were Thinking Just After the Attacks." Disaster Recovery Journal. Winter 2002.

D'Auria, Thomas. "Facilitation, Cooperation Guide New York City to a Quick Recovery." Disaster Recovery Journal. Winter 2002.

De Jesus, Audrey. "Real Life After September 11." Disaster Recovery Journal. Spring 2002.

"Disaster Takes Toll on Pubic Network." Informationweek. 17 September 2001.

"Enterprise Storage." InfoWorld. 26 February 2001.

Foley, John. "Ready for Anything?" Information Week. 24 September 2001.

Grygo, Eugene, Ed Scannell, Mathew Woollacott, and Dan Neel. "Industry Toils to Restore Trading." InfoWorld. 24 September 2001.

"IT Recovery Efforts Forge Ahead." InfoWorld. 17 September 2001.

Kempster, Linda. "2001: The Year of Vulnerability." Storage Management Solutions. Vol. 7, No. 2, 2002.

Mehta, Stephanie N. "Telco on the Frontline." Fortune. 15 October 2001.

Rendleman, John, et al. "Assessing the Impact." Information Week. 17 September 2001.--and

-- and George V. Hulme. "Security Synergy." Information Week. 22 July 2002.

Rynecki, David. "The Bull Fights Back." Fortune. 15 October 2001.

Schoeder, Michael and Kate Kelly. "U.S. May Require Backup Wall Street." Wall Street Journal. 22 October 2002.

"Terror Attack Brings Renewed Emphasis On Security." Information Week. 17 September 2001.

Whiting, Rick and Eric Chabrow. "Safety in Sharing" Information Week. 8 October 2002.

David O. Stephens, CRM, FAI, CMC, is Vice President, Records Management Consulting, at Zasio Enterprises. He may be reached at [e-mail].