What has the IT industry really learned from 9/11?

Of all human lamentations, without doubt the most common is: "If I had only known." But we can't know, and so days of death and fire so often begin no differently than those of love and warmth.--Tom Clancy

Another year has gone by since the tragedy we call 9/11 took place, and the world was transformed. We learned more about our own vulnerabilities than we ever knew, or perhaps cared to know. In the IT world, some kinds of disasters are predictable and can be provided for--given manpower and resources. California has earthquakes, the Midwest has floods or cyclones, and the East Coast has hurricanes. But 9/11 was unprecedented in many ways. The raw loss of life was like nothing in the American experience. Financial markets closed for days. Communications, transportation and public services were taxed beyond any precedent. And the information technology resources of large to mid-sized businesses were challenged and, in some cases, over whelmed. Last year, I wrote an article examining whether the industry has learned from the experience. The results this year don't look all that promising.

The commitment to data security and business continuity has always been uneven, with many companies unwilling to make the investment necessary to be sure that the business survives. The IT manager, in the face of our new recognition of vulnerability, is no longer just responsible for bringing up the data canter in all emergency, but is rightly or wrongly expected to recover business and information that was never part of the IT mandate.

Imation conducted a recent national survey of IT directors and network managers, focused on perceptions of data backup and disaster recovery, and how 9/11 impacted data backup and disaster recovery. The results are pathetic; only 26% of those surveyed had developed a disaster recovery plan. Only 21% established a disaster recovery budget (see Figure). An outside observer can only wonder what it takes to get the message across.

Old Lesson

Last year, we examined the old adage of Scouting to "be prepared." In too many cases, IT management wasn't. But since that time, even government leadership has shrank from taking the reins. Storage consultant and author Jon William Toigo (Dunedan, FL) observes: "I was disappointed that the review board of Federal agencies that oversee financial affairs failed to establish that minimum 'safe distance' between a primary IT site and its secondary site. They decided it was not within their purview to suggest a distance." It should also be noted that the administration could come under considerable criticism mandating a potentially expensive and unpopular distance, no matter how necessary.

In last year's examination, we observed that some lessons can be learned from what succeeded as well as from what failed. Modern high availability technologies received their first acid test. Those solutions, on the LAN, PC, Client/Server and Internet level, were often successful. Many of the data security plans put in place for Y2K were useful in recovering from 9/11. This includes all the documentation and drilling that Y2K inspired--employees and emergency personnel were practiced, and therefore prepared.

One lesson, then, is that contingency planning actually worked, for those who did it. Incident management teams were effective and adaptable to circumstances. The Imation survey also suggested that one area where some progress has been made is in injecting more sophistication in backup and recovery efforts. As shown in the figure, respectable numbers of respondents increased backup budgets, established regular update procedures and in many cases moved data backup off site.

Being Prepared

Last year, I had written that the new focus was on preparedness. And that preparedness is beyond just the IT center. In enterprise organizations, I thought, a new sense of urgency and awareness is present in the CFO, the CEO, and the board of directors.

This assumption could well have been false. Consultant Toigo observes: "Disaster recovery is a different kind of a sell. In the best of all cases, you are paying for something that will never be used." This is tree, but insurance is costly as well--and no business exists without the appropriate kinds of coverage. To fail in either adequate coverage or adequate preparedness in case of disaster is to risk a business beyond reason.

It was true last year, and it is true now. The problem is no longer just loss of data and loss of profits. The issues in the boardroom will be loss of life, security planning, a reassessment of traditional risk management formulas.

But while the board deals with the broader issues, IT has a number of narrower ones to consider. There has been a trend in IT for a number of years to consolidate IT resources in the data center. While distributed computing is likely to continue, management and control is the responsibility of the data center. With this trend in view, IT management will have to look at the risks involved with consolidation.

Consolidation, especially in the SAN environment, is IT conventional wisdom but might not be the right answer. Toigo tells the story of two competing insurance carriers victimized by the Kobe earthquake. The company with a centralized IT operation took 4 weeks to return to full operation. The one with distributed processing took 4 hours.

Follow The Money

The risks also demand the increased investment in high availability solutions. Storage networking, as implemented in SAN and NAS, will be looked at as more than just an easier way to manage and control storage. It is more and more to be looked at as the cornerstone of a recoverability strategy. Storage resource management (SRM) assumes a more significant role as a potential lifeline that contributes to data preservation rather than just data management. Issues of disk-to-disk backup for rapid recovery, snapshot backup techniques, storage over IP as opposed to proprietary infrastructures may become a standard since the Internet proved itself invaluable for emergency communications and emergency management during the tragedy.

But in some cases, enterprise IT has not only failed to get the message, they get the wrong message. Toigo notes: "In some cases, 9/11 has had a negative effect on preparedness. They look at the loss of life, loss of money and they ask 'What's the use?'" This kind of fatalism would be very convenient thinking for terrorists, cyber-fiends and other foes of American business, if the thought becomes widespread.

Resisting the Inevitable?

The vulnerabilities that 9/11 laid bare have run head-fore most into the current economic softness, creating a serious crisis especially for IT. As mentioned earlier, IT is going to be expected to be the lead player for overall business recovery rather than just bringing servers back on-line. And this awesome responsibility will go to a department where budgets have been eroding 2-5% per year in recent years. The result is an IT department struggling with the risk versus cost tradeoff, and the hard decisions about which technologies, old and new, are necessary to the survival of the business.

If IT management was actually empowered to independently make these decisions, there might be fewer problems. In discussing the matter of delayed investment in IT for DR with Brian Babineau, research analyst for Enterprise Storage Group (Milford, MA), it became clear that IT ills need not be at IT management's feet.

Says Babineau: "As we got further away from 9/11 and an emphasis on DR, other items took the spotlight, such as immediate regulatory requirements such as the Patriot Act and Sarbanes-Oxley. One came to be because of terrorism, the other due to corporate misconduct. Senior executives are reacting to short-term business needs of the enterprise. IT, in turn, is expected to react to short term needs ..." The senior executives in an organization are supposed to see the big picture clearly. But if many firms are exemplars, the thinking is short term. Babineau foresees a day that there will be board-level representation of the technical experts in an enterprise, possibly banking as an early implementer. If so, it might be possible to break senior managers of the notion that IT management can forestall, without resources or decision input, short-term regulatory demands as well as long-term business continuity protection with the wave of a magic mouse

All sources seem to agree that DR planning needs to be, as Toigo notes, part of the IT infrastructure plan from the very beginning. It is also incumbent on business management to give an attentive ear to IT management, who know the art of the possible. The integrator, OEM and VAR communities have contributory solutions in just about every price range. But IT management, and those who manage businesses, very literally has to get off the dime.

Figure 1--Data Backup and Disaster Recovery Practices Post 9/11

Established regular
testing procedures     56%
Moved data backup
off-site               43%
Established regular
update procedures      42%
Increased Data
backup budget          39%
Developed DR Plan      26%
Set up a data
backup budget          25%
Set up a DR budget     21%

Note: Table made from bar graph.

www.elipsan.com