When disaster strikes: IHEs are implementing wide-ranging plans to protect their technology systems in the event of natural or man-made disasters.

Campuses along America's southern coasts are familiar with the annual battering they get from tropical storms. So they put a lot of effort into ensuring the safety of their main assets--students, faculty, staff, and facilities and critical data. When the first storm of the 2005 season threatened Florida's coast at the beginning of summer, people like Bernard Chapple were able to see just how well their planning would pay off. Chapple, who is chief information officer and director of Information Technology and Telecommunications at Edward Waters College in Jacksonville (Fla.), has made a career doing disaster recovery planning and management at organizations as diverse as Merrill Lynch and Toyota.

Though all kinds of businesses and organizations will have many emergency needs in common, there are some needs that only come up in higher education. The needs can be institution-specific. For example, at Chapple's college, which is affiliated with the African Methodist Episcopal Church, collecting students in safe places to wait out a storm involves keeping the men and women in separate dorms. In terms of technology needs, however, many schools are pursuing similar strategies of outsourcing or replicating much of their data to protect it in the event something takes down the on-campus servers. Technology leaders are also working closely with their staffs to develop plans, and then run them through worst-case scenarios. They're meeting regularly with representatives from other departments, too, to make sure the entire organization would work together in times of crisis. "The chief information officer, in particular, must believe in and articulate the importance of developing and maintaining such a plan in order to earn staff buy-in," notes Linda Deneen, director of Information Technology Systems and Services at the University of Minnesota, Duluth.

People Need technology

At St. Mary's College in Notre Dame, Ind., Director of Information Technology Keith Fowlkes says a disaster recovery and business continuity plan for information technology and telecommunications--luckily unused, so far--has been in place for about 18 months.

St. Mary's plan for its IT systems "states that securing and isolating storage devices and systems is our top priority," says Fowlkes. A closed network protects and secures data and equipment, and the school has agreements with several external vendors and service providers to help out in time of a disaster, for both telecommunications and hardware repair.

Yet, he warns, "even iron-clad contracts are a variable in a regional or national disaster event." That's why having good internal staff members who are committed to the institution's mission and have the best training possible is so important. Fowlkes also advises having "as many backup components as your budget will allow. This is especially true for small schools that will almost inevitably be lower priorities to vendors who also service large institutions and businesses in their regions."

Institutional data is the top priority for Rosie Quelch, network manager at The Isle of Wight College (UK). "Software applications and server hardware can all be replaced," she says. To provide that backup service, the college relies on storage and backup consulting services by GlassHouse Technologies. The company's team designed a backup and archiving system (using IBM Tivoli storage management, xSeries servers, and Overland storage systems) that reduced backup time for the college from 14 hours to two, as well as shortened the times of other IT management tasks.

Toronto's Humber Institute of Technology and Advanced Learning also considered Tivoli but instead went with storage and backup technology from CommVault Systems. The solution delivered the same benefits Isle of Wight College got--speedier backups of critical data on a system that is easier for the IT staff to manage.

"We are now very confident in the data," says C.M. Shum, manager of Technology Services Support at Humber. His team is using the time freed up by the new system to research other technologies that will further the institution's goal of data protection and integrity.

Man with a Plan

Chapple seems to take a bit of delight in reeling off the names of the four hurricanes that he had to deal with in rapid succession in his first year at Edward Waters. (For completists, Charley, Frances, Ivan, and Jeanne slammed into Florida in August and September 2004.) Such disasters are not what-ifs; they are practically an annual event. So it makes sense that the school brought in a disaster-preparation expert. Chapple, a member of the faculty of the International Disaster Recovery Association and co-author of Information Security: The Complete Reference Guide (McGraw-Hill Publishing, 2003), joined the school in mid-2004. Immediately, he set about creating plans, holding meetings, and directing his staff to be ready for crises.

This year, he's continuing those efforts. Chapple plans to outsource the school's web systems so, for example, Edward Waters students traveling out of state can access the school's web systems and get information in the event of trouble. He's working on a plan to outsource the institution's critical data storage to an off-campus company, both for access and recovery in the event of natural disasters and to increase security of the data. Colleges are becoming juicy targets of data hackers, Chapple notes.

Tabletop rehearsals with the various business units on campus are another way Edward Waters' administration stays prepared. Chapple once created a fictional hurricane (named after his daughter) and had it moving toward the school on a specified route. "'It's going to hit Jacksonville in X hours. What are you going to do? Admissions, suppose you have recruiters on the road; what are you going to do?'" he recalls asking his team.

"It's not rocket science," Chapple says, "but you have to have dexterity."

For IT staff at University of Minnesota, Duluth, the top priority during a disaster is to restore help-desk service, "followed closely by restoring our phone and data services," says Deneen.

Enterprise data is already managed off-site, at the Twin Cities campus of the University of Minnesota. "We do have important data [in Duluth], but most of our critical data is stored at the Twin Cities' campus." Duluth's disaster plan was renewed and updated this year. Though it has not yet had to be used in a real emergency situation, administrators have run tests of the system during each of the past two years.

The Components

Key to restoring lost data and resuming business operations after a disaster strikes is a safe and sound backup system. Companies such as Blue Hill Data Services, LiveVault, and NSI Software provide backup services in the that on-site data is lost. Fusepoint Managed Services, Rackspace, and others offer a range of services such as backup, server hosting, and disaster recovery plans.

"We are very pleased with the software we use for backup, redundancy, and recovery," says Fowlkes from St. Mary's. SyncSort software, with redundant backup storage systems placed in different campus locations, and tape backup to an ADIC Scalar system get the job done. "All these storage systems are linked via their own isolated gigabit network. The functionality of the SyncSort software is great and platform independent, which is valuable to us because we support Sun Solaris, Windows, Linux, and Mac OS X server systems." Off-campus storage of media is currently handled by staff members, but Fowlkes says his department is planning to move to a local provider for off-campus storage in the near future.

Off-campus providers are central to many disaster-recovery and business continuity plans. Options range from off-site storage (almost every organization at one time or another used to have a staff member designated to take backup tapes home every day, though now most have moved to automated off-site backups and storage) to rerouting of internet and telecommunications traffic (so schools can keep in touch with all concerned parties, which often includes students on out-of-state or even international research trips).

Experts say the key to getting the right third-party support service is specificity about the needs of the institution and the capabilities of the provider in the event of a disaster. Make sure that any vendor has a package that matches the school's needs. For example, would you just have data replication and backup services with them? If so, would you be able to continue the school's business if your on-site network and servers were actually destroyed and not just taken offline for a couple days?

Once a vendor is selected, be clear about the services you expect, as well as the level of quality control you demand (and do demand it). Also, be a pessimist, experts advise. Ask questions like: What if our backup fails? If the backup service is across town, would we likely be okay if the emergency is a campus-based, man-made one? And then there are natural disasters to consider. If a hurricane or tornado devastates the community, your cross-town backup has just been an expensive lark. They're all tough questions. But asking them now sure beats an "Uh-oh!" exclamation later.

Resources

ADIC, www.adic.com

Blue Hill Data Services, www.bluehilldata.com

CommVault Systems, www.commvault.com

Fusepoint Managed Services, www.fusepoint.com

GlassHouse Technologies, www.glasshouse.com

IBM, www.ibm.com

LiveVault, www.livevault.com

NSI Software, www.nsisoftware.com

Overland Storage, www.overlandstorage.com

SyncSort, www.syncsort.com

Where's My Data? Tips for Disaster Planning 

Administrators with experience in thinking through potential disasters have learned a few things about the process. They suggest that those at other institutions:

* HAVE A PLAN. Within a week of arriving at Edward Waters College in Jacksonville, Fla., Chief Information Officer Bernard Chapple told his tech staff he wanted not just a plan, but "an actual step-by-step plan so it can be idiot-proof," he recalls. For instance: "They can bring up the AS 400 [IBM server], they can bring up the network--write that down so we can have that information together."

* CHANGE THAT PLAN. Disaster plans will change over time based on lessons learned at your own or other institutions. Also, technological assets change over time. Administrators at St. Mary's College in Notre Dame, Ind., review their plan annually. "We have not had to make additions as much as clarifications," says Keith Fowlkes, director of Information Technology. But a more significant revision is coming. "Now that our data center location has changed, we will definitely need to make some important changes to it," Fowlkes says.

* COMMUNICATE. When this summer's hurricane season began, Edward Waters College posted on its home page a hurricane update to keep students and faculty informed. Chapple suggests that, at a very minimum, schools have a contact list. "Remember the students. The students don't know anything about what we're doing," he says. "They need to feel secure. I know that feels Pollyannaish or soft, but they are the most important thing, and if the students are safe, then the parents feel safe." Fowlkes advises having several off-site options for redirection of web traffic in an emergency so families can be in touch. "Also, have vendor agreements in place for expanded cellular capabilities, backup telecom and network circuit providers, vendors for repairs and new installations, and dial-in network access--local and regional," he suggests.

* MEET. While meetings are probably the Last item anyone wants to add to their schedules, they can be critical to ensuring all departments know what they should be doing in event of a disaster, what other departments such as IT will be doing, who's in charge and who can be contacted at what numbers and e-mail, and what best practices all departments can implement. Chapple holds monthly meetings with a safety committee, which he chairs. Its members are from across campus: Residential Life, Business and Finance, Communications and Marketing, Athletics, Security, Admissions, and more. They prepare, they consider Lessons learned, they strategize, and they rehearse. After a big event such as a tropical storm or a tornado, his team also meets for a postmortem to dissect the college's performance during the emergency. "Carve out staff time to work on this regularly," says Linda Deneen, director of Information Technology Systems and Services at the University of Minnesota, Duluth. "It can be very hard to give this priority over daily pressures, and this is the first thing to slip."

* TRAIN. Make sure everyone has a very clear idea of what they should be doing in the event of a disaster. They should also be prepared to do even more. "Cross-training within the operation is paramount, given the possible staff attrition in a disaster event," says Fowlkes.

* OUTSOURCE. "Protect your data," says Chapple. While systems may remain on campus, outsourcing data replication and backup to an off-campus (and preferably out-of-state) third-party provider can ensure that the organization will be able to continue operating in the event of an emergency.

* PREPARE FOR RECOVERY. Depending on the nature of the emergency, off-site vendors or service providers may also have Lost staff or physical assets, or they may just be stow in servicing a heavy toad of requests at one time. Have a fallback provider or plan ready to deal with slower-than-expected recovery.

John Burton is the West Coast correspondent for University Business.