Disaster recovery: regulatory issues.

In response to the events of September 11th, the Office of the Comptroller of the Currency, Federal Reserve Board, and Securities and Exchange Commission jointly issued a white paper on September 5, 2002 titled Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Banking System. The purpose of this paper was to solicit commentary from the banking industry on proposed regulatory changes designed to increase the security and resiliency of the banking system in a post-September 11 environment.

The issuance of this paper combined the resiliency requirements for multiple regulatory agencies, which had not been common practice before. Issued to retail banks, clearance and settlement firms, investment banks, technology companies, and state and local officials, the paper received significant response from the collective finance industry. The original draft suggested requirements between 200 and 300 miles for separation between primary and secondary processing facilities, with an expectation of close-to-zero data loss and 2-4 hours recovery time.

The final Interagency Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System, promulgated four key practices:

* Identify clearing and settlement activities in support of critical financial markets

* Determine appropriate recovery and resumption objectives for clearing and settlement activities in support of critical markets

* Maintain sufficient geographically dispersed resources to meet recovery and resumption objectives

* Routinely use or test recovery and resumption arrangements.

While these key practices are accompanied by definitions and commentary on scope, areas remain that are subject to interpretation. The report used the language "significant distance away from primary" in lieu of an explicit requirement to refer to separation between primary and secondary processing facilities. The final paper differentiates between the larger and smaller financial firms by indicating the guidelines would be applied more stringently to "Firms that Play Significant Roles in Critical Financial Markets." The net result has been that feedback from the financial industry incorporated into the final white paper, and that the resiliency and recovery requirements--from a technology perspective--allow broad discretion on the part of the regulators with respect to their interpretation on an institution-by-institution basis.

The broad latitude afforded the regulators has engendered a condition wherein the Chief Examiners assigned to specific institutions may differ markedly from one another with respect to intent or application of the guidelines. Taking the example of a financial institution with the primary trading workforce located on the island of Manhattan with a primary data center tens of miles away in another state, the guidelines may be interpreted such that little marginal resiliency is required to meet the requirements, inasmuch as the primary processing facility is not located within a key target region (i.e., Manhattan). Alternatively, the guidelines may be interpreted such that significant marginal resiliency, viz., a full-scale secondary data center, may be required to protect the firm from the potential loss of the primary facility, regardless of whether it is in a high-risk geography or not. The crux of the issue is whether the guidelines are interpreted to apply to primary facilities in high-risk geographies such as Manhattan, or primary facilities regardless of geography. The difference in this interpretation can lead to $100MM decisions as one considers the full cost of implementing a secondary data center with appropriate system duplication.

The first key to the current regulatory focus is separation: Separation of primary and secondary processing facilities, separation of user communities from their primary processing to segregate workplace recovery from system recovery, and separation of in-region and out-of-region system recovery. The second key to the current regulatory focus is resiliency: Continued resiliency of books and records (essentially unchanged from previous regulatory guidelines (see Securities Exchange Act of 1934 (Amended), Rules 17(a)(3) and 17(a)(4)); resiliency of intraday processing to reduce exposure to lost or unsettled transactions; and overall resiliency of the financial markets as a whole through an industry-wide synchronized but continually shrinking recovery time objective (i.e., 2-4 hours). The shrinking of recovery times from days to hours, and the requirement to minimize data loss, forces firms to move from tape-based restoration procedures to data replication procedures to ensure the timely availability of data.

The paper specifically addresses risks associated with the likelihood of "wide-scale disruption." Prior to September 11th, most institutions had business continuity and disaster recovery plans based primarily on the likelihood of an institution or facility-specific event. The expansion of planning scope to include a wide-scale event has significant implications for the shared-services disaster recovery market. In the shared-services market, vendors develop system and workplace recovery capacities including system hardware and hot-desk arrangements for workers well short of the total subscription base (i.e., for 100 subscribers, there may only be sufficient capacity to recover 15-20% of the systems and workplaces for 15-20% of the subscriber base, a 1:50 - 1:100 ratio). In the event of an institution-specific disaster (such as loss of a single building, data center, or trading floor), this will provide sufficient capacity to recover the institution. In the event of a wide-scale event, however, 80-85% of the institutions, based on the vendor's business model, may be unrecoverable. This shortfall is inducing some large financial firms to move their recovery capacity in-house as well as driving changes in the shared services vendors to lower overlap ratios, viz., 1:10 - 1:25, which greatly increases the cost base.

The focus of the white paper is on "recovery and resumption of clearance and settlement activities for material open transactions in wholesale financial markets," not the resumption of business as usual retail banking or trading functions. Nevertheless, the majority of large financial services firms developing updated disaster recovery plans in response to the white paper have elected to extend the four key practices to their retail banking and trading functions.

The regulatory changes addressing the resiliency of the financial industry have driven changes into the scope of disaster recovery, the selection of suitable technology solutions, and the vendor industries that support these efforts. The confluence of these changes has engendered much planning and implementation activity within the financial industry. These planning and implementation activities begin with a Business Impact Assessment. The BIA is a business-focused discovery activity that drives out the requirements for recovery including workforce distribution and recovery, critical business processes to be recovered, and the financial risks associated with each business line. Due to the highly technical nature of the banking industry, this effort is usually executed as a joint partnership of both the business and technology teams. Once the BIA has identified the businesses and primary systems to be recovered, as well as their associated recovery time and recovery point objectives, the technology team must perform a Technology Impact Assessment to analyze and assess the ramifications of system recovery, such as associated back-end systems not visible to the business community.

Many technical and business hurdles must be overcome to meet these requirements, primarily data replication, which can occur either synchronously or asynchronously. In synchronous data replication, the write to disk of changed tracks of data must occur on both the primary and secondary sides before processing is permitted to continue. Due to latency, distance (usually between 40 and 100km) and retransmission, the time required to send, acknowledge, and write the changed track may have an impact on production processing times; a primary limitation of synchronous data replication environments. Synchronous data replication, however, ensures that no data is lost as processing does not continue until the data is known to be valid at the secondary site. By contrast, in asynchronous data replication, the "write to disk" of changed tracks must occur only on the primary side for processing to continue. Without knowledge of, or intervention from, the ongoing processing, the data replication toolset forwards the changed tracks to the secondary side while processing continues; thus, the processing and the forwarding of changed tracks to the secondary side are un-coupled, or asynchronous to each other. As processing may continue prior to successful transmission to the secondary side, asynchronous data replication unavoidably introduces the potential for lost data, but allows for greatly reduced sensitivity to latency, distance, and retransmission.

The potential for lost data does not invalidate asynchronous replication as a viable strategy. Traditional nightly backup tape-based recovery systems, for example, are asynchronous to production in that they introduce a potential of 24 hours of data loss in the event of a primary side failure. Institutions have very successfully managed disaster recovery with this limitation. The key difference, as promulgated by the regulators, is the requirement to capture in-flight transactions for same day settlement and recovery of systems within two to four hours. The key architectural element for systems planners responding to the altered regulatory environment is to balance marginal business requirements against technical efficacy and cost; for example, zero data loss may be desirable, but solutions are limited to a 40-100km limit, while five minutes of data loss may be less desirable but acceptable, and permit out-of-region recovery.

In conclusion, the shift in regulatory oversight to address more specifically the resiliency required to respond to a wide-scale event, coupled with a consistent, industry-wide shift to "business as usual," real-time replication, and restart vs. recovery disaster planning have had marked impact on the requirements, scope, and technical implementations of disaster recovery plans.

www.cstechnology.com

James Dow is director of the Technical Architecture Group at CS Technology (New York, NY)