All For One, One For All!

SCSI-FCP, FC-IP, FC-VI share a single switched fabric

This article is the second in a two-part series. The first part appeared in the September issue of CTR.

FC-VI is the Fibre Channel application of Virtual Interface (VI) Architecture, a protocol standard defined by Intel, Microsoft, Compaq, and more than 100 other industry organizations for low-latency communications between servers. The design goal of VI Architecture is to achieve low-latency, high-bandwidth communications between application processes running in separate nodes within a computing cluster. This goal is accomplished through Direct Memory Access (DMA) between the application memory and the buffer in the HBA that interfaces to another node on the Fibre Channel network. Figure 1 shows a comparison between the functional structures of TCP/IP and VI Architecture.

VI Architecture establishes a memory registration, which essentially straps down memory locations in user memory and supports the transfer of data directly from user memory to the buffer in the HBA. This data can then be sent over external media to a registered location in another server's application memory. In order to use VI, the application, database, or operating system code must employ an API obtainable from www.viarch.org DB2 Version 6.1 and Oracle 8.1 both use VI in their clustered databases.

Table 1 shows the relative latencies associated with IP over Ethernet, IP over Fibre Channel (FC-IP), and VI over Fibre Channel (FC-VI).

Latency for IP over Ethernet includes the TCP stack (CPU overhead) and the transmission delays of Ethernet. The maximum data transfer rate using 100BaseT is about l00Mbit/sec. EC-IP reduces the transmission latency associated with Ethernet and transfers at Fibre Channel speeds, providing better throughput than IP over Ethernet, but it still suffers from the TCP software latency. FC-VI eliminates the TCP stack and provides DMA transfers from application, memory to the HBA buffer. FC-VI bypasses the OS kernel and avoids OS context switching and buffer changes, leading to much higher data transfer rates than the IP implementations.

FC-VI requires a Fibre Channel HBA designed to support VI Architecture. FC-VI HBAs are radically different from Fibre Channel HBAs designed to support SCSI I/O. Troika and Finisar both offer HBAs that support FC-VI. Finisar sells a PCI-based HBA that supports FC-VI for point-to-point or switched fabric. Troika sells a PCI-based intelligent controller--the SAN 2000 Series Controller. This controller supports SCSI-FCP, FC-IP, and FC-VI in point-to-point, FC-AL, and switched fabric topologies. The Troika controller also provides multiple management options and features such as protocol priority assignments and alternate paths with path load balancing.

Multiprotocot Fabric

All three protocols (SCSI-FCP, FC-IP, and FC-VI) can be combined in one Fibre Channel fabric. Although these protocols could also work in an FC-AL configuration, the associated bandwidth sharing and arbitration contention would negate the performance advantages of FC-IP or FC-VI. Fabric switches are recommended because they provide multiple, non-blocking, 100MB/sec paths from server to server and from server to storage (Figure 2).

The Multiprotocol Business Intelligence Solution

Hitachi Data Systems has developed a multiprotocol business intelligence solution that uses SCSI-FCP, FC-IP, and FC-VI simultaneously in a switched Fibre Channel fabric. The solution, demonstrated at Comdex in November 1999, uses fabric switches from Ancor, Brocade, or both.

Integration of fabric switches in this solution allows central management of the entire data network and also provides unlimited scalability as the number of servers and storage subsystems grows. The use of fabric switches has a negligible impact on overall performance. The overhead cost for SCSI-FCP going through a fabric switch is less than 2% in most cases.

Previous Clustering Implementations

A number of leading IT organizations have implemented solutions to handle business workloads (OLTP, DSS, e-commerce) by using NT clustering and parallel database technology with multiple protocols. Some vendors implement VI Architecture over proprietary interfaces instead of Fibre Channel, creating complicated environments that usually require special expertise and complex interactions between hardware from multiple vendors.

These "butterfly" implementations (Figure 3) may improve performance, but they require management to be distributed across multiple protocols communicating over disparate networking infrastructures.

The added management overhead and the cost of implementing multiple infrastructures through multiple vendors makes the added performance very expensive, reducing price/performance and return on investment. As database needs grow beyond the initial implementation, a three-tier butterfly system may also prove to be expensive and difficult to scale. And any increase in the size and complexity of the system will only exacerbate management problems.

The Hitachi Data Systems solution is an all-in-one implementation: SCSI-FCP, FC-IP, and FC-VI share a single switched fabric. The ability of Freedom Storage subsystems to distinguish between FC-IP and SCSI-FCP frames broadcast to their fabric ports facilitates the all-in-one implementation. Storage subsystems lacking this ability may suffer a sudden "shock," crashing the storage or causing the server to reset.

Hitachi Data Systems Multiprotocol DB2 Implementation

In November 1999 Hitachi Data Systems demonstrated a multiprotocol business intelligence solution running DB2 Version 6.1 Extended Enterprise Edition on a Windows NT 4.0 platform. This MPP (Massively Parallel Processing) implementation uses a "shared nothing" approach, meaning that each server controls its own set of LUNs and sees only one portion of the database. In a 4-way NT cluster, for instance, each server accesses 15 million rows from a table with 60 million rows. Tables are partitioned equally among the servers, and clustering is administered by DB2. It is important to use servers that have exactly the same configuration, since the fastest server must always wait for the slowest server.

The following sections describe the role of each Fibre Channel protocol in this business intelligence solution when all protocols are present. An additional section discusses DB2's behavior in the absence of one or more protocols.

SCSI-FCP

The SCSI-FCP protocol has the most important role in the performance of the business intelligence solution, since it accesses the indices and records stored in the various tables of the database. When running a business intelligence query, up to 90% of the time required elapses during the performance of I/O operations. The preponderance of I/O operations makes the Hitachi Freedom Storage 7700E ideal for this solution, as it features the best I/O performance and throughput for this kind of workload.

Business intelligence requires moving as much data as possible as fast as possible. By using Fibre Channel technology, the 7700E can move up to 93MB/sec through a single interface. With multiple servers connected in a cluster environment and multiple Fibre Channel interfaces configured per server, extremely large amounts of data can be moved in parallel. The Hitachi Data Systems business intelligence solution can scale to such a high level that millions of rows per second can be fetched during a full table scan.

FC-IP

The FC-IP protocol is responsible for performing all command-related tasks between the servers.

Command-related tasks include:

* Db2start/Db2stop commands

* Connect to 'database'/Connect reset commands

* "Force applications all" command

* Propagation of SQL statements across servers

FC-IP provides the "heart beat" between all servers. MSCS (Microsoft Cluster Server) supports only 2 nodes per cluster with Windows NT 4.0 and Windows 2000 Advanced Server. Windows 2000 Data Center will support up to 4 nodes per cluster. DB2 UDB (Universal Database) under Windows NT can support up to 256 "nodes in a cluster. The largest cluster ever built in a DB2 UDB environment included 32 nodes.

In the absence of FC-IP, DB2 will use any NIC (Network Interface Controller) present on the server and connected to the other servers in the cluster. Without any form of TCP/IP communication, DB2 cluster implementation is impossible.

No special commands or registry entries are needed to allow DB2 to use FC-IP on a Fibre Channel HBA. The FC-IP HBA is managed the same as any other NIC on the server. We recommend "pinging" between all servers in order to confirm communication before issuing the Db2start command, which activates all nodes on the cluster.

FC-VI

FC-VI moves data between servers. When fairly large amounts of data need to be moved between servers, the DB2 optimizer (which decides everything) will call the Fast Communication Manager (FCM). FCM includes all the code that supports VI in a DB2 UDB environment. When FCM is invoked, the data is sent to the requesting node through the HBA that manages VI. The requesting node is the node where the SQL statement has been initiated. FC-VI is particularly effective with table joins that require large amounts of data to be transferred between servers.

In order to use VI in a DB2 UDB environment, several NT registry entries have to be coded. For example, the following entries are needed when using the Troika controller:

* DB2_VI_ENABLE=on

* DB2_VI_VIPL=vipl.dll

* DB2_VI_DEVICE=Nic0

When the Db2start command has completed successfully, an entry is created in the Db2diag.log file to confirm that VI was enabled for all nodes in the cluster. We recommend verifying this entry the first few times that VI is used in a DB2 UDB environment.

DB2 ISC and Disk Access Options

DB2 UDB offers multiple protocol options for InterServer Communication (ISC). DB2 decides which protocol to use based on:

A. Option presence or absence

B. Task to be performed

Table 2 presents those options and the priority assigned to each protocol.

Benefits

The primary benefit of the Hitachi Data Systems multi-protocol business intelligence solution is that users can reduce infrastructure costs by integrating protocols into a single data network. This allows functions like disk access, command passing, data movement, "heart beat" verification, and many others to be performed simultaneously at high speed over the same set of wires.

By employing "intelligent controllers" that support multiple protocols on the same HBA, such as the one offered by Troika, users can push this integration to a higher level. Intelligent controllers allow the three Fibre Channel protocols (SCSI-FCP, FC-IP, and FC-VI) to share the same HBA, the same Fibre cables, the same fabric switch, and the same data network. Because each controller can support all three protocols, two multi-protocol HBAs can provide full redundancy for all three protocols, as opposed to six single-protocol HBAs. A multiprotocol HBA implementation can produce major cost benefits. HBA and fabric switch costs and PCI slot, memory, and CPU usage can be reduced significantly when using an intelligent controller like Troika's SAN 2000 Series Controller.

Other Benefits Of The Solution

Fibre Channel transfer speeds make extremely high performance and throughput possible. This is true not only for the SCSI-FCP access to storage, but also for server to server communications and data transfers that would otherwise be limited to much slower 10BaseT or 100BaseT speeds.

A switched fabric with multiple non-blocking paths provides unlimited scalability as the number of users and the database sizes increase.

The integration of a multiprotocol data network that encompasses server area networks and storage area networks provides the opportunity for unprecedented availability and ease of management.

Future Directions For Data Networks

As more applications are developed and modified for VI Architecture, we will see more integration of data networks over Fibre Channel. Emerging market opportunities like Enterprise Application Interchange (EAI), which provides real time transformation and exchange of data between heterogeneous platforms and databases, require the extremely fast server to server data exchanges made possible by FC-VI. Existing applications like network attached storage could utilize VI sockets to run their network file systems at Fibre Channel speeds.

There are efforts underway to propose IP standards for access to storage. Cisco has submitted a specification for SCSI over IP to the IETF (Internet Engineering Task Force). This specification, still under development, requires separate media lines for command/control tasks and data. The overhead for flow control or transmission control is also a major concern.

Morgan Keegan, a leading investment firm and trend watcher, published a research report dated March 8, 2000 that came to the following conclusion:

"With support for concurrent operation of VI and SCSI, we believe Fibre Channel can emerge an a high speed System Area Network solution that drives the convergence of NAS, SAN and cluster networking in the data center. Furthermore, we believe this convergence over time will reach a level of functional integration that provides a common framework for access and control to System Area Network resources."

Hitachi Data Systems is committed to Freedom Data Networks, a data-centric approach to system area networks. The current availability of VI and SCSI over Fibre Channel makes Fibre Channel the media of choice for data networks. Because of the large investment in SCSI applications and data storage, any future media for data networks must provide for the seamless integration of SCSI, as the Fibre Channel architecture has done.

Hitachi Data Systems' multiprotocol Fibre Channel solutions make it clear that Freedom Data Networks can bring unprecedented performance, efficiency, reliability, and elegance to networks everywhere.

Tracy Edmonds and his team from Troika Networks, along with Patrick Wong from Finisar assisted with this article.

John Nguyen is a Data Network architect. Pierre Raymond is the director of Data Network architecture, and Hu Yoshida is the vice president of Data Network Solutions at Hitachi Data Systems (Santa Clara, CA).

Table 2
DB2 UDB ISC Protocol Priority
          ISC-Command  ISC-Data  Disk Access
FC-VI         N/A         1          N/A
FC-IP          1          2          N/A
100BaseT       2          3          N/A
(TCP/IP)
SCSI-FCP      N/A        N/A          1

[Graph omitted]