InfiniBand today.

InfiniBand is one of a few I/O architectures initially developed to address high bandwidth, low latency requirements for High Performance Computing (HPC) environments. While early HPC deployments may have used Ethernet interconnects, a certain amount of latency inherent to TCP/IP limited the overall potential performance of the clusters. Since the transport requirements for compute intensive applications do not need all the features of TCP/IP, development began on streamlined I/O architectures. The resulting solutions like Myrinet and InfiniBand support a Message Passing Interface (MPI) over high bandwidth (10 Gigabits per second, Gb/s), very low latency transport architectures.

Recognizing the broad potential for performance gains and cost savings to general data center customers, several systems manufacturers, including Dell, IBM, Hewlett-Packard, Sun, Intel, and Cisco, have participated in research that resulted in the final specification of InfiniBand. The InfiniBand technology we see today results from merging designs developed by Compaq, IBM, and Hewlett-Packard (Future I/O) and by Intel, Microsoft and Sun Microsystems (Next Generation I/O). Originally called System I/O, InfiniBand is now a trademarked term.

There are two competing protocols currently used in InfiniBand. The first, SCSI Remote Direct Memory Access (RDMA) Protocol (SRP) is the older of the two. Many of the early attempts to connect storage directly to the InfiniBand fabric used SRP as the protocol. Although SRP products are shipping today, the protocol lacks discovery and management features and is unable to accept unsolicited data. These deficiencies may have contributed to the decline in follow-on efforts to develop the SRP2 standard.

The second, and potentially more successful, protocol is an InfiniBand version of the iSCSI Extensions for RDMA (iSER). This emerging standard seems to have traction with the InfiniBand Trade Organization (IBTA) who is working to define the iSER on InfiniBand mappings.

The InfiniBand specification provides a protocol standard and physical connectivity standard with three connection models. The standards require the base 1x connection to sustain a transfer rate of 2.5 Gb/s. Also available are 4x and 12x connections. The 4x provides four "lanes" of wires for (4x2.5 Gb/s) 10 Gb/s serial data transport. Increased clock speeds in InfiniBand components have yielded Double Data Rates (DDR), so the 4x devices can achieve a transport performance of 20Gb/s.

Lanes (or channels) are created between InfiniBand Host Channel Adapters (HCAs) and Target Channel Adapters (TCAs). HCAs are similar to a host initiator device in a storage network. A target device is simply a mode change within the same type of adapter residing on the recipient server or storage device. An InfiniBand switch can be used to interconnect the various devices. When two or more InfiniBand switches are deployed, a 30 Gb/s interconnect is used between them. Collectively, the adapters, wires, interconnected systems, switches, and storage can be referred to as an InfiniBand infrastructure or fabric. InfiniBand is capable of supporting tens of thousands of nodes in each subnet, providing high performance, and enabling scalability and redundancy. This type of infrastructure is quite suitable for HPC, cluster, and grid requirements, which can require thousands of nodes. The ideal storage architecture to support such a multi-node server fabric is a Storage Area Network (SAN). SANs allow multiple nodes to access the same storage simultaneously, which is often critical for HPC applications. SANs also provide more scalable storage architecture, since the data center manager can usually add more space to the storage area network without service interruption. Typically, SANs are deployed with high bandwidth Fibre Channel arrays and accessed through a Fibre Channel switch.

However, the cost of a Fibre Channel fabric is quite high, while the cost of an InfiniBand fabric is more consistent with deploying Ethernet switches. To reduce costs and simplify management effort, customers ranging from those performing HPC simulations to those managing data with Oracle solutions have been pressing for a means to integrate their Fibre Channel storage directly to the InfiniBand fabric. Generally, customers will not choose InfiniBand specifically for storage purposes. Those who have already implemented an InfiniBand infrastructure in their computing environments will find it more efficient and cost effective to connect their storage to the existing InfiniBand infrastructure.

There are two mechanisms to connect Fibre Channel storage fabrics to an InfiniBand infrastructure: bridges and native interconnects.

Bridges. The first, Fibre Channel to InfiniBand bridges, are still somewhat costly and create a bottleneck that gates the Fibre Channel access to speeds less than the array is typically capable of delivering.

Native interconnects. A more cost-effective, easier-to-manage solution is to integrate the arrays directly to the InfiniBand fabric. This is accomplished with native InfiniBand interconnects that replace, or in some case coexist with, Fibre Channel interfaces on the RAID controller.

Connecting storage directly to InfiniBand fabrics also allows applications to leverage InfiniBand's built-in support for RDMA. This is particularly beneficial to HPC environments, allowing the applications to fetch data, compute, and put intermediate results into memory for other processes to complete the computation. InfiniBand implementations allow the server processors to post to a queue and continue processing, rather than managing I/O to a memory mapped device. Much work has yet to be done to bring RDMA to the storage level.

Today, the adoption of InfiniBand is driven largely through the IBTA and OpenIB. The IBTA governs the physical specifications of InfiniBand, while OpenIB is dedicated to a common, open source InfiniBand software stack implementing all relevant protocols. LSI Logic is a founding member of both organizations and holds a seat on the Board of Directors of OpenIB. LSI's participation reflects the evolution of this interconnect to the storage portion of the data center or cluster fabric.

InfiniBand is still a relatively new technology and today it is supported only in homogenous networks based on a Linux operating system. As the early adopters in HPC and data center environments continue to deploy it and reap the benefits of immensely increased speed and low latency, InfiniBand will eventually become more mainstream. It is expected to be adapted, in time, for use in more general purpose computing environments, and even has the potential to be a replacement for PCI bus architecture in high-end servers and PCs.

Dave Ellis is the director of HPC Architecture, Engenio Storage Group, LSI Logic

www.lsilogic.com