InfiniBand and the revolution of the DataCenter.

Today's bus-based architecture is limited in its ability to meet the needs of the evolving data center. The speed of the Peripheral Component Interconnect (PCI) bus, the 'gateway' between external communications (the Internet) and the CPU, has not increased in tandem with CPU speed and Internet traffic, creating a bottleneck. InfiniBand (Infinite Bandwidth) promises to eliminate this bottleneck. InfiniBand, a switched-fabric architecture for I/O systems and data centers, is an open standard that implements a network for I/O connectivity, thereby de-coupling the I/O path from the computing elements of a configuration (the CPU and memory). InfiniBand allows for improvements in network performance, processor efficiency, reliability, and scalability. Despite these compelling benefits, the enormous investment in PCI-based architectures will make a phased implementation of InfiniBand necessary.

Bus-Based Architecture

There is an estimated one million terabytes of online information, and this amount is growing with every new web page that is posted. However, getting this information out of the processor and to the end user has become the central challenge in today's data center.

The present bus-based architecture is limited in its ability to handle the amount of data coming in. At the heart of the data center is the uni-processor or symmetric multiprocessor server, functioning either standalone or in a cluster. Network I/O is handled via a shared PCI or PCI-X bus. The 32-bit or 64-bit PCI local bus has become standard in servers mid desktop computers for I/O devices. Historically, this shared bus has proven to be extremely reliable and capable of meeting the requirements of the data center. However, changes over the last few years have begun testing the limits of the shared bus, and have revealed it as a bottleneck.

In the last several years, Internet, intranet, and extranet traffic have increased exponentially. Simultaneously, microprocessor speeds have risen dramatically. The result? The PCI bus has emerged as a bottleneck between the processor and the network. All PCI-based architectures, also know as "shared bus architectures," are able to handle communications with only one device at any given time, creating contention between network devices. As devices are added, the CPU must wait on data, decreasing effective bandwidth.

InfiniBand Architecture and its Benefits

The InfiniBand Trade Association (IBTA), consisting of 230 member firms, including Compaq, Dell, HP, IBM, Intel, Microsoft, and Sun Microsystems, has recognized the limitation in bus-based architecture and formed the IBTA to address these concerns. The IBTA has developed a specification that is scalable and flexible; is drawn on existing technology; and utilizes a governance model that effectively balances the need to develop a specification quickly while involving the industry throughout the development process.

On June 19, 2001, IBTA released version 1.0.a of its InfiniBand Architecture (IBA) specifications. These specifications can be downloaded at www.infinibandta.org.

InfiniBand Architecture

According to the 1.0.a specifications, IBA is described as "... a first order interconnect technology for interconnecting processor nodes and I/O nodes to form a system area network. The architecture is independent of the host operating system (OS) and processor platform."

InfiniBand is much like a phone system--it is able to handle thousands of messages at any given time, as opposed to a shared bus, which is able to handle only one message at a time.

The specification defines the various nodes in a subnet (a single InfiniBand network) Nodes of a subnet include:

* Routers--interconnect components for routing traffic across subnets or to non InfiniBand networks.

* Switches--interconnect components for intra-subnet routing.

* Channel Adapters (CAs)--devices that terminate a link; execute transport-level functions to CPU nodes and I/O nodes.

InfiniBand is an open standard that implements a network for I/O connectivity, thereby decoupling the I/O path from the computing dements of a configuration (the CPU and memory). As illustrated in the Figure, InfiniBand server elements consist of CPUs and memory. Together with the server, switch technology forms a network to which I/O devices are attached. This is the core framework, or, otherwise known as an InfiniBand subnet.

InfiniBand Benefits

InfiniBand allows for greater network performance, processor efficiency, reliability, and scalability. The following outlines how each of these are achieved.

Network Performance--InfiniBand has been designed to solve the problem of meeting I/O demand, which is being generated by high-end computing concepts, such as clustering, fail-safe, and 24X7 availability. The architecture is intended to minimize disruption of existing paradigms and business practices. The specification creates three different performance classes--1x, 4x, and 12x. Each lx link can transmit 2.5Gbps in each direction. Even in its slowest configuration, InfiniBand's throughput is on par with the fastest PCI bus, SCSI, Gigabit Ethernet, and Fibre Channel technology. Thus, implementation of the highest-class InfiniBand architecture will increase throughput by twelve times or more. InfiniBand enables systems to keep up with the ever-increasing customer requirements for reliability, availability, and scalability, increased bandwidth, and support for Interact technology.

Processor Efficiency--InfiniBand's channel adapters are intelligent. This allows them to offload much of the communications processing from the operating systems and CPU. InfiniBand shifts the burden of processing I/O from the server's CPU onto the InfiniBand network, freeing up the CPU for other processing.

Reliability--Reliability is superior to today's PCI model because data can take many paths across the InfiniBand architecture. For example, a processor could have two ports; each port would connect to one of two switches. In the event one of the links failed, all traffic could be rerouted over the other operating link. By building a network of redundant pathways using multiple switches, reliability can be achieved.

Scalability--The center of the Internet data center shifts from the server to a switched fabric in an InfiniBand architecture. Servers, networking, and storage all access a common fabric. Each of these devices can scale independently based on the needs of the data center.

InfiniBand Adoption

Because of the dramatic architecture changes inherent in full-scale InfiniBand implementation, early implementation will be by brand-new data centers and technically sophisticated early adopters. Early InfiniBand adoption is likely to happen in highly technical fields, such as financial services, engineering, scientific, and where legacy hardware simply do not exist.

For the mainstream, adoption will occur slowly, and in phases. Most enterprises have enormous investments in PCI and they are reluctant to abandon this proven standard overnight. Initial InfiniBand implementations will likely be an InfiniBand card that plugs into a PCI slot. This card will connect the server to an InfiniBand switch, connecting it to an InfiniBand sub network. This will enable InfiniBand to be implemented quickly, without rendering existing architectures obsolete.

Given the current macroeconomic climate, it is impossible to accurately forecast when InfiniBand adoption will 'take off'. However, it is clear that the benefits of the technology are so compelling that the appropriate question to answer is not 'whether' the technology will become mainstream. Rather, it is a question of 'when' it will.

Tommy Sprinkle is a Senior Software Architect at BMC Software (Houston, TX).