InfiniBand--A Look Ahead.

This article is the first in a three-part series. The second part will appear in the May issue of CTR.

The leaders in the Intel Architecture (IA) and RISC-Unix open systems markets have come together to define InfiniBand, a Switched Fabric I/O Standard slated to replace the shared bus for attaching I/O to servers. A switched fabric connects processor and I/O nodes via a network of switches and point-to-point links. The shift from bus to fabric is a profound reset of server architecture with far reaching implications for the way systems are designed, built, and sold. The promise is end-user systems that are simpler, cheaper, more reliable, and easier to expand and service.

The leaders of the open systems market backed InfiniBand, which might imply broad adoption of InfiniBand as a PCI replacement. It will happen quickly. What could go wrong? Surprisingly, the answer is plenty. Getting InfiniBand defined and working is just the first step. Then, comes the real work--replacing PCI. Economics will shape the outcome and the bus may prove surprisingly resilient.

Today's Bus-Based Server Architecture

A brief review of today's server architecture, depicted in Fig 1, provides a reference in which to understand InfinBand. The basic building block is an uni-processor or Symmetric Multiprocessor server, functioning either stand-alone or as a node in a multi-server cluster. Cluster configurations are increasingly used in data center, ISP, and electronic commerce applications. The server node consists of CPUs, memory, and I/O bridges, connected via a proprietary memory bus or crossbar. The I/O bridges implement one or more PCI (or PCI-X) segments, each supporting one to five I/O slots. (The higher the segment frequency, the fewer the slots.)

Fig 1 depicts the three main server I/O functions: external communications (e.g., the Internet), data storage and retrieval, and, in the case of a cluster, server-to-server Inter-Processor Communications (IPCs). Typically, all are implemented via PCI adapters, using either a Host Adapter Board (HAB) or chip on the motherboard.

A wide variety of PCI HABs exists because of the wide variety of external communications interfaces such as Ethernet, T1, and ATM. HABs attach the server to LAN/WAN infrastructure elements like switches and routers. Storage is a bit simpler. Two interfaces, SCSI and Fibre Channel, dominate the IA and RISC/Unix server markets. A SCSI or PCI RAID adapter connected directly to hard drives may suffice in a small, standalone server. Fig 1 depicts Fibre Channel HABs connected to a SAN.

IPC networks vary significantly, depending on cluster usage. In the simplest case, a point-to-point Ethernet link is used to exchange heartbeat messages in a two-node availability cluster (each backs the other in case one fails). Generally, however, multiple servers must work together effectively on parallel or partitioned applications, which puts tremendous demands on the IPC network in terms of data throughput and server-to-server message latency. These demands are beyond the capability of standard LAN technologies.

Unlike LAN/WAN and storage, there has been little in the way of broadly adopted IPC standards or third-party merchant solutions. OEMs have, instead, relied on internally developed solutions like Compaq's ServerNet. Recently, the Virtual Interface (VI) standards initiative defined an architecture that allows applications on different server nodes to exchange messages and data directly without going through the operating system, eliminating much of the overhead associated with standard LAN protocols. VI has been mapped to a variety of hardware layers. Also, third-party solutions explicitly targeting cluster IPC are beginning to surface. One example is GigaNet's cLAN (for Cluster LAN) family, a line of VI-based PCI HABs and switches.

Switched fabrics are not new. Fibre Channel, ServerNet, and cLAN are all switched fabrics. Proprietary fabrics have existed in enterprise markets for years. The InfiniBand vision, though, is much broader than that of any of its switched-fabric predecessors.

Tomorrow's InfiniBand Switched Fabric Architecture

Now, compare Fig 1 with the InfiniBand architecture depicted in Fig 2. The first thing to notice is that there are no PCI slots or HABs. An InfiniBand bridge has replaced the PCI bridge in the host chipset. The server has collapsed to nothing more than processors, memory, and InfiniBand I/O ports. (Each server node needs local I/O like boot drives to function.)

To replace PCI, InfiniBand must, by definition, be a true unified systems area network, not just a Storage Area Network like Fibre Channel. Like PCI, a unified network can transport any/all traffic types: LAN/WAN, cluster IPC, and storage. A user may allocate specific InfiniBand channels to specific functions, but it's a configuration rather than a technology choice.

The nature of buses is such that PCI slots and HABs must physically reside in the server enclosure with CPUs and memory. In contrast, InfiniBand decouples I/O from the server. The PCI HAB is replaced by the Target Channel Adapter (TCA) board, which plugs into a switched fabric, rather than a shared bus backplane. TCAs do not have to physically reside in the server enclosure. Separate enclosures just for TCAs, especially in rack mount environments, will emerge. Of course, standalone servers with everything in one cabinet will continue to exist, but it will become a packaging choice.

In contrast to PCI, InfiniBand is defining TCA slots and boards for easy, one-dimensional insertion and removal. This will allow boards to be added and removed without having to open the enclosure, similar to the way individual drives are accessible in a disk array.

In some cases, there will be no need for a host or target board-level adapter at all. In Fig 2, no adapter exists between the server host chipset and the RAID system or LAN switch with a native InfiniBand interface. This will become increasingly true as the InfiniBand market matures.

Tom Heil is the senior systems architect for the storage components division at LSI Logic (Fort Collins, CO).