Enabling Technology For The Data Center OF The Future: DAFS And The InfiniBand Architecture.

I/O is the lifeblood of computing. You can execute all the compute cycles you like, but if you can't get the relevant information into and out of the processor and to the end user on a timely basis, what's the point? This has always been true, but never so much as in the Network Economy.

Researchers estimate that there are now about one million terabytes of online information, an amount that grows daily, with every Web page posted and every online transaction executed. If data isn't smoothly moving among storage, servers, applications, and users, then networks and users are just spinning their wheels. Avoiding I/O bottlenecks and getting data where it needs to be is the central challenge for the e-business data center of the future. The InfiniBand and Architecture and the Direct Access File System (DAFS) protocol are key to accomplishing this critical job.

Getting Data Where It Needs To Be

A variety of I/O technologies are used today. Ethernet and the TCP/IP protocol drive LANs, WANs, and ultimately the Internet. In the data center we also see specialized networks, such as Fibre Channel storage area networks (SANs) and specialized cluster links. At the system level the PCI bus and the SCSI bus are prevalent. The PCI bus occupies a special place, because it has typically been the foundation for all the others. Gigabit Ethernet, Fibre Channel, and SCSI links are usually implemented as PCI adapter cards or PCI components, so all of their traffic is also PCI traffic.

But as performance, reliability, scalability, and density demands increase, shared buses have become increasingly challenged to meet the needs of the new data center. This is especially true as the expansion and I/O requirements of individual servers increasingly take a back seat to the requirements of connecting multiple servers and storage.

The typical data center for Web applications or B2B exchanges, for example, has tens or hundreds of servers arranged in 19-inch racks, all requiring shared access to terabytes of file storage. The workload is defined by network requests coming in through routers, switches, firewalls, load balancers, caching appliances, and the like. Since file-sharing by multiple servers is a fundamental requirement of this environment, storage is usually aggregated into shared storage pools, accessed by the servers using a file access protocol such as NFS.

The result? A complex and sophisticated infrastructure that has exploded in importance in just the past few years. Consequently, the issues surrounding how individual servers connect to "the outside world" have been replaced with issues focused on how such large complexes of discrete units can be organically composed.

The historical I/O solution uses Fast and Gigabit Ethernet networks running TCP/IP for the majority of inter-unit links, with limited numbers of specialized Fibre Channel SANs or proprietary clusters for specific back-end needs. But that's a lot of diversity for solving essentially one interconnect problem. Diversity leads to complexity, introducing more risk of configuration errors, requiring more skilled staff to install and manage, and souring the economic picture by lowering product volumes and raising costs. Even when carefully planned, the combined wiring for racks of servers, disks, and network equipment is ugly and error-prone. And even if you could solve the basic rack wiring mess, network data centers still have a number of critical scaling problems.

Take latency and efficiency. The TCP/IP protocol suite has proven itself a wonderful foundation for communications of all kinds over essentially unreliable networks. But that very fact makes it inefficient. TCP/IP and TCP/IP-based protocols have a complex, layered design, with lots of inter-layer dependencies that can easily demand thousands of lines of code and significant buffer memory to implement. Handling gigabit-class network traffic, servicing interrupts, moving data through long code-paths, and numerous kernel-to-application context switches are all expensive operations. Taken together, they yield long message latencies and use up a significant percentage of available processor power.

InfiniBand Architecture Revealed

The InfiniBand Architecture is a new I/O architecture for the data center, designed specifically to support a clean message-passing paradigm, multiple parallel channels, intelligent I/O controllers, high-speed switches, and "just works" RAS (reliability, availability, serviceability). It is "switched fabric" technology, designed to deliver a point to point, scalable interconnect infrastructure, as well as hot-plug flexibility and true wire-speed communications. In addition, its baked-in notions of link availability, fault isolation, multi-path I/O across redundant links, zoned/partitioned switches, and quality of-service make it ideal for fast-changing and fast-growing configurations.

The InfiniBand Architecture spec creates three different performance classes of link, known as 1x, 4x, and 12x. The lx link will be the high- volume, low-cost InfiniBand Architecture implementation. Each lx link can transmit 2.5Gbps in each direction, or about 500MBps overall--on par with a very fast PCI bus, and well in excess of the latest SCSI (320MBps), Gigabit Ethernet (100MBps), and Fibre Channel (200MBps) designs. But the real kicker is that in addition to having more bandwidth on each link, it's quite straightforward to support multiple parallel links.

And bandwidth isn't the only challenge to which the InfiniBand Architecture rises. It also surpasses other interconnects in terms of efficiency, availability, and flexibility.

The InfiniBand Architecture transitions system I/O from a handful of shared bus expansion slots to a flexible switched network design, enabling efficient, fast I/O at a distance. No longer must each server in a rack devote precious space and cost to having its own hard disk, CD-ROM, and PCI expansion slots. A few processors, memory modules, and support chips can be packaged with a power supply, fans, and an InfiniBand Architecture HCA into much less pace than today's 1U servers--forming a compute module. The computer has been generalized into an InfiniBand Architecture fabric of compute, storage, network/router, and similar modules. This modular approach provides high purchase efficiency and excellent incremental scalability, making it a very attractive package for enterprises and service providers alike.

Clustering will particularly benefit. Clusters need low-latency, high-bandwidth connections in order to support seamless load, balancing, rapid application failover, and parallel database optimizations. But while such links have been desirable, they have been expensive and proprietary. But with the InfiniBand Architecture, such speedy, cluster-class links will soon be part of virtually every server.

DAFS

DAFS is a file access protocol based on NFS, which is designed specifically to take advantage of the Direct (RDMA) and asynchronous I/O features of InfiniBand and VI-compliant interconnects in high-performance clustered environments.

DAFS is a lightweight protocol that enables applications to directly access transport resources. Consequently, a DAFS-enabled application can transfer data from its application buffers to the network transport, bypassing the operating system while still preserving file semantics. This translates into high-performance "native" file I/O over the InfiniBand fabric, with significantly improved CPU utilization, greatly reduced system overhead (due to fewer data copies, context switches, and interrupts), and far less network protocol processing.

Also, since it is designed specifically for 24x365 data center environments where clusters of application servers need low latency access to shared pools of file storage, DAFS provides data integrity and availability features such as consistent high-speed locking, graceful failover of clients and servers, fencing, and data recovery.

DAFS is a perfect complement to the InfiniBand Architecture as a key technology for the data center of the future, enabling independently and massively scalable storage and compute power, high-performance file I/O, data sharing, resiliency to application server and file server failure, ease of management, and a heterogeneous, standards-based environment that promotes cost efficiency and vendor independence.

The Data (enter of the Future starts Here

The InfiniBand Architecture has built up steam, and real products are just now getting underway. With version 1.0 of the specification released in October 2000 companies have shipped prototypes and samples in near-record time. Servers, peripherals, switches, and other infrastructure are expected to market in late 2001 and early 2002, followed by IT appliances, cluster hubs, networking equipment, storage arrays, and Web switches.

To take advantage of this fundamental change in data center designs, OEMs, IHVs, and ISVs began building last year and are continuing their development efforts. By developing prototypes, planning a market-entry strategy, and gaining the experience necessary to win in the next generation data center, more infrastucture players are following suit.

In the Network Economy, effective I/O is the ticket to success and the InfiniBand Architecture is one of the keys that will make it possible.

Jim Pappas is a co-chair of the DAFS Collaborative and director of technology initiatives at Intel (Hillsboro, OR). David Dale is a co-chair of the DAFS Collaborative and industry evangelist for Network Appliance (Waltham, MA).