Has Ethernet's time come for storage systems?

The convergence of all network systems onto a single infrastructure has been the "holy grail" of enterprise SAN managers--long sought after, but never attained. Fibre Channel and InfiniBand are firmly entrenched as storage and compute interconnect technologies, and the common wisdom is that the way forward is to bridge these independent systems into the IP/Ethernet network using expensive and performance-robbing gateways.

But recent enhancements to Ethernet may challenge this view. Improvements in transmission latency--a key factor in network and application performance--mean Ethernet is now well suited to become the universal interconnect technology that can deliver the performance, low cost and interoperability that will drive the next-generation data center.

This vision--enabled by the 10 Gigabit per second (Gbps) performance of Ethernet and its ubiquity for every other networking purpose in the data center--can deliver big benefits to enterprises. First off, the higher performance of 10-Gigabit Ethernet means support for more users and faster response. Additionally, the storage and compute sub-systems become services within an enterprise-wide Ethernet "cloud," giving network managers the flexibility to apportion storage to servers throughout the enterprise without regard for physical connectivity, and to give access to any user in the enterprise as long as they are connected to the network.

If the reader is skeptical of this grand vision, it's understandable. Ethernet has been knocking on the high-performance interconnect door for a long time, but up to now has not met the needs of the application. High CPU processing requirements and a highly "chatty" Internet-strength transmission protocol have introduced a latency that has interfered with real-time performance needed by storage and compute interconnects.

Latency's Impact on SAN Performance

Latency--the fraction of a second it takes for the first bit of a packet to reach its destination from its source--is often not widely recognized as a key component in SAN performance, taking a backseat in many minds to bandwidth measurements. In fact, both are key figures and the suitability of Fibre Channel for SANs can be tied to its low latency level as well as its bandwidth performance.

Network latency calculations take into consideration the processing time required by network adapters to process and transmit (or receive) the packet, as well as the latency in the interconnect network, which includes cable length, load on the network and the processing latency of the interconnect switch. Latency becomes an issue in storage-dependent applications because the small delays in the network are compounded as data is assembled from various servers and storage devices to answer a query.

Distributed databases, for example, give a vivid example of how latency can have a dramatic impact on performance. Distributed databases generate large numbers of short messages between nodes, which are used to communicate pending transactions and other important information to other nodes in the cluster.

A database will typically gain exclusive access to a record by locking it before modifying the record. This prevents multiple databases from corrupting data by trying to simultaneously modify the record. To gain exclusive access, a database will send a lock request message to other databases in the cluster. Then, once the request is granted, the database will modify the record, and send another message unlocking the record for others to access it.

A typical database query, such as "show me the last 12 months of sales of cars sold in the US by model, ranked by color", could generate hundreds-of-thousands--if not millions--of messages. Taxing each message with tens or hundreds of microseconds of interconnect latency would severely limit the practical ability to distribute and scale the enterprise database.

In fact, it is estimated that without a low-latency interconnect, a distributed database application can effectively be limited to only four servers, curtailing the promise of scaling these systems using a large number of low-cost commodity servers. Scaling would then be limited to a small number of expensive servers blocking the paradigm-changing promise of these servers.

Attacking Ethernet Latency at the End Points

Squeezing the latency from Ethernet has meant efforts by both the standards bodies and by vendors that have attacked both the network end points and the interconnect switch. New technology enhancements resulting from more than nine different technology standards initiatives are serving to comprehensively reshape Ethernet to better serve the data center as an interconnect.

Two of these standards efforts, in particular, are designed to help reduce latency at the end points by lessening the impact of the TCP/IP transmission protocol that sits atop Ethernet. Today's Ethernet adapter cards have latency of about 30 microseconds, which includes the latency of moving data from the application to the wire. By comparison, InfiniBand adapter cards accomplish the same handoff in about 3 microseconds.

With remote direct memory access (RDMA), an addition to the IP standard, computers can directly place information onto another computer's memory with minimal demand on the memory bus and on the CPU, replacing TCP/IP copy requirements that add latency and consume CPU and memory resources.

Another solution to the problem is a TCP offload engine (ToE), which is a specialized protocol processor on a network interface card that accelerates all TCP operations and offloads them from a host processor. New adapter cards with both RDMA and ToE hit the market in 2004, with vastly improved latency performance. Second- and third-generation cards, which are beginning to hit the market, offer measured latency performance that is equivalent to that of InfiniBand and Fibre Channel.

The Last Bottleneck: The Switch

Today, an Ethernet switch introduces multiple microseconds of latency to the link. Compared to the end points, the importance of the switch might be understated, because by fixing the end points, you address more than two-thirds of the problem. However, interconnect networks are built using a hierarchy of switches expanding the impact of the switch latency. In a two-tier federation of switches, data would touch three switches between any endpoint, making the switch responsible for nearly half of the total end-to-end latency.

The switch latency issue is being addressed by vendors who are starting to look anew at this market and release new switch chips that can serve as the heart of an interconnect. These switches are growing in their data capacity to remove congestion as a source of latency, and are implementing new flow control mechanisms that smooth the flow of packets through the system. Novel methods for implementing intelligent cut-through switching dramatically reduce the delay through the switch.

Use of these techniques has resulted in a drop in the latency of an Ethernet switch chip--the key component in interconnect latency--by a factor of 10 to around 200 nanoseconds. Putting this chip into the switch used to build the two-tier interconnect network described above results in a latency of less than a microsecond, which, when added to an RDMA or ToE-supported NIC means a total latency of less than 10 microseconds. This is a dramatic improvement over the last generation switches, and provides acceptable SAN interconnect performance.

Convergence Realized

With these advances, Ethernet is now on par with Fibre Channel and InfiniBand--but is that enough to cause the market to change? Normally, it would take more for a new technology to oust a well-performing incumbent, but history has shown that when Ethernet is equal to a task, other considerations of interoperability and operating efficiency take over and push decisions in its favor.

While latency is the key to Ethernet's suitability for SAN interconnect, it is the realization of the long-held dream of network convergence and lower costs that will make it a success.

Mike Zeile is vice president of marketing for Fulcrum Microsystems. He can be reached at [e-mail].

www.fulcrummicro.com

Opening shots in continuing stories ...