InfiniBand comes into its own.

Computer interconnect technologies do not appear very often. Ethernet has dominated this segment for over two decades. The first standard for Ethernet was approved in 1983; "Fast Ethernet" in 1995; Gigabit Ethernet (Gig-E) in 1999 and 10 Gigabit Ethernet (10Gbe) in 2002. Token Ring was an Ethernet competitor and made a strong showing for a while, but did not have the staying power to last. There were a few others (ex. FDDI), but Ethernet has close to 100 percent market share and is continuing to develop. In 2000, a new high-speed interconnect, InfiniBand, was announced. The initial InfiniBand standard was for 10 Gigabit (Gb); companies are already delivering 60Gb products and 120Gb InfiniBand is just around the corner. Should Ethernet providers be worried?

Sound Investment

Many detractors of InfiniBand will say that most applications don't use all of their existing gigabit Ethernet or Fibre Channel (FC) bandwidth so why move to InfiniBand? Not every technology purchase is about performance. For example, when 2Gb FC hit the market, most people weren't using the full potential of their 1Gb FC infrastructures. Today, it is very difficult to find new 1Gb products. 2Gb FC products will become scarcer now that 4Gb FC products are on the market and are priced at 2Gb FC levels. In the next 6 months it will become harder and harder to buy 2Gb FC products. So do you buy 4Gb FC today because it's the performance leader? While performance is a very good reason to buy it, the main reason should be because an investment in 2Gb FC is an investment in a technology that will disappear soon. 10gbe is the latest-greatest offering of an aging technology while 10gbe InfiniBand is just the beginning of InfiniBand. Sun Microsystems has announced that its Utility Computing Grid will be based on InfiniBand; IBM's blade-center is based on InfiniBand; and SGI, one of the pioneers and most important participants in the HPC market, has also integrated InfiniBand in its Altix servers and TP9700 storage device. These are just a few examples. InfiniBand is not going away; it is a very sound investment to buy InfiniBand-based technologies.

So why not just buy 10Gbe; it's not going away any time soon either? 10Gbe is very expensive. The infrastructure (switches, routers, etc.) is very expensive. Even today's most powerful servers would have a hard time driving a 10Gbe without the assistance of a TCP/IP Offload Engine (TOE) card. 10Gbe TOE cards are quite costly and are required. An "upgrade" to 10Gbe is more of a complete infrastructure replacement than an upgrade. InfiniBand, on the other hand, is relatively cheap. Host Channel Adapters (HCA's) and switches are less expensive than 10Gbe TOE cards and 10Gbe switches. Many servers have native InfiniBand support built in, eliminating the need for an HCA, and many blade servers have InfiniBand-based blade chassis. If the decision is to replace infrastructure with InfiniBand or 10Gbe, InfiniBand would be the less expensive solution.

InfiniBand has been in existence two years longer than 10Gbe; there are now millions of InfiniBand-based products deployed today; operating systems have native InfiniBand support available; major companies are developing technologies around InfiniBand.

Besides investment protection attractive pricing, InfiniBand offers another level of savings: less complexity. A typical production server can have three or more network connections. Every server has a network connection (Ethernet) to the production LAN so that other computers can communicate with it; that's a given. Many servers will also have a network connection (Ethernet) to support backup traffic. A third, common connection will be a FC connection to a Storage Area Network (SAN) for access to high-speed, intelligent disk storage devices. Clustered servers will usually have yet another dedicated network connection to support the inter-cluster communication. That is a lot of connections to support. Add to that the fact that servers are becoming slimmer and slimmer and those networking ports take up more and more valuable real-estate within the server.

With InfiniBand, all of those connections can be replaced with one InfiniBand port. For servers that have a native InfiniBand port, there is no added cost to connect into the InfiniBand network. InfiniBand will also limit the amount of network infrastructure; a server that used to use three Ethernet switch ports and one FC port will now use one InfiniBand port. That is a dramatic savings in switches, cables, power and cooling.

Fitting in

InfiniBand is not for everyone. There are many datacenters that don't require more performance and may be perfectly satisfied with their existing infrastructure and have no reason to upgrade. With that said, anybody considering an upgrade to 10Gbe should consider InfiniBand. Too often, InfiniBand is dismissed as only being relevant for high-performance computing environments. Nothing could be further from the truth. A few examples:

Disaster recovery

Any solid Disaster Recovery Plan starts with the fast, reliable backup. Most backups still flow from the application server over a LAN, many times a dedicated LAN, to a backup server. Backup servers will typically have multiple network adapters so that they can process backups from more servers. Implementing InfiniBand to replace the production LAN and backup LAN can be a cost savings (by eliminating host and switch network ports), increase the performance of the application servers (by utilizing a faster, lower latency network) and decrease the time it takes to perform backups. For those who have implemented a disk-to-disk backup solution over a SAN, the increased performance associated with InfiniBand-enabled storage devices can further decrease the time needed to complete backups.

Enterprise-class data replication solutions have made a large splash in the mid-range market over the last few years. More and more companies now replicate between storage devices to provide another level of protection as part of their Disaster Recovery Plans. Storage devices with native InfiniBand support can replicate between each other much more efficiently than over today's FC SANs.

Clustering

There are two reasons to implement a clustered server solution. The first, most prevalent reason is to protect against the loss of a single server. When one server fails, the remaining servers in the cluster take over. The second reason is to increase the performance of the application on the cluster. Today's clusters usually have a dedicated network to carry the data associated with the intercommunication of the cluster nodes. This communication doesn't have a high-bandwidth need but does require low latency. With latencies at a fraction of Ethernet, InfiniBand is the clear choice for this network. Smaller clusters (two nodes) probably won't require InfiniBand but with larger and larger Oracle RAC solutions appearing every day, the need for InfiniBand-level latencies will become much more evident.

Another trend in the market is the concept of a shared file system. NFS has dominated that market for quite some time; today, SAN-based shared file system deployments are on the rise. While not technically clusters, shared file systems do allow servers to share a resource and like clusters rely on a TCP/IP network (typically Ethernet) to manage this sharing. Replacing this network with a low-latency InfiniBand solution will greatly improve the performance of the shared file system brining these solutions closer to non-shared file system performance.

Grid computing

Grid computing is an emerging technology that is gaining in popularity as more and more enterprises realize the benefits of grid-based solutions. Grid computing is not one product but is more of a concept focused on sharing compute resources. In very general terms, it is an environment where all the compute resources (servers and storage) are shared in a large pool. A grid solution is essentially a large cluster with shared file systems and while it certainly fits in a high performance computing world, it is a great solution for companies looking to better utilize their computing resources. This is a perfect solution for InfiniBand: low-latency connectivity with minimal infrastructure (cabling, switches, etc.).

Do not dismiss InfiniBand as a High-Performance Computing solution. While InfiniBand performance starts at 10Gb/sec, with 60Gb/sec solutions currently available and 120Gb/sec solutions just around the corner, performance is only part of the appeal. InfiniBand also provides attractive cost savings and very attractive ROI potentials. Many people are very comfortable with Ethernet solutions and feel that 10Gbe is a good next step for them to take; some analysts feel the 10Gbe is all anyone will ever need; that's also what people said about Gig-E. Today, NFS and iSCSI solutions are pushing Gig-E to its limits. It won't take long for users to fill their 10Gbe pipes either. To rely on Ethernet, which is slow to improve, when a newer, faster, more robust solution is available, may not be the best decision for everybody. InfiniBand offers a single unified wire interconnect for clustering, communication and storage. If you need more performance, are looking to reduce networking complexity or are considering a move to 10Gbe, make sure you take a look at InfiniBand. You may find it meets or exceeds your needs.

Jim McKinstry is a senior systems engineer at Engenio Storage Group, LSI Logic

www.lsilogic.com