VI: Enabling High-Performance Data Access.

This is the fourth in a series of columns authored by members of the DAFS Collaborative, an industry group formed to create a protocol specification for direct, memory-to-memory data networking.

The Virtual Interface (VI) Architecture, released in 1997 by Compaq, Intel, and Microsoft, is a specification for a high-performance networking interface which dramatically reduces the overhead of traditional network architectures. To accomplish this, VI defines mechanisms which allow applications to access VI network adapters without operating system intervention and carry out bulk data transfers directly to or from application buffers.

First generation VI implementations typically featured parallel database applications running in clustered environments. The second generation VI Network Adapters discussed in this article enables a new generation of applications deployed in high-performance standards-based clustered environments.

What Is A Virtual Interface?

Several components are brought together to create a 'Virtual Interface', as shown in Figure 1.

A Virtual Interface is created by a VI "Provider" (VI Network Adapter) when requested to do so by a VI "Consumer" (application).

A Virtual Interface consists of a Send Queue and a Receive Queue, which hold Descriptors of data movement requests. Descriptors contain all of the information the VI NIC needs to process the requests. However, data transfer does not occur until VI pairs are connected. A connected pair of Virtual Interfaces forms a bi-directional, memory-to-memory virtual circuit. Doorbells are used to notify a VI NIC that work (new Descriptor) has been placed on one of the Work Queues. A Completion Queue holds information about completed Descriptors from multiple Work Queues.

A VI Provider initiates a Virtual Interface using a VI NIC and a VI Kernel Agent. The VI Kernel Agent is a device driver for the VI NIC that registers application memory and manages VIs. The VI Consumer, consisting of an application program, communications facility and a VI User Agent, communicates using a VI. The VI User Agent is the software that abstracts the underlying VI NIC in accordance with an interface defined by the Operating System communication facility. An implementation of the VI User Agent is called a Virtual Interface Provider Library (VIPL).

VI-Enabled Applications Provide Improved Use Of Resources

Parallel databases, including Oracle Parallel Server, IBM DB2 UDB EEE, and Microsoft SQL Server 2000 were early adopters of VI, exploiting the benefits of direct memory access. Native support for VI allows these applications to place data directly on the wire without copying data to kernel buffers. VI has enabled these applications to achieve benchmark results previously obtained only on mainframe class systems.

Oracle Net8 (and its predecessor for Oracle7 called SQL*Net) was another early adopter, using VI to speed network communications. Although focused initially on providing low-latency message passing for distributed applications, VI has garnered growing acceptance as a mechanism to accelerate data access. One example is Computer Associates International's use of VI in its ARCserve 2000 Workgroup/Advanced Edition Backup Agent for Microsoft SQL Server, doubling performance over its 100BaseT Ethernet offering.

VI Network Adapters

Leading providers of VI network adapters include Giganet and Troika Networks.

Giganet's cLAN solutions have been enabling high performance VI-based data center networks since 1998. The cLAN product suite includes host adapters and cluster switches supporting server farm networks in Windows and Linux environments. Giganet customers have built server farms ranging in size from two nodes to greater than 100 nodes, deployed in business critical environments such as e-commerce, high availability computing, data warehousing, parallel databases, and scientific and technical computing.

Troika pioneered the notion of simultaneously running multiple protocols over a common wire transport. The Troika Zentai Controller supports industry standard networking and storage protocols (FC-VI, IPFC, FCP-SCSI) over networks based on the Fibre Channel standard. Coupled with management software (employing an industry standard HBA API) and unique dynamic multipathing capabilities, the Zentai Controller has enabled Internet data centers to consolidate their storage and cluster networks.

Next generation network adapters will leverage VI to embrace other networking technologies, including Gigabit Ethernet and InfiniBand.

Giganet has recently introduced a second product family called VI/IP By incorporating both VI and TCP/IP into a single architecture, Giganet's new VI/IP products will combine the speed and efficiency of VI with the standard structure of the IP transport. Supported on existing Gigabit Ethernet infrastructures, Giganet's VI/IP products will bring the power and benefits of VI to an Ethernet standard environment.

InfiniBand defines a switched communications fabric for connecting multiple end nodes with high bandwidth and low latency in a protected, remotely managed environment. It defines the host behavior (verbs) and memory operation such that the channel adapter can be located as close to the memory complex as possible. The InfiniBand point-to-point switched fabric supports reliable messaging (send/receive) and memory manipulation (remote DMA) without software in the data path, providing enhancements to VI.

Emerging VI Applications

The benefits of low client overhead and zero copy technology are gathering support for using VI throughout the data access path. A framework for understanding how applications access data is illustrated in Figure 2. Most applications use the services provided by a file/record subsystem to access data. This subsystem in turn uses the services of a block subsystem to access data stored as blocks on physical media. Functions in both subsystems can be performed in the host, the network, the storage device, or combinations thereof.

File systems themselves may be divided into client and server components and distributed across a network. The advantage of this approach is that multiple clients, running on different host operating systems, can share files by using the same file server. The tradeoff of this approach has been a relative performance penalty associated with client/server communication over a network. VI and the related communication mechanisms specified by InfiniBand verb behavior, provide an innovative way to not only speed client/server communication, but to return more application server resources to the application.

The Direct Access File System (DAFS) is a lightweight file access protocol optimized for high-performance, mission-critical data center environments. Based on NFSv4, DAFS is enhanced to take advantage of standard memory-to-memory interconnects. DAFS supports client implementations suitable for database and operating system vendors as well as native DAFS applications.

Other developments focus on mapping traditional protocols over Remote DMA networks. Microsoft Windows 2000 Datacenter Server allows applications to automatically leverage VI networks using its Winsock Direct interface. The Direct Access Sockets (DASockets) open source standardization effort is also working to provide interoperable traditional sockets emulation on other open system platforms. And a SCSI RDMA Protocol (SRP) project in the ANSI T10 Technical Committee is standardizing a SCSI protocol mapping onto cluster protocols such as InfiniBand.

In 1997, several leading companies, including Intel, Compaq, and Microsoft, envisioned the day when traditional networking architectures would no longer process data movement requests fast enough to keep pace with CPU and memory technology, thereby becoming the bottleneck to overall system, storage, and networking throughput. That day has come. The Virtual Interface Architecture was designed to solve this critical problem.

VI provides a technique for direct memory-to-memory transfer of data between networked nodes. VI applications have direct access to the network, bypassing the operating system, thus greatly reducing CPU network processing requirements and freeing up CPU cycles to perform application processing.

The emergence of second-generation products from leading providers of VI hardware and software is enabling a growing number of application, database, hardware, and software vendors to embrace VI and develop robust products.

Mike Dutch is the director of business development at Troika Networks (Westlake Village, CA) and Marc Shelley is the business relations/product marketing manager at Giganet (Bolton, MA). Both Dutch and Shelley are co-chairmen of the DAFS Collaborative marketing working group.