DAFS And NISv4 Could Become Semantically Identical.

This is the third in a series of columns authored by members of the DAFS Collaborative, an industry group formed to create a protocol specification for direct, memory-to-memory data networking.

The Direct Access File System (DAFS) is a file access protocol, based on NFSv4 that is being designed to take advantage of new standard memory-to-memory interconnect technologies such as VI and InfiniBand in high-performance data center environments.

DAFS enables an application to directly access transport resources, and transfer data from its application buffers to the network, bypassing the operating system while still preserving file semantics. This translates into high-performance file I/O, significantly improved CPU utilization, and greatly reduced system overhead due to fewer data copies, context switches and interrupts, and far less network protocol processing.

DAFS also provides data integrity and availability features such as consistent high speed locking, graceful fail-over of clients and servers, fencing, and enhanced data recovery, to meet the needs of 7x24 machine room environments where clusters of application servers need low-latency access to shared file storage. This article examines the relationship between DAFS and NFS-v4.

Starting Point For DAFS

NFSv4 was the logical starting point for DAFS because it is an open standard, has good failure recovery characteristics, ACL support, high-performance locking, and accommodates both Unix- and Windows-style access mechanisms.

However, NFS evolved with an increasingly wide-area focus and lacks some of the semantics required in our target "local sharing" environment. Another problem was that NFS is designed with synchronous procedure calls in mind, whereas we wanted to take advantage of the asynchronous nature of VI to allow fully pipelined behavior that is critical in high-throughput environments.

To understand the performance potential of VI, we performed experiments using NFS over a sockets-like protocol on top of VI. Given that VI can reduce network processing overhead by orders of magnitude, the results were disappointing.

The multiple data buffering mechanisms throughout the protocol stack are independent and not easily coordinated. These additional protocol layers were providing little value, while preventing access to useful VI features. Tuning did not help much so we needed a new approach.

We then prototyped a (DAFS) "native" VI data transfer (i.e., no RPC or sockets layer) and saw a dramatic improvement (25 [micro]sec/CPU op-three-and-a-half times better than NFS over sockets on VI (Table I).

Our approach, therefore, has been for DAFS to closely follow NFS-v4 semantics but make fairly radical changes in how requests are represented and exchanged to improve performance using memory-to-memory networking technologies.

The Transport Mapping

NFS tolerates low bandwidth, high latency, connectionless, unreliable networks at the cost of significant complexity. The RPC layer provides synchronous procedure call semantics (Table 2).

DAFS, in contrast, runs over high-bandwidth, low-latency, reliable, connectionoriented VI transports. DAFS communication has more in common with high-performance messaging systems. Asynchronous requests are the primary mode of access, preserving the asynchronous nature of VI requests and allowing high performance and low overhead without extra threads or processes. In highperformance applications asynchronous communication is actually simpler tha;n RPC.

In addition, the :DAFS protocol is adapted to memory-to-memory networking at a fundamental level, providing, for example, both direct and inline variants of read and write operations. These allow low-latency for small operations and efficient data transfer for large operations.

DAFS Performance Improvements

DAFS performance improvements to leverage VI include: use of RDMA, chaining, client-specified byte order, read ahead, and batch writes.

DAFS supports RDMA (remote direct memory access), for requests that might have large data areas, to avoid bulk data copying on those requests, and to limit the size of the control portion of requests and responses. This allows simple efficient message handling implementations on the client and server.

DAFS chaining provides the benefits of NFSv4 COMPOUND in a low-latency environment. Rather than combining multiple operations into a single request, each chained operation retains it own identity within a series of requests. This takes advantage of the low latency network and minimizes buffer space requirements, yet preserves the performance and semantic benefits of pipelining.

In a traditional network stack, XDR performs byteswapping to a standard network byte order. For memoryto-memory networking, a better model is to allow the client to send simple, fixed layout data structures laid out for direct access by languages like C or C++. Whenever the client and server are of the same byte-order, no conversion is necessary. When they differ, the client uses its natural order, and the server performs the conversion. This avoids the complexity of serialized message encoding and decoding for both the client and server.

To promote efficient message processing, DAFS data items are aligned on natural boundaries, avoiding potential inefficiencies of processing misaligned data. Variable-length items are avoided or placed at the end of the message. Allowing fixed length fields to be created and analyzed independently of variable length fields becomes increasingly important as network speeds increase.

DAFS provides a read-ahead mechanism allowing the client to use server memory for read-ahead buffers in preparation for anticipated future use. Once the server has cached the data, the client can retrieve the data buffers without server intervention.

Write batching provides asynchronous request and completion for batches of RDMA write operations. This can be used to implement a periodic write-back cache mechanism allowing the server to smooth out demand spikes and optimize throughput.

"Local Sharing" Semantics

DAFS semantic improvements to NFSv4 focus on environments where clusters of application servers have shared access to pools of file storage. These new capabilities include flow control, a bounded response cache, recovery locks, fencing, fail-over, and migration.

When multiple high-performance application servers share access to each file server, there is the possibility that a flood of requests from a single application server could cause congestion at the file server, locking out other requests. DAFS addresses this by providing flow control directly within the protocol.

Recent requests are tracked in a "response cache" which is used to avoid multiple executions of the same request. DAFS pre-allocates response cache space on a per-session basis, with limited space per session. This eliminates cache overwrite problems and allows DAFS to achieve "exactly once" semantics.

Although NFSv4 introduced major improvements in locking, NFSv4 clients cannot detect that a file lock has been released abnormally, possibly causing the underlying file data to be corrupted. DAFS adds persistent and auto-recovery locks to help clients identify and recovery from this situation.

In clustered environments the cluster manager requires a mechanism whereby processes ejected from a cluster can be prevented from accessing shared file storage. DAFS "fencing" provides this.

In addition to tools supporting client fail-over, DAFS provides for server fail-over, as well as migration mechanisms to accommodate failures, growth, and upgrades in 7x24 environments.

The Direct Access File System protocol is based on NFSv4, but it reorganizes how requests are presented to the transport protocol for transmission. This reflects our observations that traditional network processing inefficiencies are becoming intolerable as transfer rate and operation rate requirements increase.

The DAFS changes involved in this remapping of how requests are transcribed are extensive, but they don't affect NFSv4 semantics at a fundamental level. Today, DAFS and NFSv4 can be viewed as complementary technologies with different target environments. We hope that in the future, they will become semantically identical.

Steve Kleiman is the vice president and Chief Technology Officer of Network Appliance (Sunnyvale, CA) and Mark Wittle is a DAFS architect who works out of Research Triangle Park, NC.

Table 1
Implementation             Client
                             CPU
                        [micro]sec/op
                          (4k read)
User NFS over sockets
on top of VI                 90
Raw access to directly
attached storage             40
User DAFS prototype          25