Wide area file sharing across the WAN.

Distributed enterprises virtually cover the globe. Remote offices are everywhere and remote office workers now far outnumber those who work out of central office locations. With this distribution of resources, today's companies must manage development efforts across multiple remote locations which means that they must also somehow enable all remote office workers and team members, worldwide, to collaborate on the same shared files and data at the same time. Add to this the fact that file sizes and data storage requirements are increasing year after year, and the efficient sharing of files across distributed enterprises over the wide area network (WAN) has become a Herculean task.

The problem is that although gigabytes of data can easily be shared over a local area network (LAN) using standard file server technology, they cannot so easily be shared across remote offices connected over the WAN. In truth, standard file server protocols provide unacceptably slow response times while opening and writing files over the WAN and this forces remote office IT managers to make some unappealing choices. IT managers and network users must either live with reduced productivity due to poor network performance at remote offices or they must use replication schemes that waste storage and inhibit global collaboration.

Recently, however, a new class of product known as wide-area file services (WAFS) has showed remarkable results in solving the problem of remote office collaboration for distributed organizations. WAFS allows companies with remote offices to utilize the WAN to share files as if it were a virtual LAN, enabling real-time, read/write access to shared files while also guaranteeing the consistency and coherency of all file data.

The most successful WAFS systems address inherent WAN file sharing issues through a multi-layered technology approach. In designing this technology, WAFS vendors begin by carefully looking at the file sharing protocols used in today's enterprise infrastructures. This is a key starting point, which bears a closer look.

Problems With File Sharing Over the WAN

All major file sharing protocols, including NFS (Network Filesystem for Unix/Linux environments), CIFS (Common Internet Filesystem for Windows environments), and IPX/SPX (Internetwork Packet Exchange/Sequenced Packet Exchange for Novell environments) were designed for LAN environments where clients and servers were located in the same building or campus.

The assumption that the client and the server would be in close proximity led to a number of design decisions that do not scale across WANs. For example, these file-sharing protocols tend to be rather "chatty", which means that they send many remote procedure calls (RPCs) across the network to perform operations.

Let's take a look at a closer look at the NFS protocol to show an example of this type of "chatty" behavior. For certain operations on a filesystem using NFS (such as an synchronization of a source code tree), almost 80% of the RPCs sent across the network can be access RPCs, while the actual read and write RPCs typically comprise only 8-10% of the RPCs. Thus, 80% of the work done by the protocol is simply spent trying to determine if the NFS client has the proper permissions to access a particular file on the NFS server, rather than actually moving data.

In a LAN environment, these RPCs do not impact performance significantly, but when combined with the high latency typical of WANs, these RPCs can be deadly to performance. Worse, remote clients often end up timing out and retransmitting the RPCs, compounding the inefficiency. Furthermore, because data movement RPCs make up such a small percentage of the communication, increasing network bandwidth will make no difference to the aggravated end user. Like NFS, CIFS and IPX/SPX suffer from issues of "chattiness" that negatively impact performance over the WAN.

Workarounds and Attempted Solutions

Various solutions and workarounds have been proposed to the WAN file-sharing problem, including replicating file copies and implementing distributed file systems, but neither approach has provided a complete solution. Enterprise content delivery networks (eCDNs) tried to mitigate this problem by caching copies of files at each remote office. But eCDNs, like web caching infrastructure, only provide a read-only copy of data at the remote office. If remote office users wanted to modify the file, they either had to go across the WAN to access the original copy and incur a major performance penalty, or update the local copy and create multiple, out-of-sync versions of the same file.

Filesystems developed over the last 15 to 20 years such as AFS attempted to solve the WAN file-sharing problem using a distributed filesystem architecture which unites disparate file servers at remote offices into a single logical filesystem. The problem with these technologies is that they require substantial changes in IT architecture to work properly and also require remote-office applications to use entirely new protocols because they do not export data using industry standard protocols such as NFS or CIFS. With over 1 billion computers deployed in the world that access data using either CIFS or NFS and billions of dollars invested in current file server and NAS infrastructure, filesystem solutions are clearly untenable.

The bottom line is that for any WAFS solution to gain traction, it must be able to integrate itself with existing infrastructure rather than requiring new infrastructure to be built.

A Whole New Option

In spite of the failures of both caching technologies like eCDNs and distributed filesystems to address the central issues in WAN file sharing, these technologies do provide important components for solving the WAN file-sharing problem. New WAFS products combine distributed filesystems with caching technology to allow real-time, read-write access to shared file storage from any location, while also providing interoperability with standard file sharing protocols such as NFS and CIFS.

WAFS products enable transparent worldwide design collaboration on the same data set, without complicated replication schemes or slow network performance. WAFS products will cache files in a read-write mode at remote locations, thus speeding up data access for remote users tremendously. WAFS enables LAN semantics for file access to be extended to the entire enterprise.

WAFS systems usually consist of edge file gateway (EFG) appliances, which are placed at remote offices, and one or more central server (CS) appliances that allow storage resources to be accessed by the EFGs (see Figure 1).

Each EFG appears as a local fileserver to remote office users. Together, the EFGs and CS implement a distributed filesystem and communicate using a WAN-optimized protocol. This protocol is translated back and forth to NFS and CIFS at either end, to communicate with centralized storage and remote user applications.

3 Key Design Questions

When building a WAFS system, three key design questions that must be addressed include:

* What are the features of the optimized protocol run between the EFGs and CSes across the WAN?

* What specific optimizations have to occur in the system design for reading files?

* What is the specific architecture for writing files and moving updates back to central storage resources?

The protocol used between the remote offices and the datacenter should incorporate fileware differencing technology, data compression, streaming, and other technologies to improve performance and efficiency in moving data across the WAN. File-aware differencing is especially important because it can detect which parts of a file have changed, and only move those parts across the WAN. Furthermore, if pieces of a file have been rearranged, only offset information will be sent, rather than the data itself. These techniques result in tremendous, order-of-magnitude bandwidth reduction across the WAN and time savings in accessing files by remote users.

Read performance is governed by the ability of the EFG to cache files at the remote office, and the ability to serve cached data to users while minimizing the overhead of expensive kernel user communication and context switches, in effect enabling the cache to act just like a high-performance file server. If the WAFS system is architected correctly the remote cache should mirror the data center exactly and only a few WAN round trips are required to check credentials and availability of file updates, but read requests will be satisfied from the local cache. Thus, regardless of how many NFS/CIFS read RPCs come into the EFG, it should hardly translate into any WAN traffic.

Unlike read performance, write performance is governed by the write caching mechanism that is used in the WAFS system. The two main types of mechanisms are known as write-back and write-through.

In a write-through approach, data written to a file is sent immediately over the WAN to the datacenter, while in write-back the data is written to the EFG and then sent over the WAN. Either approach, in isolation, has certain associated tradeoffs. Write-through is very safe, because all file updates are stored in the datacenter, but it suffers from poor performance and does not survive WAN disruptions. Write-back is very fast, but is riskier if the EFG fails before updates are sent to the datacenter.

The optimal combination involves using a write-back approach for maximum performance, coupled with synchronous logging of file updates to persistent storage, ensuring no data loss in case of filesystem crashes or WAN outages. Write-back caching is typically very difficult to implement correctly with logging, but has superior performance and reliability characteristics.

Figure 2 depicts representative performance for both read and write operations over a WAFS system and over a standard WAN. Opening a 5-Mbyte file over the WAN takes about 122 seconds, while a high-performing WAFS system will fetch the file in 11 seconds the first time it is accessed, and at essentially LAN speed on subsequent (warm) accesses because the file is cached locally. Writing a 2-MB file over the WAN takes 81 seconds. A write-back WAFS system achieves the same result in about 4 seconds.

In all the tests shown in Figure 2, the WAN latency was 60ms (representative of actual conditions between San Francisco and Houston) and the bandwidth allotted was 1,544 Mbit/s (T1 Line). Clearly, WAFS products can enable near-LAN speed read-write access to data in a WAN environment.

[FIGURE 1 OMITTED]

Maintaining Data Coherency and Consistency

Data coherency and data consistency are important properties of WAFS implementations, because they ensure that file updates are safe (cannot be written over) and available throughout the network of edge devices--crucial features for supporting engineering collaboration.

Data coherency means that file updates (writes) from any one remote office are guaranteed never to conflict with updates from another remote office. Properly designed WAFS implementations guarantee this by maintaining a system of file leases. Leases are defined as a particular access privilege to a file from a remote office.

If a user at a remote office wants to write to a cached file, the EFG at that office must obtain a "write lease", i.e., a right to modify the document before it can do so. WAFS solutions guarantee that at any time there will be only one remote office that has the write lease on a particular file thus guaranteeing coherence. Also, when a user at another office tries to open the file, the EFG that has the write lease flushes its data first and optionally can give up the write lease if there are no active writers to the file. This mechanism ensures that writers at different offices do not collide with each other and that file updates are safe.

Data consistency implies that file updates made at one office are always available enterprise-wide, and well-architected WAFS system do this immediately after the update is made. Again, for collaboration, this is supremely important because remote designers want to be sure they are working on the most current version of any file, no matter where it was worked on last.

Scalable Implementation

Any WAFS implementation should be capable of handling large files and large numbers of files (particularly important for CAD), as well as large numbers of concurrent users. A WAFS product that cannot scale beyond a few hundred MB of files or 10 users is not of much use.

The issue of write-through and write-back architectures mentioned earlier figures into how well a WAFS implementation scales. A write-through WAFS implementation that cannot scale to enterprise levels as synchronous data transfers across the WAN quickly becomes a bottleneck as the number of files and users increase. Systems that are based on write-back architectures, that incorporate differencing and compression technologies, scale much better.

Additionally, scalable systems recognize temporary files that applications (e.g. Microsoft Word) may create during the normal course of operation and do not send these files over the WAN, instead only sending over the final revision, which accelerates performance. Changes are reflected back to the datacenter in a consistent and coherent manner.

Summary

For companies with remote offices, sharing data in real time between team members at distributed locations has been a challenge. The culprit has been the poor performance and reliability of file-sharing protocols such as NFS and CIFS when used over the WAN. To date, workarounds to the problem have only exacerbated the issues. WAFS technologies have entered the industry to "save the day" by enabling enterprises around the world to share data over the WAN with the same performance, reliability, and peace of mind that they would have over a local area network.

[FIGURE 2 OMITTED]

Figure 2: Diagram showing performance of a WAFS system

Time to Open a 5 MB File *

WAFS System:   3 seconds
Warm Cache

WAFS System:   11 seconds
Cold Cache

Standard       122 seconds
Network Share

Note: Table made from bar graph.

Figure 2: Diagram showing performance of a WAFS system

Time to Save a 2 MB File *

WAFS System:   4 seconds

Standard       81 seconds
Network Share

Note: Table made from bar graph.

www.tacitnetworks.com

Noah Breslow is vice president of marketing at Tacit Networks (South Plainfield, NJ)