Increasing network performance using Molecular Sequence Reduction technology, Part 2.

The first part of this article, which appeared in the February issue of CTR, described how Wide Area Network (WAN) links represent the most common performance bottleneck within data communications networks. We also demonstrated how much of this limited WAN capacity is wasted by frequent repetitions ranging in size from a few bytes to megabytes. In part two, we examine a new technology to remove these repetitions, Molecular Sequence Reduction (MSR).

The throughput, latency, and reduction limitations of traditional compression have been solved by a new technology, Molecular Sequence Reduction (MSR). Molecular Sequence Reduction operates by finding variable sized repeating patterns anywhere in the data stream, across multiple packets, applications, or sessions. MSR adaptively learns the patterns that occur most frequently in the data and assigns to each pattern a unique label. MSR then uses this table of patterns to encode the data stream in a highly efficient format by replacing each instance of the pattern with the corresponding label (See Figure). The numerous redundancies that waste network resources are thus stripped from the data stream before it is transmitted across the network's most capacity-constrained links. A second MSR engine receives the reduced and highly optimized data stream and converts it back into its original form by replacing each label with the original data pattern. By performing this data reduction at the IP layer, MSR remains completely transparent to the network, applications, servers, and clients.

MSR's pattern discovery algorithm is one of the key differentiators of this technology. The speed and granularity with which the pattern discovery algorithm searches for new patterns is critical to the high data reduction rates provided by MSR and the high network speeds at which this technology can operate. The MSR pattern discovery algorithm is able to search the data stream destined for the WAN link in real-time and with very low latency. MSR can scale this search buffer size arbitrarily without impacting the performance and throughput of the algorithm. Thus, at high data rates MSR can increase the search buffer size to find patterns that are more widely separated while maintaining the same high processing through put. MSR's ability to search this large buffer at high speeds is therefore a key strength of the algorithm and the source of the consistently high data reduction rates that it achieves.

MSR operates directly on the data path by encoding individual packets based on its pattern discovery algorithm. This process is carried out on each packet independently of all other packets, without imposing any dependencies between processed packets. This allows MSR's data reduction process to operate at high speeds and to minimize processing latency. Thus packets that are encoded by an MSR device can be transmitted to the decoding device in a connectionless fashion without imposing any ordering or reliability constraints on the encoded data stream.

While MSR is obviously very powerful, it operates transparently to other devices in the network, and does not require any changes to the network or existing applications. It also does not require any tuning. In other words, MSR is a core networking technology that operates transparently like all other network devices (e.g. switches, routers) on all network traffic.

Adding rapacity Without Network Buildout 

Enterprise wide area links are typically the most expensive and most capacity-constrained component of the networking infrastructure. Enterprise networks are littered with repetitive data generated by applications, business processes, and commonly used strings or objects. Removing these repetitions can eliminate more than half of the data currently running on these networks. The traditional solution of compression is not effective nor widely used in most high speed enterprise networks. Molecular Sequence Reduction technology can identify repeating patterns in real time on high speed networks resulting in data reduction rates of 60-90 percent. Thus MSR can deliver effective throughput gains of 3 to 10 times the original link capacity.

The health and fitness of a network may not be apparent at first glance or measurable via traditional performance metrics. Looking forward, what network managers need is a new metric to measure the undiscovered additional capacity they already have in their networks that is currently being wasted by repetitive data. That, however, is a topic for another discussion.

www.peribit.com

Amit P. Singh is co-founder and CTO of Peribit Networks (Santa Clara, CA).