Switch-fabric requirements for next generation storage directors. (Storage Networking).

Currently, storage equipment vendors are faced with a problem: products of ever-increasing complexity requiring a long and expensive product development cycle. Typical figures are two to three years of development requiring massive teams of R&D and incurring tens to hundreds of millions of U.S. dollars of R&D investment. The end products, however, suffer from a short life span, typically three years and sometimes less. This results in a short period of sale where revenues can be accumulated in order to justify the massive investment in R&D. This forces a cycle of equipment development, introduction to market, and retirement that leaves very little room for profits.

A look at voice networks, a more mature and profitable section of the communications industry, presents a different picture. Voice-products life cycles are seven to twelve years, and sometimes more. The products are a-priory built to scale, which enables the networks to evolve gradually, meeting the requirements to higher port count, higher port rate and more comprehensive services.

Reasoning Behind the Problem

The reason for the problem is that current director architectures are inherently constrained in their growth potential, just like current data-centric architectures. Specifically, they embody "hard" limitations on the number of ports, on the ports rates, and the services that they provide.

If we look deeper at the products' architectures we can discern that often the limiting factor is the switch fabric that lies at the heart of storage directors. Stripping away all the upper level protocol processing, the switch fabric is responsible for the "raw" data switching from the input port to the output port.

The architectural paradigm of most switch fabrics that exist or are being designed today is such that they inherently limit the scope of the products' future growth. As an example, most of the current switch design can accommodate only the "port-rate of the day" and the "port-count of the day." A system design that starts today around an "aggressive" 320 FC 1/2G fabric, will hit the market in early 2007. Even a moderate five-year life cycle would require that system to be attractive in 2012. Will the number of ports be sufficient in 2012? Will the port rate? It is likely, looking at the storage market predicted growth, that neither the number of ports nor the rate indicated today, will be sufficient in 2012.

As a second example, consider the service scheme offered by the fabric. Any design that fulfills the current director service scheme, which is hardwired within the switch-fabric and does not reside only on the interfaces line cards, will force the end-customer to replace this centralized element (e.g., switching card) upon the introduction of line cards that require new scheduling disciplines and inter-mixing of other types of line cards. Even a farsighted equipment vendor, when designing a product to be released in two to three years, can hardly anticipate the environment its product will need in order to operate in six to ten years.

To grasp the implausibility of such foresight, we can examine the evolution of directors. A few years ago, a storage network built with a fabric supporting up to 32 ports of 1Gbps FC ports was considered advanced. Such a fabric supported a single, or at best two levels, service scheme. Today, within less then seven years, the latest directors offer up to 320 of 1Gbps and 2Gbps FC ports and are offering at least a 4level class-based service scheme. There is also talk of a 10Gbps FC ports and convergence with data networks in the future. This evolution spans at least three generations of equipment architecture at the leading equipment vendors.

A new switch-fabric architecture is needed, one that is built to scale and, hence, to last.

Requirements for Switch-Fabric Solution

In order to last, the switch fabric must have the following properties:

Port Count Scalability: Since a switch-fabric solution is the "heart" of directors, it needs to scale in the number of ports it supports. The architecture must enable the director switching systems to start with few ports, grow to tens, hundreds and thousands.

Without port count scalability, the only way to increase the port count without replacing the equipment is to cascade directors with proprietary Inter Switch Links (ISLs). The problems with simply using ISLs are numerous. Networks constructed out of several directors interconnected with ISLs always exhibit blocking behavior that is traffic pattern dependent. Also, cascading directors increase the delay through the network, since more hops may be required for a data packet traversing the network. For storage networks, latency has a big impact on performance, just as bad as blocking. Last, this involves separate devices where each of them has to be maintained, operated and managed separately.

The end user should be able to start with any desired port count and progressively increase the director capacity (as they acquire more customers). Directors should be able to grow beyond a single physical chassis without any impact to performance. In addition, the extension of the system port count should entail merely the insertion of additional line cards and fabric cards without disrupting the service of existing ports (live upgrade).

A system vendor may take advantage of the same chip-set and design a whole scalable product line, from a low port-count "pizza box" to a medium port-count chassis-based director that can scale to a high port-count multi-chassis system.

Port Rate Scalability: A switch fabric solution needs to scale in the port rates it supports. The switch fabric must embody a scheme for seamlessly connecting ports of increasing rates; from 1G FC ports, 2G to 10G and so on. Supporting ports of larger rates should require only the development of new line cards, leaving the rest of the system intact. Furthermore, the fabric should be able to accommodate any diverse mix of port rates. This required port rate scalability should easily enable migration from old and current to next generation directors.

Service Awareness: Service awareness in data-centric systems has the following aspects: the service model, distributed scheduling, and the division of labor between the switch-fabric and the traffic manager. Current switch fabrics supporting storage systems typically have a rough service model. Current systems typically support two to four rough priorities. Next generation storage applications require that storage directors support more sophisticated service schemes. For example, guaranteed bandwidth and rate-based scheduling are a must to enable mission critical applications to coexist with other storage network traffic in a highly utilized network. The convergence of data networks and storage networks is another driver for differentiated treatment of traffic. More generally, one should expect that the requirements for the directors' service scheme will evolve. In order to be able to support future service requirements, the switch fabric solution must first support a more elaborate and finer-grained service model as well as enable service scheme scalability. Service scheme scalability dictates that scheduling, the foundation of service creation, be distributed.

In current switch fabrics, scheduling decisions are often centralized or hardwired into the switching devices. As a result, increasing the number of ports and the service disciplines with this approach requires changing the switching/arbitration devices that comprise the switch fabric. This amounts to a major product re-design.

A scalable switch-fabric solution must distribute its scheduling decisions among the egress ports. Each port independently summons traffic across the fabric to meet its egress-bound bandwidth and service requirements. As a result, the introduction of a new service scheme involves only the replacement/addition of a new type of line card.

As an example, consider a switch fabric offering four priorities per fabric-port that are served in a strict/weighted priority model. This switch fabric typically incorporates hardware hooks in its switching/arbitrating devices; e.g., its switching devices may incorporate four queues per output port and some mechanism to control it. Such a fabric cannot support protocols and service schemes that require min-max rate guarantees or tighter jitter/delay requirements. The imposition of these new requirements will require a replacement of the entire system. On the other hand, a truly scalable fabric is expected to accommodate such a change with only the addition of the new type of line card. This includes a new "fabric access device" that enables such services while cohabiting with the rest of the line cards.

Data-centric networks experience poor ROI due to the products' short life cycle. This same short life cycle is anticipated in younger new generation storage systems. In order to develop a product with a much longer life cycle that can support the high bandwidth requirements of this growing market, storage platform vendors need a new switch fabric architecture paradigm. This switch fabric architecture should be highly integrated to enable the system to grow over the years at the customer premises by the virtue of its number of ports, port rate, and service scheme.

Ofer Iny is the CTO of Dune Networks (Agoura Hills, Calif.)

www.dunenetworks.com