Expanders: an indispensable component of the SAS architecture.

As the SCSI world began to embrace the successful market migration to Low-Voltage Differential (LVD) SCSI signaling, SCSI expanders emerged as a new class of SCSI devices. The market decided that the robust, yet costly High-Voltage Differential (HVD) or the limited distance, low reliability connections offered by Single Ended (SE) signaling were simply inadequate to serve the SCSI market moving forward. SCSI expanders were introduced as a way to bridge these various signaling domains.

Originally, these devices were created to ease the migration from the legacy SCSI installations to the significantly improved LVD physical connection schemes. These expander devices allowed for logical SCSI domains to be comprised of two or more physical SCSI segments, each with its own signaling capability (HVD, SE or LVD).

These early expanders were often called converters, because they converted the downstream SCSI signaling scheme and were simply modeled in the system as a slight cable delay. It was quickly realized that while SCSI expanders did a great job of converting from one signaling domain to another, they also had the impact of greatly improving SCSI signal integrity by repeating the signal and reducing the impact of the variable bus loading, inherent with the distributed transmission line nature of parallel SCSI. Even in situations where signal conversions were not required, expanders became an excellent way of reducing the interaction between internal and external SCSI buses.

As RAID and clustering became prevalent in the mainstream server market, expanders steadily increased in system usage. Their ability to effectively extend cabling distances, increase the number of devices attached to each parallel SCSI bus and to isolate failed segments of clustered systems (which were often used to provide failover routing in redundant storage configurations), made expanders an essential component in a large number of installations (see Figure 1).

SCSI Goes Serial

Serial Attached SCSI (SAS) is a natural evolution of SCSI beyond Ultra320 SCSI. It is inherently multi-initiator, supports full-duplex transfers, and features dual-ported capabilities--functionality that is not found in desktop-class drives. SAS is designed exclusively as a point-to-point serial interface and provides improved addressing to remove the imposed physical addressing limits of parallel SCSI. Moving to this new connection scheme also reduces the size of the physical connection, making it more appropriate for smaller form factor drives. SAS provides all of these new capabilities, while preserving compatibility with over 20 years of legacy SCSI software.

One of the most significant attributes of Serial Attached SCSI is that it allows SCSI enterprise class drives to coexist with lower cost/gigabyte Serial ATA drives. This important capability of SAS provides an unprecedented level of customer choice and will impact how systems are provisioned to serve different segments of the storage market.

To support this co-existence of Serial ATA and SAS drives, SAS defines three transport protocols:

* Serial SCSI Protocol (SSP): A mapping of SCSI supporting multiple initiators and targets

* Serial ATA Tunneled Protocol (STP): The method by which a SAS host makes a connection to a Serial ATA drive, when using a SAS expander

* Serial Management Protocol (SMP): A management protocol

SAS Expanders

Many of the capabilities of Serial Attached SCSI described above are enabled or enhanced by a new generation of SCSI expanders, called SAS expanders. These SAS expanders are an indispensable component of the SAS architecture. While the point-to-point topology native to the SAS architecture changes some aspects of the traditional SCSI expander, they perform similarly in term of providing the capability to incrementally expand upon the base architecture, while preserving the cost and software advantages of the traditional direct attached storage model.

SAS ports that are physically a part of a SAS host controller may directly address either SAS or SATA drives. The number of drives addressed would be limited by the number of physical ports integrated into the host SAS controller itself, if there were no mechanism of readily expanding the architecture. Integrating a large number of SAS ports into one device could be costly for systems not requiring all the ports in the base controller, yet not providing enough SAS ports would dramatically limit the utility of certain systems. Serial Attached SCSI deploys SAS expanders as the method used for extending the device addressing to the complete range (16K) of devices specified in the SAS standard. It also uses expanders to incrementally expand in-box and near-box storage capabilities in systems requiring greater bandwidth connections, as additional expanders provide redundancy and also provide for addressing large numbers of devices.

[FIGURE 1 OMITTED]

Expanders, in general, offer the ability for a host port to establish an operating connection with the desired device, be it a SAS device, a SATA device or another expander. In order to accomplish these connections, all expanders support an addressing mechanism for routing requests, a means for managing the connections between devices or other expanders and the ability to broadcast primitives across all of the expanded connections it supports.

Without going into too much technical detail on how expanders are implemented, an expander device contains the following components:

* An expander connection manager (ECM)

* An expander connection router (ECR)

* A broadcast primitive processor (BPP)

* Two or more physical expander links

* An expander port available per physical connection

* An SMP target port

An expander device may also choose to contain SAS devices with SSP ports, STP ports, and/or SMP ports.

Edge Expanders Devices

SAS expander, much like the traditional SCSI expanders, offer a greater degree of flexibility and provide an important mechanism for incrementally scaling storage subsystems. Inherent in the notion of SAS expanders is the concept of serving multiple market segments, without unduly taxing less demanding segments of the enterprise storage market. The SAS standard accomplishes this by defining two different classes of expanders; edge expander devices and fanout expander devices.

Edge expanders are implemented without using internal memory to buffer SCSI payloads. In addition, these cost-sensitive edge expanders limit the number of devices that can be addressed, which serves to constrain the size of the routing tables, thereby reducing complexity of these devices. Edge expanders are a low-cost way of expanding SAS architecture, and are used in much the same way as the traditional SCSI expanders (see Figure 2).

[FIGURE 2 OMITTED]

Edge Expander Device Sets

Edge expanders may be combined to form an "edge expander device set." An edge expander device set may be attached to no more than one other edge expander device set or one fanout expander device (described below). Each edge expander device set may address up to 128 devices.

In order to manage all the connections between initiators and targets, SAS devices and SATA device, and to avoid deadlock and live-lock conditions within the topology, the SAS protocol requires that each expander have an expander connection manager (ECM). The expander connection manager maps a destination SAS address in a connection request to a destination device using any of the acceptable addressing methods, it arbitrates and assigns (or denies) path resources for connection requests, and it is used to configure the expander connection router (ECR).

[FIGURE 3 OMITTED]

Fanout Expanders

An edge expander device set may only communicate with one other edge expander device set. This sets the upper limit of SAS scalability, without adding additional SAS ports in the host controller or by deploying fanout expander devices. Fanout expanders allow multiple edge expanders or edge expander device sets to communicate with each other. These devices have more extensive routing tables and may provide facilities to support topology discovery and self-configuration. Additional expander ports and increased routing complexity make these fanout devices more complex, but may be necessary in systems designed to connect large numbers of storage devices. Even without fanout expanders, SAS topologies can support up to 256 device connections, far beyond the 15 physical devices supported with parallel SCSI (see Figure 3).

SAS Expander Summary

As with traditional SCSI expanders, SAS expanders are essential components for scaling mainstream enterprise storage. These expanders are valuable in extending SAS cabling distances, isolating internal and external storage domains, and providing fail-over isolation in fault tolerant environments. By offering two classes of SAS Expanders, system designers can deliver the capabilities demanded by their targeted markets without unduly taxing less demanding systems. Like traditional SCSI expanders, SAS expanders will prove themselves to be indispensable in delivering mainstream enterprise storage solutions.

Harry Mason is president of the SCSI Trade Association, and director of Industry Marketing at LSI Logic Corporation (Milpitas, CA)

www.scsita.org

www.lsilogic.com