Optimizing serial attached SCSI with PCI Express.

Data centers have relied on the parallel SCSI bus interface as the workhorse of disk drive interconnections for almost 25 years. SCSI, as well as other parallel bus technologies, remains in widespread use and continues to meet the performance requirements of many of today's enterprise systems. However, significant technical challenges such as increasing the clock skew and crosstalk have made it economically impractical to retrofit these parallel bus technologies to meet the rising demand for more robust data integrity, higher system performance, greater storage flexibility and scalability, and smaller system designs. Serial Technologies Deliver New Capabilities for Next-Generation Server Storage Serial architectures, specifically Serial Attached SCSI (SAS), Serial ATA (SATA) and PCI Express, have emerged to deliver higher performance by allowing more bandwidth per device pathway than their parallel counterparts.

Serial architecture connections consist of a single pair of transmission signals that contain an embedded clock for self-clocking, enabling the clock speed to be easily scaled and to wrap data bits into individual packets to achieve data transfers up to 30 times faster than what parallel technology can provide.

[FIGURE 1 OMITTED]

Serial bus architectures also support a network of dedicated point-to-point device connections--versus the multi-drop architectures of parallel buses--to deliver full bandwidth to each device, eliminate the need for bus arbitration, reduce latency and greatly simplify hot-plug and hot-swap system implementations. This dedicated serial connection also eliminates the single point of failure found in today's parallel environments.

Serial ATA extends the parallel ATA technology roadmap by delivering disk interconnect speeds starting at 1.5 Gb/sec (150 MB/sec) with second generation speeds of 3.0 Gb/sec (300 MB/sec). Due to its lower cost-per-gigabyte, SATA will continue as the prevalent disk interface technology in desktop PCs and in sub-entry-level servers. In addition, SATA disk drives will continue to be deployed in networked storage systems where cost and capacity are more critical than performance and reliability, such as nearline or virtual tape applications.

Serial Attached SCSI, the successor to the parallel SCSI interface, leverages SCSI functionality and features while expanding SCSI's proven performance, scalability and reliability for enterprise storage. SAS offers many features not found in today's mainstream storage solutions such as drive addressability for up to 16,256 devices and reliable point-to-point serial connections (at first generation speeds) of up to 3.0 Gb/sec. SAS improves drive addressability and connectivity by using an expander to connect one or more SAS host controllers to up to 128 ports, which may include other host connections, other SAS expanders for even greater scalability, or hard disk drives. This highly scalable SAS connection scheme enables enterprise-level topologies to easily support multi-node clustering for automatic failover availability or load balancing.

The full-duplex, point-to-point nature of SAS enables simultaneous active connections among multiple initiators and high-performance SAS targets. Since SAS devices can transfer data in both directions at once, they effectively double the useable bandwidth of the link rate. As a result, these wide SAS ports support multiple links that enable the aggregation of multiple SAS or SATA targets to increase total available bandwidth (see Figure 1).

In addition to its full-duplex, point-to-point model, SAS' small connector supports full dual-port connections on 2.5-inch hard disk drives. This capability was only previously found on larger 3.5-inch Fibre Channel disk drives. These full, dual-port connections have become essential for applications that require redundant drive spindles in a dense server form factor such as those required in blade server configurations.

[FIGURE 2 OMITTED]

The SAS interface is also compatible with lower cost-per-gigabit SATA drives, giving system storage designers/builders the flexibility to integrate either SAS or SATA devices. This capability substantially reduces procurement, inventory and other costs associated with supporting two interfaces (see Figure 2).

PCI Express (PCIe), a new serial host bus architecture, is designed to address a wide range of current and future system interconnect requirements by delivering the flexibility, scalability and performance bandwidth needed to support upcoming technologies like 10 Gigabit Ethernet (10GbE) and SAS. As a point-to-point architecture with hot-plug and hot-swap support, PCIe is software-compatible with PCI and PCI-X to simplify next-generation serial computing system designs.

PCIe uses a dual simplex serial data stream with an embedded clock to overcome many of the performance limitations of parallel bus architectures. The PCIe link consists of two low-voltage, differentially driven pairs of signals (a transmit pair and a receive pair) and similar to SAS, uses an embedded data clock with the 8b/10b encoding scheme to produce very high data rates while maintaining signal integrity. Each point-to-point interconnection can have 1, 2, 4, 8, 12, 16, or 32 dual simplex 2.5 Gb/sec lanes (or an effective rate of 2.0 Gb/sec in each direction), that provide scalable bandwidth up to 128 Gb/sec (12.8 GB/sec) between nodes, compared to 1.064 GB/sec of bandwidth for a typical 64-bit, 133 MHz PCI-X 1.0 device.

Scaling SAS Bandwidth

The first generation SAS link rate is 3.0 Gb/sec (300 MB/sec) and supports full-duplex data transfers for up to 600 MB/sec bandwidth. And because SCSI protocols are not restricted to half-duplex operation, SAS will also support full-duplex transfers, allowing data to be transferred simultaneously in both directions to maximize bandwidth. For example, a device can simultaneously transfer data from a previously queued read operation while receiving data for a write operation. Although full duplex will not be used during all transfers, it can double the useable bandwidth of the link rate. In contrast, Ultra320 SCSI's shared bus architecture is restricted to 320 MB/sec for all attached devices.

SAS' full-duplex, point-to-point architecture also supports simultaneous active connections among multiple initiators and high-performance SAS targets. Under this scenario, SAS devices can transfer data in both directions simultaneously, with mixed protocol control, to effectively double the useable bandwidth of the link rate to 6.0 Gb/sec. These multiple links, in turn, can be combined into wide ports that enable system designers to aggregate the performance of SAS initiators and expanders, increasing total available bandwidth. By grouping four or eight links together (which is a typical requirement), SAS can produce bandwidth of 24 Gb/sec or 48 Gb/sec, respectively.

Currently, a 15,000-RPM disk drive can achieve maximum throughput of around 100 MB/sec and can sustain data rates of around 75 MB/sec. At these sustained data rates, two disk drives will saturate a SATA 1.5 GB/sec bus. The shared Ultra320 SCSI bus supports a total of 320 MB/sec or the sustained data rates of four to five disk drives. By contrast, a 4-wide SAS port can support as many as 16 hard drives before becoming saturated. (See Figure 3)

SAS uses expander hardware as a switch to simplify configuration of large external storage systems that can be easily scaled with minimal latency while preserving bandwidth for increased workloads. This expander hardware enables highly flexible storage topologies of up to 16,256 mixed SAS and SATA drives.

[FIGURE 3 OMITTED]

One type of expander, a fan out for example, can connect up to 128 devices per each PHY, including initiators, SAS and SATA drives and edge expanders with either narrow or wide links. These additional initiators and edge expanders can, in turn, be linked to other hosts and drives, providing additional connection nodes. The SCSI Management Protocol (SMP) within SAS manages the point-to-point connections in the topology.

SAS' ability to aggregate bandwidth through the use of wide ports will support the performance scalability required by next-generation servers and storage systems. However, while SAS can supply the bandwidth for next-generation storage I/O, it requires a proficient host interconnect to optimize total system performance.

Maximizing Server Storage Performance

System architects typically optimize performance by eliminating bandwidth bottlenecks--a goal typically met by matching interleaving technologies with complementary efficiency and availability levels.

Like SAS, PCI Express delivers scalable performance by combining multiple data links to create wide data paths. This common capability enables maximum system performance between SAS and PCI Express.

[FIGURE 4 OMITTED]

The key to maximizing storage system throughput with PCIe and SAS, is to match bandwidth and reduce any I/O bottlenecks on the PCIe link and the SAS link, as well as in internal buffers and the management of internal bridges. Since PCIe and SAS links support different bit rates, internal buffers are required for temporary storage of maximum frame data payloads to help manage the difference. Interface flow control is also needed to ensure proper buffer sizes.

By combining SAS with PCI Express, a system designer can easily supply sufficient bandwidth for eight hard disk drives or more, with neither technology bottlenecking the performance of the other. With today's SAS drives typically generating a maximum sustained data rate of about 100 MB/sec, eight drives will require about 800 MB/sec bandwidth. A SAS port configured as 4-wide supports 1200 MB/sec while a PCI Express slot configured for 4-wide supports 1024 MB/sec, easily enough bandwidth to supply full access to each drive without any restriction.

Conclusions

As data centers are called on to serve more clients, IT managers must choose technologies that optimize the capabilities of their server and storage components. Deploying complementary components within the system is a vital step in that direction.

New serial technologies are emerging to overcome the bandwidth limitations of today's parallel architectures and deliver highly scalable performance for next-generation systems. SAS delivers a 3.0 Gb/sec per link data transfer rate, the unprecedented flexibility to deploy either low cost-per-gigabit SATA drives or high-performance SAS drives in the same system, point-to-point connections for high reliability, and highly scalable connectivity to more than 16,000 devices in a single domain. PCI Express offers similar benefits, with each PCI Express lane supporting 2.5 Gb/sec performance with scalability up to a 32-wide lane configuration.

PCI Express is ideally suited for maximizing SAS bandwidth because both technologies are highly scalable and therefore able to support performance improvements well into the future. In addition, by combining SAS wide ports with PCI Express wide lanes, a system designer can maximize total storage system performance.

Paul Griffith is a strategic marketing manager at Broadcom Corporation (Irvine, CA) and currently serves as secretary on the SCSI Trade Association (STA) Board of Directors.

www.broadcom.com

www.scsita.org