The challenges of testing SATA and SAS: part 3: testing wide SAS.

One of the first obstacles to testing SAS devices is getting an analyzer connected into the point-to-point bus. The similarities between SAS and FC have made it possible to borrow from Fibre Channel techniques, but making sure the analyzer can adapt may still require special cables.

SAS specifies four different types of cables and connectors. There are single-link, dual-link and four-link cables designed for use inside an enclosure. There is also a four-link connector specified for external use, such as connecting a server to an external RAID box. Since SAS analyzers today use the single-link connector, connecting an analyzer to a dual-link or four-link bus requires special adapter cables. Fortunately, these are the same cables used in standard SAS storage applications so they are starting to become available from cable manufacturers.

[FIGURE 1 OMITTED]

The single-link connector (Fig. 1) is the same as the SATA connector and is manufactured by many different companies. The dual-link connector (Fig. 2) is new and physically similar to the SATA/SAS single-link connector, but adds additional pins on the unused opposite side, providing for a redundant set of connections between the drive and host in case the primary physical connection goes bad.

SCSI marketing departments have always maintained that a significant differentiator between ATA and SCSI drives is reliability, and the redundancy of the new dual-link connector for SAS is further proof of this difference going forward. The internal four-link connector (Fig. 3) is just like a very-wide SATA connector. RAID applications inside the box will use the internal four-link connector. The external four-link connector is based on the same MicroGiga connector used by 4x Infiniband, but uses jackscrews instead of the quick-release latches that Infiniband cables use. External four-link SAS applications include expanders and RAID boxes.

Testing "What If" Scenarios

SAS provides an unusual level of flexibility when it comes to some of its handshaking using primitives. Table 1 shows some areas where SAS state machines are more flexible than their FC equivalents.

The flexibility of SAS primitives makes it difficult to verify that a SAS device is robust enough to support the primitives, no matter when they occur. Typically, those designing SAS silicon will make or buy the simulation code they need to verify the RTL (Register Transfer Level) code, ensuring that the different scenarios have been verified.

[FIGURE 2 OMITTED]

One company that develops and sells SAS protocol verification tools is Perfectus (Santa Clara, CA). Their verification engine generates the packets and primitives in software to test all the possible timing and sequence possibilities. The Perfectus SAS verification tests are also translated to the Data Transit PacketMaker hardware for verification of the final device--not just a simulation. Figure 4 shows the sequence for verifying RTL prior to finalizing the hardware, and later verifying the final hardware with the same tests.

[FIGURE 3 OMITTED]

Perfectus tests are categorized according to the different protocol layers of PHY, Link, Transport, and Command. They give the developer the ability to adjust a variety of test 'knobs' which vary the timings and occurrences of SAS Errors, OOB, Primitives, and Frames within legal specifications, or beyond specifications if the user chooses. The test case generation is coupled with a protocol checker, which analyzes a device's response to the generated traffic to determine if the device followed protocol. The protocol checker is implemented in software and verifies both the simulated results of the pre-silicon design, and the actual results of the final hardware as received by the PacketMaker.

[FIGURE 4 OMITTED]

Table 2 shows the tests performed in each protocol category by the combination of the Perfectus verification engine and the Data Transit PacketMaker.

[FIGURE 5 OMITTED]

Obstacles to Analyzing Wide SAS

Wide SAS uses multiple links in order to handle multiple commands simultaneously. In this multi-link environment, each link carries the data for a different command. The links cannot be aggregated to achieve higher bandwidth for a single command, but multi-lane performance gains can be achieved at the application layer when a host adapter breaks a data request into multiple commands, using a different link for each command. Thus, the data can be sent down multiple lanes simultaneously even though commands can only be issued on one lane at a time.

Certain features of wide SAS make it impossible to analyze each link as a distinct SAS bus using a single-link SAS analyzer. On a wide SAS bus, a command will be issued on whichever link is available, rather than waiting for a specific link, which would add more latency.

One of the difficulties of analyzing commands on a wide SAS bus is that the connection on one link can be closed before the data transfer for that command is complete, such as when the drive empties its buffer and closes the connection in order to free-up the link while it refills its buffer.

In this example, the original link may be busy when the drive is ready to continue, so a different link is opened in order to continue the transfer. Thus, the command was executed using different links at different times, preventing a single-link analyzer from capturing all the data associated with a command.

Special analyzers, such as the Data Transit 4x-SAS Analyzer (Fig. 5), can simultaneously capture and correlate all the links, putting commands back together even if they used multiple links before completing. The following trace display (Fig. 6) shows how the analyzer correlates the data from multiple links into a single display. The display allows the user to display the traffic in a time-ordered, command-ordered, or connection-ordered format.

DVDs, CD-ROMs on SAS/SATA Bus

Hard drives supporting SATA are shipping in volume, but CD and DVD drives are just now in development. The new CD/DVD drives for SATA will continue to use the ATAPI protocol as they did on parallel ATA, even though ATAPI adds a significant amount of protocol complexity.

The ATAPI protocol is necessary for compatibility with existing host drivers, just as it was necessary when the first SCSI CD-ROM drives made the transition to the low-cost parallel ATA bus. The history of CD and DVD interfaces is interesting because it demonstrates how much protocol baggage current devices must carry in the name of driver compatibility.

The original CD-ROM drives were SCSI devices, and using them on a PC required the purchase of a SCSI host adapter. Later, in order to eliminate the extra cost of the host adapter, the ATAPI specification was added to the ATA standard as a method for transmitting SCSI commands across ATA hardware by encapsulating or packetizing much of the SCSI protocol into standard ATA data transfers.

[FIGURE 6 OMITTED]

CD-ROM drives were then able to use their tried-and-true SCSI command set on the pervasive PC architecture while eliminating the cost of a SCSI host adapter. The encapsulation adds overhead, but for optical drives in the PC market, price is much more important than performance.

[FIGURE 7 OMITTED]

Figure 7 shows the interface migration of low-cost CD-ROM drives from SCSI to ATA to SATA. Today, even though SATA is already a packetized interface and it would be a much simpler protocol to discontinue the encapsulation of SCSI commands into ATA data transfers, the ATAPI encapsulation will still be used so that the OS and its drivers won't see a difference between a Parallel ATA DVD drive and a SATA DVD drive.

Protocol Analyzers used with the new generation of SATA optical drives must support the ATAPI protocol with special decoding and triggering functions. In the future, CD/DVD drives designed for performance instead of low cost will become available with a SAS interface instead of ATAPI over SATA. Figure 8 shows a state listing of an ATAPI protocol SCSI command encapsulated into a SATA data transfer.

Conclusion

SAS presents a number of special challenges and obstacles to engineers working with this new storage technology: Cabling and the multi-lane data paths characteristic of SAS being just two of them. Overcoming these challenges takes knowledge and the right test tools.

[FIGURE 8 OMITTED]

Table 1

                 Feature                           SAS

Primitives start with pos or neg disparity  YES

Allows deletable primitives in frames       YES-ALIGNs

Allows credit primitives in frames          YES-R--RDYs

Handshake primitives in frames              YES-ACK, NAK

Zero primitives between frames              SOF can follow EOF

                 Feature                             Fibre Channel

Primitives start with pos or neg disparity  NO-Must start with neg
                                            disparity

Allows deletable primitives in frames       IDLE not allowed in Frames

Allows credit primitives in frames          R--RDY not allowed in Frames

Handshake primitives in frames              ACK Frame instead of
                                            primitive

Zero primitives between frames              Must transmit 6 IDLEs
                                            between

Table 2

      PHY               LINK           TRANSPORT

OOB Signals       Primitives        FIS construction
SASRESET,                           and decomposition
SASINIT and
SASWAKE

Host Phy          Disparity         All FIS types
Initialization                      transmission and
State Machine                       reception checking

Device Phy        Data and CRC      Host and Device
Initialization                      Transport State
State Machine                       Machines

Power Management  Link Idle,        Transport Layer
State Machines    Transmit,         Error Handling
(Host/Device)     Recieve and
                  Power Management
                  State Machines

Phy Layer Error   Link Layer Error
Handling Testing  Handling Testing

      PHY             COMMAND

OOB Signals       Power-on,
SASRESET,         Deviceidle and
SASINIT and       Reset protocol
SASWAKE

Host Phy          PIO & DMA data-in
Initialization    and data-out
State Machine     protocol

Device Phy        Read and Write
Initialization    DMA-queued
State Machine

Power Management  Command Layer
State Machines    Error Handling
(Host/Device)

Phy Layer Error
Handling Testing

Parts 1 and 2 of this article appeared in the January and February issues of CTR. To obtain copies of CTR, please contact the circulation dept.: [e-mail]

Dale Smith is CTO and founder of Data Transit Corporation (San Jose, CA)

www.datatransit.com