The challenges of testing SATA and SAS; Part 2: primitives and frames.

Serial ATA (SATA) and Serial Attached SCSI (SAS) are quickly replacing their parallel counterparts. SATA is already the undisputed heir to the ATA throne, and SAS is poised to replace SCSI by offering an economical roadmap to higher bandwidth, while providing backward compatibility with SATA.

[FIGURE 1 OMITTED]

Most SAS hosts can connect to SATA drives directly using the native SATA protocol, sacrificing the advanced features of SAS and limiting the connection to a single drive. SAS hosts can also connect to SATA drives through a SAS Expander using the SATA Tunneled Protocol, which encapsulates SATA frames within the SAS addressing frames. This allows a SAS host to connect to multiple SAS and SATA drives, combining the multi-drive capabilities of SAS with the low cost of SATA.

Analyzing the traffic on a mixed SAS/SATA bus is best done with a SAS analyzer that supports both protocols, although for the native SATA connections a SATA analyzer will work fine. Figure 1 shows where in a mixed SAS/SATA system a SAS analyzer is required, and where a SATA analyzer will suffice.

SATA and SAS: By the Numbers

With all of its advanced features. SAS is significantly more complex than SATA. SAS devices will be more expensive but the price is easy to justify in an enterprise storage environment where increased versatility and reliability (MTBF) are critical. The Table provides a SATA/SAS comparison of the additional feature complexity that comes with SAS.

Handshaking on a Serial Bus

While individual signals were often used for handshaking and signaling special events on Parallel ATA and SCSI, on the new serial busses it is the 'primitives' that perform the handshaking and event signaling.

A 'primitive' is a defined sequence of 4 bytes, beginning with a special 10-bit character such as a K28.5 or K28.3, SATA and SAS primitives are used for a variety of signaling and handshaking functions, and perform most of the low-level, time-critical functions. Frames (also called packets) perform the high-level functions on a serial bus by carrying commands, data, status, and configuration information.

[FIGURE 2 OMITTED]

One of the biggest differences between SATA and SAS is in the handshaking between the host and drive. While a frame is being sent, SATA uses the return direction for handshaking, thus the host and drive handshaking is tightly interlocked. SAS, on the other hand, provides a loose handshake and thus frames can be transferred in both directions simultaneously. Transferring frames in two directions at once provides higher bus utilization in some circumstances, such as a command queuing environment where data can be returned while new commands are being sent.

The trace from a SATA system in Figure 2 shows how the handshaking in the return direction is interlocked with the frame being transmitted. Notice the delay between when the frame is sent on the Device>Host side, and the SATA_R_IP (SATA_Receive_InProgress) primitives on the Host>Device side.

The trace from a SAS system in Figure 3 shows frames being transferred in both directions at the same time.

Frames and Connections

SAS Frames can be categorized as either 'Outside' a connection, or 'Inside' a connection. A SAS connection defines the same Initiator_Target (I_T) connection that parallel SCSI defines. Address Frames are used to establish a connection between two addresses, similar to the arbitration and selection phases of parallel SCSI. Address Frames are outside the connection, since the connection isn't established until the Open Accept primitive is sent back. Address Frames use SOAF (Start Of Address Frame) and EOAF (End Of Address Frame) primitives to begin and end each frame, while other SAS frames use different primitives, called SOF and EOF. The inside category consists of three protocols: SMP (Serial Management Protocol), SSP (Serial SCSI Protocol), and STP (SATA Tunneled Protocol).

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

A typical power-up sequence consists of an Out-Of-Band (OOB) COMRESET and COMSAS, followed by speed negotiation, followed by Device Discovery consisting of the 'Identify' Address Frame which gives the worldwide unique address of the device and identifies the protocols it supports (SMP, SSP, STP). After the discovery, the 'Open' Address Frame can be used to open a connection. The connection is officially 'open' when the addressed destination responds by sending the Open Accept primitive back. Once a connection is opened for a specific protocol (SMP, SSP, or STP) frames can now be sent 'Inside' the connection. The DONE and CLOSE primitives are issued to close the connection once the transaction(s) are complete, or if a device needs to free up the connection while it executes the command.

A 'Frame Type' field is present at the beginning of the frame that identifies the type and format of the frame, SAS Address Frames are usually referred to by the designator 'AF' followed by the type, such as AF 0, which is an Identify, and AF 1, which is the 'Open' frame. Address Frames have a 4-bit type field. Other SAS frames have 8-bit type fields and are called Information Units, or IU frames. SATA frames also have an 8-bit type field and are referred to as a FIS, short for Frame Information Structure.

SMP includes the REQUEST frame (IU 40) and the RESPONSE frame (IU 41). These frames are used for management functions such as controlling expanders.

SSP is the SCSI protocol that carries COMMAND frames (IU 06). TASK MANAGEMENT frames (IU 16), XFER_RDY frames (IU 05), Read/Write DATA frames (IU 01), and the RESPONSE frames (IU 07) that carry SCSI Status. Figure 5 shows many of these frames in a typical command sequence.

STP is used to encapsulate SATA frames between the SAS host and a native SATA device. SATA frames include the Host-to-Device Register Transfer frame (FIS 27) used most often for sending commands, the Device-to-Host Register Transfer frame (FIS 34) used for reading register status back, the Data frame (FIS 46) used for carrying payload data, the DMA Setup frame (FIS 41), the DMA Activate frame (FIS 39), the PIO Setup frame (FIS 5F), and the rarely used Built-In-Self-Test (BIST) frame (FIS 58) and Set Device Bits frame (FIS A1). An expander is typically used to handle the routing of SATA frames to one of many possible SATA devices. Figure 6 shows many of these frames in a common SATA command sequence.

Timeouts

SAS specifies four times as many timeouts for error recovery purposes as SATA does, SAS specifies timeout periods for many of the handshaking sequences. The list below shows the variety of SAS timeout conditions, some of which cause a primitive to be sent notifying the other side of the timeout.

* SMP Frame Receive Timeout

* Initiator Response Timeout (programmable via Mode Select)

* Partial Pathway Timeout (programmable via SMP Request/Response frames)

* Open Timeout

* Break Timeout

* Close Timeout

* Identify Timeout

* COMSAS Timeout

* DONE ACK/NAK Timeout (Primitive)

* DONE Credit Timeout (Primitive)

* Reserved Timeout 0 (Primitive)

* Reserved Timeout 1 (Primitive)

Defined SATA timeouts:

* Response Timeout

* COMWAKE Timeout

* FIS 34 Sent after Init

In some cases, it is easy to tell that a timeout occurred, such as when a timeout primitive is issued. In other cases an operation will simply be aborted. Verifying that these timeouts are handled correctly can be difficult without special equipment that allows the user to take total control over the SAS traffic and force the timeout. The Data Transit SAS and SATA PacketMakers allow the timing of frame and primitive handshaking to be adjusted with Dword granularity (27ns) by inserting idle data between frames or primitives. Although the PacketMaker uses SAS or SATA state machines for handshaking, these state machines can be disabled mid-stream allowing the user to force illegal handshakes or slow handshakes onto the bus, and then reenable the handshaking to continue normal operation.

[FIGURE 5 OMITTED]

The PacketMaker stream in Figure 4 shows the process for disabling hardware handshaking and inserting user-defined handshakes. The 'Modify Behavior' packets are directives to the PacketMaker hardware to disable and enable hardware handshaking. The user can then insert primitives and Idle data as shown.

Top-Down Troubleshooting

The preferred approach to troubleshooting a deeply layered bus like SAS or SATA is to start at the higher protocol layers, then drill down when a high-level sequence didn't complete normally. Fortunately it's fairly easy for those familiar with SCSI and ATA to recognize the same high-level sequences they are used to in SAS and SATA. For SAS, the 'Open' packet is the equivalent of the Arbitration/Selection phases in SCSI. The Command frame will contain the SCSI CDB, just like the Command Phase in SCSI. SAS Data frame(s) will contain the data that SCSI transfers in the Data Phase, and the SCSI Status is contained in the SAS Response frame. Although the handshaking is much different, at the frame level SCSI and SAS carry mostly the same information. The Figure 5 trace file shows a Frame Listing of a normal command sequence, making it easy for SCSI engineers to identify the same information they have always looked for. From this view, it is easy to drill down to the further detail if Check Condition status is returned in the response, or the sequence doesn't complete normally, such as when a timeout occurs. Figure 5 shows a Frame Listing of captured SAS traffic. The description field shows the SCSI commands, status, etc.

[FIGURE 6 OMITTED]

Similarly, ATA engineers will be able to easily recognize the same Task File information in a Frame Listing that they are accustomed to. The same registers that existed in parallel ATA are now contained in the SATA frames including the Error/Features, Sector Count. CHS/LBA, Command/Status, and Device Control registers. Even the concepts of PIO transfers. DMA transfers, and 28-bit and 48-bit addressing still apply to SATA. Figure 6 shows a Frame Listing of a SATA transfer and SATA traffic. The Task File information is easily recognizable.

SATA                  SAS

18 Primitives         70 Primitives
 8 Frames             17 Frames
 3 Timeouts           12 Timeouts

Table

Part III of this article will appear in the March issue of CTR, covering Wideport SAS. ATAPI over SATA, and other advanced features.

Part I of this article appeared in the January edition of CTR. To obtain copies of CTR, please contact our circulation dept.: [e-mail]

Dale Smith is CTO and founder of Data Transit Corporation (San Jose. CA)

www.datatransit.com