Fibre Channel Building Blocks: Physical Layer Elements.

This article is the second in a two-part series. The first part, Interconnect Elements, appeared in the June issue of CTR.

In the first installment of this article, the diverse copper and optical interconnect technologies supported by Fibre Channel were detailed. In this second part, an exploration of the physical layer, or PHY layer, semiconductor components of Fibre Channel systems are described so that users understand the functionality of these building blocks, as well as key parameters impacting system performance. Five functions are discussed: Serializers, Deserializers, Port Bypass Circuits, Repeaters, and Retimers. These building blocks are used throughout the Storage Area Network (SAN) in disk drives, host adapters, switches, hubs, routers, JBODs, and RAID systems. An understanding of these functions is critical for implementation of Storage Area Networks.

Serializes

Fibre Channel SANs are comprised of a network of point-to-point duplex links at 1.0625Gbps and 2.125Gbps. Serializers and Deserializers, often packaged together as "SerDes," are located in host bus adapters, disk drives, switches, and RAID systems. Consequently, their operation is key to the performance and robusmess of SANs.

Raw data bytes from a protocol controller are encoded into 10 bits using 8b/10b encoding to provide fixed run length serial data that maintains proper DC balance to facilitate copper and fiber optic transmission, In Fibre Channel, 100Mbps data is encoded and framed to provide an overall serial data rate of 1.0625Gbps, which includes the 20 percent overhead penalty of encoding. It is

quite common for implementations to use external 10-bit Serializer/Deserializers, where the protocol controller passes 10-bit data at 106.25MHz to the Serializer, which serializes the 10-bit data onto a differential high speed output buffer at 1.0625Gbps. A key element within the Serializer is the Clock Multiplier Unit (CMU), which takes a reference clock at 106.25 or 53.125MHz and multiplies it by l0x or 20x to generate the bit rate clock used to serialize the data. The data on the parallel transmit bus is synchronous to the reference clock; however, some SerDes provide an additional clock, called the Transmit Byte Clock (TBC), which is used to synchronize the bus. This is frequency-locked to the reference clock. Using TBC allows protocol controllers generating TBC to more easily meet the setup and hold time constraints of the Serializer.

There are several factors to be aware of regarding the performance of Serializers. First, all Serializers transfer data at line rate. Differences in system I/O bandwidth are due to the protocol controller, not the Serializer. Secondly, the quality of the serial signal is of critical importance for robust system-to-system interoperability. Measurements of signal quality are Amplitude, Rise Time, Fall Time, and Jitter (Random and Deterministic). Industry standard specifications (see www.T11.org) detail the signal quality criteria required by Serializers to ensure interoperability with Deserializers. Vendors should supply proof of compliance to these signal quality standards as a requirement for adoption.

Perhaps the most difficult issue for system designers is jitter, which is the time displacement of the signal's edges from their ideal location. Traditionally, eye diagrams are available that show the signal quality at the output connectors of the device. Although jitter is a very complicated issue, it should be proven that Serializers meet the Random, Deterministic, and Total jitter specifications at the output.

Deserializers

A Deserializer receives serial data at the opposite end of the link from the Serializer. A high speed Clock and Data Recovery unit (CDR) uses sophisticated Phase Lock Loop (PLL) techniques to monitor the edges of the incoming serial data and extract a recovered clock at the bit rate. This clock is used to resynchronize the data to the recovered clock. Data is, then, deserialized into a 10-bit word. Byte alignment occurs within the Deserializer so that the serial data can be mapped onto the 10-bit bus properly. Deserializers monitor the incoming data for a seven-bit "comma," which is a "0011111" pattern found only within three special control characters: K28.5, K28.1, and K28.3. The byte aligner within the Deserializer identifies this pattern within the serial data and places the first "0" at bit 0, the second "0" at bit 1, and so forth. In general, stretching the recovered clock so that it is properly aligned with the data bus does this.

There are two key elements of concern when working with Deserializers: receiver sensitivity and jitter tolerance. Industry standards specify the minimum differential receiver sensitivity at 400mV peak-to-peak. However, most Deserializers must operate below 200mV to ensure robust interoperability, especially with long copper cables having significant attenuation.

Jitter tolerance is the ability of the Deserializer to properly recover data in the presence of large amounts of Random, Deterministic, and Total jitter. Early standards (e.g., FC-PH Rev. 4.3) specified a 24 percent eye opening. Later standards have modified the requirements and complicated the measurements of jitter. Again, it should be proven that the Deserializer is compliant with jitter tolerance specifications in order to ensure robust interoperability.

Building Blocks For Hubs And JBODs

The circuits used in JBODs and hubs differ substantially from SerDes since, for the most part, these systems do not deserialize data, but rather route serial data within the system. Consequently, three building blocks are used throughout these systems: Port Bypass Circuits (PBCs), Repeaters, and Retimers.

Port Bypass Circuits are gigabit speed muxes that route serial data around systems. The most common of a PCB is the routing of Serial Input data (SI) route to an external drive (via TX) and the PBC multiplexer. The mux can select between this data or data from the drive (on RX) as the output of the PBC (SO). Daisy chaining these PBCs allows a disk array to be built that can include a functioning drive on the loop or exclude a non-functioning drive from the loop.

Repeaters

At the end of a Fibre Channel link, the incoming signal typically has reduced amplitude and significant jitter. If this signal is to be passed to downstream devices, it must be amplified and the jitter eliminated or attenuated. A repeater is a CDR that resynchronizes the incoming data to the CDR's locally recovered clock. This function can be implemented with a high performance, but traditionally architected, analog PLL or with an all-digital CDR optimized for Fibre Channel applications. In either case, the repeater must reduce the amount of jitter in its outgoing signal to ensure that downstream devices can recover it error-free.

It is important to note that repeaters cannot eliminate all jitter from the incoming signal. PLLs have a jitter transfer function that passes low frequency jitter while attenuating high frequency jitter. Consequently, the loop bandwidth of the repeater's PLL must be low enough so that any jitter that is transferred is at a frequency that can be reliably tracked by the downstream receiver's CDR. Typically, this is between 500KHz and 2MHz. In traditionally architected CDRs, reducing the loop bandwidth of the PLL also reduces the CDR's jitter tolerance, which requires a compromise between jitter transfer and jitter tolerance.

There are several issues to be aware of regarding repeaters. If a repeater is used at the output of a JBOD or hub, output of the system must be compliant with all industry standard specifications in all configurations. For instance, a JBOD with repeaters at the input and output may transfer jitter when drives are not installed. Similarly, hubs may transfer jitter, which can impact the ability of hubs to be cascaded. An additional issue impacting cascading is whether the CDR suffers from jitter "peaking" where certain frequencies of jitter are actually slightly amplified. Peaking in Fibre Channel repeaters should not be tolerated since it reduces the ability of a system to be cascaded. Information should be requested concerning the repeater's loop bandwidth and peaking characteristics to determine that robust operation is ensured in all configurations.

One solution to eliminate this problem is a digital, two-stage non-peaking repeater. Here, the input PLL is a high loop bandwidth CDR that has excellent jitter tolerance but rather poor jitter transfer. A second CDR with low PLL loop bandwidth is added to this first stage PLL in order to reduce jitter transfer resulting in an overall loop bandwidth of approximately 75KHz. This provides maximal jitter tolerance and minimal jitter transfer with low latency in a digital circuit, which behaves identically over voltage, temperature, and process.

Retimers

In order to eliminate jitter transfer, a retimer must be used. A retimer is similar to a repeater but resynchronizes the recovered serial data to a local reference clock. This ensures Fibre Channel compliance at the output of the retimer regardless of the amount of jitter on the incoming signal. This function is significantly more complicated than a repeater since the recovered data must be stored in an add/drop FIFO synchronously to the recovered clock and pulled out of the FIFO timed to the local reference clock. In order to accommodate rate differences between the recovered clock and the reference clock, the FIFO must add/drop Fibre Channel fill words between frames, per the rules delineated in Fibre Channel standards. Retimers are embedded in all host adapters, switches, and disk drives so that frames, which are not addressed to that node, are passed to the downstream node while meeting Fibre Channel jitter specifications.

Although retimers sound ideal, they should be used sparingly in systems. The latency of a retimer is significantly higher than a repeater because the data passes through the FIFO. Typically, repeaters have latencies less than ten or twenty bits, whereas retimers may have up to 180 bits to 250 bits of latency. This adversely impacts network efficiency if a large amount of retimers are used in the SAN network. Fibre Channel Arbitrated Loop (FCAL) devices have a clock skew budget calculated by assuming each of the 126 devices in a loop has local oscillators with +/- 100ppm reference clocks. Consequently, each retimer in a system should be counted as one of the 126 devices in the clock skew budget, which potentially reduces the number of FC-AL devices available in a loop. In practice, few installed SANs have 126 devices and, even if they did, additional retimers would not likely cause errors since this is a conservative worst-case analysis.

As a general rule, retimers should only be used on the output side of links at bulkheads where compliance to Fibre Channel signal quality specifications must be met. Repeaters should be used everywhere else, since any jitter transferring through the repeater is low frequency and benign to downstream devices. The availability of retimers will significantly improve the interoperability of complex JBODs and hubs.

The functional building blocks of Fibre Channel systems: SerDes, PBCs, repeaters, and retimers are used throughout the SAN in order to generate, receive, and steer gigabit speed serial signals. The system designer should request proof of Fibre Channel signal quality compliance for each component within the SAN. Retimers should be requested at the output of all JBODs and hubs to ensure interoperability.

In general, these building blocks are getting very mature at 1.0625Gbps. However, many of the newly developed 2.125Gbps implementations are still somewhat untested and should be treated with healthy skepticism in order to speed the adoption of reliable 2Gbps SANs.

Bob Rumer is the VP of SAN Products at Vitesse Semiconductor Corporation (Camarillo, CA).