DWDM: its time has come; it offers high bandwidth and reliability for demanding storage applications. (Tape/Disk/Optical Storage).

There are four storage-connectivity approaches that are all available in some form today. These include:

* Coarse/dense wavelength-division multiplexing (CIDWDM) storage connectivity (lambda) solutions: This approach provides the highest bandwidth and reliability for the most demanding storage applications.

* Legacy access solutions (frame relay, ATM, T1/T3, etc.): This approach provides a low-cost solution for connecting storage systems that have low bandwidth requirements.

* Storage/Fibre Channel(FC) to SONET-based networks: This is becoming a popular service offering from service providers for low-bandwidth (subgig data rate) storage requirements. As SONET is a well-established metropolitan connectivity option, this carrier-grade storage connectivity approach is well suited to meeting lower-bandwidth requirements.

* Storage/Fibre Channel to an IP network: This approach is just emerging with multiple TCP/IP-protocol implementation approaches. These are lowcost solutions that take advantage of the prevalence of IP. The drawback is that IP quality of service (QoS) is inconsistent across networks and can rarely be implemented across every IP router in the end-to-end storage path. This reduces the overall reliability and predictability of IP storage traffic. This IP-based approach increases the overall operational-management expense of the network to ensure storage traffic is properly prioritized across the IP network.

Overview of DWOM

DWDM is a technology that puts data from different sources and protocols together on an optical fiber with each signal carried on its own separate and private light wavelength. This is commonly referred to as a lambda. Using DWDM technology, up to 80 and, theoretically, more separate wavelengths of data can be multiplexed into a light stream transmitted on a single optical fiber. On the receiving side, each channel is then de-multiplexed back into the original source.

A set of wavelengths has been allocated by the ITU standards body, establishing more than 50 optical channels (lambdas) within the wavelength range of 1520-1570 nm, with a nominal channel spacing of 100GHz (-0.80 nm). Within this range, wavelengths are grouped together in what are referred to as bands.

Optical add/drop multiplexers (OADMs) are used to add and drop wavelengths onto a DWDM network. For transmitting data, these OADMs take in various optical input sources (Fibre Channel, ESCON, HCON, GigE, etc.) and perform an optical-electrical-optical (O-E-O) conversion, converting each channel into wavelengths that are added onto the DWDM network. When receiving data, the OADMs perform the inverse O-E-O conversion to de-multiplex (demux) the DWDM wavelengths into their original data stream.

The process takes microseconds to complete since no error-detection, protocol-conversion, or switching decisions are involved. Data integrity, error detection, and error correction are left to the end devices (e.g. host bus adapter, network interface card, storage device). OADMs are bit-rate and protocol independent, enabling multiple channels with different data formats and different data rates to be transmitted together as separate wavelengths. Fibre Channel, ESCON, FICON, SONET, IP and ATM data can all be traveling at the same time within the same optical fiber, each on a different wavelength. TDM technology can additionally be combined with DWDM to make better utilization of wavelengths. For example, a wavelength may have a total capacity of 1.25Gbps, however, the traffic to be carried may be ESCON, which is rated at 200 Mb/sec. To make more efficient use of a wavelength, multiple ESCON channels can be aggregated within a single wavelength using cards equipped with multiple, lower-speed ports.

DWDM Storage Network Planning and Design Considerations

DWDM topologies: DWDM networks can physically be deployed in a number of common network topologies, including point-to-point, ring, and meshed. Point-to-point and ring configurations are commonly utilized for extending SANs. Ring topologies are appealing due, in part, to the ease of adding nodes (locations) to the ring. In addition, SONET-like capabilities, such as high resilience and automatic restoration within 50 msec are often standard features of ring topologies. Mesh topologies provides the ability to add or drop any band or channel at a number of locations, as opposed to being restricted to having the source band in one location and the entire destination band in another. For example, it may be useful to have a band span multiple locations where individual wavelengths within that band terminate in different locations. This capability provides more granular and flexible allocation of DWDM wavelengths. Another topology that supports storage networking is a huband-spoke architecture. This approach is popu lar with carriers that offer enterprise customers the ability to lease a managed wavelength from carriers like SBC, AT&T, and others. The carrier will offer one or more lambdas to the enterprise, depending on their storage bandwidth requirements.

Dark fiber: Dark fiber refers to unused (unlit) fiber optic cable. Dark fiber is crucial to the implementation of a metropolitan SAN using DWDM, therefore finding it needs to be the first step. The quality of the fiber is critical to ensuring a successful DWDM deployment. The minimum operating specifications for the transport fiber is standard, single-mode fiber, or NDSF (non-dispersion shifted fiber). The dispersion grading of the fiber affects optical signal attenuation.

The type of the fiber will assist in estimating the amount of loss incurred on the path, which is a significant factor for distance and link budget determination. If possible, optical time domain reflectometer (OTDR) readings from the actu al fiber path are recommended for engineering the network. if the actual losses from fiber characterization are not available, an approximate figure of 0.3-0.4dB/km can be used. Next, add an additional 10% safety margin to the estimated loss to accommodate for distance estimation errors, additional losses resulting from poor splices, patch panels, and allowing for future maintenance (splices, etc.). Engineering the optical network without a margin is never recommended, as it can cause problems in the future if a fiber break occurs and splicing that introduces additional losses is required.

The last issue to consider with dark fiber is the physical paths or route of the fiber. To build a resilient, fault-tolerant, and secure network, the fiber paths between locations should be diverse. By ensuring diverse fiber routes, if a fiber break should occur in one path, the backup (redundant/protected) path is immediately switched to preventing an outage from occurring. This diversity includes entrances to the building or site complex and the last mile. The actual fiber path plays a big part in how the topology of the network will look.

Distance considerations

At the basic level, a digital electrical signal is converted to an optical signal and transmitted across an optical fiber using a laser. It is received by a light concentrating device (LCD) that in tam converts the optical signal back into a digital electrical signal. There are several factors that affect the receiver's ability to correctly interpret the contents of the signal. These factors are very important for determining the maximum distances between nodes in a DWDM network. The three main factors are attenuation, vendors link budget, and dispersion. Now to review each one independently:

Attenuation: This refers to the loss in the optical signal strength. If the signal is too weak, the receiver is unable to interpret the signal. The link loss budget refers to the amount of loss in signal strength before the receiver can no longer interpret the signal. Attenuation can be attributed to a number of elements, including fiber distance, number of patch panel connections (typically 0.5dB of loss) and splices, dirty fiber connectors.

Link loss is often expressed in terms of the decibel (dB) loss. Decibel loss is different than decibel milliwatts (dBm). Decibel loss represents the amount of signal gain or loss and does not represent an absolute value. For example, it is correct to say that there is 10dB of loss in a particular connection, but it is incorrect to say that that the signal strength is 10dB.

Loss in decibels is described on a logarithmic scale. If a signal P0 is input to a fiber and a signal P is transmitted, the loss in decibels is expressed as dBloss = 10 log10 (P0/P).

The attenuation also varies with the actual wavelength selected. The longer the wavelength, the lower the fiber loss that is incurred per kilometer. For example, at a wavelength of 850 nm the fiber loss is approximately 2dB per km while at a wavelength of 1550 nm, the loss is approximately 0.2dB per km. As a result, DWDM systems tend tooperate in the 1550 nm window.

Manufacturer's Link Budget: Link loss budgets define the amount of acceptable loss within nodes on a DWDM network. Each DWDM vendor's equipment has its own loss budget that is dependent on the actual hardware in use as well as other factors, such as the number of OADM nodes.

Operating the optical network within the manufacturers link budget is critical for establishing a reliable and stable network. Link budgets are vendor specific and determined by the overall design of the network. To assist with the overall link engineering of the network, it is highly recommended to ensure the equipment vendor provides an optical-network-modeling tool. These modeling tools remove the complexity of DWDM network design and, in turn, permit network designers to readily review multiple design approaches for design optimization.

Dispersion: Light dispersion is another important aspect of physics that affects the attainable distance. Each wavelength of light travels at its own particular speed that is different from other wavelengths. For example, when white light passes through a prism, some wavelengths of light bend more because their refractive index is higher, i.e. they travel slower. This is what gives us the spectrum of white. light. The red and orange light, travel slowest and so are bent most, while. the violet and blue travel fastest and so are bent less. All the other colors lie in between. This means that different wavelengths traveling through an optical fiber also travel at different speeds.

As distances increase, this dispersion can result in the inability of the .receiver to distinguish one wavelength from another. Dispersion can be minimized through the use of different types of fiber or through hardware.

Other Factors for Increasing Distance

There are a number of additional factors that affect the attainable distance between DWDM nodes. External hardware devices can increase the distance that an optical signal can travel between endpoints. The type of device to use depends on what element of physics you are trying to counteract. Two devices that are typically used to extend distances. are. an optical fiber amplifier (OFA) or a regenerator (Regen).

Optical fiber amplifier: This can be used to overcome attenuation issues on the fiber. Amplification differs from regeneration in the way an increase in power is achieved. With an OFA, light (power) is added (amplified) to an existing wavelength (or wavelengths) without taking the signal back to an electrical state. The amount of gain (power added) is dependent on the efficiencies of the amplifier, the number of wavelengths being amplified and the strength of the wavelengths entering the amplifier. There are several types of OFAs available in the industry. The most popular for metro DWDM systems is the erbium doped fiber amplifier or EDFA.

Using OFAs is a cost-effective way to extend distances when required. The disadvantage to this approach is that each amplifier introduces unwanted signals (noise), in addition to amplifying the designed signals. As a result, there is a limit of how many times an OFA can be used to extend distance before the more costly method of regeneration is used.

Regeneration: Regeneration can compensate for both dispersion and attenuation, but can be costly to a design. Regeneration works by translating the optical signal back into its original electrical format, processing it to clean up jitter, and then translating it .back to an optical signal for re-transmission. This process provides us with what is referred to as 3R regeneration.

The first "R" is "re-shape." This is where the wavelength is restored back to a well-defined signal pattern. The second "R" is "re-time." This is where the electrical signal is put through a jitter buffer and the signal timing is re-aligned. The third "R" is "re-amplify," which is where the wavelength is launched with a new signal strength.

Storage Topologies

DWDM technology has been very popular with disaster-recovery and data-replication deployments. Extended-SAN configurations, such as host-channel extension or backup over a metro region, are. feasible through DWDM technology. In a Fibre Channel SAN, DWDM equipment will directly connect into the Fibre Channel switches to bridge storage islands. To the Fibre Channel switches, the DWDM equipment is invisible. The Fibre Channel switches see only the inter-switch link (ISL) connection between .one another. The DWDM topologies can include any of the configurations mentioned above (point-to-point, ring, mesh, hub and spoke).

DWDM natively extends the SAN (i.e. no protocol conversion required), maximizes the investment in leased or purchased dark fiber, and scales to support other storage and networking equipment. When DWDM is utilized to interconnect storage sites, it is not only possible to secure data simultaneously in multiple locations but also share equipment resources such as servers and storage systems thereby enabling consolidation and reducing operational cost. It: is extremely common that DWDM systems are initially cost justified for storage requirements, and then used to consolidate interoffice LAN (data) and voice traffic.

Performance Considerations

Host applications tolerance to latency: A common requirement in a. clustering or data mirroring environment is for each transaction to be secured by the DWDM-extended SAN in multiple geographical locations in real-time. This may include requiring acknowledgement to the primary data site: that transactions where received by one or more secondary sites. Each application's tolerance to latency must be examined to determine the feasibility of this mode of operation. Although DWDM networks are very, deterministic and introduce little, latency, signal propagation over long distances (such as hundreds of kilometers) increases response times and therefore may impact the host application. Typically, 5 [micro]sec per kilometer is used to calculate the one-way latency. For example, a 100 km fiber link would introduce a one-way delay of 500 [micro]secs.

Fibre Channel link over-subscription: With DWDM systems, it is a common practice to over-subscribe the DWDM or ISL being utilized for storage. A common practice is to use an over-subscription ratio of 7:1. Depending on the applications being utilized, however, these ratios can readily vary from 2:1 to 20:1.

Fibre Channel buffer-to-buffer credits: The Fibre Channel protocol supports sending multiple frames or data units, from one. point to another, before receiving acknowledgments (called R_RDY) to previous frames from the remote point, thus filling up the link with frames and .increasing the overall link efficiency. This aspect is dependent on the FC class of service being implemented.

The number of buffers is referred to as buffer credits or BB_Credits. The number of buffers at the receiving end of the link determines the number of frames that can be sent down the fiber link to the other end. The sender must have a credit available with the target device. When the sender reaches the BB_Credit number, the transmission must stop until the receiver acknowledges the data and frees up some available credits. Throughput on long links, such as those enabled [by DWDM equipment can quickly degrade if not enough frames are on the link. The longer the link, the more frames that must be sent down the link to prevent this degradation. If you add additional buffer credits to the port, the link efficiency can readily be extended over distance.

The amount of buffer credits required is dependent on the link distance plus other factors such as host application requirements. To determine the amount of buffer credits needed to maximize link efficiency, strong performance management tools are needed in the DWDM equipment and the Fibre Channel switches deployed. Appropriately engineering Fibre Channel links is critical to DWDM-extended SANs.

The approximate buffer credits calculation as published by EMC for a 1Gbps Fibre Channel link is as follows: Buffer Credits = (2 * fiber link length in meters)/4311+1.

For example, the buffer credits needed for a 100 kin, 1Gbps link would be: (2 * 100,000)/4311 + 1 = 48 buffer credits (always round up any fraction).

Propagation-delay calculations: Long link distance affects the response time of each input/output (I/O) as a result of a propagation delay. In a Fibre Channel network, every write I/O requires two round trips (depending on class of service)--one for the I/O command and another for the transmission of the data. Therefore, the response time is computed by multiplying link propagation delay by 4. Other storage protocols may have different characteristics that need to be considered when calculating propagation delay.

DWDM enables SANs to be natively extended to great distances, efficiently utilizes fiber resources, introduces minimal latency, supports multiple protocols simultaneously, provides significant bandwidth and is completely future proof. Storage networks require the efficient, high bandwidth, deterministic networks that are provided by DWDM. Availability of dark fiber, network topology, link distance, signal propagation delay, Fibre Channel buffer credits and host applications tolerances to latency are among the items that storage networking engineers must consider.

Enterprise organizations have the option today of deploying their own DWDM network for storage connectivity, or they can optionally purchase. a wavelength connectivity service from providers such as SBC, AT&T, and others.

Although other new storage protocols like iSCSI, iFCP, InfiniBand and many others are under development, these are all still very immature and will require 12 to 18 months minimally to mature while the initial deployment problems are resolved. Unfortunately these multiple new approaches tend to create market confusion and paralyze IT organizations from acting, however companies can no longer afford the operational expense and associated business risk of continuing in a "wait and see" mode. Companies need to act now with proven storage connectivity design approaches that are future proof and will readily accommodate future technologies as they mature.

The best network designs tend to be the simplest. These designs reduce the capital expense, the network complexity and the operational expense associated with maintaining the network. The availability of new DWDM modeling tools from vendors today removes the complexity of designing DWDM networks, making storage connectivity via DWDM a simple way to natively extend the storage network. C/DWDM storage connectivity networks are. proven and ready for prime time deployment.

Rene Dufrene is a senior manager in the metro optical alliance for Nortel Networks and Al Lounsbury is a senior manager for metro optical technical marketing, also for Nortel.

www.norteInteworks.com