Practical considerations for iSCSI target deployment.

Since the iSCSI specification was officially advanced onto the standards track by the Internet Engineering Task Force (IETF) six months ago, many vendors have begun shipping iSCSI initiator and target products. Storage administrators can now choose among software iSCSI initiators and iSCSI HBAs for most of the popular operating systems and target products covering a wide range of features and capabilities. Those planning an iSCSI deployment face a myriad of choices as they decide what features they want and need and how to deploy these products in their networks. To make sense of these issues, it helps to understand a few of the key features that target products provide. This paper gives an overview of five key feature areas--security, performance, network design, interoperability and robustness--and explores how these features affect deployment decisions.

Security

Storage administrators must consider preventing unauthorized access to iSCSI targets and their data. iSCSI standards currently provide two levels of security--authentication between initiator and target during login negotiation and privacy by encrypting data.

During iSCSI login, the initiator and target can agree to participate in authentication by exchanging the authentication method in the security negotiation stage. Several authentication methods are available while configuring authentication in an iSCSI network if they are supported in an iSCSI product, including Kerberos V5 (KRB5) [RFC 1510], Simple public-Key GSS-API Mechanism (SPKM) [RFC 2025], Secure Remote Password (SRP) [RFC2945], Challenge Handshake Authentication Protocol (CHAP) [RFC 1944] and No Authentication (NONE). The authentication method and associated information may be configured administratively on the initiator and the target or through the use network services such as RADIUS.

iSCSI standards mandate support for CHAP authentication in iSCSI products. The spec recommends that administrators select random CHAP secrets, which may be up to 16 bytes in length. The iSCSI standard recommends using randomly generated CHAP secrets at least 12 bytes in length when privacy is not in use.

Data privacy is attained by the use of IPSEC with iSCSI sessions. However, IPSEC is optional in iSCSI standards and not universally used or implemented. iSCSI deployments in a data center over a dedicated gigabit Ethernet LAN have little need for IPsec, but it becomes important when iSCSI is run over a WAN whose traffic might be intercepted. An alternative for protecting iSCSI traffic when the initiator and target do not support the feature is to use a VPN tunnel.

Some target products allow administrators to use access control lists (ACLs) to control initiators' access to target volumes with finer granularity. For example, using ACLs one might allow one initiator to have write access to a volume while several other initiators have only read access.

Performance

The performance of an iSCSI target is usually characterized in terms of throughput, expressed in megabytes per second (MB/s) and I/O operations per second (IOPS). Like a disk drive, the performance of an iSCSI target is greatly affected by the I/O access pattern. Sequential I/O that can be streamed to or from the target's disks will run faster than random I/O that causes those disks' heads to seek. Moreover, most targets provide an ample disk block cache, and I/O that can be satisfied from that cache will be faster than I/O that must go all the way to the target's disks. Unlike a disk drive, however, the performance of an iSCSI target is also affected by the characteristics of the network path dealing with available bandwidth and latency between the initiator and target.

Since iSCSI is layered above TCP, it operates over virtually any IP path from gigabit links to wireless. While it has specific applications running over WANs, iSCSI is most often deployed over high-speed, low-latency networks. In a data center deployment of servers sharing a common iSCSI storage array as shown in Figure 1, for example, the initiators and targets are typically connected via a single low-latency gigabit Ethernet switch. Most enterprise class target products provide one or multiple gigabit Ethernet links and can be expected, with the right I/O pattern, to saturate those links, providing throughput comparable to Fiber Channel.

[FIGURE 1 OMITTED]

Most targets allow the administrator to configure volumes that stripe the data across multiple disks (RAID-0), mirror the data across two or more disks (RAID-1), employ parity disks (e.g. RAID-5), or allow combinations of these configurations. Stripe and mirror configurations typically improve performance for read access because consecutive I/Os can be spread across multiple disks, reducing the load on each individual disk. On the other hand, a mirror configuration may result in lower performance for write access since the target must incur the extra overhead of writing all of the data to all of the disks in the mirror set.

Currently, there is no generally accepted iSCSI benchmark, although the Storage Performance Council's recently announced SPC-1 benchmark might hold some applicability. Most users characterize the performance of targets using simpler tools such as IOMeter and XDD. Both of these tools allow a user to synthesize access patterns--by selecting such parameters as request size, the number of simultaneous outstanding I/Os, range of blocks to access, and randomness of access pattern to model the behavior of the applications they are interested in running. Applying a combination of well-defined access patterns can give a good prediction of how a particular application will perform against a target. A basic "four corners" test measuring sequential I/O throughput to cached and un-cached volumes and random I/O IOPS to cached and un-cached access patterns gives a feel for how a target will perform in a wide variety of applications.

Network Design

Designing a network to carry iSCSI depends a great deal on the needs of the applications--database, video, e-mail server, etc.--that will be using it. Many iSCSI applications are latency sensitive, so building the network with the fewest number of hops and the shortest possible links is usually a key consideration. The other important consideration is to provision adequate bandwidth for the application. When deploying on an existing network, this means understanding the available bandwidth of the links, switches and routers the iSCSI traffic will be carried over, and understanding the characteristics of the competing traffic. Administrators deploying iSCSI over WANs, such as the one shown in Figure 2, may wish to employ QoS to ensure adequate bandwidth is reserved for their iSCSI traffic.

[FIGURE 2 OMITTED]

Even in data center deployments such as the one shown in Figure 1, where colocated initiators and targets are interconnected via a dedicated gigabit Ethernet LAN, it is important to understand the performance characteristics of the switches. In a bladed switch, the bandwidth available between ports on the same blade or the bandwidth across the backplane might be the bottleneck. In a multi-switch LAN the inter-switch trunk bandwidths are a consideration. Administrators will want to consider whether the aggregate iSCSI throughput across these potential bottlenecks exceeds their capacities and, if so, reposition equipment or expand the network for more capacity.

Interoperability

Since iSCSI products are still relatively new, interoperability issues may surface during initial deployments. iSCSI provides a login stage during which parameters and features are negotiated. Convergence of this stage of negotiation will determine the features that get used for the iSCSI session. Examples of such negotiable features include unsolicited data usage, number of connections per session, maximum size of iSCSI PDU, header and/or data digest computation and fixed interval markers and any other vendor specific features. Most iSCSI products do not support all options, so the negotiation may result in a successful session but may not be optimal.

In order to ensure that interoperability problems are found in the vendor's labs and not on the customer's network, iSCSI target vendors typically qualify initiators for use with their systems. All initiators should interoperate but vendors have tested qualified initiators to ensure interoperability. Microsoft provides a test suite under their Windows Logo Program Qualification Service (WLPQS) to obtain iSCSI interoperability certification for Windows 2000, XP and 2003 platforms. Sun Microsystems also provides iSCSI certification program on their Solaris operating system. In addition to vendor qualifications events such as University of New Hampshire's (UNH) "plugfest," which occur every 6 months, offer product developers an opportunity to refine their interoperability. UNH also provides a reference implementation and test suites that have helped to improve interoperability.

Robustness and Reliability

Because an iSCSI target often serves a mission critical function, enterprise storage administrators usually consider robustness of the product a paramount concern. It is a given that a storage server must never lose or corrupt data. It is nearly as important that the availability of the system be as high as possible. Robustness features are usually first apparent in the hardware. Things such as redundant power supplies, field replaceable fans and multiple network interfaces are often supplied in target equipment intended for enterprise deployments. Though not as immediately evident to the user, the robustness of the target's software and its resistance to crashing are just as important. Another important consideration is the restart time if a target does happen to crash. Since most initiators try to reconnect and recover the session for a brief period of time after losing a TCP connection, a target crash can be transparent to the application if the target can manage to restart within that short interval. TCP's retransmission mechanism usually allows an iSCSI session to survive a brief loss of network connectivity such as a switch reboot. Session timeout, reconnect and recovery times are not dictated by the iSCSI spec, however, so these tolerances do vary from initiator to initiator.

Some vendors deliver high availability (HA) targets with two or more redundant controllers. If one controller fails, another can take over its load. iSCSI's session redirect feature makes possible a particularly cleaver HA solution, implemented by vendors like Intransa, that also provides load balancing. Initiators can begin their session by talking to any controller in the cluster, which will redirect the initiator to one that will serve its traffic. If that controller fails, the initiator can again connect to any node and be redirected to the controller that has taken over for the failed node.

Advanced Features

Most first generation iSCSI products do not support all features of the iSCSI standard, which includes advanced performance and reliability enhancing features such as:

* Fixed Interval Markers (FIM) inserted into TCP stream for out of order processing

* Multi-connection sessions

* Error recovery upon iSCSI header and/or digest failures resulting in retransmission requests (iSCSI Error Recovery Level 1)

* SCSI command level recovery (iSCSI Error Recovery Level 2)

* Vendor private or public keyword support during login

* As time progresses, more implementations will support these features.

The iSCSI standard is now ratified and first generation products are being deployed. As with any first generation product in a new technology, these deployments present many opportunities and risks.

While most implementations do not implement all of the protocol's features, the login negotiation feature ensures that they interoperate. The interoperability forums and vendor certification programs help product developers to find and repair most interoperability problems before customers see them.

Before choosing the products to deploy in your organization, one should understand the technology, standards and feature sets supported in available products. Factors that should be taken into account in an iSCSI product evaluation include security, performance, scalability, interoperability, reliability and robustness.

As first generation iSCSI products are deployed, the experience gained will help vendors design the next generation iSCSI products that will support new classes of applications.

Bob Gilligan and Raj Velpuri are senior software engineers at Intransa (San Jose, CA)

www.intransa.com