Pitfalls and promises: will IP storage supplant Fibre Channel? (Storage Networking).

Fibre Channel-based SANs are firmly established in high-end data centers. But smaller enterprises and midtier businesses have not adopted SANs at this same rate, often citing unfamiliarity with Fibre Channel, distance issues, and high cost as reasons for not adopting storage networking. Nevertheless many IT administrators understand the benefits of SANs and would like to get over the adoption obstacles.

Fibre Channel is not as complex as it appears, and in some ways is easier to administer than IP networks. But the high replacement and management costs are real, as are the distance issues. Many companies are looking to IP-based storage, which primarily runs using the iSCSI protocol. iSCSI is based on the familiar SCSI protocol, a parallel transport type. Fibre Channel came along after SCSI as a highspeed serial transport. Fibre Channel retained the SCSI controller interface code so can run the SCSI command set, which helps vendors to migrate from parallel SCSI to serial Fibre Channel without undue difficulty.

iSCSI is the only protocol that natively addresses IP storage, but is not the only protocol that transmits storage 110 traffic over IP networks. Fibre Channel already preserves the SCSI command set intact, which allows it to interface with SCSI-enabled storage devices. Other protocols based on Fibre Channel also integrate IP traffic, including FCIP, iFCP, and mFCP. FCIP is a transport protocol to link Fibre Channel SANs over IP networks. iFCP and mFCP replace Fibre Channel with IP while allowing some Fibre Channel devices to attach to the SAN. Like Fibre Channel, iSCSI preserves the SCSI command set and maps it to IP, but iSCSI dispenses with Fibre Channel and uses only itself as the storage networking protocol. iSCSI mapping requires intensive coding, which can lead to overhead and latency issues. iSCSI implementations largely require specialized iSCSI chip sets to achieve the desired speed and efficiency.

IP-based storage actually hit the market a few years ago, but was not successful. Earlier implementations lacked TOE (TCP/IP offload engine) technology, did not have a standardized iSCSI protocol, and targeted the corporate data center at the top of the storage pyramid. Mike Wentz, senior director of customer engineering at 3Ware, notes that the market is different now.

1 TOE card: "The good news is that there are now TOE cards that can solve one of the biggest issues we had, what we called 'application transparency," said Wentz. This obstacle centered around IP storage appearing as block level devices on Windows networks. The network-based block-level storage showed up as removable media, which many applications simply refused to access. TOE cards also allow systems to boot off the storage arrays.

2 Standard iSCSI protocol: iSCSI existed as a proposed standard, but it was not developed enough for industry adoption. IP storage vendors had to code their own, making proprietary protocols that were unpopular in open environments.

3 Targeting different markets: Wentz said that IP vendors "tried to go into data centers and say, look, there is this IP-based storage that's faster, better, and cheaper than your Fibre Channel systems.' But the data centers were famously reluctant to dump their Fibre Channel investments for IP. Although is still the case today, Wentz believes that IP-based storage will work well for the mid-tier markets.

Distance and IP Storage

A common claim from IP storage vendors is that IP is a much better wide-area network, since Fibre Channel has severe distance limitations. To a point this is true, since Fibre Channel was developed to meet the demands of centralized storage applications. As a brand-new protocol for block-level storage, with very high bandwidth, low latency, and loss-free conventions, it was ideal for localized storage networking. But it was never optimized for long-distance data transfers--when Fibre Channel switches are too far apart, fabric communication breaks down and data must stop sending while it waits for a response from the last packet. Storage networking's transport mechanisms lack IP's sophistication level at finding new data paths and retries, because it was never designed to need it. Since Fibre Channel was designed to do loss-free applications, re-transmission has a severe impact on the performance of end-user applications.

There are some solutions in place, most commonly linking Fibre Channel SANs across metro areas. According to Matthew Williams, product marketing manager at Akara, the best way to further increase distance between Fibre Channel SANs is to ensure a loss-free WAN environment. He suggests the way to do this is by shortening the flow control between Fibre Channel SANs using such technologies as fiber-optic cabling (dark fiber) and boosting signals with DWDM. In this scenario, flow-control moves from one center's flow control switch to the second, creating a flowcontrol loop between Fibre Channel elements. Williams suggests installing multiple flow-controls at the data centers and the buildings where they're housed, though Akara prefers using SONET as a lossfree medium instead of Fibre Channel flow controls.

Williams believes that IP can also handle block-level data that requires re-transmission, but points out that TCP/IP high-bandwidth transfers put a large load on the server. This usually means that IT must use TCP offload engines, usually handled by specialized network cards to take the load off the host CPU. In most cases he does not believe this approach is adequate for high performance storage transfers. "IP storage has its fit into the lower end of the market, but it's the Fortune 1000 performance that matters. Disk mirroring, tape backup, performance matters. IP just isn't the right choice,' said Williams.

In the end, the success of distance, transfers depends on how much money a business is willing to spend and what their data-protection needs are. Businesses must have dedicated high-bandwidth lines at the very least, and may need to invest in services like SONETbased transport. Price is an issue, since boosting Fibre Channel signals over distance can be extremely expensive. IP supporters point out that dedicated lines and private networks such as T1, T3, and VPNs are usually less expensive than Fibre Channel channel extensions. Data-protection requirements will also influence remote connection choices, since even the speed of light suffers a measurable delay going cross-country. Some backup procedures are not sensitive to distance data movement. Asynchronous remote copy is an example, where the primary server mirrors locally and copies its transactions to a remote secondary server. Latency does not matter, since the primary server does not wait for the secondary's confirmation to continue its processing. How ever, if a company is doing synchronous operations such as remote mirroring, where the primary server must wait on the confirmation from the secondary server before continuing, fiber will often experience an unacceptable latency level.

IP has its own distance limitations. What works well in a data network--sophisticated retry and timeout algorithms--is unacceptable in storage networking, which does not brook latency and delays. However, not everyone agrees that packet loss is such a large problem anymore. Well-known IP protocols such as NFS used to use a basic data transport mechanism called UDP (user datagram protocol), which was only semi-unreliable. This caused NFS to work well on a LAN but to fail over long distances, requiring multiple routers between locations and even then experiencing severe packet loss. But the most recent incarnations of NFS are TCP/IP based, which solves packet-reassembly issues.

Bob Rogers, chief storage technologist at BMC, said, "The moral of the story here is, that IP is getting better, faster, more reliable just like everything else.' He believes the main question for businesses to ask about IP vs. Fibre Channel is not which one works better, but which one will allow them to converge their data and storage networks. Many companies would rather not run separate networks, but many were forced to in order to handle their distributed backup traffic. When distributed backup products first hit the market and attempted to run over LANs, they did not do well. Networks were configured for message traffic with packet sizes of 512 bytes. But when IT administrators started to blast gigabytes of distributed backup data through networks configured for message traffic, the networks were overloaded. This led to laying down separate networks beside their 'data network to do their backups, Fibre Channel storage networks that were configured for high availability and high-bandwidth traffic. Faster Ethernet eventually allowed IP networks to improve their bandwidth, enabling storage administrators to route storage traffic at different levels of priority. But current network bandwidth is. still not. sufficient .to handle .large scale combined messaging and storage data. TOEs will be the. next' step in converging data and, storage networks.

iSCSI uses a lot of compute power to process TCP/IP commands. In high-performance environments, the performance hit will be. unacceptable without TCP/IP offload engines, to relieve the host CPUs. TOE implementations, which range from Intel's dedicated server to Alacritech's specialized chips, offloads storage traffic from the main servers and provides' dedicated processing to iSCSI. Less performance-intensive environments may not need TOE at first, but the more data companies push through distributed backup procedures, the more they will need TOE-enabled tools.

Barry Hasser, vice president of marketing at Alacritech, believes that TOEs can protect large storage investments up. front and fuel better storage management and near-line storage procedures. "We see them as key for [small enterprises] dipping their toes into iSCSI. Even in the data center,' we see products coming into market that will address previously un-addressed needs. Over the past year we've gained a much keener understanding of the issues surrounding. disaster recovery, and the need for fast rescue operations.' This includes not only disk, but linear tape in the data center. "Linear tape does represent some challenges there due to its linear nature. We're seeing products which will address a near-line environment. There you're going to see Fibre Channel and iSCSI in close, cooperation with each other, where Fibre Channel provides the high-end storage management and iSCSI the near-line storage management."

The Market

BMC's Rogers thinks that IP will not take Fibre Channel's place in the big data centers for the next five to six years, but it will happen. "I'm one of these people that thinks that IP probably will take over the world," said Rogers. "If you look at the problems IP has solved over the years, it's an amazingly resilient and very effective networking protocol. So eventually I think IP will take over. But it's got a long way to go. Fibre Channel is still on the upswing, and will be for a number of years."

If Fibre Channel is firmly ensconced in the corporate data center, does IP storage have a market today? Haaser said about IP storage adoption, "The early take is going to be a lot of the people that haven't moved to Fibre Channel SANs, that currently have direct attached. Certainly it's going to be departmental level in the enterprise customer and small and medium-level enterprise."

Larry Cormier, vice president of marketing and business development at Comm Vault, said that the downturn in the economy has forced storage vendors to pull back development and continue with already profitable products, which includes Fibre Channel. "I think storage over IP is inevitable, but I think it's been put off for over a year." He believes the market will see shipments generating significant revenue late 2003 or early 2004.

A strong argument for adopting IP storage now is that most IT administrators already know IP. However, Fibre Channel apologists point out that storage networking has its own knowledge set apart from the network type. Cormier commented, "Storage networking is more about storage and less about networking, more about data management and less about network management."

Jeffrey Schnabel, vice president of marketing at StoneFly Networks, believes that there are more similarities than differences between data and storage networks, and data network administrators will not have that large of a learning curve around storage networks. Data network administrators already segment data networks, configure ports and switches, balance server loads, and manage quality of service (QoS) requirements. Most are also familiar with multi-homing: putting two NICs on one server with different MAC addresses but sharing an TIP address. Schnabel believes the Fibre Channel storage vendors have overstated storage complexities: "The storage vendors and their high margins, they have to make it look complicated. It's not."

Learning curves are not the only consideration when deciding between Fibre Channel and IP storage. John Barry, director of product marketing at Comm Vault, said, "Applications that need the bulk of data movement, like big databases, they must move data quickly. This is what Fibre Channel does well. Large data centers will stay on Fibre Channel for a very long time. If you want to do serverless backup of an Oracle database, you won't do it over iSCSI -large data movement is not the forte of IP." He added, "We're committed to wide area block level movement. I wish iSCSI were farther along so we could take advantage of that, but we really believe that's the way to go."

Price is not yet an advantage for IP as a Fibre Channel replacement or network convergence strategy. Network-wide P storage adoption needs 1OGbE which will be a "rip and replace" procedures. IT must also add new IP-based storage devices and infrastructure, all of which is in short supply. But if IP starts selling well into the storage space, its usual economy of scale will kick in. The resulting volume would result in lower prices than the more specialized Fibre Channel equipment. However, 1OGbE is not essential for IP storage adoption for remote connections and in secondary storage environments. Peter Wang, founder and CTO of Intransa said, "Beyond the function of bridging between FC-SANs, the real benefit of IP-SANs will be to make enterprise class networked storage available at a substantially lower cost, both initial acquisition and start-up costs, as well as total cost of ownership. For instance, IP' SANs can enable backup of remote data stores to central sites. And low-cost disk farms on an IP-SAN can a lleviate, and potentially obviate, the tape backup window problem, while making backup data readily available for disaster recovery."

There are advantages of using IP-based storage networking today. Overcoming Fibre Channel distance limitations is one of them, with many companies looking to iSCSI as a wide-area connection for Fibre Channel SANs. Cormier said, "iSCSI is particularly good as a wide-area protocol for connecting Fibre Channel SANs, with lots of promise for disaster-recovery solutions."

Some vendors don't believes the customer cares about their storage network type. Nick Tabellion, CTO of Fujitsu Softek, believes that most businesses care very little about that, but hey do care about availability, price and performance. He said, "No matter what your storage is, it has to work. I believe it's a given that whether it works over IP or Fibre Channel protocols, the availability issue will be solved. They're going to work. Price and performance move in sync, you have to get the performance you're willing to pay for. Those are what matters to the customer. The whole notion of P or Fibre connectivity, I don't believe the customer cares at all. It's more a vendor religious war."

Tabellion believes that it will be difficult for P to ever match the price/performance equation in the high-end market. "At the mid-tier and lower end, it might be very viable," he said. "But at the mid-range and lower end, you have to convince your customer that he has to network his storage anyway."

Cisco believes it can convince these markets. The networking company co-authored the iSCSI protocol, and was the first to introduce an iSCSI networking product, a gateway between iSCSI networks and Fibre Channel SANs. Cisco has since announced the 5428, a multi-protocol SAN device that attaches directly to disk arrays and tape libraries, designed to replace direct-attached storage at mid-tier businesses. Doug Ingraham, manager of product marketing for Cisco's Storage Technology Group, said that IP is "a technology we believe in, and are spending a lot of development money and putting a lot of effort behind." He agrees that business's first priority is not choosing Fibre Channel or IP, but, "at the end of the day, really what the customer is tying to solve here is not a media-protocol issue. They're trying to increase their disk utilization, they're trying to lower their storage backup time, they're trying to find the time to allocate storage dynamically to servers. Those are their high level problems. Whether it's Fibre Channel or iSCSI or InfiniBand, or any infrastructure, the customer to a large degree almost doesn't care and doesn't want to know. They want to solve those problems at a price point that's affordable to them, and that's different from low-end to mid-tier to high-end customers."

Other vendors think less of market size and capitalization than application-driven IP storage adoption. Intransa's Wang said, "The move to IP will be primarily driven by the need for companies to make much more data readily available online and accessible over their network. I'm referring specifically to functional data, which we see as the sweet spot for IP storage and the fastest growing segment in the storage arena. Functional data covers the range of scientific, engineering, medical, entertainment media, business historical data, and the like. This data is often geographically distributed yet needs to be managed centrally. Such data is found within the largest organizations to the smallest enterprises." He expects departments and workgroups to use P storage first, while the data centers adopt it for secondary online storage and near-line backup.

IP storage will experience some success as a long-distance transport for remote Fibre Channel SANs in larger enterprises. It will likely see its first large-scale adoptions as mid-tier SANs in smaller businesses. To ensure market success, it will need to offer lower prices through innovative products and economies of scale, offer a larger product selection, and simplify storage networking for Windows and .NET environments.

Alacritech's Haaser said, "We certainly see Fibre Channel viable for a long, long time. Once you deploy a network technology it tends to live for a long time. It may be surpassed by another technology at a later time, but no one does a big cut-over overnight. They peacefully co-exist for some time."

www.akara.com

www.alacritech.com

www.bmc.com

www.cisco.com

www.commvault.com

www.fujitsusaftek.com

www.intransa.com

www.illuminata.com

www.stoneflynetworks.com

www.3ware.com