The economics of large-scale data transfer: the benefits of Fibre Channel and SONET for high-performance, cost-effective data transport through WAN. (Storage Networking).
The Internet and IP networks, in general, were designed to provide large-scale interconnection in·ter·con·nect
v. in·ter·con·nect·ed, in·ter·con·nect·ing, in·ter·con·nects
To be connected with each other: The two buildings interconnect.
v.tr. of computers and networks. This goal has been achieved with incredible success. The Internet provides rapid access for millions of client machines to hundreds of thousands of servers (mostly providing HTTP HTTP
in full HyperText Transfer Protocol
Standard application-level protocol used for exchanging files on the World Wide Web. HTTP runs on top of the TCP/IP protocol. , FTP FTP
in full file transfer protocol
Internet protocol that allows a computer to send files to or receive files from another computer. Like many Internet resources, FTP works by means of a client-server architecture; the user runs client software to connect to and email services See Internet e-mail service. ). Yet, for all of its size and scalability, predictable quality of service (QoS) has continued to elude e·lude
tr.v. e·lud·ed, e·lud·ing, e·ludes
1. To evade or escape from, as by daring, cleverness, or skill: The suspect continues to elude the police.
2. the Internet. As individual users continue to increase their requirements for wide bandwidth and low-latency interconnections, alternative solutions must be developed that provide higher performance and greater cost efficiency in the movement of large amounts of data.
The Price of Scalability in IP Networks
Given the tremendous success that IP routing has enjoyed in the development of the Internet, it is natural to assume that it is an ideal medium for interconnecting high-performance data centers. Nothing could be further from the truth. IP routing protocols A formula used by routers to determine the appropriate path onto which data should be forwarded. The routing protocol also specifies how routers report changes and share information with the other routers in the network that they can reach. , including the random early discard (RED) algorithms of routers, were optimized to service the needs of millions of users who wished to move a small amount of data each. IP routing was not optimized to provide high throughput performance for a small number of users. This becomes clear when a detailed throughput analysis of TCP/IP TCP/IP
in full Transmission Control Protocol/Internet Protocol
Standard Internet communications protocols that allow digital computers to communicate over long distances. routing is conducted.
One of the most-often-cited research reports on the performance of TCP/IP is "The Macroscopic macroscopic /mac·ro·scop·ic/ (mak?ro-skop´ik) gross (2).
mac·ro·scop·ic or mac·ro·scop·i·cal
1. Large enough to be perceived or examined by the unaided eye.
2. Behavior the TCP Congestion Avoidance Algorithm The TCP uses a network congestion avoidance algorithm that includes various aspects of an additive-increase-multiplicative-decrease (AIMD) scheme, with other schemes such as slow-start in order to achieve congestion avoidance. ." In this seminal seminal /sem·i·nal/ (sem´i-n'l) pertaining to semen or to a seed.
Of, relating to, containing, or conveying semen or seed. report, the behavior of the TCP (1) (Transmission Control Protocol) The reliable transport protocol within the TCP/IP protocol suite. TCP ensures that all data arrive accurately and 100% intact at the other end. in the face of latency (1) The time between initiating a request in the computer and receiving the answer. Data latency may refer to the time between a query and the results arriving at the screen or the time between initiating a transaction that modifies one or more databases and its completion. (the time delay between the sender and the receiver) and packet loss rate are carefully analyzed an·a·lyze
tr.v. an·a·lyzed, an·a·lyz·ing, an·a·lyz·es
1. To examine methodically by separating into parts and studying their interrelations.
2. Chemistry To make a chemical analysis of.
3. . The result of this analysis is a relatively straightforward equation that explains the relationship between bandwidth, packet size (referred to as maximum segment size), round trip time and packet loss rate.
* (BW<=1.31 x MSS MSS - maximum segment size )/RTT x [PkLoss.sup.5] x PkLoss (5)
* BW = bandwidth (bps)
* MSS = maximum segment size (bits)
* RTT RTT - Round-Trip Time = round trip time (seconds)
* PkLoss = packet loss rate
As we model the effects of real-world packet loss rate, we reach the inescapable conclusion that throughput (the total amount of data that is moved from Point A to Point B in the WAN) is much more dependent on latency and packet loss rate than on the native bandwidth of the pipe. To demonstrate this point, let's look at an interconnect (1) To attach one device to another.
(2) A physical port (plug, socket) or wireless port (transmitter, receiver) used to attach one device to another. pipe with infinite bandwidth that connects a sender and a receiver (see Figure 1).
Even if we have a pipe that has infinite bandwidth, the fact that we must stop and retransmit Verb 1. retransmit - transmit again
channel, transmit, carry, impart, conduct, convey - transmit or serve as the medium for transmission; "Sound carries well over water"; "The airwaves carry the sound"; "Many metals conduct heat" data every time a packet is dropped quickly becomes the dominant factor that limits throughput. Furthermore, while we are re-transmitting lost data, all of the successfully transmitted data that came after the lost data must wait until the lost data has been retransmitted to prevent misordering the data.
Based on this analysis of TCP performance, we can plot a family of curves that describe the behavior of TCP as a function of latency and packet loss rate. In Figure 2, each curve shows the expected actual throughput when operating over an infinitely fast connection with a specific error rate. The maximum length of an IP packet has been assumed to be 1,500 bytes. This is the standard MTU (1) (Maximum Transmission Unit, Maximum Transfer Unit) The largest frame size that can be transmitted over the network. For example, an Ethernet MTU is 1,500 bytes. Messages longer than the MTU must be divided into smaller frames. (maximum transfer unit) for Ethernet. Note that as the error rates increase, the throughput decreases. Increasing the distance between the sender and the receiver also decreases the throughput.
Remember that the bandwidth of the pipe in this model is infinite! The limiting factor A factor or condition that, either temporarily or permanently, impedes mission accomplishment. Illustrative examples are transportation network deficiencies, lack of in-place facilities, malpositioned forces or materiel, extreme climatic conditions, distance, transit or overflight rights, in throughput is based on the need to retransmit dropped data and the amount of distance over which the data must be sent. To calibrate To adjust or bring into balance. Scanners, CRTs and similar peripherals may require periodic adjustment. Unlike digital devices, the electronic components within these analog devices may change from their original specification. See color calibration and tweak. latency in terms of distance, we have assumed that the sender is in New York New York, state, United States
New York, Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of and the receiver is located in Boston, Chicago or Denver. Packet loss rate is a critical parameter (1) Any value passed to a program by the user or by another program in order to customize the program for a particular purpose. A parameter may be anything; for example, a file name, a coordinate, a range of values, a money amount or a code of some kind. in this analysis. The average packet loss rate for the Internet at large is between 1% and 2%. Service level agreements (SLAs) with major carriers typically guarantee only 0.5% packet loss. For the purpose of this analysis, we will assume a packet loss rate of 0.1% (five times better than premium SLA (1) (StereoLithography Apparatus) See 3D printing.
(2) (Service Level Agreement) A contract between the provider and the user that specifies the level of service expected during its term. ). With a packet loss rate of 0.1%, data that is sent between New York and Chicago can only achieve a maximum throughput of approximately 29Mbps (even if the actual pipe that interconnects the two sites has infinite bandwidth).
This performance limitation of TCP/IP over lossy See lossy compression.
(algorithm) lossy - A term describing a data compression algorithm which actually reduces the amount of information in the data, rather than just the number of bits used to represent that information. networks is nothing new to the supercomputing industry. For years, it has been trying to move large amounts of data across TCP/IP-routed networks with little success. Los Alamos Los Alamos (lôs ăl`əmōs', lŏs), uninc. town (1990 pop. 11,455), seat of Los Alamos co., N central N.Mex. It is on a long mesa extending from the Jemez Mts. The U.S. National Labs, for example, has resorted to sending data via tapes and Federal Express to its partner facility at Sandia National Labs. The real-world throughput limitations of TCP/IP over lossy networks are so extreme that it is faster for them to send data via truck than to use their OC-3 IP wide-area access.
So, why do the carriers operate their networks with such a high loss rate and thwart the desire of large-scale single users? The answer is simple. They wish the SONET pipes that power their networks to be as full as possible. To make sure that happens, they oversell o·ver·sell
tr.v. o·ver·sold , o·ver·sell·ing, o·ver·sells
1. To contract to sell more of (a stock or commodity) than can be delivered.
2. To be too eager or insistent in attempting to sell something to. the bandwidth. Statistically, there is great variation in the amount of traffic that passes through an IP network. The carriers sell bandwidth based on their long-term averages--not on the short-term bursts of traffic that occur.
Major airlines operate the same way. They regularly oversell the number of seats on a given flight. Statistically, they know that some passengers will cancel at the last moment (without providing notice). In the event that all of the passengers that actually purchased tickets show up, they are forced to bump a passenger. Just like in the case of IP carriers, most customers never know that this took place. Only a few are inconvenienced.
This over-subscription business model works well if there are many little users whose requirements cannot be predicted in advance. However, if a single user attempts to move large amounts of data (passengers), this model begins to fail rapidly. Let's return to our airplane airplane, aeroplane, or aircraft, heavier-than-air vehicle, mechanically driven and fitted with fixed wings that support it in flight through the dynamic action of the air. analogy. Suppose we have an entire football team that must be moved from one city to another. What if the airline decides to bump one person (say, the quarterback)? The rest of the team will arrive at the destination at the correct time. However, since the team cannot function until the quarterback is safely delivered, the entire team is forced to delay its game until he arrives. As the size of the team increases, the chances of being disrupted dis·rupt
tr.v. dis·rupt·ed, dis·rupt·ing, dis·rupts
1. To throw into confusion or disorder: Protesters disrupted the candidate's speech.
2. also increase. The same effect occurs with large-scale data transfer.
When single users attempt to transfer large amounts of data across the Internet, their chances of losing data increase, as well. If the carriers drop one out of a thousand packets and the users are small, 999 customers will see great performance and one customer will have difficulty. On the other hand, if a single user is moving large amounts of data, every time he sends a thousand packets, he will get hit. TCP windowing For Northcoast
Where we call someone over and then roll our window up on them. Bassline preference.
For Example: "Hey, Andi." *insert window being rolled up* "HAHAHA. will force him to reduce his bandwidth and he will constantly experience frustration (retransmission Retransmission might refer to:
Quite simply, today's Internet is designed to let the carriers manage the aggregate behavior In economics, aggregate behavior refers to relationships between economic aggregates such as national income, government expenditure and aggregate demand. For example, the consumption function is a relationship between aggregate demand for consumption and aggregate disposable of many little users that need to route data to any destination. It is not well suited to the transfer of large amounts of data by single users.
Using Fibre Channel Over SONET
Fibre Channel over SONET (FC/SONET) represents an excellent solution that utilizes the available infrastructure and yet provides very high performance. Before we can talk about the benefits of FC/SONET as a data movement solution, let's review what SONET is and how it works.
SONET (synchronous optical network (networking) Synchronous Optical NETwork - (SONET) A broadband networking standard based on point-to-point optical fibre networks. SONET will provide a high-bandwidth "pipe" to support ATM-based services. ) is a network of fiber optic cables Noun 1. fiber optic cable - a cable made of optical fibers that can transmit large amounts of information at the speed of light
fibre optic cable
transmission line, cable, line - a conductor for transmitting electrical or optical signals or electric power that carry data and voice throughout the United States United States, officially United States of America, republic (2005 est. pop. 295,734,000), 3,539,227 sq mi (9,166,598 sq km), North America. The United States is the world's third largest country in population and the fourth largest country in area. . Data is passed through dedicated channels at rates from 155Mbps up to 10Gbps. Data paths are not changed rapidly like they are with routed IP networks. Rather, they are provisioned and remain very static. Each dedicated channel is given a guaranteed fixed bandwidth for the data. A good way to think about SONET would be to imagine a dedicated pipe with water flowing from one location to another. Every drop of water that enters the pipe on one side will emerge from the other side of the pipe. The water is never lost along the way and the water always emerges in the same order that it entered the pipe. The same is true for SONET channels. All the data that enters the pipe will emerge from the far side of the pipe and it will exit in the same order that it entered the pipe.
SONET provides the universal infrastructure used by every carrier in the United States to transport data and voice. ATM and IP all ride on top of SONET. Intuitively, it makes sense to use the lowest layer of transport available that can accomplish a particular goal. This eliminates unnecessary overhead and streamlines the transport process. Because it is so universally deployed, SONET-based signals are readily passed from carrier to carrier to move across country.
Another benefit of SONET-based networks is that they are generally deployed in redundant rings. In the event that there is a failure (say, due to a backhoe digging up a cable), SONET add-drop multiplexers Not to be confused with optical add-drop multiplexer.
An add-drop multiplexer (ADM) is an important element of optical fiber networks. A multiplexer combines, or multiplexes, several lower-bandwidth streams of data into a single beam of light. (ADMs) will automatically re-route the traffic within 50 milliseconds. This level of reliability is one of the key reasons that the telecom and datacom infrastructure of the world has worked so well for so many years.
SONET is like Fibre Channel in the sense that it inherently offers in-order, lossless See lossless compression.
(algorithm, compression) lossless - A term describing a data compression algorithm which retains all the information in the data, allowing it to be recovered perfectly by decompression.
Unix compress and GNU gzip perform lossless compression. delivery of data. Because of this and because of its ready availability and ability to transit from carrier to carrier, SONET is the ideal mechanism to carry high-performance FC data over distance.
A crucial aspect of a Fibre Channel-based storage system is credit buffering. Credit buffering accomplishes two things--both related to the concepts of congestion The condition of a network when there is not enough bandwidth to support the current traffic load.
congestion - When the offered load of a data communication path exceeds the capacity. and flow control. First, it prevents the switches in the fabric from becoming congested con·gest·ed
Affected with or characterized by congestion.
congested ENT adjective Referring to a boggy blood-filled tissue. See Nasal congestion. and dropping data. The receiving switch controls the pace of traffic as it moves from the sender to the receiver by issuing credits back to the sender. If the receiving switch is congested, it will not issue credits to the sender and the sender will slow down the pace of traffic. Since the receiving switch controls the pace, it can never be overwhelmed o·ver·whelm
tr.v. o·ver·whelmed, o·ver·whelm·ing, o·ver·whelms
1. To surge over and submerge; engulf: waves overwhelming the rocky shoreline.
a. by traffic. This is a crucial difference between Fibre Channel-based and IP-based networks. At any instant, an IP router A router that is set up to route IP packets. See router and IP. can be suddenly overwhelmed with traffic. If this occurs, it will be forced to drop data to protect itself.
The second attribute of Fibre Channel credit buffering is that it allows the actual storage devices to communicate their levels of congestion to each other. Let's consider the example of a server that is suddenly faced with a large number of interrupts interrupts - interrupt and is temporarily unable to service the needs of an in-process SCSI SCSI
in full Small Computer System Interface
Once common standard for connecting peripheral devices (disks, modems, printers, etc.) to small and medium-sized computers. SCSI has given way to faster standards, such as Firewire and USB. transfer from a disk. In this case, it will slow down the pace of credits that it issues to the Fibre Channel switch In a computer storage field, a Fibre Channel switch is a network switch compatible with Fibre Channel (FC) protocol. It allows the creation of a Fibre Channel fabric, that is currently the core component of most storage area networks. to which it is connected. By reducing the pace of credits that are issued, the server communicates backpressure back·pres·sure
Residual pressure opposing the free flow of a gas or liquid, as in a pipe or an exhaust system. to the Fibre Channel fabric A Fibre Channel fabric (or Fibre Channel switched fabric, FC-SW) is a switched fabric of Fibre Channel devices enabled by a Fibre Channel switch. Fabrics are normally subdivided by Fibre Channel zoning. Each fabric has a name server and provides other services. . That backpressure will propagate prop·a·gate
1. To cause an organism to multiply or breed.
2. To breed offspring.
3. To transmit characteristics from one generation to another.
4. through the fabric and ultimately cause the transmitting disk to slow its pace of traffic. The important theme here is that congestion is transmitted back through the network without dropping data. This is fundamentally different from the process used in TCP/IP routed networks that must drop data to communicate congestion.
These attributes of Fibre Channel are extremely important when Fibre Channel is extended outside the data center and onto the WAN. Once distance (latency) is added to the system, proper credit buffering becomes an even more crucial aspect of the system design. The WAN side of the extension gateway must use a credit-based system that has been optimized for the distance and bandwidth of the link extension. The LightSand S-600 and S-2500 gateways were specifically designed to support loss-free transmission of large amounts of data over long distances. They deploy WAN-optimized buffering systems that ensure line rate data transfer for thousands of kilometers.
SONET has the most important features necessary to enable a high-performance FC extension system. It has deterministic 1. (probability) deterministic - Describes a system whose time evolution can be predicted exactly.
2. (algorithm) deterministic - Describes an algorithm in which the correct next step depends only on the current state. (low) latency and loss-less transmission of data. When coupled with a properly designed gateway, FC over SONET provides an optimal method of transferring large amounts of data over long distances.
For this analysis, we will look at the LightSand S-600 and S-2500 gateways, which encapsulate en·cap·su·late
1. To form a capsule or sheath around.
2. To become encapsulated.
en·cap Fibre Channel over SONET. The S-600 uses one or two Fibre Channel inputs and a SONET OC-12 WAN port (622Mbps line rate). In practice, the actual data rate will be approximately 580Mbps because of the SONET encapsulation (1) In object technology, the creation of self-contained modules that contain both the data and the processing. See object-oriented programming.
(2) The transmission of one network protocol within another. , so we will use that number as the maximum single-user data rate for the system. In this case, the single user will saturate sat·u·rate
v. Abbr. sat.
1. To imbue or impregnate thoroughly.
2. To soak, fill, or load to capacity.
3. To cause a substance to unite with the greatest possible amount of another substance. the SONET link that interconnects the sites.
The S-2500 has three Fibre Channel inputs (800Mbps each). These are multiplexed into a SONET OC-48c (2.488Gbps line rate). In practice, the gateways will transmit approximately 2.32Gbps of usable USable is a special idea contest to transfer US American ideas into practice in Germany. USable is initiated by the German Körber-Stiftung (foundation Körber). It is doted with 150,000 Euro and awarded every two years. data rate. The maximum single-user bandwidth in this case is 800Mbps (limited by the data rate of the Fibre Channel input port). A single-user is defined as any given flow of data from one site to another (i.e. from one disk to another disk or from a disk to a server).
Let's take a closer look at the real-world performance that can be expected when a standard IP WAN is compared with a FC/SONET system. We'll take a hypothetical Hypothetical is an adjective, meaning of or pertaining to a hypothesis. See:
The Town of Boston is an interior town of the county and one of the county's "Southtowns. to Chicago, and New York to Denver. For each example, we will estimate the cost of service (standard ISP (1) See in-system programmable.
(2) (Internet Service Provider) An organization that provides access to the Internet. Connection to the user is provided via dial-up, ISDN, cable, DSL and T1/T3 lines. or dedicated SONET pipe). We will estimate the single user throughput and then calculate the cost efficiency of the link in dollars per megabit per second A megabit per second (abbreviated as Mbit/s, Mbps, or mbps) is a unit of data transfer rate equal to 1,000,000 bits per second. Because there are 8 bits in a byte, a transfer speed of 8 megabits per second (8 Mbps) is equivalent to 1,000,000 bytes ($ per Mbps). We will also look at the cost efficiency of data transfer when the links are fully utilized (with multiple concurrent transfers).
All of the analysis in this article is based on the assumption that SCSI block size is not an issue in the transfer of data. While SCSI block size can seriously affect the performance of data transfer if not handled properly, it applies equally to any form of storage over IP-based and storage over FC/SONET-based WAN implementations. SCSI block size should be set as large as possible when transferring data over the WAN.
Before we get started, let's look at the how monthly recurring re·cur
intr.v. re·curred, re·cur·ring, re·curs
1. To happen, come up, or show up again or repeatedly.
2. To return to one's attention or memory.
3. To return in thought or discourse. costs are estimated.
Cost Basis for Service
In this article, we are addressing the recurring costs of service for interconnecting two sites. We have not addressed the capital costs of the equipment that would be used at the customer premises. In general, the cost of equipment such as IP routers and FC gateways is comparable. Regardless, the monthly cost of service is the dominant factor when compared with capital outlay capital outlay
See capital expenditure. .
As a basis for estimating the price of Internet service, we referred to the website www.broadband-internet-provider.com/research-information.htm. The company is a broadband Internet See broadband. service provider (ISP) and publishes average pricing information for T1, T3 and OC-3 service. While it does not publish rates for OC-12 Internet Service, we can estimate OC-12 pricing from the numbers for T1, T3, and OC-3.
Since there is a clear trend of decreasing cost as data rate increases, we will reduce the price per Mbps by another factor of two in estimating the monthly recurring cost of OC-12 service. $316/2 = $158 per Mbps per month for OC-12 Internet Service (estimate).
Multiplying mul·ti·ply 1
v. mul·ti·plied, mul·ti·ply·ing, mul·ti·plies
1. To increase the amount, number, or degree of.
2. Mathematics To perform multiplication on. this number by the 622Mbps line rate yields a monthly recurring cost of $98,000 for OC-12 service (four times faster than OC-3 and twice the price). We will use this estimate as part of our comparisons below.
Note that Internet access See how to access the Internet. is not distance sensitive in its pricing. These estimates are based on providing Internet access to the customer premises. Since the customer can transmit his IP packets anywhere, there is no concept of distance in this price model. However, we will see below that distance does become a significant factor in calculating the effective throughput that a single user can achieve using IP transport.
Unlike the pricing of Internet access, the monthly recurring cost of SONET service is distance sensitive. Since the customer is contracting for a dedicated point-to-point link at a committed rate, the monthly recurring charge will be a function of the distance between the sites as well as the bandwidth. SONET price models are also complicated by differing rate structures that depend on the distance from the enterprise to the Carrier's POP (point-of presence), as well as the distance between the POPs. SONET links that move across telephone LATAs (local access and transport areas Local access and transport area (LATA) is a term used in U.S. telecommunications regulation. It represents a geographical area of the United States under the terms of the Modification of Final Judgment (MFJ) that precipitated the breakup of the original AT&T into the "Baby Bells" ) may also be subject to additional charges. We used a pricing model that is representative of that used by the major carriers. We assumed that the enterprise was located 10 miles from a carrier's POP in each of the target cities. We checked the results of our model against several recently issued contracts for OC-48 service to validate that the model accurately represents typical rates for SONET service.
The figures that are used for this analysis are just estimates (both Internet access and point-to-point SONET). The actual pricing will be highly dependent on many variables such as the distance between sites, the availability of fiber in the local loop and the distance between the enterprise and the carrier's POP.
Moving Data from New York to Boston
To examine the difference in performance between IP routing and FC over SONET, we first need to know the distances involved so that we can estimate the throughput of the network layer. Boston is 306 kilometers (190 miles) from New York. If we add 50% to allow for the non-straight path that fiber typically takes, we can use 459 kilometers as an estimate for the working distance. The speed of light in fiber is approximately 5 microseconds per kilometer kilometer
one thousand (103) meters; 3280.83 feet; five-eighths of a mile; abbreviated km. . Thus, the one-way latency between the sites is approximately 2.3 milliseconds (459km x 5ms/km).
To interconnect two sites using the IP WAN of a major carrier, each site would require broadband Internet access Broadband Internet access, often shortened to just "broadband", is high speed Internet access—typically contrasted with dial-up access over modem.
Dial-up modems are generally only capable of a maximum bitrate of 56 kbit/s (kilobits per second) and require the full use of a . This will be factored into our cost model below. The actual costs for Internet service will thus be twice the monthly recurring costs shown above. Once service is obtained, each site is free to transmit as much data as desired (up to the data rate of the ISP access).
Using the equation (BW <= 1.31 x MSS)/RTT x PLLoss. (5), we can estimate the bandwidth that a single user can actually achieve between New York and Boston. For the purposes of this analysis, we assume that the round-trip-time (RTT) is 4.6ms, the packet loss rate is 0.1% and the maximum data segment size is 1,500 bytes. Note that the 1,500 byte limit is based on the MTU limitations of Ethernet. Since all of this data originates at a server that is connected using Ethernet, this MTU is reflected throughout the system. For this combination of RTT, loss ratio, and MTU, the maximum single-user bandwidth is 108Mbps. This limitation applies regardless of the how much faster the basic Internet access speed is. (i.e. The limitation stems from the packet loss rate and retransmission-- not the maximum bandwidth of the Internet access.) By comparison, FC/SONET uses the bandwidth of the channel more effectively than TCP/IP. Data is never dropped to manage congestion, so precious bandwidth and time are never consumed with re-transmitting lost data.
Once the SONET encapsulation is removed, the achievable data rate of the SONET link itself is approximately 580Mbps for the OC12c SONET link and 2,320Mbps for the OC-48c SONET link. As discussed above, we will use 580Mbps as the maximum single-user bandwidth for the S-600 and 800Mbps for the S-2500. The 800Mbps limitation stems from the maximum speed of the FC input to the FC/SONET gateway. Figure 7 shows how these different options compare.
Note the difference in the amount of time that it would take a single user to transmit a terabyte One trillion bytes. Also TB, Tbyte and T-byte. See tera and space/time.
(unit) terabyte - 2^40 = 1,099,511,627,776 bytes = 1024 gigabytes or roughly 10^12 bytes.
(Note the spelling - one 'r'). See prefix. of data using OC-12 Internet Access (23 hours) and an OC-12 FC/SONET link (4.2 hours). Note also that the cost of service is significantly less. The result is that the cost per Mbps of single-user bandwidth is more than an order of magnitude A change in quantity or volume as measured by the decimal point. For example, from tens to hundreds is one order of magnitude. Tens to thousands is two orders of magnitude; tens to millions is three orders of magnitude, etc. higher for IP service over OC-12 when compared with FC/SONET using the same OC-12 line rate! Clearly, the IP-based solution leads to very wasteful utilization of the bandwidth for the single user that simply wishes to move large amounts of data from one location to another.
In reality, a carrier (or the large enterprise) would not ignore all of the remaining bandwidth not consumed by the single-user. It would re-use the remaining bandwidth to service other users. Let's assume that the carrier re-uses it completely. If we take the bandwidth of the EC/SONET case (580Mbps) as the maximum achievable bandwidth for the SONET link, we can divide it by the single-user data rate using IP (108Mbps). Thus, we can increase the effective number of users by a factor of 5.4 (580/108). If we adjust the cost-efficiency number the same way, we can determine the cost-per-Mbps for a fully utilized link. Based on this calculation, we see that OC-12 Internet access can provide service for $338 per Mbps (196,000/580) when the link is fully utilized. Note that this is still significantly higher than the cost-per-Mbps for FC/SONET. The case of FC/SONET for OC-48 access is similarly adjusted to reflect three users. OC-48 FC/SONET now becomes the most cost effective method of transferring data from one site to another (with three simultaneous users).
From the above analysis, it is clear that FC/SONET is, by far, the most cost-effective method of moving data from site to site for the single user. Even when the link is fully utilized by other users, FC/SONET still maintains a cost-effective edge. What are the implications of this analysis for FCIP (Fibre Channel over IP) A protocol for tunneling Fibre Channel data across an IP network. Fibre Channel was designed for local storage area networks (SANs), but FCIP extends the distance to remote locations via any IP network. See Fibre Channel, iFCP and IP storage. ? Since FCIP operates as a single tunnel through the WAN, it is subject to the single-user analysis above. Even though there may be multiple flows of FC data through the FCIP tunnel, all of these flows will transit the same IP tunnel IP tunneling is the process of embedding one IP packet inside of another, for the purpose of simulating a physical connection between two remote networks across an intermediate network. . FCIP will have significant difficulty in achieving the performance required by large-scale users when running through the public network. Furthermore, its cost-effectiveness will never approach that of FC/SONET.
Moving Data From New York to Chicago
Let's look at the economic considerations for moving data from New York to Chicago. We can base our analysis on the above case for New York to Boston but substitute the additional latency for the longer distance. The distance is 719 miles (1,158 kilometers). With our 50% fudge factor fudge factor - A value or parameter that is varied in an ad hoc way to produce the desired result. The terms "tolerance" and slop are also used, though these usually indicate a one-sided leeway, such as a buffer that is made larger than necessary because one isn't sure exactly how for cable length, we can use 1,737 kilometers as the distance. The one-way latency will be 8.7 milliseconds. With this increase in latency (and the same packet loss rate identified above), we see the single-user data rate drop substantially to 29Mbps. Notice that the monthly recurring cost for Internet access has remained the same and that the monthly cost for SONET has increased. This is to be expected since internet access is blind to distance, while SONET pipes are leased by the mile.
Although the cost of service has remained the same for Internet access, the single-user data rate has plummeted to 29Mbps. The time required for a single-user to move a terabyte of data is now measured in days instead of hours. For FC/SONET, the performance has remained the same but the pricing has increased to reflect the increased distance in the SONET link. The disparity dis·par·i·ty
n. pl. dis·par·i·ties
1. The condition or fact of being unequal, as in age, rank, or degree; difference: "narrow the economic disparities among regions and industries" between the cost-effectiveness of the IP-based solutions and the FC/SONET solutions has increased significantly for single-user data movement.
As with our example from New York to Boston, let's add back in the other users that would naturally fill the remaining bandwidth of the pipe.
Notice that the gap between the cost-effectiveness of IP networks and EC/SONET networks is closing, although FC/SONET remains the more cost-effective method of transferring data between sites. This is to be expected since the monthly costs for Internet access are constant and the monthly costs for SONET circuits will increase with greater distance. More important, however, is the fact that we have had to increase the number of users to 20 in order to fully utilize the bandwidth of the link. For the enterprise that wishes to move large amounts of data through the WAN, this significantly complicates the issue. In order to move large amounts of data, we must break the data into 20 different flows and manage each one as a separate TCP connection. At the receiving end, we must reassemble re·as·sem·ble
v. re·as·sem·bled, re·as·sem·bling, re·as·sem·bles
1. To bring or gather together again: reassembled the band for a reunion tour.
2. the results from the 20 separate TCP connections in order to restore the data.
Moving Data From New York to Denver
Let's continue this process to an example with even greater latency between the sender and the receiver. The distance between New York and Denver is approximately 1,629 miles or 2,621 kilometers (one way). Applying our 50% fudge factor, we will use 3,930 kilometers as the basis of our analysis. This translates into a one-way latency of 19.5 milliseconds. Recalculating the expected single-user throughput for the IP network, we can see that it has dropped to a mere 12.6Mbps. FC/SONET throughput has remained the same since a properly designed FC gateway (such as the LightSand S series) provides credit buffering and flow control specifically designed for long-haul communication. As before, Internet costs remain the same while SONET costs have increased in proportion to distance.
The trends that we observed from New York to Chicago continue as we look at moving data from New York to Denver. Notice that the "Time to Move a Terabyte of Data" has now increased to more than 8 days for the IP-based WAN. On the other hand, the FC/SONET interconnection remains constant at 4.2 hours and 3.1 hours for the OC-12 and OC-48 cases. The cost-effectiveness of IP-based transport for a single user has now plummeted even further and no one would consider dedicating an IP based link to moving a single-user's data. On the other hand, a single-user FC/SONET pipe could easily be established to move large amounts of data. It would remain very cost effective (even at large distances). It would also be a manageable process since the servers at each end would still only be managing a few connections.
Let's look at the multi-user model that would fill the remaining bandwidth of the links and see what has happened to our cost efficiency and manageability. Since a single-user can now only expect 12.6Mbps, we must have 46 simultaneous connections to fully utilize the available bandwidth across the IP WAN. This model is perfectly normal for the Internet at large (many smaller users coexisting co·ex·ist
intr.v. co·ex·ist·ed, co·ex·ist·ing, co·ex·ists
1. To exist together, at the same time, or in the same place.
2. on the same links). However, it has dramatically increased the management overhead that would be required for a single user to realistically send data from one site to another. Imagine trying to push a fire hose of data through 46 drinking straws. It might be possible to break a data stream into 46 smaller threads and reassemble it but it certainly is difficult.
Using the New York to Denver case, IP WAN transport cost efficiency (for a fully utilized link) has finally reached the same close order of magnitude as FC/SONET. Even so, the FC/SONET data transfer model still provides the most cost-effective method of moving large amounts of data across country.
Comparing the Options
Let's summarize sum·ma·rize
intr. & tr.v. sum·ma·rized, sum·ma·riz·ing, sum·ma·riz·es
To make a summary or make a summary of.
sum the options that we have examined for moving data from New York to Boston, Chicago and Denver. The chart below contains the single user cost-efficiency estimates for OC-12 Internet access as well as the single-user cost efficiency estimates for FC/SONET (OC-12 and OC-48 dedicated channels). Note that the vertical scale is logarithmic logarithmic
pertaining to logarithm.
when the logs of two variables plotted against each other create a straight line. .
As we discussed above, FC/SONET offers a significant advantage for the single-user that wishes to transfer large amounts of data from one site to another. Although it becomes more expensive (on a cost per Mbps basis) as the distance increases, the cost-per-Mbps of FC/SONET increases much less rapidly than Routed IP transport over the WAN.
The FC/SONET OC-12 link is more cost-effective than the FC/SONET OC-48 link because the OC-12 link is fully saturated saturated /sat·u·rat·ed/ (sach´ah-rat?ed)
1. denoting a chemical compound that has only single bonds and no double or triple bonds between atoms.
2. unable to hold in solution any more of a given substance. and a single user is consuming all the bits-per-second being purchased.
If we look at the cost efficiency of fully utilized links, we see some other interesting trends, as well.
Here, we see the cost efficiency of routed IP transport remaining constant (on a cost per Mbps basis). This makes sense since the pricing for Internet access is independent of distance. The price paid per Mbps for FC/SONET, of course, increases since the cost of a dedicated SONET channel goes up with the increasing distance between the cities. Even with the increase in price-per-Mbps, FC/SONET remains more cost effective than routed IP transport. Furthermore, in order to obtain the high utilization factor for IP routed networks, it is necessary for the enterprise to maintain numerous simultaneous connections (20 for a fully utilized link to Chicago and 46 for a fully utilized link to Denver).
Recent WAN-based demonstrations of IP storage 6 seem to indicate that IP can provide wide-area connectivity with high data rates at transcontinental distances. A closer look at the results indicates that this might not be so. The test results include a snapshot (1) A saved copy of memory including the contents of all memory bytes, hardware registers and status indicators. It is periodically taken in order to restore the system in the event of failure.
(2) A saved copy of a file before it is updated. of the IP carrier's performance statistics during the time of the test. At the instant of the testing, the carrier's network was operating with 0% packet loss. Since this packet loss rate is artificially low, it produces an unreasonably high estimate for the real throughput of the system. The same chart also shows that the 7-day average packet loss rate was 0.1%, the 30-day average packet loss rate was 0.07% and the 90-day average packet loss rate was 0.58%. Had these tests been conducted when the network was operating at its long-term average packet loss rate, the results would have shown a significant reduction in available throughput. Note that these numbers are averages. By definition, the packet loss rate will be even worse than these numbers dur ing a large portion of the time. This reduces the system throughput even further.
We have examined the impacts of real-world packet loss rates on the large-scale transfer of data across the IP WAN. As we have seen, routed IP networks provide amazing a·maze
v. a·mazed, a·maz·ing, a·maz·es
1. To affect with great wonder; astonish. See Synonyms at surprise.
2. Obsolete To bewilder; perplex.
v.intr. flexibility and scalability but they do so at the cost of single-user performance. Single users who wish to move large amounts of data are continuously thwarted thwart
tr.v. thwart·ed, thwart·ing, thwarts
1. To prevent the occurrence, realization, or attainment of: They thwarted her plans.
2. in their attempts to do so. Although IP routing offers the illusion of inexpensive data transfer over long distance, we have actually seen that the ability of a single user to move data significantly decreases as the distance and the amount of data are increased.
On the other hand, Fibre Channel over SONET provides a high performance and cost effective method of transferring large-scale data across the WAN. The secret to Fibre Channel's success really isn't such a secret--don't drop data. If you haven't dropped the data in the first place, there is no need to waste the precious bandwidth and time of the system by re-transmitting it.
In the end, as the requirements for large-scale data transfer increase, economics and practicality will dictate the solutions chosen by the end-user. Fibre Channel over SONET provides single user performance that dramatically exceeds anything available over the IP backbone. Furthermore, because it does not waste bandwidth and time on retransmission, it provides a significantly more cost-effective method of transferring large-scale data across the WAN.
[FIGURE 2 OMITTED]
[FIGURE 13 OMITTED]
[FIGURE 14 OMITTED]
Figure 6 Monthly recurring costs for Internet access. Monthly $ per Mbps Source Service Recurring Cost (line rate) T1 1,200 777 Published Price T3 28,000 626 Published Price OC-3 49,000 316 Published Price OC-12 98,000 158 Estimated Price Figure 7 Cost effectiveness of data transfer over the WAN (single-user, NYC to Boston). Monthly Effective 1TB Recurring Single-User of Cost of Line Rate Data Rate Data Service OC-3 155Mbps 108Mbps 23 Hours $ 98,000 Internet Access OC-12 622Mbps 108Mbps 23 Hours $ 196,000 Internet Access OC-12 622Mbps 508Mbps 4.2 Hours $ 39,000 FC/SONET OC-48 2,488Mbps 800Mbps 3.1 Hours $ 109,000 FC/SONET $ per Mbps (Single-User Data Rate) OC-3 907 Internet Access OC-12 1,815 Internet Access OC-12 67 FC/SONET OC-48 136 FC/SONET Figure 8 Cost effectiveness of data transfer over the WAN (fully utilized link, NYC to Boston). Aggregate Data Rate Number of Monthly (for Fully Users of Fully Recurring Utilized Link) Utilize Link Cost OC-12 580Mbps 6 $196,000 Internet Access OC-12 580Mbps 1 $39,000 FC/SONET OC-48 2,320Mbps 3 $109,000 FC/SONET $ per Mbps (for Fully Utilized Link) OC-12 338 Internet Access OC-12 67 FC/SONET OC-48 47 FC/SONET Figure 9 Cost effectiveness of data transfer over the WAN (single-user, NYC to Chicago) Monthly Effective 1TB Recurring $ per Mbps Single-User of Cost of (Single-User Line Rate Data Rate Data Service Data Rate) OC-3 155Mbps 29Mbps 3.5 Days $ 98,000 3,380 Internet Access OC-12 Internet 622Mbps 29Mbps 3.5 Days $ 196,000 6,760 Access OC-12 FC/SONET 622Mbps 580Mbps 4.2 Hours $ 87,000 150 OC-48 FC/SONET 2,488Mbps 800Mbps 3.1 Hours $ 277,000 346 Figure 10 Cost effectiveness of data transfer over the WAN (fully utilized link, NYC to Chicago). Aggregate Data Rate Number of Monthly $ per Mbps (for Fully Users to Fully Recurring (for Fully Utilized Link) Utilize Link Cost Utilized Link) OC-12 580Mbps 20 $196,000 388 Internet Access OC-12 580Mbps 1 $87,000 150 FC/SONET OC-48 2,320Mbps 3 $277,000 119 FC/SONET Figure 11 Cost effectiveness of data transfer over the WAN (single-user, NYC to Denver). Monthly Effective 1TB Recurring $ per Mbps Single-User of Cost of (Singl-User Line Rate Data Rate Data Services Data Rate) OC-3 155Mbps 12.6Mbps 8.1 Days $ 98,000 7,800 Internet Access OC-12 622Mbps 12.6Mbps 8.1 Days $ 196,000 15,600 Internet Access OC-12 622Mbps 580Mbps 4.2 Hours $ 168,000 290 FC/SONET OC-48 2,488Mbps 800Mbps 3.1 Hours $ 568,000 710 FC/SONET Figure 12 Cost effectiveness of data transfer over the WAN (fully utilized link, NYC to Denver). Aggregate Data Rate Number of Monthly $ per Mbps (for Fully Users to Fully Recurring (for Fully Utilized Link) Utilize Link Cost Utilized Link) OC-12 580Mbps 46 $ 196,000 338 Internet Access OC-12 580Mbps 1 $ 168,000 290 FC/SONET OC-48 2,320Mbps 3 $ 568,000 245 FC/SONET
Andy Helland is director of product management at LightSand (Milpitas, Calif.)