Multi-Homing Provides As Many Internet Routes As You Have ISPs.

This article is the second in a two-part series.

Multi-homing connects a site to the Internet through more than one link, whether an Ethernet, T-1, T-3, or other pipe. Sites choose multiple connections for three basic reasons: redundancy, load balancing, and performance tuning, listed here in the order of management difficulty. A site may be multi-homed to a single ISP, which achieves only link redundancy. To attain ISP redundancy as well as the ability to load balance and tune performance a site must connect to multiple ISPs; of course, the connections to each of those may be multi-homed as well.

To qualify for multi-homing to multiple ISPs, a site will generally be required to obtain its own Autonomous System (AS) number and apply for at least a /24 block of its own P addresses (256 addresses, or what used to be called a Class C address block), rather than using addresses owned by the ISPs it's connecting to. The complications that can ensue from trying to announce addresses owned by one ISP through another are beyond the purview of this article.

The Ins And Outs Of Multi-Homing

While a site with its own AS and set of IP addresses may announce routes to attempt to control the way traffic reaches the site, this aspect of BGP management is somewhat arcane, and subject to interference or denial by the practices of the ISPs supplying the connectivity to the Internet. Their filtering policies can effectively mask a site's routes by aggregating them into a larger address block, and usually do to avoid the strain on the global routing tables. An aspect of BGP called communities--a kind of subgrouping within an AS--can give sites more control over how their route announcements are distributed and acted upon, and many ISPs offer communities for this purpose.

Nonetheless, the degree of autonomy available for sites wishing to announce their own routes to balance incoming traffic is limited, and with web sites, at least, traffic is so asymmetric (often 20:1 in favor of outgoing traffic due to the nature of HTTP), that such balancing is of limited benefit. Some excellent information on route announcements with BGP (aimed at ISPs) can be found at http://avi.freedman.net.

The options for controlling the routes taken by outbound traffic are limited too, of course, since multi-homed sites can only see the routes announced to them by the border routers assigned to them by each ISP they multi-home to: a single route for each destination reachable through a given ISP. Nonetheless, this gives a multi-homed site as many different paths through the Internet to a given destination (visitor, customer, business partner, employee) as it has ISPs. This ensures that should one ISP go down, the others will pick up the slack (assuming the links to them have sufficient capacity for the diverted traffic), giving a multi-homed site full redundancy. The fail-over is automatic with BGP or one of the hot standby protocols (HSRP and VRRP), so this redundancy comes at very little cost.

Leave The Driving To Us

The other benefits of multi-homing--load balancing and performance tuning--require more skill and management time, and many sites opt to rely on the fact that it's rare for a well-managed ISP with good connectivity (lots of peering and/or transit arrangements with backbones) to fall completely. For instance, XOR (www.xor.com) is a provider of customized e-business solutions that is itself multi-homed via T-3 links to five different Tier One (national or global) backbones as well as InterNAPs routing service (discussed below), and offers its customers redundant connections in the data center. Ned McClain, director of infrastructure engineering, notes that the company's engineers adjust the BGP tables daily to maintain optimal connectivity to the Internet for customers whose applications they have developed and host. "Many multi-homed sites will opt to balance for cost," he says. "But we're focused on routing around slow spots on the Internet, which requires routing and BGP expertise many customers don't have o r can't afford."

Likewise, e^deltacom (www.edeltacom.com), a hosting and managed services provider in the Southeast, offers BGP management as a managed service to customers in its 367,000 square foot data center. "We peer to four major backbones at three points around our regional packet-over-SONET ring," says Dave McGirt, vice president of engineering and CTO of the company. "Each customer in our data center has redundant Ethernet links to the Internet, and we manage their routing for them using the Keynote Global 50 benchmark and our own tools to monitor their connectivity to points around the world."

The tools used by companies like e^deltacom are generally home-grown scripts based on trace-route and ping (ICMP-based applications that use TCP fundamentals like the Time-To-Live variable in packets to discover routes and the condition of network elements) to probe the Internet, usually in conjunction with synthetic traffic-based probes from companies like Keynote. "We use Keynote and scripts that trigger every 60 seconds to gain a broad overview of Internet conditions," says Kevin Martin, founder and CEO of Pair Networks (www.pair.com), a hosting provider that maintains 4 DS-3 and OC-3 link to the five backbones it is multi-homed to. He notes that Pair generally makes BGP changes to respond to problems as needed, and then goes through the BGP tables in depth about every three months to tune performance more closely based on what they've seen during that period.

Tools For DIYs

Those sites that want to do it themselves can avail themselves of a growing number of products and services aimed at enabling their customers to optimize Internet routing. These range from tools aimed at giving better visibility into Internet routing to full-service offerings that manage routing decisions for a self-hosted site.

As an example of routing tools, consider the offerings from CAIMIS Inc. (www.caimis.com), a spin-off from the Cooperative Association for Internet Data Analysis (CAIDA) effort at the UC San Diego Supercomputing Center. CAIMIS is developing a suite of traffic engineering tools that includes Skitter and RouteReporter. Skitter is a distributed system for monitoring and reporting on network latency, packet loss, and path stability. RouteReporter is a tool that gives users deep visibility into the BGP status of a network. Although Route Reporter is actually more intended for network providers than multi-homed sites, it is still an example of the new generation of tools being offered to address the growing importance of Internet routing. In addition, a sister company, CAIMS Geo, supplies software (IPMapper) for geographically identifying IP addresses by country, state, city, latitude-longitude, and postal codes where available, which can be useful when making routing decisions.

Method Networks (www.methodnetworks.com), on the other hand, offers, among other products, the Method Path Director, for outbound route management and Method Domain Director, for inbound routing control. In sum, Method Network's offerings comprise a managed service plus peering points that allow customers to monitor connectivity to the ASs their site visitors use to reach them, with control of both inbound and outbound routing. They can choose from several different transit providers (ones with whom Method has connectivity) for each major visitor AS.

Method is going up against an established competitor in Internap Network Services Corp. (www.internap.com), which has established an overlay network by peering arrangements with many major backbones, along with a network operations center that monitors backbone conditions in real time and controls their customers' routing accordingly. As noted above, some sites consider Internap a backbone to buy transit from in its own right, underscoring the power of routing management.

One limitation common to services like Method and Internap is, of course, that a site must be able to connect to a local point-of-presence for the service. This is not a problem for sites located in the major cities where these services have connectivity, but the expense of a leased line to the POP from any distance can eliminate the advantages of a routing management service (connecting through the Internet is obviously rather self-defeating). There are other companies that appear to be attempting to develop routing solutions that don't require an overlay network or POP, but all of these are currently holding their cards close to their vests, so little information is available about them.

Among these may be NetVMG Inc. (www.netvmg.com), which apparently intended to offer a managed service plus a Flow Distribution Platform (FDP) that might have been a box for the customer site, controlled by a central NOC. However, their business plan apparently changed suddenly this spring, and their current plans aren't known. Even more stealthy are Sockeye (www.sockeye.com), spun out of Akamai, and Speedtrak (www.speedtrak.com). Watch their web sites for future announcements.

The first part of this article appeared in the June issue of CTR.