LanProbe fights fires; remote monitoring of 9000-user Carnegie-Mellon network.

LANPROBE FIGHTS FIRES

Frank Kietzke has a word for the network he manages at Carnegie-Mellon University.

Diversified.

"We have equipment from just about every vendor imaginable," he says.

As home to one of the nation's premier academic computer-science and engineering departments, CMU has an installed base of upcoming resources that covers the gamut of ever-evolving technologies, with mainframes, minicomputers and workstations.

Now, with one of academia's most sophisticated multivendor data communications systems, Carnegie-Mellon is a network model not only for higher education but commercial enterprises as well.

The reason: it works, and it works well.

Composed of hundreds of Ethernet, Appletalk and Token-Ring local area network subsystems, the campus-wide communications utility--known as ANDREW--connects some 2000 faculty and staff and 7000 students over an Ethernet backbone.

Centralized File

The system itself is based on what Kietzke, CMU's network manager, describes as a centralized file/distributed computing architecture.

Under this arrangement, users at workstations throughout the university's 50 buildings have minimal local storage capabilities. Instead, they rely on remote access to personal data and to executable programs which are stored on central file servers tied in over a LAN.

The "inverted" Ethernet backbone to which the various file servers and all subnetworks are connected is actually made up of two LattisNet chasses linked together over a bridge from Digital Equipment Corp. These LattisNets are equipped with interface cards to the various LANs that serve individual departments and workgroups.

Looking at a picture of the network, one might be reminded of two sets of railroad tracks that run parallel to one another. Each of these tracks contains spurs or subnetworks to various buildings on campus and also back to the main network management center.

This hypothetical "railway system," like CMU's network, is inverted; rather than extending the main railway route or backbone network to all campus buildings, a portion of each network is brought back to the main network management center.

"This way," Kietzke explains, "all network monitoring, maintenance and troubleshooting can be done from a central site. I never need to leave this room."

From his seat at a workstation in the university computing center, Kietzke's overall system management tasks include tracking not only the established network, routing and transport protocols of the various LANs in place, but also monitoring the array connectivity devices that link them together.

In this category, CMU's network hardware includes Kinetic Systems' Fastpath routers which link Appletalk nets to the Ethernet backbone, IBM AT-based routers that tie in Token Ring nets to the backbone, and "homegrown" 6800-based routers for Ethernet-to-Ethernet links.

Software snags can be disruptive. A malfunctioning network management interface card on one of the two LattisNet chasses can cause excessive packet collissions, resulting in a network-wide performance slow down or idling an entire network segment by misrouting correctly addressed packets.

To diagnose and troubleshoot these and other network problems, CMU relies on an intelligent network management product for Ethernet networks: Hewlett-Packard's LanProbe distributed analysis system.

An advantage of this system, says Kietzke, is that he and other network administrators can monitor critical aspects of all campus networks from one central site.

Each LanProbe segment monitor--CMU has two, one on each LattisNet chassis--comes with an internal 2400 baud modem, allowing remote access.

The management system software operates with Windows and contains a network mapping feature that lets users diagram network segments targeted for monitoring. These segments, as well as routers and gateways connecting them, also can be mapped.

Once this network map is generated and the LanProbe segment is attached to the end of an Ethernet segment, network administrators can collect data such as network statistics and error and collision rates.

Shows Traffic Stats

At CMU, the ProbeView Manager software runs on an IBM AT and can simultaneously display traffic statistics and segment activity in multiple windows, as all the network analysis tools run concurrently

Two tools in particular--Quick View and Packet Trace--have been used extensively at the university, especially in what Kietzke calls "firefighting situations."

He recalls: "In one case, we had a machine on an Ethernet answering address resolution protocol packets for everybody on the network. What would happen is the machine that answered all of these calls would simply throw away traffic meant for other nodes."

To find the problem node, Kietzke used packet tracing which can look at all packets or capture only specified packets.

Kietzke explains, "I got two answers back. One was from the machine that was supposed to be answering the address resolution protocol packet; the other was from the culprit, the machine that had been answering all calls.

"Without the packet filter tool, it might have taken three or four hours of tearing apart the network to try and figure out where the culprit machine was. With the packet trace, we had the network back up in less than half an hour."

As it turned out, the problem was in a router. To emphasize just how valuable LanProbe proved to be in this particular firefighting situation, Kietzke explains what would have been involved in pinpointing the trouble had he not had access to packet tracing.

"We would have needed to disconnect the network a piece at a time until we could determine that the problem was on the leg we had just pulled out. In the case of the router, we would have had to first disconnect it from the backbone, wait for things to clear, reconnect the router, then disconnect the network from the backside of the router. In this case, it would not have cleared; the router was the problem. But you can see how long it would have taken to establish that.

"A lot of times, nailing down a problem is a matter of how lucky you are at grabbing the right thing first," Kietzke notes. "But on the average, you'll need to take half the devices off of a network before you can figure out where the problem is. That's what makes a packet trace so valuable."

The simultaneous availability of packet trace and all LanProbe tools is especially helpful, says Kietzke. "If you're looking for one kind of problem and have another kind, you don't miss it."

Recently, for example, he was using LanProbe's trace facility to track down recurring Ethernet runt packets. At the same time, the LanProbe Trends graph (one of the statistics tools) was up and running to monitor for potential packet collisions.

Kietzke also awards high marks to LanProbe's QuickView feature, a bar chart type tool that shows on one screen the real-time accumulated average and peak of a segment's traffic in terms of number of packets, LAN use, bytes, broadcasts, errors and collisions.

LanProbe allows users to set thresholds and alerts for all of these parameters.

For example, a network administrator tracking down an intermittent fault could have ProbeView furnish an alert whenever CRC errors exceeded a certain limit. That alert would sound on the PC, be logged, and also show on QuickView's screen.

At CMU, QuickView has been used extensively to track down the source of collision problems, such as the one that arose recently due to a faulty interface card in one of the LattisNet chasses.

"We moved LanProbe from port to port to watch for collisions," says Kietzke.

"Then we got a report of a couple of ports with no collisions. We knew that wasn't accurate because we were still having collisions on the network. As it turned out, one of the cards for the LattisNet chassis was not reporting collisions. We had been transmitting messages not knowing that we were all walking all over other people's traffic."

Problems have occurred, but solutions have been swift. Last year, a new release of system firmware crashed upon being installed. A phone call to H-P resulted in a technician sent to the school to fix the problem, which turned out to be compiler error.