The SQL Slammer worm: how two organizations survived the attack.

The Oakland Raiders offense wasn't the only thing that got slammed on Superbowl weekend. Fans who needed twenties to cover their office pool bets got shut out of their accounts as the "SQL Slammer" worm shut down most of Bank of America's 13,000 ATMs the day before the big game. When the dust settled, this attack once again demonstrated the precarious nature of organizational security measures.

While antivirus protection and firewalls are an essential part of any organization's resources, they are essentially reactive. Software vendors are quick to release patches or new antivirus definitions as soon as one hits, but they can't necessarily predict what attack someone will come up with next. By the time they discover the problem, work out a fix, and get customers to install it, the damage is done. SQL Slammer was no exception. It took only three minutes to reach a rate of conducting 55 million scans per second as it sought to locate and spread to vulnerable computers. After that, its growth slowed only because it tied up so much bandwidth that it couldn't continue to expand.

"The successful management of malicious-code threats is a more complex enterprise initiative than simply installing and maintaining antivirus software with signatures issued by antivirus vendors," says Gartner, Inc. research director Arabella Hallawell. "Organizational processes and effective governance decisions are more important than technology 'fixes.'"

Although the worm attacked servers running Microsoft products, for once the company escaped the scathing attacks usually directed its way whenever such a virus or worm hits. This time it was administrators that took the hit, because Microsoft had issued a security bulletin on the vulnerability and a patch six months earlier. Those system affected simply hadn't had the patch installed. Ironically, some units at Microsoft's Redmond campus hadn't installed the patches and were targeted by the worm.

"It is tempting to blame enterprise system and security administrators for not addressing this issue, but it is also unfair," says Hallawell. "Microsoft has released so many patches for security vulnerabilities in SQL Server that administrators can't reasonably be expected to keep up with them all."

Patching SQL itself is one small part of the job. Every other piece of software in the enterprise has its own steady stream of updates and patches. It's a next-to-impossible task to keep up with all of them. Hallawell advocates asset management and network and systems management tools to address the problem.

Self-Service Survival

Enterprise management tools, in fact, do more to ensure reliability and security than just making it easier to deploy patches. Take the cases of Computer Associates, Inc. and Brigham Young University-Idaho. Both were hit by SQL Slammer but each organization used network management software to quickly detect the problem, isolate it, repair it and bring systems back online.

Computer Associates (CA), relied on its own tools. CA Unicenter Network and Systems Management (NMS) initiallydetected a pattern of abnormal behavior similar to what had occurred with the Code Red II worm. This matched up with alerts being generated by the CA eTrust Intrusion Detection software.

"It became clear that we were seeing a great deal of activity on a specific port, indicating some sort of SQL worm," says William Taub, security team manager for CA's Global Information Systems (GIS). "The first goal was to identify and contain the worm, free up the network and disable SQL network traffic by closing the appropriate ports."

The reports generated by Unicenter Asset Management showed that the bulk of the production systems had already been patched, but that certain other machines, including ones in the lab, remained vulnerable. GIS stopped the SQL processes on these machines to prevent the spread of the worm. It then used Unicenter Software Delivery to patch the affected machines.

"When Microsoft made available an updated SQL roll-up patch to address the emergency, we immediately deployed it to the systems that were not yet patched," says Taub.

Most organizations, though, are not multi-billion dollar operations that spent the last two decades producing enterprise-class management software. Nevertheless, the same principle of using network management software to mitigate the damages caused by cyber-attacks applies to other entities.

Brigham Young University-Idaho's IT infrastructure is much simpler than CA's. The school has a 3000-node network with gigabit Ethernet in the backbone core and 10 or 100Mbits going out to the desktops. It also is in the process of rolling out 802.11b wireless nodes. The servers are primarily Windows 2000, with some Linux back end apps and an IBM iSeries server running OS/400. Desktops are mainly Windows 2000/XP with some Linux and Mac machines. The university uses Microsoft Exchange, but the main enterprise apps such as admissions and finances are all homegrown.

But it wasn't until this year that BYU-I had any type of management software in place. After reviewing various products such as Tivoli and HP Open View, the university opted to go with a lower-cost, web-based network and systems management package from Somix Technologies, Inc. called WebNM.

"We looked at various other network management platforms and software packages, but they have long implementation times and can be expensive," says BYU-I network manager Michael Rydalch. "WebNM has most of the functionality we needed.

BYU-I bought the software in the nick of time. A Somix technician spent four days installing and configuring the management software, finishing the job on January 23, one day before SQL Slammer hit. The technician set up WebNM to monitor CPU and disk utilization on all the servers. On switches and routers, the software monitors CPU utilization and the interface to the ISP. In addition to providing real-time information on hardware or service status and performance, it also generates graphs showing long- and short-term trends for any of the monitored parameters. In addition, the university uses WebNM's hardware and software inventory module known as OStivity.

SQL Slammer hit a server in the university's DMZ on January 24 at 11:30 pm, a little more than a day after the Somix technician left. The network operations analyst on call that weekend received an alert from the university's response center about problems coming from an unidentified source. After coming into the office, the BYU-I staff checked the WebNM graphs and narrowed the problem down to excessively high utilization on a perimeter--router--probably caused by a virus or worm. After checking the emails and alarms, the server administrator logged onto CNN and found out what was known about the worm at that--time--that it was attacking databases. Based on this information, they next used OStivity to rapidly identify all systems running either SQL Server 7.0/2000 or the Microsoft Data Engine (MSDE). They shut down the necessary TCP and UDP ports at the perimeter and cleaned up all infected servers.

Constant Vigilance

As can be seen having network and systems management software in place didn't prevent the attack. Despite the billions spent annually on antivirus software, firewalls, intrusion detection systems and packet sniffers, attacks will keep occurring, and people will continue to find methods to exploit the security holes. But management software helps provide another line of defense by alerting personnel of unusual patterns and helping them to repair the damage or contain the threat.

Drew Robb is the senior writer at Robb Editorial (Los Angeles, CA)