HUMAN ERROR DISABLED AOL : ENGINEERS THOUGHT UPGRADE CAUSED PROBLEM.

Byline: Daily News Wire Services

America Online Inc.'s 19-hour blackout was caused when an operator transferred erroneous information to its main computer while the system was being upgraded and couldn't detect the mistake.

Wednesday's trouble was compounded because engineers thought the problem was caused by the upgrade they were performing, and they were looking in the wrong place.

The system outage stranded AOL's 6.2 million subscribers in the most notorious glitch an on-line company has experienced.

``This is not a systemic type problem,'' chief financial officer Len Leader said. ``This is a result of a sequence of events we don't think will be repeated and also involved human error.''

In the wake of the crash, rivals moved quickly to lure away angry subscribers.

Still, things weren't all bad for the on-line giant: The company reported its profits more than tripled to $16.1 million in its fiscal fourth-quarter.

But the crash was the talk of cyberspace Thursday, and here's how AOL and consulting engineers recounted the story:

America Online closes down every other week on Wednesday from 1 to 4 a.m. PDT for maintenance. This week, the company planned to install new software and replace switches within the internal network of its main data center in Virginia. That work proceeded routinely.

At 2 a.m., an AOL subsidiary called Advanced Network Services was performing maintenance on its computer and erroneously changed routing instructions that it sends to the main AOL system. Routing instructions are like maps that tell units of data where to go, steering electronic mail to the right person, for instance.

AOL's system was still down when the erroneous changes came from Advanced Network Services. It was thus unable to diagnose that there was trouble in the new routing instructions.

So, when AOL's technicians tried to get their system running again, none of the data went anywhere, because suddenly it didn't have the right maps.

Thinking the trouble was caused by the changes that had been made by their new software, AOL's technicians began to remove it.

``At 6:30 in the morning when you're supposed to go back up at 7, you don't ask what the problem is, you just reverse your steps and return the network to the state it started in,'' said Matt Korn, the company's vice president for operations.

That turned out to compound the trouble. When the new software had been pulled off and the system returned to what they believed was its original state, it didn't work.

The technicians meticulously went over their backout procedure, making sure they'd done everything correctly. Technicians from companies that provide equipment to AOL arrived to look at subsystems.

The engineers desperately tried to figure out what they possibly could have done wrong to knock out the system.

They had not considered that new instructions could have come in from elsewhere, in this case, the Advanced Network Systems computer.

A California engineer who sometimes consults with AOL and asked to remain anonymous said the AOL group acted on the assumption that's always made when trouble occurs during a system upgrade: ``The thing you changed is probably the thing that's broken.''

But after many hours, ANS technicians found the trouble in their routing information. ``Once they corrected it, we were on the road to having a stable network,'' Korn said.

Service was restored at 7:45 p.m. PDT.

Fledgling services - especially direct-access Internet and e-mail providers that are eager to gain a foothold - jumped on AOL's misfortunes.

``We would hope that people would look at us as an alternative for their e-mail needs,'' said Sabeer Bhatia, CEO of HoTMaiL, a Web-based e-mail service.

``Being on the Internet, we don't have to worry about the infrastructure folding because of lots of people accessing our service.''

Another small firm, 1.800 eMail, which lets users load software on a laptop to do business anywhere without a local access number, pushed itself as a reliable alternative.

Rival Prodigy highlighted its backup systems, which it said would prevent a similar crash. ``There's only so long that a consumer will tolerate `busy, try later,' '' said Prodigy spokeswoman Carol Wallace.

In its earnings report, AOL announced that it posted profits of 14 cents per share, slightly below Wall Street forecasts of 16 cents per share.

Earnings would have been higher, except for the $8 million cost of settling lawsuits that questioned its former practice of rounding up charges to the next highest minute.

Excluding the special legal charge, which came to $4.9 million after taxes, the Virginia-based company earned $21 million, or 19 cents a share.