Anchorage Water and Wastewater Utility's Year 2000 Remediation Project.

Anchorage Water and Wastewater Utility has taken the Y2K threat very seriously and implemented a project to prevent computer collapses and interruptions.

Anchorage is a mid-size, modern city with approximately 250,000 residents and a climate comparable to upper Michigan, Wisconsin, and Minnesota. Since Anchorage's geographic location precludes the opportunity to participate in regional water and wastewater systems that could provide redundancy for backup, the Anchorage Water and Wastewater Utility (AWWU) is a stand-alone enterprise that must be able to provide consistent and uninterrupted service to its customers without external assistance. The Anchorage Water and Wastewater Utility serves approximately 50,000 accounts consisting of single and multiple family dwellings, businesses, industrial, and military customers.

The Y2K Problem 

The Year 2000 (Y2K) computer glitch is a date problem. It can cause a computer to mistake the year 2000 for 1900. This is probably the most pervasive problem but not the only problem and possibly not the most serious. Some computers may incorrectly determine that the year 2000 is not a leap year thereby causing additional problems. Furthermore, certain programming conventions may lead to problems around the turn of the century.

These are all real problems with a dramatic scope. The date rollover presents an impossible situation for many computers. If the computer's operating system does not intercept the problem and terminate the offending program, the whole computer will shut down. The computer could be a microprocessor in a piece of equipment that the owner never suspected was computerized. Indeed, the problem ranges far beyond what are traditionally thought of as "computers."

AWWU's Y2K Project

AWWU's Y2K project formally began in 1996, but programmers had known of the issue for some time. The project began by evaluating systems for Y2K compliance. It was determined that older systems were not compliant, while those developed recently were compliant. Even though AWWU originally planned to hire consultants to develop the Y2K plan, it became clear that internal resources could better and more economically develop this plan. AWWU is currently using a combination of outside consultants and internal resources to implement the plan. In 1997, AWWU inventoried its computer systems and began the final Y2K readiness assessment and remediation project.

There has been an obvious evolution of thinking regarding the Y2K problem. Initially it seemed like a simple, although extremely large problem. As industry understanding of the problem evolved, it turned out to be anything but simple. It now appears that even the industry's pessimists underestimated the pervasiveness of the problem.

As remediation of the problems discovered in the 1997 assessment nears completion, problems subsequently identified are still being addressed. These include issues such as problems with microprocessors in equipment that was not known to be computerized and dealing with failure of business partners, some who may have failed to adequately address their problems. As AWWU moves into 1999, it's Year 2000 program is on track and ahead of schedule.

The Y2K Compliance Strategy

Strategy Development. Formulating a Y2K strategy for AWWU was not an easy task. An information systems professional might hope to be able to rely on industry publications and the mass of information available on the Internet for insight. It seems the Y2K problem has brought out the technical tabloid writers en masse. The valuable information is diluted in a sea of doom prophesies, generalizations and redundant information. Everyone has something to say; most of which is very repetitive. Opinions range from "the sky is falling and not only that - it's worse, there are two layers and they will both fall," to "what's the big deal?"

It was incredibly difficult to determine which advice was credible. Moreover, no one knows what will happen on January 1, 2000. What is necessary and prudent, and what is just hype? AWWU determined that these decisions had to be made internally. These did not always align with standard industry thought, but AWWU felt a responsibility to choose internal strategies in which there was great confidence.

AWWU's Y2K program addresses the following objectives:

* finding and fixing potential problems;

* responding to evolving understanding of the Year 2000 problem;

* addressing user-developed systems - spreadsheets and PC databases;

* developing contingency options;

* testing;

* spending the correct amount; and

* developing a Year 2000 Statement.

Finding and Fixing Potential Problems. As its first line of defense, AWWU has adapted a strategy of locating and correcting Y2K problems. When the formal year 2000 effort commenced, the initial task was to inventory and evaluate its systems - itself an evolving task. This process refined understanding the issue and the overall strategy. AWWU went through several iterations before arriving at the final inventory and plan. In some areas, the inventory approach eventually took a backseat to a more global strategy. For example, with desktop workstations the concept of inventorying and certifying individual software products was abandoned and replaced with the development of a standard office software set to support. Ultimately, a strategy was implemented for identifying and addressing problems in each class of product.

Response to Evolving Understanding. Nationally, a tremendous amount of money is being spent on Y2K issues and problems. Much of the information obtained from these efforts has become available on the Internet, through industry periodicals, and from vendors. This has prompted a continuous evolution in thinking about the Y2K issue. As problems continue to be found in previously unsuspected places, the understanding of the year 2000 issue advances. A critical part of the AWWU's Y2K strategy is to remain abreast of the evolution of the worldwide understanding of the problem and adjust efforts accordingly. The project manager is taking the lead on monitoring Y2K information on the Internet and in periodicals.

User-Developed Programs

With the evolution of PC-based spreadsheets, databases, and many other office software products, it now seems that there are two ways to develop a system: 1) have a contractor or the IT Division develop it at a substantial cost or 2) develop it yourself for a small fraction of that cost. Even though a system developed in a few hours can have some major limitations, it allows employees to quickly develop a functional, cost justified information system. Prior to the spreadsheet and database packages, that possibility did not exist. As part of the Y2K inventory, all network storage devices were searched for unregistered spreadsheets and PC databases. Approximately 2,000 unregistered spreadsheets and 200 PC databases were found.

AWWU has taken a common-sense approach in addressing user-developed systems. What do end users do when a problem occurs in one of their spreadsheets? They either fix it or retire it. The bottom line is that the Y2K issue is not a justification for interfering with the users' ability to build and maintain their own economical system.

AWWU's strategy for this problem is to ensure that the underlying software is at least Microsoft Office 97, that all applicable patches have been applied, and to ensure that users are aware of the Y2K problem. The strategy also includes offering the support of the Information Technology Division to those who request it. A few complex spreadsheets within the utility are considered to be critical to the utility's operation. These are considered to be enterprise information systems, not end user spreadsheets, and have been addressed individually.

Developing Contingency Options.

* AWWU does not expect to prevent or correct all Y2K problems but it does expect to find and correct or mitigate most of the problems.

* AWWU expects to prevent undetected problems from causing interruptions in the service to customers.

* AWWU expects to find a high enough percentage of the problems that the few that arise in the year 2000 can be quickly addressed. However, there is no reasonable way to guarantee that a problem will not escape detection. A hardware or software component that was certified to be Y2K compliant could fail. There is no way to fully test bugs out of computer components for which the source code is not available.

* AWWU is vulnerable to the potential mistakes of the computer equipment providers, as well as to internal human limitations.

AWWU is developing contingency plans to ensure effective operation even in the face of some Y2K-related equipment failures. This includes having extra employees on duty on January 1, 2000, inspecting remote sites, and verifying that everything is functioning correctly. Most importantly, AWWU plans to continue to provide a safe, adequate, and uninterrupted supply of drinking water, and effectively collect and treat wastewater.

Testing. Much of the available Y2K literature suggests that both public and private entities should be spending massive amounts of money on testing. This is another area where AWWU developed its own strategy based on proven systems analysis and testing concepts. AWWU takes an item of hardware or software, spends some money developing a test plan, and then spends some money testing it. The product of the testing is an indication, and the indication has some degree of reliability. Simply because the testing revealed no problems, one should not conclude that there are no problems. There are some expenses and some results and AWWU needs to make sure that the results justify the expenses. And, of course, there is the possibility that the test itself will cause more damage than the Y2K problems.

Except for custom software for which AWWU possesses the source code, AWWU is relying primarily on methods other than in-house testing to assess year 2000 readiness and compliance. These are discussed elsewhere in this document.

Spending the Correct Amount

From the beginning AWWU has treated the Y2K remediation project as any other project with respect to cost/benefit justification. How much should be spent to fix a problem? Some problems are cheaper to prevent than to fix, but some are cheaper to fix than to prevent. The Y2K problem must not be a license for irresponsible spending. Cost justification has been a critical factor in the strategy development, but cost has never been allowed to override public health and safety concerns.

Developing Year 2000 Statements

Just as AWWU must be concerned about the failure of business partners, they must be concerned about AWWU's reciprocal ability to operate effectively into the Year 2000. AWWU receives inquiries from customers about year 2000 readiness. Some even demand statements certifying Year 2000 compliance. AWWU customers need to be reassured that Y2K problems are being addressed.

Organizations are growing reluctant to certify their systems to be Year 2000 compliant. A favored term now is Year 2000 Ready. This is usually defined to mean that the entity thinks it is Year 2000 compliant, but it might not be. With the help of the Municipal Attorney's Office, AWWU has developed a standard Year 2000 Statement. AWWU offers the statement in response to inquiries from customers and has posted the statement on its Web site.

Implementation

Custom Business Software. When the Y2K program began, this category was called custom software; however, it was soon discovered that that the custom software was really business software. For software developers, this is the easiest area of the Y2K issue to attack, although not necessarily the cheapest.

For the few client/server systems built in recent years with modern development tools, the task of certifying Y2K capability was delegated to the vendors. Since the software used internal database date storage formats and functions for processing dates, AWWU relied on the strategy of keeping the database software and licenses current so that it would be in a position to receive any patches that might be necessary for Y2K compliance. For the remaining systems, which were mostly COBOL-based (one FORTRAN), the decision about Y2K compliance was based exclusively on source code analysis rather than testing. It became obvious that none of the COBOL or FORTRAN systems were Y2K capable; if this software looked Y2K capable, testing would have been appropriate. However, that was not the case.

The next decision point for each system was whether to replace it or upgrade it. Luckily, almost all of the noncompliant custom software was already scheduled for replacement before the year 2000. Although this was a relief to the Year 2000 team, it added some constraints to the software replacement projects. They were now part of the year 2000 solution and could not be allowed to slip into January of the year 2000. Unfortunately, AWWU's most complex business system, the Customer Information System (CIS), was neither year 2000 compliant nor was it scheduled for replacement.

A separate project was initiated for the CIS upgrade, under the leadership of the Y2K project manager. The upgrade was treated like a new software development project. AWWU spent a month performing high level analysis, gathering metrics, modifying the database definition, and developing a detailed workplan. An interesting twist occurred in the design process. AWWU went into the project with the attitude that this is a legacy system that the utility wants to replace in the near future but could not replace it fast enough to meet the year 2000 deadline. The team agreed that the appropriate action would be to do the minimum required to produce a system that would be stable and dependable into the year 2000. There was unanimous agreement within the development team that the database structure would not be modified. Instead, AWWU would use a common programming technique called windowing that assumes dates are in the range 1950 through 2049 instead of assuming they are all in the 1900s. But as the process was analyzed, the thinking changed. The system has almost a thousand modules and the team concluded that it would be far easier to modify the database than to insert windowing code everywhere a date was manipulated.

Five months were spent analyzing each program and developing specifications detailing each change that this would require. Each subsystem was modified individually, each program was tested, and then the subsystem was tested. System testing is scheduled for completion by December of 1998. The project is now about six weeks ahead of schedule and costs are coming in at about 80 percent of the estimates. The upgraded system is scheduled for production after the database upgrade in the first quarter of 1999.

Telemetry and SCADA 

Telemetry is the mechanism for communicating between the control or sensing equipment and the monitoring and control panels. This equipment controls and monitors the water distribution and wastewater collection and treatment systems.

The Supervisory Control and Data Acquisition (SCADA) equipment is one of the greatest areas of concern because problems could have immediate and direct effects on the service to customers. Also, the potential for disastrous results is greater than with business systems. A bug in the billing system could destroy a database and cause considerable cost by requiring reconstruction from backup tapes. However, the consequences of a pressure-regulating valve malfunctioning and causing a main water input line to burst or a chemical regulating system failure have far more serious implications.

Certain problems simply cannot be allowed to happen under any circumstances. AWWU needed a comprehensive problem abatement program, but that was not enough. AWWU adopted the position that there is no way to be sure that every Y2K problem will be detected and fixed. Additional measures were needed. The likelihood of problems as well as the potential severity of (even unlikely) problems was reviewed.

1) AWWU needed to look at each class of equipment and determine the worst thing that could happen if it failed.

2) AWWU needed to determine how to mitigate the problems.

As AWWU worked through the above issues, it became apparent that two obvious concepts had been missed. First, the equipment can fail, and in some cases already has failed, regardless of the Y2K issue. AWWU has continued to provide consistent and uninterrupted service in spite of these failures. Secondly, wherever there is potential for catastrophic failure, it really does not matter if the failure is caused by the Y2K date problem. It is the failure that is the problem and this could happen anytime. So, to some extent, AWWU broadened its perspective to include a complete review of dependability and contingency issues surrounding the SCADA equipment. At the same time, AWWU has continued to emphasize the timing aspect of the year 2000 and the necessity of having mitigation measures and contingency plans in place by then and preferably sooner.

The SCADA area presented another challenge. The amount of equipment that could not be reliably certified as Y2K compliant exceeded the ability to replace all of it before the year 2000. This called for some creativity. For some of the equipment, reporting an incorrect date could be tolerated though it would cause inconvenience. For this equipment, AWWU will set the clocks back. Preliminary tests have been conducted and a final, comprehensive test will be conducted this winter when the system is at the lowest point on the load cycle.

All of the SCADA systems can be bypassed to allow manual operation. Some equipment is thought to be Y2K compliant, but could produce highly undesirable results if it failed. This equipment will be switched to manual operation before January 1, 2000, and then, item by item, switched back to automatic operation after January 1, 2000, and closely monitored until it has been determined that it is functioning correctly.

Network Hardware and Software

This is another area where testing turned out to take a back seat to other methods. AWWU is dealing with equipment that can be tested much more meaningfully by the manufacturer. The strategy consists of using trusted vendors, then monitoring their Y2K efforts. AWWU is making sure that all hardware and software is certified Y2K capable by the vendor and is reasonably current (generally not over three years old as of January 1, 2000). AWWU ensures that there is a maintenance contract, reads the vendor's Y2K literature and applies any appropriate Y2K patches. Two older servers that could not be certified to the satisfaction of the AWWU are being phased out. SCADA servers are addressed in the SCADA and Telemetry plan and are not included in this equipment class.

Desktop Hardware and Software. AWWU's year 2000 strategy with office hardware and software is synonymous with the overall strategy for dependability. Although the utility does not endorse any single product it believes that diversity can have detrimental effects in the areas of training, maintenance, and document sharing when it comes to office software. There may be a number of functionally equal products but we selected the Microsoft Office package as the standard. On January 1,2000, all of the utility's desktop operating systems will be Windows NT Workstation 4.0 or 5.x. No office software older than Office 97 will be in use on January 1,2000. Patches will be downloaded on July 1 1999, and always kept current. In the last quarter of 1999, IT staff will make the rounds of all workstations and apply all patches.

A PC software inventory has been performed and considerable diversity of software was found. AWWU could easily spend more money evaluating the Y2K compliance of products than the products are worth to AWWU. Reducing diversity and bringing existing software and licenses up to the current levels were major parts of the Y2K strategy. AWWU's Year 2000 Project will not address Y2K compliance of products that are alternatives to the Utility's supported products. If these packages fail to operate properly because of Y2K problems, the users will need to switch to products supported by AWWU.

Commercial Off-the-shelf Software. AWWU performed an inventory of commercial off-the-shelf software, excluding office software, and found relatively few products in this category. None of them could be certified Y2K compliant to the satisfaction of the Year 2000 Project Manager. Each system has been evaluated and strategies have been adopted for resolving Y2K problems. Resolution strategies include upgrade of license, replacement, and phasing out. For systems to be retained, the solution is to obtain a maintenance agreement that automatically keeps the software current into the Year 2000 so that, for any problems identified and solved by the vendor, the fix will automatically be applied.

Non-IT Equipment

Traditional computer systems were the original source of concern for Y2K issues. However, equipment such as lighting, heating, elevators, and telephone systems sometimes contain computer chips that are date sensitive. Y2K problems in unanticipated places are continually being brought to light. AWWU needed to inventory and assess all equipment and systems that could conceivably contain a microprocessor that works with dates, then assess the potential impacts of failure of the equipment and, where appropriate, upgrade or replace equipment that is not Y2K compliant with equipment that is. The utility's Operations and Maintenance Division has been collecting manufacturers' data to determine where problems may exist so as to get them corrected.

Business Partners

Year 2000 project managers realize that preparing their internal systems for the year 2000 is not enough. Current industry data suggests that a significant number of organizations have waited too long and will not be ready to achieve Y2K compliance in time. Some authors suggest that entities should be evaluating not only their own Y2K readiness, but also the Y2K readiness of the rest of the world. What others do (or do not do) with regards to Y2K preparedness affects everyone.

AWWU is very concerned about its business partners and is trying to pursue a rational approach for self-protection to the extent that is possible. These seem to fall into two categories. The major suppliers are the power and communication suppliers and other important suppliers such as those who supply and deliver chemicals and equipment. The major suppliers are the ones whose problems could have the most immediate and drastic effects on us. But these are the easiest to monitor because they are local, fixed entities and several are municipally owned.

Other suppliers include a delivery chain which is probably the most impossible to assess and one of the most susceptible to failure because of the large number of small, independent shippers. AWWU's strategy has been to gradually build up the supply inventory to enable bridging some minor worldwide disruptions in the supply chain. Presumably some supply shortages will likely start to develop long before the year 2000 rolls in. This has been factored into the contingency planning as well.

Resource Availability

For the most part, the year 2000 gloom and doom - shortage of programmers and sky-hi prices - never materialized. At the end of 1998, AWWU continues to be able to secure the services of competent analysts and programmers at reasonable prices. The number of computer professionals is adequate to support the demand.

Conclusion

AWWU considers the Y2K problem to be very real and has taken it very seriously. This Y2K project manager feels that many have overstated the problem with gloom-and-doom prophesies and irrational recommendations; however, this may not be all bad. The problem is very pervasive and perhaps more organizations are underestimating than overestimating it. AWWU's Y2K team has complete support of the general manager and management team and has been provided with the resources to do the job. With the continuing effort by all, AWWU expects to be ready for January 1, 2000, and continue to function effectively into the next century.

LARRY W. BRYS is the Anchorage Water and Wastewater Utility Year 2000 Project Manager. Questions about AWWU's Year 2000 project may be addressed to [e-mail].