Automatic electronic laboratory-based reporting of notifiable infectious diseases at a large health system. (Research).

Electronic laboratory-based reporting, developed by the UPMC Health System, Pittsburgh, Pennsylvania, was evaluated to determine if it could be integrated into the conventional paper-based reporting system. We reviewed reports of 10 infectious diseases from 8 UPMC hospitals that reported to the Allegheny County Health Department in southwestern Pennsylvania during January 1-November 26, 2000. Electronic reports were received a median of 4 days earlier than conventional reports. The completeness of reporting was 74% (95% confidence interval [CI] 66% to 81%) for the electronic laboratory-based reporting and 65% (95% CI 57% to 73%) for the conventional paper-based reporting system (p>0.05). Most reports (88%) missed by electronic laboratory-based reporting were caused by using free text. Automatic reporting was more rapid and as complete as conventional reporting. Using standardized coding and minimizing free text usage will increase the completeness of electronic laboratory-based reporting.

**********

Public health surveillance of infectious diseases is crucial for detecting and responding to illnesses that may represent potential outbreaks or bioterrorism events (1,2). The Centers for Disease Control and Prevention (CDC) is collaborating with state health departments to improve current disease surveillance by using a standards-based information architecture through the National Electronic Disease Surveillance System (NEDSS), which includes electronic laboratory-based reporting of certain diseases to local, state, and federal public health authorities (3-5). Automatic reporting at private clinical laboratories in Hawaii has been shown to be more rapid and complete than conventional reporting (6).

Allegheny County (population 1,348,000) is located in southwestern Pennsylvania and includes the city of Pittsburgh. Incidences of notifiable diseases in the county are required by law to be reported directly to the Allegheny County Health Department (ACHD). Each notifiable event is recorded on a case report form that is mailed or faxed to the ACHD by laboratory personnel, physicians, nurses, or infection-control staff; this procedure constitutes the conventional paper-based reporting system (Figure 1). A notifiable event is considered reported when received and confirmed by the health department.

[FIGURE 1 OMITTED]

The UPMC Health System is a large university-based health-care network consisting of approximately 20 hospitals and hospital affiliates (http://www.upmc.edu) in western Pennsylvania and is affiliated with the University of Pittsburgh School of the Health Sciences. UPMC established real-time, electronic laboratory-based reporting. This system is based on an existing hospital communications infrastructure designed to improve the speed and completeness of reporting (Wagner MM et al., unpub, data). ACHD personnel estimate that 40% of all notifiable infectious diseases reported to the ACHD come from UPMC. We evaluated the accuracy, completeness of coverage, and timeliness of electronic laboratory-based reporting before its integration into the conventional paper-based reporting system (7).

Background

Eight UPMC hospital microbiology laboratories in Allegheny County are capable of electronic laboratory-based reporting by using Health Level 7 (HL7), an electronic messaging standard for data exchange and communication between health-care information systems (http://www.h17.org). Laboratory personnel and health-care providers who obtained results from these laboratories were required to report through the paper-based system and were unaware of the establishment of new electronic reporting (Figure 1). Once a laboratory technician obtained a test result, he or she entered the information into the hospital laboratory computer, which generated an HL7 message. Although laboratory workers could enter test results by using preprogrammed codes or free text (non-coded, nonstandardized text entered by laboratory personnel), the electronic laboratory-based reporting system monitored only coded organism names in each HL7 culture message.

The processing occurred in real time, i.e., messages were checked as they were received. Instead of a batch mode in which data are extracted from sets of reports at predetermined times, extraction of information occurred whenever data were received by the electronic system. The electronic system extracted the specific laboratory specimen, procedure, and result from the HL7 records. The data were then interpreted; a laboratory result was positive or negative based on the code in the result section, which was compared with a data dictionary developed by the UPMC Health System. Some duplicate records were recognized by the electronic system. The extracted three- or four-letter coded organism name, defined by the UPMC laboratory system data dictionary, was then convened to its full name through a translation table maintained in the Oracle (Oracle Corporation, Redwood Shores, CA) data storage; computer personnel could use the Oracle data storage to add or remove an organism they wanted to be monitored. To obtain complete patient demographic information, if not provided in the laboratory HL7 message, the electronic laboratory-based reporting system queried the Medical Archival Retrieval System at UPMC based on the patient's medical record number in the message (8). Simultaneously, an electronic mail message containing the laboratory test result was sent by the electronic reporting system to selected UPMC personnel. Of note, the loop for direct automatic reporting between UPMC and ACHD was not completed at the time of this evaluation and, thus, no notifiable events were reported by the electronic system to the ACHD.

Methods

We conducted a comparison evaluation of the UPMC electronic laboratory-based reporting and the ACHD conventional paper-based reporting systems. From the eight UPMC hospital or affiliated microbiology laboratories with HL7 links in Allegheny County, we compared all disease reports in the UPMC electronic and the ACHD paper-based systems (derived from the National Electronic Telecommunications System of Surveillance) databases with dates of positive culture from January 1 to November 26, 2000, for 10 infectious organisms: Campylobacter, Cryptosporidium, Escherichia coli O157:H7, Giardia, Listeria, Legionella, Neisseria meningitidis, Salmonella, Shigella, and Yersinia. The diseases caused by these organisms are notifiable to ACHD, requiring specific laboratory findings to meet the CDC case definition for notifiable diseases (9,10). Reporting of Legionella was evaluated for the period June 21-November 26, 2000, because the UPMC electronic laboratory-based reporting did not capture reports of diseases caused by this organism before June 21. Duplicate records and cultures performed in the context of research studies not notifiable to ACHD but included in the UPMC electronic database were excluded. Case reports in each database were matched manually by the investigator. A match was defined as a report in the UPMC electronic database that had the same patient name, date of birth, and type of notifiable infectious disease as a report in the ACHD paper-based database. After matching, the case reports that were found in both databases, as well as cases found in only one of the two databases, were entered into a separate Excel (Microsoft Corp., Redmond, WA) spreadsheet.

Completeness of reporting was defined as the total number of unique, notifiable events identified independently through each surveillance system (UPMC electronic laboratory-based and ACHD conventional paper-based systems) divided by the estimated total number of reports available for reporting at the laboratory level (N) (Figure 2). To estimate the total number of reports available, we used the Chandra Sekar-Deming capture-recapture method (12). Since both the UPMC electronic and ACHD systems may not have captured all notifiable events, capture-recapture provided an approximation of the true total number of notifiable cases based on samples from these two independent, parallel surveillance systems (12,13). The overall completeness of reporting, the completeness of reporting by disease and by hospital, and the 95% confidence interval (CI) for completeness of coverage calculations were determined by using a resampling analysis based on the capture-recapture method (11). To date, no methods for the calculation of the 95% CI for completeness of coverage have been published. When SAS version 8.1 (SAS Institute, Inc., Cary, NC) was used, the resampling was based on the assumption that the distribution of the data observed for the three cells (e.g., C, n l, n2) of the contingency table followed a uniform distribution; the 5% and 95% values of the distribution from this analysis yielded the 95% CI for completeness. Completeness could not be calculated for diseases and hospital laboratories with zero values in the 2 by 2 contingency table cells (Figure 2).

[FIGURE 2 OMITTED]

An electronic false-positive result was defined as a case that was incorrectly detected by the electronic system; a missed report (or false negative) was defined as a notifiable case that was not detected through the electronic system. The completeness of reporting of both systems was estimated after excluding false positives and duplicate reports.

The chronologic sequence of events for the reporting of an infectious disease or condition consists of exposure to an infectious agent, followed by symptom onset after an incubation period, and then the seeking of medical attention (Figure 3). Although a presumptive diagnosis could be made by interpretation of the clinical syndrome at this point, the ability of electronic laboratory-based reporting system to detect a notifiable disease or condition begins at the time the laboratory result has been entered into the data system.

[FIGURE 3 OMITTED]

To determine the timeliness of the two surveillance systems, three time points were defined. [T.sub.1] was the date/time when the laboratory result was obtained and entered into the UPMC laboratory computer. [T.sub.2] was the date/time when the laboratory result was reported to ACHD by the conventional paper-based system. [T.sub.3] was the date/time the automatic electronic laboratory-based system notification was generated at UPMC. The timeliness of electronic and paper-based systems was defined as [t.sub.3] - [t.sub.1] and [t.sub.2] - [t.sub.1], respectively. The difference between [t.sub.3] and [t.sub.2] represented how much sooner or later the electronic system identified notifiable diseases than the paper-based system. Median differences were expressed with an interquartile range. The timeliness calculations were performed with SAS version 8.1.

Before matching individual records in both systems and removing duplicate records, we calculated the completion rates for the data fields common to both the UPMC electronic and the ACHD paper-based databases by using Epi-Info 2000 (Centers for Disease Control and Prevention, Atlanta, GA).

We identified the specific reasons for the electronic system's false positives and missed reports by using a traceback error analysis. Reports that were found in the UPMC electronic database but not in the ACHD paper-based database were identified, and case-patient information was reviewed from the laboratory computer reports and their HL7 messages (electronic false positives). Reports that were found in the ACHD paper-based database and not in the UPMC electronic database were identified after reviewing case-patient information, laboratory case reports, and archived computer files (electronic laboratory-based reporting missed reports). To further assess database accuracy, we also reviewed the paper reports and logs at the Allegheny County Health Department and compared these with data in the ACHD conventional paper-based reporting system database.

Results

A total of 141 unique reports were identified; 116 (82%) were reported through the UPMC electronic laboratory-based system, and 94 (67%) were reported by ACHD conventional paper-based reporting system. Forty-seven (33%) of the notifications were received through the UPMC electronic system only, 25 (18%) through the ACHD paper-based system only, and 69 (49%) through both (Figure 4). The estimated total number of reports calculated by the capture-recapture method was 144.

[FIGURE 4 OMITTED]

After excluding electronic laboratory-based reporting false positives, the overall completeness of reporting was 74% (95% CI 66% to 81%) for the UPMC electronic system and 65% (95% CI 57 to 73%) for the ACHD paper-based system (p>0.05), showing no significant difference in completeness of reporting between the electronic and paper-based systems (Table 1). Table 1 also lists the completeness of coverage and 95% CI by disease and by hospital. Most of the cases missed by electronic reporting were from one hospital (UPMC Hospital C).

Timeliness was calculated by using the 69 records common to both databases. The timeliness of paper-based reporting was a median of 5 days (interquartile range 4 days). The timeliness of the electronic reporting was a median of 1 day (interquartile range 0 days). Electronic alerts were reported a median of 4 days (interquartile range 4 days) sooner than through paper-based reporting. We discovered a trend in the UPMC electronic reporting: the time difference between the date/time the laboratory result was obtained and entered, and the date/time the HL7 message was sent was almost exactly 24 hours to the second. After extensive discussions with the UPMC laboratory administration and informatics personnel, the reasons for this finding are unknown.

Eleven data fields were common to both the UPMC electronic and the ACHD paper-based databases (Table 2). Of these, six fields were 100% complete in both. Of the remaining five, two were more complete in the UPMC electronic system (date of birth and age), whereas three were more complete in the ACHD paper-based system (address, zip code, report status [i.e. final results]); these differences were significant (p<0.001).

Electronic laboratory-based reporting generated 10 reports that were found to be false upon investigation (Table 3). Since the electronic reporting system can only capture diseases that are entered with the preprogrammed UPMC disease codes, test results entered with free text could not be extracted correctly into the UPMC electronic database. Most of the identified errors were, in fact, caused by the use of free text in combination with the UPMC code for the organism (i.e., the combining of the free text "No" with an organism code when laboratory technicians entered results). For example, a result entered by laboratory technicians as "no" "SALM" (the UPMC code for Salmonella) was recognized and incorrectly detected by electronic system as a positive case of Salmonella. Other errors involved the inability to retract preliminary reports of the isolation of notifiable organisms that were not subsequently confirmed and data extraction from the incorrect part of the result field. In the latter instance, a case of legionellosis was reported as listeriosis because the data field included the phrase "Specimen Delivery: UNIT LIST," and LIST is the disease code at UPMC for Listeria.

Data-entry errors, such as the incorrect use of free text, led to missed reports in the electronic system. Typically, these errors occurred when laboratory technicians entered the name of the organism as free text rather than with the preprogrammed UPMC disease codes. These errors accounted for 22 (88%) of 25 electronic missed reports, whereas the remaining three missed reports were found in the hospital computer systems but were not detected by the UPMC electronic system for reasons that remained unclear after investigation. Of the 47 cases in the UPMC electronic system not reported by the paper-based system to ACHD, 37 should have been reported to ACHD ("ACHD false negative").

Discussion

This is the first report of an evaluation of an existing health-system-based electronic notifiable disease reporting system. The electronic laboratory-based reporting was as complete as conventional paper-based reporting. The estimated completeness (74%) is similar to the recent report of 80% completeness of the electronic laboratory-based reporting from commercial clinical laboratories to the Hawaii Department of Health (6). The incompleteness and inaccuracy of UPMC electronic reporting were caused mainly by the use of free text, rather than standardized organism codes, by laboratory personnel at one hospital. Similarly, most of the electronic false positives were caused by the use of free text.

The magnitude of the difference in completeness between electronic laboratory-based reporting and conventional paper-based reporting may have been greater if it had been possible to review reports coming exclusively from laboratories to the ACHD; paper-based systems receive reports from sources other than laboratories. Data specifying if a case record originated from a laboratory or health-care provider were not available in the paper-based database. Hence, a bias favoring completeness of reporting by the paper-based system existed in our analysis. However, most reports received by health departments originate from clinical laboratories (14). The capture-recapture method used to calculate completeness required that the two surveillance systems (UPMC electronic laboratory-based reporting and ACHD conventional paper-based reporting system) operate independently. However, some interaction between the systems existed; the laboratory director used the generated electronic e-mail message, containing the laboratory test results, to check for potential false positives before a report was falsely sent conventionally to ACHD. This interaction was thought to be minimal (Figure 1). Other capture-recapture assumptions, such as the surveillance being performed on a stable population and only true matches and events being identified by the systems, were fulfilled (12,13).

Maximizing electronic laboratory-based reporting sensitivity is important for detecting diseases, while maximizing specificity enhances the likelihood that cases are reported correctly. Theoretically, electronic reporting has the potential to be both sensitive and specific, with few false negatives and false positives. The specificity of electronic reporting could be particularly high for diseases diagnosed by laboratory tests with a low rate of false positives (e.g., culture for enteric organisms); the diseases caused by the organisms used in this study met this qualification. Notifiable diseases based on other types of tests (e.g., serology for syphilis) would require clinical criteria to enhance specificity (information not available by electronic reporting). In this evaluation, we found that the inability to retract preliminary positive laboratory reports that were subsequently confirmed to be negative reduced the specificity of electronic reporting. However, in some instances, the benefit of early detection might supersede an occasional false-positive report. For example, early detection is paramount for some organisms, such as Bacillus anthracis--the release of which could represent a potential bioterrorist event. Nonetheless, a substantial amount of public health effort might be expended unnecessarily if such a laboratory finding were found to be a false positive. One must balance the tradeoffs between sensitivity, specificity, and timeliness when deciding to allow these preliminary laboratory results to be reported.

The UPMC electronic reporting has the potential to serve as a prototype for use nationally because it uses hospital-based laboratory information systems already in place to capture cases of disease that may be representative of the population at large. However, several findings from our analysis have implications for large health systems attempting to establish electronic laboratory-based reporting. The use of standardized disease codes should be encouraged because it maximizes both the sensitivity and specificity of electronic laboratory-based reporting. At UPMC, the incorrect use of free text at a single hospital substantially reduced the overall completeness of electronic laboratory-based reporting reporting. However, eliminating the use of free text may not be desirable from a laboratory personnel standpoint. As such, training laboratory personnel in the correct use of free text is important (15). Moreover, UPMC computer personnel could relegate the free text option only to a note field that provides useful information to health-care providers without generating a report; in this regard, a properly constructed result code entered through a correctly designed data entry method would be useful. The use of standardized codes has broad implications for electronic laboratory-based reporting in general. To effectively enhance the unification of NEDSS, CDC recommends the implementation of standardized coding schemes for disease names (Systematized Nomenclature of Medicine [SNOMED]; http:// www.snomed.org) and laboratory test names (Logical Observation Identifier, Names, and Codes [LOINC]) (http:// www.regenstrief.org//loinc/). Unfortunately, UPMC and many other health centers are not using standardized coding schemes. The lack of standardized coding requires the creation of a translation table, a process that requires refinements to maximize accuracy and completeness.

A mechanism for retracting of preliminary reports not subsequently confirmed is essential to reduce false-positive reports. Retraction is the ability to both remove an incorrect or preliminary report from the database as well as to notify the recipients of the information of the change. If the sending system does not explicitly label the message as a correction or a retraction, then the electronic laboratory-based reporting system must have logic to detect it. The detection logic simply compares the previously reported preliminary reports in a cached table with a new one. If the logic finds a match but the new report does not have any notifiable organisms, the logic will send a retraction alert to officials at the local health department or hospital laboratory administrators and remove the false-positive report from the cached table. At the time of this evaluation, such retraction capability did not exist at UPMC because the UPMC laboratory sending system did not explicitly label the message as a correction or a retraction. In the future, one option may be to label preliminary reports as "preliminary" or "suspect." Currently, two authors have been working on the retraction capability and expect to have such functionality available soon. However, the risk versus benefit of reporting preliminary laboratory results should be weighed in making the decision to retract such reports. The best approach might be to report preliminary results for diseases that require immediate notification, while reporting confirmed results for others.

Decisions to remove certain duplicate records that were not detected by electronic laboratory-based reporting should be made before integration of automatic reporting to ACHD. Caution should be used in the removal of some type of duplicates, as this decision may need to be disease specific. For example, repeated positive sputum cultures for tuberculosis from a patient who has received the recommended course of therapy may represent persistent, active infection and drug resistance, both of which are substantial public health concerns.

UPMC electronic laboratory-based reporting had lower completion rates of data fields with important contact information (specifically, address and zip code fields) compared with ACHD conventional paper-based reporting system, which may hamper efforts by public health personnel to contact patients quickly. The implementation of automatic demographic data extraction by electronic laboratory-based reporting from other resources such as epidemiologic or administrative databases (e.g., billing records), could substantially improve the data field completion rate for electronic laboratory-based reporting.

UPMC should continue to refine its electronic laboratory-based reporting before implementing direct automatic reporting to ACHD. Electronic laboratory-based reporting should not replace conventional reporting as observations made by astute clinicians are valuable in the timely reporting of certain notifiable syndromic illnesses (Figure 3). Instead, electronic laboratory-based reporting should become integrated with and complement the existing conventional reporting system to ensure the most complete capture of notifiable disease events.

The findings from this evaluation indicate that direct automatic reporting from a health system is feasible and as complete but more rapid than conventional reporting. An error analysis showed many correctable problems; better control of the use of free text and an ability to retract preliminary reports were key areas for improvement. Standard coding schemes should be used. Health departments need to evaluate electronic surveillance systems before integrating the data into existing reporting systems. CDC and state health departments should collaborate to develop a consensus on the goals for an electronic laboratory-based reporting system intended for public health laboratory-based disease reporting. Once these goals have been determined, guidelines may be created that would assess if the system achieves these desired goals. The methodology in this evaluation may be used by health departments when evaluating other electronic surveillance systems, taking into consideration the different design issues of such systems.

Table 1. Completeness of coverage for UPMC electronic and conventional
reporting systems by the notifiable infectious disease and hospital
laboratory (a)

                                         Conventional reporting (ACHD)

                           Total no.      No. of
                          of available   reports      Completeness of
                          reports (b)    received    coverage (95% CI)

Notifiable infectious
  disease
    Campylobacter              37           25      0.68 (0.49 to 0.85)
    Salmonella                 35           32      0.91 (0.83 to 0.97)
    Escherichia coli
      O157:H7                  17           10      0.59 (0.33 to 0.86)
    Giardia                    22           13      0.59 (0.39 to 0.77)
    Neisseria
      meningitidis              9            5      0.58 (0.30 to 0.88)
UPMC Hospital
  laboratory
    A                          26           16      0.62 (0.46 to 0.80)
    B                          52           29      0.55 (0.42 to 0.65)
    C                          35           24      0.69 (0.43 to 0.90)
    D                          13           11      0.85 (0.64 to 0.92)
    E                          10            9      0.90 (0.70 to 0.90)

                                           Electronic reporting (UPMC)

                           Total no.      No. of
                          of available   reports      Completeness of
                          reports (b)    received    coverage (95% CI)

Notifiable infectious
  disease
    Campylobacter              37           18      0.49 (0.32 to 0.65)
    Salmonella                 35           34      0.95 (0.91 to 0.97)
    Escherichia coli
      O157:H7                  17            7      0.41 (0.19 to 0.67)
    Giardia                    22           17      0.77 (0.58 to 0.90)
    Neisseria
      meningitidis              9            7      0.72 (0.46 to 0.88)
UPMC Hospital
  laboratory
    A                          26           24      0.92 (0.81 to 0.96)
    B                          52           47      0.91 (0.79 to 0.96)
    C                          35            9      0.26 (0.12 to 0.40)
    D                          13           12      0.87 (0.71 to 0.92)
    E                          10            9      0.90 (0.70 to 0.90)

(a) UPMC Health System; ACHD, Allegheny County Health Department; CI,
confidence interval.

(b) Estimated total number of reports available by using
capture-recapture (N in Figure 2).
Table 2. Data field completion rates on common data fields for cases
in UPMC Health System electronic and conventional reporting system
databases (a)

                           No. (%) of conventional
                          reported cases with field
Data field                    completed (n=534)

Patient information
  Patient ID                      534 (100)
  Name                            534 (100)
  Sex                             534 (100)
  Date of birth                   462 (86.5)
  Age                             518 (97.0)
  Address                         533 (99.8)
  Zip code                        533 (99.8)
Specimen information
  Organism name                   534 (100)
  Time result obtained            534 (100)
  Time result reported            534 (100)

Other information
  Status of report                534 (100)

                            No. (%) of electronic
                          reported cases with field
Data field                    completed (n=582)

Patient information
  Patient ID                      582 (100)
  Name                            582 (100)
  Sex                             582 (100)
  Date of birth                   582 (100)
  Age                             582 (100)
  Address                         306 (52.6)
  Zip code                        306 (52.6)
Specimen information
  Organism name                   582 (100)
  Time result obtained            582 (100)
  Time result reported            582 (100)

Other information
  Status of report                220 (37.8)

(a) All rates before matching and duplicate record removal.
Table 3. Electronic false positives and missed reports in UPMC Health
System reporting system

                                      No. (%) of electronic or paper-
Errors                                       based only reports

Electronic false positives
    Incorrect use of free text
      with organism codes                          6 (60)
    Inability to retrieve sent
      false reports                                3 (30)
    Failure of logic detection                     1 (10)
Total                                             10
Electronic false negatives (missed
  reports)
    Incorrect use of free text                    22 (88)
    Unknown (failure of
      transmission?)                               3 (12)
Total                                             25

Errors                                       Nature of problem

Electronic false positives
    Incorrect use of free text
      with organism codes             Culture report reads "No [free
                                      text]" followed by organism ID
                                                   code
    Inability to retrieve sent
      false reports                    Unable to retrieve preliminary
                                                  reports
    Failure of logic detection       Data extracted from wrong portion
                                     of result field by logic detection
Total
Electronic false negatives (missed
  reports)
    Incorrect use of free text        Organism name typed out as free
                                            text in result field
    Unknown (failure of
      transmission?)                   Found to be in UPMC hospital
                                     computer terminal system by using
                                     organism ID code properly but not
                                     found in UPMC electronic database
Total

Acknowledgments

We thank the staff of the Allegheny County Health Department, including Nancy Felton and Mary Jane Walicki, for their interview time and data collation efforts.

This evaluation was partially funded by a Centers for Disease Control and Prevention Cooperative Agreement with the BioMedical Security Institute, a collaboration of the University of Pittsburgh and Carnegie Mellon University for research on bioterrorism issues.

References

(1.) Thacker SB, Berkelman RL, Stroup DF. The science of public health surveillance. J Public Health Policy 1989; 10:187-203.

(2.) Institute of Medicine and National Research Council. Chemical and biological terrorism: research and development to improve civilian medical response. Washington: National Academy Press; 1999.

(3.) Centers for Disease Control and Prevention. Supporting public health surveillance through the National Electronic Disease Surveillance System. Available from: URL: http://www.cdc.gov/od/hissb/docs/NEDSS% 20Intro.pdf

(4.) Godes JR, Hall WN, Dean AG, Morse CD. Laboratory-based disease surveillance: a survey of state laboratory directors. Minn Med 1982;65:762-4.

(5.) Jernigan DB. Electronic laboratory-based reporting: opportunities and challenges for surveillance. Emerg Infect Dis 2001;7:538.

(6.) Effler P, Ching-Lee M, Bogard A, Ieong M, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from clinical laboratories. JAMA 1999;282:1845-50.

(7.) Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems. MMWR Morb Mortal Wkly Rep 2001;50(RR13): 1-36.

(8.) Yount R, Vries J, Councill C. The medical archival retrieval system: an information-retrieval system based on distributed parallel processing. Infect Proc Manag 1991;27:1-11.

(9.) Centers for Disease Control and Prevention. Case definitions for infectious conditions under public health surveillance. MMWR Morb Mortal Wkly Rep 1997;46(RR10):1-55.

(10.) Council of State and Territorial Epidemiologists. Table 2. Reporting requirements for health care providers and laboratories diseases and conditions under national surveillance. Available from: URL: http:// www.cste.org/nndss/ndtable1a.html

(11.) Simon JL, Burstein P. Basic research methods in social science. 3rd ed. New York: Random House; 1985.

(12.) Sekar CC, Deming WE. On a method of estimating birth and death rates and the extent of registration. J Am Stat Assoc 1949;44:101-15.

(13.) Teutsch SM, Churchill RE. Principles and practices of public health surveillance. 1st ed. New York: Oxford University Press; 1994. p. 136-49, 158-74.

(14.) Vogt RL, LaRue D, Klaucke DN, Jillson DA. Comparison of an active and passive surveillance system of primary care providers for hepatitis, measles, rubella, and salmonellosis. Vermont. Am J Public Health 1983;73:795-7.

(15.) Hogan WR, Wagner MM. Free-text fields change the meaning of coded data. Proc AMIA Symp 1996:517-21.

Dr. Panackal is an officer with the Epidemic Intelligence Service at the National Center for Infectious Diseases, Centers for Disease Control and Prevention. His recent field epidemiology research experiences have included outbreak investigations of Rift Valley fever in Saudi Arabia and Yemen and invasive aspergillosis in renal transplant recipients in California.

For additional information on corrections, send e-mail to [e-mail].

* Centers for Disease Control and Prevention, Atlanta, Georgia, USA; ([dagger]) Pennsylvania Department of Health, Harrisburg, Pennsylvania, USA; ([double dagger]) University of Pittsburgh, Pittsburgh, Pennsylvania, USA; and ([section]) Allegheny County Health Department, Pittsburgh, Pennsylvania, USA

Anil A. Panackal, * Nkuchia M. M'ikanatha, ([dagger]) Fu-Chiang Tsui, ([double dagger]) Joan McMahon, ([section]) Michael M. Wagner, ([double dagger]) Bruce W. Dixon, ([section]) Juan Zubieta, * Maureen Phelan, * Sara Mirza, * Juliette Morgan, * Daniel Jernigan, * A. William Pasculle, ([double dagger]) James T. Rankin, Jr., ([dagger]) Rana A. Hajjeh, * and Lee H. Harrison ([double dagger])

Address for correspondence: Lee H. Harrison, Infectious Diseases Epidemiology Research Unit, University of Pittsburgh, 521 Parran Hall, 130 De Soto Street, Pittsburgh, PA 15261, USA; fax: [phone]; e-mail: [e-mail]