Do proficiency testing participants learn from their mistakes? Experience from the EXCEL throat culture module.

With the implementation of the Clinical Laboratory Improvement Amendments of 1988 (CLIA), proficiency testing (PT) was required for designated laboratory tests in all laboratories seeking a CLIA license. (1) Although the primary purpose of PT was to assess current laboratory performance, a secondary benefit of PT was postulated to be an improvement in quality of testing services as laboratories received feedback on their performance relative to their peers and gained experience. (2) The EXCEL PT program is a program of the College of American Pathologists (CAP) that is designed for small hospitals, clinics, and physician office laboratories. It offers a number of challenges in hematology and clinical microscopy, chemistry, and microbiology and immunology. Specimens are chosen as challenges that assess a specific competence important to the testing to be evaluated. Shortly after CLIA became effective, a problem in a throat culture competence, the ability to differentiate group A from non-group A streptococci, especially the recognition of group C streptococci, was identified. This was believed to be a clinically important competence because group C streptococci are isolated from a significant number of individuals with acute pharyngitis, (3,4) are not associated with preventable secondary consequences such as rheumatic fever, and hence do not need to be treated with antibiotics. (5) The effect that PT experience, with the consistent provision of feedback on performance relative to peers and optimal laboratory practices, has on the ability to successfully differentiate group C from group A streptococci was evaluated throughout a 6-year period.

METHODS

A challenge using a standard group C streptococcus (Streptococcus equisimilis) was presented in 1 of the 15 challenges of the throat culture module in each year from 1996 through 2001. The culture characteristics of the organism, specifically that it Was bacitracin disk resistant, that it did not hydrolyze pyrrolidonyl arylamide (PYR test), and that latex typing was group C and not group A, were confirmed by 3 high-complexity hospital laboratories that agreed to pretest test isolates. Participants received their specimens on loops per EXCEL PT program procedures, performed throat culture tests in their laboratories, and recorded and returned their response to CAP for evaluation. These results were analyzed, tabulated, and collated into summary reports from which the data on overall performance on the group C streptococcus specimen were abstracted.

For each testing event, an individual performance report was returned to the participant, which in addition to individual and group performance data contained a discussion of the results that addressed problems in performance. In the case of group C streptococci, the reason it should be differentiated from group A streptococci was discussed and recommendations to avoid pitfalls in differentiation (such as trying to interpret bacitracin disk results on a primary plate) were addressed.

Data were analyzed for significance of differences in performance using a computer program (SPSS-PC), which yielded a value for Pearson [chi square] probability, likelihood ratio [chi square] probability, and a Mantel-Haenszel test for linear association. (6)

RESULTS

The performances of EXCEL PT program participants on the yearly group C streptococcus challenge are summarized in the Table. Each year approximately 20% of participants failed to recognize that the [beta]-hemolytic streptococcus isolated was group C and not group A. There was no significant improvement in performance noted over time, and no trend toward improved performance was evident; in fact, the largest percentage of unsatisfactory performance was noted in 2000, the fifth year of monitoring.

COMMENT

To support the hypothesis that peer feedback and experience gained over time result in improved laboratory practices (as inferred by improved PT results), performance in the differentiation of group C streptococci from group A streptococci should have improved during the 6-year period analyzed. Unfortunately, no significant improvement or trend toward improvement was observed. Understanding the cause or causes for this apparent lack of benefit of PT experience is important to the way PT is viewed in efforts toward laboratory quality improvement.

One explanation could be that the population evaluated each year was not the same because of high participant turnover; hence, there was no experience factor. EXCEL enrollment data suggest that participant turnover is low and loyalty to the program is high. In fact, trends suggest that very few new laboratories are entering PT programs, rather participants are leaving as the selection of waived tests expands. Data were reviewed at the participant pool level, not the individual participant level, and so it cannot be established whether failures were attributable to the same individuals over time.

Another possible explanation is that the specimen submitted to participants behaved in an inconsistent manner, which caused some participants to fail. Consistently good performance by laboratories that performed pretesting and a cohort of "referees" (historically competent performers who act as a comparison standard) was observed, making this unlikely. Further, EXCEL monitors and responds to complaints and concerns forwarded by participants that their particular specimens did not behave in the manner intended (this was the author's responsibility during these years), and such complaints and concerns were rare.

A single unacceptable result in a testing event does not result in failure of that testing event. It could be argued that since only a single group C streptococcus challenge was encountered each year, this unsatisfactory performance was viewed as a random event and not addressed. If such a phenomenon accounts for the lack of improvement observed, it should still be viewed as a failure of PT to improve laboratory practice, not because of an intrinsic inability of PT to engender improvement, but because of a failure of participants to use PT results appropriately. Proficiency testing programs such as EXCEL have emphasized the important role that evaluation of PT performance should play in a laboratory's quality assurance program. For quantitative tests, evaluation of differences of one's results from peer means can point to the presence of bias and imprecision. (7) For qualitative tests, such as throat cultures, review of unsatisfactory results over time can point out a particular weakness in a laboratory's testing method. Proficiency testing programs have traditionally not monitored compliance with these suggestions; rather accrediting bodies review the quality and appropriateness of response to individual PT failures at the time of accreditation inspections.

Participants are required to treat PT specimens like they treat patient specimens. The consistent 20% failure to distinguish group C streptococcus from group A streptococcus might reflect a decision on the part of some clinician directors of laboratories that making a distinction between [beta]-hemolytic streptococci is not necessary. Performing additional testing adds cost, and since multiple types of [beta]-hemolytic streptococci can cause acute pharyngitis, (3) and treatment, although not necessary to prevent complications, (5) is of symptomatic value, adoption of a pragmatic position that equates [beta]-hemolytic streptococci observed with group A streptococci observed is understandable. In such a case, the laboratory method in use might be less than optimal for making a distinction between types of [beta]-hemolytic streptococci, (8) but this method meets the clinical need. In such a case, a response of "[beta]-hemolytic streptococci isolated" would be a more appropriate and an accurate PT response. The option "[beta]-hemolytic streptococci isolated, sent for identification" is available and would be graded as "acceptable" performance, but is not a proper response because it does not truly reflect the way patient specimens are handled. This scenario again suggests that the lack of improvement is not due to an intrinsic failure of PT to support improvement, but rather to a failure of the way the PT process is structured. To resolve this would require an effort to arrive at agreement of what competencies are truly necessary for quality laboratory services among clinicians, laboratory professionals, and regulators, a consensus that does not currently exist.

In summary, it is observed that PT performance has not improved for a specific competence as expected over time. This failure indicates a less-than-optimal use of PT results in improving laboratory quality.

Performance of EXCEL Proficiency Testing Participants
on Group C Streptococcal Challenges

           Good         Acceptable        Unacceptable
       Performance,    Performance,       Performance,
Year       % *         % ([dagger])    % ([double dagger])

1996       80.4       Not applicable          19.6
1997       68.3            15.5               16.7
1998       56.1            24.1               19.5
1999       63.1            18.1               18.2
2000       63.2            16.3               20.8
2001       65.7            14.4               19.0

* Participant indicated "no group A streptocci isolated."

([dagger]) Participant indicated "[beta]-hemolytic streptocci isolated,
send for identification." This category was lumped with good
performance in 1996 data.

([double dagger]) Participant indicated "group A streptocci
isolated" or "presumptive group A streptocci isolated."

We thank the EXCEL PT committee chairs, Jane Dale, MD, and Ricky Reeves, MD, and the CAP staff, Linda Palicki, MT(ASCP), for their efforts and assistance throughout the years.

References

(1.) Clinical Laboratory Improvement Amendments of 1988 (CLIA). Pub L 100-578, 42 USC 201 (1988).

(2.) Shahangian S. Proficiency testing in laboratory medicine--uses and limitations. Arch Pathol Lab Med. 1998;122:15-30.

(3.) Turner JC, Hayden FG, Lobo MC, Ramirez CE, Murren D. Epidemiologic evidence for Lancefield group C beta-hemolytic streptococci as a cause of exudative pharyngitis in college students. J Clin Microbiol. 1997;35:1-4.

(4.) Bisno AL. Acute pharyngitis. N Engl J Med. 2001;344:205-211.

(5.) Schwartz B, Marcy SM, Phillips WR, Gerber MA, Dowell SF. Pharyngitis--principles of judicious use of antimicrobial agents. Pediatrics. 1998;101:171-174.

(6.) Munro BH, Page EB. Statistical Methods for Health Care Research. 2nd ed. Philadelphia, Pa: JB Lippincott; 1993:82-86.

(7.) Cembrowski GS, Anderson PG, Crampton CA, Carey RN. Pump up your PT IQ. MLO Med Lab Obs. 1996;32:46-50.

(8.) Roddey OF, Clegg HW, Martin ES, Swetenberg RL, Koonce EW. Comparison of throat culture methods for the recovery of group A streptococci in a pediatric office setting. JAMA. 1995;274:1863-1865.

Accepted for publication September 12, 2001.

From the Department of Pathology, Children's Hospital Medical Center, Akron, Ohio, and Northeastern Ohio Universities College of Medicine, Rootstown, Ohio.

Reprints: Robert Novak, MD, Department of Pathology, Children's Hospital, One Perkins Square, Akron, OH 44308 (e-mail: rnovak@ chmca.org).