Printer Friendly

Establishing and evaluating QC acceptability criteria.

If quality is as important as speed in your lab, then QC acceptability criteria must be properly established to assure detection of medically important errors. Here's how.

CLIA'S REGULATIONS for quality control require laboratories to "establish criteria for acceptability used to monitor test performance during a run of patient specimen(s)."|1~ In establishing QC acceptability criteria, labs can follow a manufacturer's instructions when these have been cleared by the Food and Drug Administration (FDA) and when the laboratory verifies that its observed analytical performance is consistent with the manufacturer's claims.

In the absence of QC clearance from FDA, labs must be sure that their practices for establishing QC acceptability criteria are valid and appropriate for meeting their own quality requirements. For regulated tests, the minimum requirement for analytical quality should be the total error allowed by CLIA proficiency testing (PT) criteria. If clinical requirements are more demanding, QC criteria should be based on medical needs.

* Current practices. A variety of practices are currently used for establishing quality control acceptability criteria. We recommend using power function graphs to assess the probabilities for rejecting analytical runs that have medically important errors.

These critical-sized errors are calculated from the analytical or clinical quality requirement and the imprecision and inaccuracy observed for the measurement procedure. Appropriate QC rules and numbers of control measurements

(N) are then selected by comparing the probabilities for detecting critical-sized errors, as illustrated by Koch et al.|2~

Other current practices are to impose clinical or fixed control limits directly on QC charts with the hope of detecting clinically significant, instead of statistically significant, errors. The ways of doing this are not well documented in the literature, but one practice is to use a medically allowable standard deviation|3~ (s, SD) to calculate control limits for a Levey-Jennings control chart (mean |+ or -~ 2s or 3s).

Another practice is to set control limits directly on the QC chart using fixed limits (mean |+ or -~ fixed limit), which may be the total error based on medical usefulness or a total allowable analytical error based on analytical goals, such as the CLIA PT criteria. Other related practices are to use a manufacturer's claim for method performance as the SD to calculate control limits or a manufacturer's "acceptable range" as a fixed control limit.

* Valid QC procedures. Laboratories need to assess whether these practices for establishing QC acceptability criteria are valid, particularly in the context of the FDA's new guidelines for implementing CLIA QC requirements. The FDA describes a valid QC procedure as ". . . one that adequately maintains and monitors stated analytical performance characteristics and, at the same time, alerts the analyst to unsatisfactory performance when known and/or unknown technical variables are introduced. These procedures should adequately address the critical performance parameters of accuracy and precision within the reportable range of the test."|4~

These FDA guidelines are aimed at manufacturers, but in the absence of FDA clearance of manufacturers' QC instructions by Sept. 1, 1994, laboratories performing moderate and high complexity tests will be responsible under CLIA for documenting the QC procedures in use:|1~

"...the laboratory must evaluate instrument and reagent stability and variance in determining the number, type, and frequency of testing calibration or control materials and establish criteria for acceptability used to monitor test performance during a run of patient specimen(s)."

493.1218 (b)

* Example QC. Consider a potassium method that has an actual standard deviation (|s.sub.meas~) of 0.10 mmol/L (or a CV of 2% at a concentration of 5.0 mmol/L), a medically allowable SD (|s.sub.allow~) of 0.2 mmol/L (or 4% at a concentration of 5.0 mmol/L), and a minimum level of analytical quality defined by the CLIA PT criterion as 0.5 mmol/L (or 10.0% at a target value of 5.0 mmol/L). Similar performance characteristics apply to other analytes such as glucose, calcium, albumin, total protein, urea nitrogen, bilirubin, and cholesterol--all with CLIA criteria of 10% at appropriate target values or decision levels. Five possible practices for establishing acceptability criteria for a Levey-Jennings control chart are described in Figure 1. Statistical QC limits can be set as the mean |+ or -~ 2 |s.sub.meas~ or |+ or -~3 |s.sub.meas~, where |s.sub.meas~ is the standard deviation observed for the measurement procedure. Clinical control limits can be set as the mean |+ or -~ 2 |s.sub.allow~ or |+ or -~ 3 |s.sub.allow~, where |s.sub.allow~ is the clinically allowable standard deviation. Fixed control limits can be set as |+ or -~ TE, where TE is a total error criterion such as provided by the CLIA PT requirement. Depending on the practice chosen, the actual control limits used for our potassium method could vary from |+ or -~ 0.2 to |+ or -~ 0.6 mmol/L.

* Actual statistical rules. Regardless of how the control limits are established, a Levey-Jennings chart still operates as a statistical control. Restating these practices in terms of the actual statistical control rules we use is essential for understanding their performance.

The statistical control limits are determined by dividing the resulting control limits by the observed standard deviation (|s.sub.meas~). The actual control rule depends on how many control measurements must exceed these limits to reject an analytical run as being out of control--generally one for all these practices. The statistical control rules in the first two cases are the |1.sub.2s~ and |1.sub.3s~ rules that have been used traditionally with Levey-Jennings control charts. In the last three cases, the actual statistical control rules used are |1.sub.4s~, |1.sub.5s~, and |1.sub.6s~.

* QC performance. The performance of these different control rules can be assessed using power function graphs to determine the probabilities of rejecting runs having critical-sized systematic and random errors. In our potassium example, a systematic error equivalent to 3.35 |s.sub.meas~ and a random error equivalent to 3.03 |s.sub.meas~, need to be detected to satisfy the CLIA PT requirement. Figures 2 and 3 show the critical error graphs that were generated using a microcomputer program, QC Validator (WesTgard QC, Ogunquit, Me.). The charted lines represent the different acceptability criteria or different control rules, all with two control measurements per run (from top to bottom, |1.sub.2s~, |1.sub.2.5s~, |1.sub.3s~, |1.sub.3.5s~, |1.sub.4s~, |1.sub.5s~, and |1.sub.6s~, respectively). Detection of a systematic shift equivalent to 3.35 times the observed standard deviation (|s.sub.meas~) is important to assure that the analytical quality required by CLIA proficiency testing is achieved. Random errors that are equivalent to a 3.03-fold increase in the observed standard deviation need to be detected.

* Run rejection. The probabilities for false rejection (|P.sub.fr~) and error detection (|P.sub.ed~) are shown in the keys at the right side of the graphs in Figures 2 and 3. For example, the probabilities for error detection for the critical systematic error in Figure 2 are 0.99, 0.93, 0.83, 0.63, 0.41, 0.11, and 0.01 for the |1.sub.2s~, |1.sub.2.5s~, |1.sub.3s~, |1.sub.3.5s~, |1.sub.4s~, |1.sub.5s~, and |1.sub.6s~, control rules, respectively. This means the chances of detecting medically important errors range from 99% to 63% for statistically established acceptability criteria and 41% to 1% for clinical and fixed limit acceptability criteria. Finding random errors is even more difficult. * Inaccuracy and imprecision. Figure 4 shows an operational process specifications (OPSpecs) chart for 10% |TE.sub.PT~ with 90% analytical quality assurance (AQA) for systematic errors (SE). The different lines on this chart describe the limits of inaccuracy and imprecision allowable when using different acceptability criteria or control rules.

The necessary error detection is available from any QC procedure whose operating limits are above the method's operating point (x = |s.sub.meas~, y = |bias.sub.meas~).|5,6~ From the OPSpecs chart, the best choice of an acceptability criterion for the potassium example is a |1.sub.2.5s~ control rule (or control limits set as the mean |+ or -~ |2.5s.sub.meas~), which should provide approximately 90% detection of medically important errors with only 3% false rejections. Use of 2s control limits would also provide the desired error detection, but at a higher cost due to the expected false rejection rate of 9%. * Limitations. It is not advisable to establish QC acceptability criteria TABULAR DATA OMITTED based on the practices evaluated here that employ medically allowable SDs to calculate control limits or impose fixed clinical or analytical control limits directly on control charts. The performance of such QC procedures will most likely be inadequate for detecting medically important errors. Also suspect would be related practices where QC acceptability criteria are based on a manufacturer's claimed standard deviation instead of the observed SD, or on a manufacturer's recommendation for an acceptable range instead of on a calculated statistical range.

If such practices are used, we recommend that each individual application be validated by assessing the expected error detection capabilities of the actual statistical control rule that is implemented. Power function graphs, critical error graphs, or OPSpecs charts are useful for making this assessment and documenting the appropriateness of the selected acceptability criteria.

An analogy might be helpful for viewing this problem in a new light. Consider the similarities between driving an automobile and operating an analytical process. Such key operating characteristics as speed, steering, and brakes can be related to such characteristics of the testing process as turnaround time, calibration, accuracy, and precision. Think of statistical QC as the light needed to detect obstacles in the road.

Operate safely: Just staying on the road is not the point--it's driving safely and avoiding accidents. This means being able to stop in time to avoid crashing into another car crossing an intersection. We must have enough light or a good detection system to catch a problem when it occurs (or better, anticipate the problem before it occurs). Similarly, we need the ability to detect medically important errors so we can operate our testing processes with a proper margin for safety.

Plan ahead: In planning analytical testing processes, we often set goals or criteria for precision and accuracy without considering the need for QC. This is the equivalent of assuming that a car will be driven under only ideal weather and road conditions and during daylight. For most cars, this is not the case. Unfortunately, no lights have been designed for the laboratory testing process, but the illumination is provided when the laboratory defines its QC acceptability criteria. When these criteria are finally evaluated, we are often surprised to find they are inappropriate for the precision and accuracy observed for the method. The lack of adequate planning allowed that to happen. Recognize dangerous situations: We can't determine what size problems are important to detect unless we define the quality that needs to be achieved. This means knowing the clinical or analytical quality that is required for each test and calculating the critical-sized errors that need to be detected. Must we stop the car to avoid a train, truck, car, tree, limb, branch, or leaf? What size problem will likely cause damage? For regulated analytes, it would seem that the CLIA PT criteria define the minimum level of quality that should be achieved. If we are to perform PT in the same manner as patient testing (as required by CLIA rules), it follows that we must achieve at least that minimum level of quality in patient testing. Using the PT criteria as total error requirements, we can calculate the critical sizes of errors that must be detected, then establish QC acceptability criteria that have the appropriate sensitivity for detecting these critical errors. If clinical quality requirements are more demanding, we should use them in a proper way to establish QC acceptability criteria by designing quality control into the testing process.

Understand performance limitations: Key performance characteristics of the automobile itself influence the distance the lights need to penetrate into the darkness to prevent accidents. High-performance steering and brakes, for example, make it easier to maintain control in an emergency and to stop in time, perhaps with the use of just low beams instead of high, whereas a car without such equipment might not be able to stop in time, even with the high beams on. Similarly, the accuracy and precision of the testing system affect the QC that is needed. Analytical systems with high precision and accuracy should be able to use simpler control rules and lower numbers of control measurements and still detect critical-sized errors. Methods with poor precision and accuracy may not be controllable within the quality we try to achieve.

Use safety devices: Different kinds of lights provide us with varying ability to identify obstacles in the dark that require us to slow down or stop. We can drive with high beams, low beams, parking lights, or no lights. We can evaluate the performance of the lights quantitatively in terms of power.

In much the same way, different statistical QC procedures provide different powers for detecting analytical problems (probabilities for error detection) that require us to stop the testing process. We can have high, moderate, or low error detection--or none at all. Clearly, the sorts of clinical- and fixed-limit QC procedures studied here provide error detection in the "low to none" range. At best, they are the lab equivalent of parking lights.

Adjust to conditions; When driving, the number and type of problems encountered often depend on the terrain and the condition of the road. Eliminating dangerous intersections, straightening the curves, and leveling the road surfaces reduce hazards. Some light is still needed to drive safely at night, but moderate or low illumination is adequate when the road itself is predictable and free of problems. Hypothetically speaking, lights wouldn't be needed at all if the car was under the control of a complete guidance system. Similarly, there are highly automated analytical systems that control their own environments. They automatically correct and adjust for changes in variables, providing more stable and predictable testing systems that stay relatively free from problems--even using moderate- or low-power QC procedures. These self-controlling, high-stability systems are likely to be more expensive and, consequently, are most often found in large central laboratories. They are needed even more, however, in decentralized testing applications where there is limited expertise in lab testing, little interest in doing quality control, and great difficulty in establishing appropriate QC acceptability criteria. * Avoid hazardous practices. Some of the current practices for establishing acceptability criteria are tantamount to driving in the dark on a narrow, mountainous road in the middle of avalanche season with bad brakes and no lights. Despite this knowledge, we're still surprised when we run into a problem. We continue to think that the public is wrong in its perception of the quality of laboratory testing.

It is always dangerous to argue by analogy; however, a parallel situation that is easier to understand often stimulates more thinking and discussion leading to improved practices. There is enough technical information in the scientific literature to guide us in the right direction for managing the quality of our testing processes.|7~ QC acceptability criteria can be quantitatively evaluated based on performance characteristics that are documented and available in the scientific literature.|8-10~ Microcomputer programs such as QC Validator should help to make this assessment much easier and more practical for manufacturers as well as laboratory users.

* Working smarter. Instead of gaining better control of quality, we often seem driven by the need to get more work done faster. Using hazardous practices for establishing QC acceptability criteria helps us to obtain that goal--finding fewer errors means fewer runs rejected.

If speed or maximum production is our only goal, then quality control should be eliminated completely to maximize process yield. On the other hand, if quality is as important as speed, then QC acceptability criteria must be properly established to assure detection of medically important errors.

References

1. Medicare, Medicaid, and CLIA Programs: Regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA) and Clinical Laboratory Improvement Act program fee collection. Department of Health and Human Services, Health Care Financing Administration and Public Health Service. January 19, 1993: 58: 5213-5237.

2. Koch DD, Oryall JJ, Quam EF, et al. Selection of medically useful QC procedures for individual tests on a multitest analytical system. Clin Chem. 1990; 36: 230-233.

3. Skendzel LP, Barnett RN, Platt R. Medically useful criteria for analytical performance of laboratory tests. Am J Clin Pathol. 1985: 83: 200-205.

4. Draft FDA guidance to manufacturers of in vitro analytical test systems for preparation of premarket submissions implementing CLIA. Rockville, Md: Division of Small Manufacturers Assistance, Center for Devices and Radiological Health, Food and Drug Administration: December 17. 1992.

5. Westgard JO. Charts of operational process specifications ("OPSpecs charts") for assessing the precision, accuracy, and quality control needed to satisfy proficiency testing performance criteria. Clin Chem. 1992; 38: 1226-1233. 6. Westgard JO. Assuring analytical quality through process planning and quality control. Arch Pathol Lab Med. 1992; 116: 765-769.

7. Burnett RW, Westgard JO. Selection of measurement and control procedures to satisfy HCFA requirements and provide cost-effective operation. Arch Pathol Lab Med. 1992; 116: 777-782.

8. Westgard JO, Groth T. Power functions for statistical control rules. Clin Chem. 1979; 25: 394-400.

9. Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington, DC: AACC Press; 1986.

10. Cembrowski GS, Carey RN. Laboratory Quality Management. Chicago, Ill: ASCP Press; 1989.

Westgard is professor and associate director of clinical laboratories--quality assurance, Quam is supervisor of quality control in the clinical chemistry section, and Barry is supervisor of quality assurance for the clinical laboratories in the Division of Laboratory Medicine, Department of Pathology and Laboratory Medicine, Medical School, University of Wisconsin, Madison, Wis.
COPYRIGHT 1994 Nelson Publishing
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1994 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:quality control
Author:Westgard, James O.; Quam, Elsa F.; Barry, Patricia L.
Publication:Medical Laboratory Observer
Date:Feb 1, 1994
Words:2992
Previous Article:Administration tips hand on new CLIA changes.
Next Article:Fighting the TB dragon.
Topics:


Related Articles
Quality control in the new environment: microbiology.
Keeping two automated cell counters in calibration synch.
Closing the loop on quality.
Instituting a multiple-site urinalysis QC program.
Assessing linearity the easy way: laboratories can comply with new Federal regulations for linearity verification by applying simple calculations and...
Walking the straight and narrow on quality control.
Use the LIS to monitor TAT.
Countdown to success: a fresh approach to quality in the laboratory.
CLIA 2003's new concept: equivalent quality control.
QC assigned values from peer data.

Terms of use | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters