# Selecting statistical procedures for quality control planning based on risk management.

The main objective of the statistical QC (SQC) [2] planning process is to select QC procedures that allow laboratory personnel to detect instabilities in the analytical system and to prevent reporting of patient results with medically important errors. According to the traditional approach, this objective can be achieved by selecting the appropriate control rule and number of control samples to be analyzed per run for detecting, with a high probability ([P.sub.EDC]), critical size errors that would result in 5% of test results with errors exceeding the quality requirement for the test, while maintaining a low probability of false rejections (1).However, probability of critical error detection ([P.sub.EDC]) does not allow for the comparison of QC strategies that define analytical runs differently because it does not depend on the frequency of QC testing (2), nor is it influenced by different laboratory testing modes. To overcome this limitation, the expected increase in the number of unacceptable patient results reported during the presence of an undetected out-of-control error condition [E([N.sub.UF])], has been recently proposed (2) as an alternative QC performance measure. E([N.sub.UF]) depends on the magnitude of the increase in the analytical error that produces the out-of-control condition but has a defined maximum [Max E([N.sub.UF])[, corresponding to the worst-case situation for the associated QC strategy. Obtaining a Max E([N.sub.UF]) value acceptable for the laboratory has been suggested as a goal to guide the SQC planning process (3, 4).

In comparison with [P.sub.EDC], Max E([N.sub.UF]) is more directly related to the probability of causing harm to a patient due to the report of an incorrect result, and it is more related to the recent introduction of risk management concepts for QC planning in the clinical laboratory (5). However, the exact relationship between these performance measures is unknown, and the outcome of the traditional approach to SQC planning in terms of the number of erroneous patient results reported is not well established.

On the other hand, it is difficult for a laboratory to develop a SQC plan based on Max E([N.sub.UF]) because, in contrast with the traditional approach, neither graphical nor computer tools are readily available for the estimation of this performance measure and little guidance is available regarding how to use it in practice.

The purpose of this study was to investigate the relationship between [P.sub.EDC] and Max E([N.sub.UF]) for simple QC procedures widely used in clinical laboratories and to construct charts relating Max E([N.sub.UF]) with the capability of the analytical process that allow for SQC planning based on the probability of harming a patient due to the report of an incorrect result.

Materials and Methods

All investigations were conducted assuming a continuous-mode testing process with bracketed QC, as described previously by Parvin (3). In brief, patient samples were assumed to be analyzed in a continuous stream with periodic analysis of a set of controls. QC results were evaluated and a decision was made about all samples analyzed from the last accepted QC. It was assumed that an out-of-control condition can occur with equal probability anywhere within the stream of samples being analyzed, and that it persists until it is detected and that the test results are not reported until the next set of control results are accepted.

The capability of the analytical process is characterized by its [sigma] value which determines the number of patient results with unacceptable errors [>allowable total error ([TE.sub.a]), the quality requirement for the procedure in terms of a total allowable error] that are produced while the process is stably operating. The [sigma] metric is calculated as ([TE.sub.a] - [absolute value of [bias.sub.meas])/[s.sub.meas]-where [bias.sub.meas] the analytical bias and [s.sub.meas] is the analytical SD of the procedure during stable operation.

The number of erroneous results produced will increase with the occurrence of additional errors such as a systematic shift or an increase in the imprecision of the analytical process, which leads to an out-of-control condition. In this study, only systematic errors were investigated. The increase in systematic error ([DELTA]SE) should be detected by the QC procedure to prevent the report of erroneous results. A QC procedure is defined as the combination of the QC rule applied and the number of QC samples tested in each QC event (e.g., l2s n = 3).

When the size of [DELTA]SE is sufficient to produce 5% of patient results with error (i.e., exceeding the [TE.sub.a]), a critical systematic error ([DELTA]S[E.sub.C]) has occurred, and the probability to detect it ([P.sub.ED]C) has been used to characterize the performances of the QC procedures according to the traditional approach of the SQC planning process.

[DELTA]S[E.sub.C] is calculated as multiples of [smea.sub.s] by using the following equation (Eq. 1):

[DELTA]S[E.sub.C] = [([TE.sub.a] - [absolute value of ([biasmea.sub.s]])/[smea.sub.s]] - 1.65 (1)

where 1.65 is the z-value that corresponds to 5% of the area in 1 tail of a gaussian distribution, or:

[DELTA]S[E.sub.C] = [sigma] - 1.65

The expected increase in the number of unacceptable patient results reported during the existence of an undetected out-of-control error condition, E([N.sub.UF]), is computed, according to (3), as in Eq. 2

E([N.sub.UF]) = [DELTA][P.sub.E] [M(AR[L.sub.ED] - 1) - (1 - 1/AR[L.sub.ED])(M/2)] (2)

where, [DELTA][P.sub.E] represents the increase in the probability of producing unacceptable patient results because of the presence of the out-of-control error condition, ARLPD is the average run length to error detection (the average number of batches that are processed before a QC rule rejection occurs), and M is the expected number of patient samples tested between QC events.

The average run length to error detection, ARLPD, is computed as 1/[P.sub.ED]) (2), where [P.sub.ED] is the probability of error detection by the QC procedure, computed by integration of the density function of the normal distribution, and [DELTA][P.sub.E] is computed as the probability that the error for a result exceeds [TE.sub.a] in the presence of an out-of-control error condition minus the probability the error for a result exceeds [TE.sub.a] when the process is in control (6)

The probability of a false rejection, PFR, is the value taken by [P.sub.ED] when [DELTA]SE = 0.

All results were obtained by numerical analysis and most were confirmed by Monte Carlo simulation.

Results

The probability of a given QC procedure to detect an increase in the analytical error, [P.sub.ED], depends on the magnitude of the increase, [DELTA]SE, and the capability of the analytical process, cr.

Fig. 1 plots [P.sub.ED] as a function of [DELTA]SE for a 4cr analytical process when 2 QC samples are analyzed and an l3s rule is applied in each QC event. The probability of error detection increases as does [DELTA]SE, leading to a typical sigmoidal curve (power function curve). In this example, the critical systematic error [DELTA]S[E.sub.C] = 2.35 ([sigma]r - 1.65, as multiples of the analytical SD) and the probability to detect it [P.sub.EDC] = 0.450.

The [P.sub.EDC] value for a QC procedure is the same when used to control analytical processes with the same [sigma] and is independent of the number of patient samples analyzed between QC events and the laboratory testing mode.

The expected increase in the number of unacceptable patient results reported during the existence of an undetected out-of-control error condition, E([N.sub.UF]), depends on the same variables as [P.sub.ED], but also on the frequency of QC and the laboratory testing mode (3).

In Fig. 1, E([N.sub.UF]) is plotted as a function of [DELTA]SE for a 4[sigma] analytical procedure when 100 patient samples are analyzed between QC events, 2 QC samples are analyzed in each QC event, and a [1.sub.3s] QC rule is applied, assuming a continuous-mode testing process with bracketed QC (see Materials and Methods for description). The value of E([N.sub.UF]) starts at zero, increases as the size of the systematic error increases until it reaches a maximum, Max E([N.sub.UF]), and then decreases to values close to zero, resulting in a bell-shaped curve.

The value of E([N.sub.UF]) increases at the initial part of the curve because [DELTA]SE is not large enough to be detected efficiently by the QC procedure (low [P.sub.ED]) so that the few erroneous results that are produced are also reported. At the maximum, the error is large enough to produce many erroneous results but not large enough to be readily detected, which corresponds to the worst-case situation for the QC strategy used. The decline observed afterward occurs because [DELTA]SE is so great that it is immediately detected by the QC procedure (high [P.sub.ED]) such that, of the large number of erroneous results that are produced, relatively few are reported.

For the example shown in Fig. 1, Max E([N.sub.UF]) = 4.77 at [DELTA]SE = 2.18 when [P.sub.ED] = 0.367. It is noteworthy that the [DELTA]SE at which Max E([N.sub.UF]) is reached usually does not match with the corresponding [DELTA]S[E.sub.C] value.

For analytical processes with the same capability, controlled with simple QC rules of the form [1.sub.ks] (k = 2, 2.5, 3, 3.5, 4), analyzing the same number of controls per QC event, and the same number of patient samples between QC events, the calculated Max E([N.sub.UF]) value depends on the exact combination of imprecision and bias that characterizes the [sigma] of the analytical process. This effect is more apparent at lower values of [sigma] and higher values of k for the QC rule.

In Fig. 2, Max E([N.sub.UF]) is plotted as a function of the [sigma] of the analytical process under control for a laboratory that works according to a continuous-mode testing process with bracketed QC and 100 patient samples analyzed between QC events and that uses simple QC procedures with different numbers of QC samples tested in each QC event (from 1-3) and different control rules (from [1.sub.2s] to [1.sub.4s]).

Each QC rule in the nomograms is represented by 2 lines, with the upper line corresponding to the analytical method with bias = 0.9 [TE.sub.a] and the lower line corresponding to the unbiased method. The lines are practically superimposable, except at [sigma] values below 3, where the effect of bias on the computed value of Max E([N.sub.UF]) is more evident. For analytical methods with an intermediate bias, the corresponding value of Max E([N.sub.UF]) is located between these 2 lines.

As expected, for a given QC procedure, the Max E([N.sub.UF]) values are lower for analytical processes with higher [sigma] values. This is because the magnitude of the increase in systematic error that must occur to produce a significant number of unacceptable patient results is greater for higher [sigma] values and will be detected more easily by the QC procedure, avoiding the report of erroneous results.

For a given value of [sigma], the computed Max E([N.sub.UF]) values are lower for QC procedures with tight limits (i.e., [1.sub.2s]) because they are more sensitive to increments in the systematic error, although they also have a higher probability of false rejections.

The graphical representations in Fig. 2 can be used as nomograms for selecting the appropriate QC procedure to obtain the desired Max E([N.sub.UF]), when the [sigma] of the analytical process is known. For example, a laboratory that uses an analytical process with [sigma] = 4 and works according to a continuous-mode testing process with bracketed QC and an average of 100 patient samples analyzed between QC events will report a maximum of 4.8 unacceptable patient results, on average, whenever an out-of-control condition occurs if it selects a QC procedure [1.sub.3s] n = 2. Because Max E([N.sub.UF]) is proportional to the number of patient samples analyzed between QC events, M, the Max E([N.sub.UF]) value could be lowered below 1 simply by making M = 20 or changing the QC procedure to [1.sub.2.5s] when n = 2 and M = 65, but in this last case, the probability of false rejection will increase.

It should be noted that, by definition, the Max E([N.sub.UF]) value represents the expected increase in the number, but not the total number of erroneous results, reported during the existence of an undetected out-of-control condition, and that all analytical processes in stable operation produce a number of erroneous results according to their own [sigma] values. For these reasons, the total number of erroneous results reported when using analytical processes of poor performance (low [sigma] values) can be significantly higher than the corresponding Max E([N.sub.UF]) value. For example, for a 2[sigma] analytical process in stable operation that is controlled with a [1.sub.2s] QC procedure with n = 2, a total of 46.6 erroneous patient results, on aver age, would be reported between false QC rejections, whereas the Max E([N.sub.UF]) value is only 23.4 for this combination of QC procedure and [sigma] (if bias = 0), according to Fig. 2.

Each QC procedure has its own values of Max E([N.sub.UF]) and [P.sub.EDC] when used to control an analytical process with a given [sigma] value. In Fig. 3, Max E([N.sub.UF]) is plotted as a function of [P.sub.EDC] for all QC procedures in Fig. 2. The plot shows that the relationships between these performance measures are very similar for all rules considered, especially for rules with the same N value, so that one can estimate the value of one from the other with a relatively small margin of error, almost regardless of the QC procedure considered. For example, the computed Max E([N.sub.UF]) value for a QC procedure that has a probability of 0.90 to detect critical systematic errors ranges from 0.30 for [1.sub.2s] when n = 1-0.41 for l4s when n = 3. Conversely, to obtain an average of less than 1 unacceptable patient result as a consequence of an out-of-control condition, the plot shows that a QC procedure with a [P.sub.EDC] [degrees]f about 0.8 is needed, assuming that 100 patient samples are analyzed between QC events in both examples.

Discussion

The analytical quality of a clinical laboratory depends primarily on the number of erroneous results that are reported to health providers, and it is important to ensure that wrong decisions are not made in the management of patients.

According to the risk management techniques that are currently being introduced in clinical laboratories, the analytical quality of the laboratory can be improved by identifying and evaluating potential causes of failure in the testing process, estimating the risk of harm that they represent to the patient, and designing strategies to prevent failures and to detect errors before incorrect results are reported, through the development of a QC plan (5).

These strategies include, but are not limited to, QC procedures using stable control materials. An increasing proportion of errors affecting the quality of the analytical results has its origins outside the analytical phase of the testing process. These errors are usually random and unpredictable, tend to affect individual samples, and are not detected effectively by QC procedures based on the analysis of control materials (7). Nevertheless, these strategies still have a central role for detecting errors caused by excessive process variation (such as an increase in the systematic error) that affect patient and control samples in a similar fashion. In this study, only systematic errors are addressed; however, increases in analytical imprecision also occur frequently, are difficult to detect and can result in the erroneous reporting of patient results.

Because the risk can never be completely eliminated, the goal of these strategies is to reduce the probability of harming the patient to a residual level that could be considered acceptable from the point of view of patient safety.

The Max E([N.sub.UF]) value represents the expected number of final erroneous results that could be reported in the worst case scenario, results that are not subjected to further correction and that could be used for making erroneous clinical decisions. Therefore, Max E([N.sub.UF]) is directly related to the probability of patient harm and can be used to estimate it, provided that both, the mean number of patient results examined between failures and the probability that an incorrect result lead to patient harm, are known (8). The probability of harm can be lowered to an acceptable level by minimizing the Max E([N.sub.UF]) value through the selection of appropriate QC procedures and the frequency of QC events in the SQC planning process.

We have shown that the Max E([N.sub.UF]) value can be easily estimated, at least for simple QC procedures, either from the [sigma] value that characterizes the analytical process by using nomograms as shown in Fig. 2, or from the probability of detection of critical systematic errors [P.sub.EDC], when a traditional approach to the SQC planning is used, as shown in Fig. 3. The same value of Max E([N.sub.UF]) can be achieved by using different combinations of QC procedures and numbers of patient samples analyzed between QC events.

There is an inverse relationship between the limits of the SQC rules applied and the corresponding value of M that should be used to obtain an acceptable Max E([N.sub.UF]) value. Rules with wider limits, and thus lower PFR, can be used by decreasing the M value accordingly, and vice versa.

The exact combination selected will determine the efficiency of the SQC strategy, i.e., the number of results reported per sample (patients and controls) analyzed. Low values of M tend to decrease efficiency because the number of controls analyzed per patient sample increases. In contrast, QC procedures with low PFR tend to improve efficiency because they minimize the repetition of patient and control samples.

However, it should be noted that efficiency is a complex issue that depends not only on the capability of the analytical process, but also on other factors such as the reliability of the analytical system (9) and the cost of testing QC material vs the costs associated with false QC rejections.

Max E([N.sub.UF]) only represents the expected increase in the number of erroneous results reported in the presence of an out-of-control error condition and does not reflect accurately the total number of erroneous results reported when low-[sigma] ([sigma] < 3) analytical processes are used. In these cases, many of the reported erroneous results are produced because the inherent variability of the analytical process, whether the process is out-of-control or not, and Max E([N.sub.UF]) underestimates the total number of erroneous results that are being reported, and thus, the patient risk. Appropriate selection of QC procedures can reduce the expected number of unreliable final patient results owing to an out-of-control condition but in no way can a QC strategy, regardless of how stringent it is, compensate for the bad quality of an analytical method (10).

The selection of SQC procedures has been done classically with the objective of having 90% detection of critical systematic errors, while maintaining false rejections as low as possible (11). Fig. 3 shows that values for Max E([N.sub.UF]) in the range from 0.3-0.4 are obtained when simple QC procedures with [P.sub.EDC] = 0.90 are used to control an analytical process. This means that up to about 300 patient samples could be analyzed between QC events for reporting less than 1 erroneous result during an out-of-control condition, which represents a typical daily workload for many tests performed in a small or medium-sized laboratory. Therefore, the use of [P.sub.EDC] = 0.90 as a general criterion appears to be quite reasonable for selecting QC procedures that reduce the probability of harming the patient to a level that could be considered acceptable for many tests conducted in a laboratory.

For the simple QC rules investigated here, obtaining [P.sub.EDC] = 0.90 while maintaining an acceptable rejection rate is only possible in analytical processes with fairly high [sigma] metrics. For processes with lower [sigma] values, the laboratory should select the appropriate number of patient samples to be analyzed between QC events to obtain an acceptable value for Max E([N.sub.UF]) given the [P.sub.EDC] that can be reasonably achieved.

Regardless, we have shown that there is a practically biunivocal relationship between Max E([N.sub.UF]) as a percentage of the average number of patients tested between QC events (M), and [P.sub.EDC] over a range of simple QC rules of the form Iks (k = 2, 2.5, 3, 3.5, 4); therefore, the value of [P.sub.EDC] can be estimated from the value of Max E([N.sub.UF]) and vice versa. In this way, the [P.sub.EDC] value may be used for selecting appropriate QC procedures during the development of a QC plan based on risk management.

Since the introduction of the Six Sigma concept into clinical laboratories (12), several recommendations have been made for selecting optimal SQC procedures considering only the [sigma] of the analytical process (13-15 )? Unless there is some exception (13), none of these recommendations take into account the QC frequency or the laboratory testing mode, but most of the SQC procedures recommended for control analytical processes with a certain [sigma] show low Max E([N.sub.UF]) values (typically below 1 in the nomogram of Fig. 2) because they have been selected to provide a high probability of detection of critical errors ([P.sub.EDC] > 0.90).

In summary, there is a close relationship between [P.sub.EDC] and Max E([N.sub.UF]) for simple QC procedures generally applied in clinical laboratories that allow the use of [P.sub.EDC] for estimating the probability of patient harm in SQC planning based on risk management. QC procedures selected by their high probability of detection of critical errors, according to the traditional approach for SQC planning, are also characterized by a low number of erroneous patient results reported during the presence of an out-of-control error condition reconciling these 2, apparently different, approaches.

Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 3 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; and (c) final approval of the published article.

Authors' Disclosures or Potential Conflicts of Interest: No authors declared any potential conflicts of interest.

Role of Sponsor: The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, or preparation or approval of manuscript.

References

(1.) Westgard JO. Internal quality control: planning and implementation strategies. Ann Clin Biochem 2003;40: 593-611.

(2.) Parvin CA, Gronowski AM. Effect of analytical run length on quality-control (QC) performance and the QC planning process. Clin Chem 1997;43:2149 -54.

(3.) Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem 2008;54:2049-54.

(4.) Yundt-Pacheco J# Parvin CA, Validating the performance of QC procedures. Clin Lab Med 2013;33:75-88.

(5.) Njoroge SW, Nichols JH. Risk management in the clinical laboratory. Ann Lab Med 2014;34:274-8.

(6.) Ye JJ, Ingels SC, Parvin CA. Performance evaluation and planning for patient-based quality control procedures. AmJ Clin Pathol 2000;113:240-8.

(7.) Hinckley CM. Defining the best quality-control systems by design and inspection. Clin Chem 1997;43:873-9.

(8.) Parvin CA. QC for the future: risk assessment in laboratory practice. https://wwwyoutube.com/watch?v=o7 Ja2868flM&index=12&list=PLVMFW6ATuGNhsQFRu 8y1 LM-hdluSTubj A. (Accessed July 2015).

(9.) Hatjimihail AT. Estimation of the optimal statistical quality control sampling time intervals using a residual risk measure. PLoS 0ne2009;4:e5770.

(10.) Bayat H. QC-dependent risk reduction: clinical chemistry journal club comments, http://wwwwestgard.com/ qc-risk-reduction.htm. (Accessed July 2015).

(11.) Koch DD, Oiyall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system. Clin Chem 1990;36: 230-3.

(12.) Gras JM, Philippe M. Application of the SixSigma concept in clinical laboratories: a review. Clin Chem Lab Med 2007;45:789-96.

(13.) Cooper G, DeJonge N, Ehrmeyer S, Yundt-Pacheco J, Jansen R, Ricos C, Plebani M. Collective opinion paper on findings of the 2010 convocation of experts on laboratory quality. Clin Chem Lab Med 2011;49:793802.

(14.) Schoenmakers CHH, Naus AJM, Vermeer HJ, van Loon D, Steen G. Practical application of Sigma Metrics QC procedures in clinical chemistry. Clin Chem Lab Med 2011;49:1837-43.

(15.) Westgard JO. Best practices for "Westgard Rules". https://wwwwestgard.com/lesson74.htm. (Accessed July 2015).

Martin Yago [1] * and Silvia Alcover [1]

[1] Laboratory of Biochemistry, Hospital General de Requena, Valencia, Spain.

[2] Nonstandard abbreviations: SQC, statistical QC; [P.sub.EDC], probability of critical error detection; E([N.sub.UF]), expected number of unreliable final patient results; Max E([N.sub.UF]), maximum E([N.sub.UF]); [TE.sub.a], allowable total error; [DELTA]SE, increase in systematic error; [DELTA]S[E.sub.C], critical [DELTA]SE.

* Address correspondence to this author at: Hospital General de Requena, Servicio de Laboratorio, Paraje Casablanca s/n 46530 Requena, Valencia, Spain. Fax +962339291; e-mail martinyago.lopez@gmail.com.

Received December 31,2015; accepted March 24,2016.

Previously published online at DOI: 10.1373/clinchem.2015.254094

Caption: Fig. 1. [P.sub.ED] (dotted line) and E([N.sub.UF]) (solid line) as a function of the magnitude of a systematic out-of-control error condition, for a 4[sigma] analytical process.

The process is controlled by using a bracketed QC with a [1.sub.3s] rule (n = 2) and 100 patient samples analyzed between QC events. [DELTA]SE is expressed as multiple of the analytical SD.

Caption: Fig. 2. The maximum expected increase in the number of unacceptable patient results reported during the existence of an undetected out-of-control condition, Max E([N.sub.UF]), as a function of the [sigma] of the analytical process under control for simple QC procedures based on the testing of 1 (n = 1), 2 (n = 2), or 3 (n = 3) QC samples by QC event.

In each of the graphs, the curves correspond, from bottom to top, to the control rules [1.sub.2s], [1.sub.2.5s], [1.sub.3s], [1.sub.3.5s], and [1.sub.4s], respectively. The statistical model assumes a continuous-mode testing process with bracketed QC and 100 patient samples analyzed between QC events. Each QC rule in the Fig. is represented by 2 partially overlapping lines, 1 of them corresponding to the analytical method without bias and the other to the method with bias = 0.9[TE.sub.a].

Caption: Fig. 3. The maximum expected increase in the number of unacceptable patient results reported during the existence of an undetected out-of-control condition, Max E([N.sub.UF]), as a function of the probability of detection of critical systematic errors for all QC procedures represented in Fig. 2.

Printer friendly Cite/link Email Feedback | |

Title Annotation: | Informatics and Statistics |
---|---|

Author: | Yago, Martin; Alcover, Silvia |

Publication: | Clinical Chemistry |

Article Type: | Report |

Date: | Jul 1, 2016 |

Words: | 4660 |

Previous Article: | Recombinase polymerase amplification for diagnostic applications. |

Next Article: | Iterative outlier removal: a method for identifying outliers in laboratory recalibration studies. |

Topics: |