# A quick, simple protocol for evaluating clinical chemistry methods.

A quick, simple protocol for evaluating clinical chemistry methods

Would you like to be able to test the initial performance of a new method or instrument quickly and inexpensively? Or get quick insight into an analytical problem? If so, the National Committee for Clinical Laboratory Standards has a protocol that can really help you.

Guideline EP10-T, "Preliminary Evaluation of Clinical Chemistry Methods," from the Evaluation Protocols Area Committee of the NCCLS provides a written procedure that uses only 55 assays over five days. The protocol allows you not only to evaluate precision, linearity, and bias without statistical manipulation (unless the data suggest a problem) but also to evaluate carryover and short-term drift. Like all NCCLS standards and guidelines, EP10 is open to public comments as part of the consensus process. Comments may be sent directly to the NCCLS.

While EP10 does not answer all evaluation questions and its statistical "power" is limited because of the small number of assays, it provides useful information with a minimal expenditure of time, money, and materials. The protocol can be used to provide documentation when you put a new method or instrument into service. One of the main advantages of the various NCCLS protocols is that each provides a procedure to follow for evaluating certain types of analytical problems.

If you find a problem, you can compensate for EP10's limited statistical power by using additional protocols prepared by the NCCLS's Evaluation Protocols Area Committee (Table I). You may also increase the power by using more specimens, as described in the appendix to EP10.

Because data are evaluated visually, most users will not have to calculate the statistics. If you prefer to proceed, you can easily obtain statistical verification of analytical problems by using the convenient data sheets provided.

Since few clinical or analytical criteria, analytical goals, or allowab le errors are universally accepted, users must determine their own. These allowable errors are used to test the significance of the observed analytical problems. This means that you must set your own values for allowable imprecision, bias, nonlinearity, drift, and carryover. Or you may use the manufacturer's stated performance claims to test calculated statistics that appear significant.

If the calculated statistics exceed the allowable goals or you suspect a problem, you can evaluate statistical significance at that time. The optional and more complicated calculation and interpretation of statistical significance is simplified by the written instructions and data sheets.

* Protocol summary. The secret to using EP10 efficiently is to follow the recommended analytical sequence of specimens precisely. You begin by preparing two well-mixed stable pools representing the high and low concentrations to be tested. The high pool may be wliked with appropriate materials. A third sample is prepared by intermixing the high and low pools in an even ration of 1:1. The resulting concentration, called the mid-pool, must be exactly midway between those of the two extreme pools.

Take care to insure matrix compatibility with your method. This usually means that human-based material, representing the desired state of health or disease to be tested, is best. If the materials for the high and low pool have known concentrations that are traceable to reference methodology, then bias can be estimated. Following the analytical sequence as specified in the protocol--mid, mid, hi, low, mid, mid, low, low, hi, hi, mid--is crucial, since it is from this sequence that statistical estimations will be made.

As in all statistical models, outliers are very destructive in this model. An outlier is a detached point that represents an error. Unless recognized and eliminated from the data, outliers, which seriously skew calculated statistics, will invalidate your interpretation.

To simplify this protocol, the user determines by visual inspection whether a data point is an outlier. This step is easier after the data have been plotted as shown in the example accompanying this article. If you suspect a point to be an outlier but aren't sure, allow the data it represents to remain in the study. If you do identify an outlier, try to determine the reason for the error so that you can document it.

All outliers, including the other assays in the run (11 in all), are eliminated from the study. The run involved is then repeated. Finding more than one unexplained outlier provides enough evidence of analytical problems to consult the manufacturer. to follow up, you may wish to run one of the other Evaluation Protocols from Table I

Most of the statistical tests for identifying an outlier require that a point be detached by a distance equal to or greater than multiple within-run standard deviations (3 or more) from the rest of the data. If your within-run standard deviation was 2 mg/dl, for example, each outlier would have to be detached by more than 6 mg/dl. Visual evaluation is a simple yet highly sensitive method for eliminating data outside the legitimate population.

The 55 data points gathered over five days represent two primes and three replicates of each pool per day. Recording the data on summary and calculation sheers and ploting points facilitate analysis. Commercial software packages to assist you are available, but many users complete this step with a hand calculator.

The plotted data make it easy to evaluate precision, linearity, bias, and outliers visually. Only if you suspect an unacceptable variation do you continue with the calculations. You may want to determine an uncorrected total standard deviation with a calculator, using all the data points in each pool (n = 15). If the uncorrected total imprecision is acceptable, you won't have to calculate total corrected imprecision or evaluate such components of total imprecision as within run or day to day.

EP10 calculates total imprecision with corrections for the effect of within-run variance by reducing its contribution to one-third. A common result is to end up with lower estimates of total imprecision than when the uncorrected calculation is used, especially if within-run variance is high.

While linearity is also evaluated visually, statistical verification and significance can be calculated. The current edition of EP10 contains an error in the calculation of statistical significance; this will be corrected in the next edition. If you suspect nonlinearity from your visual inspection, investigate it with EP6-P, "Evaluation of the Linearity of Quantitative Analytical Methods."

Bias is calculated as the difference between the grand mean (n = 15) of each of the three pools and their known values. If the values of your original pools are in doubt, you won't be able to calculate valid bias values. Take care not to have matrix interferences, changes in the analysis values caused by constituents in the specimens that behave differently from the specimens you normally analyze.

For example, some methods provide different values if different proteins are found in the specimen or if the electrolyte concentration is inconsistent. Compare your calculated bias with the bias that you will allow or that the manufacturer claims to be acceptable.

Be aware of the danger of over-interpreting statistical significance when precision is very high (SD is small) and deviation or error wgp jically insignificant. The opposite danger exists when imprecision is large and deviations are significant, in which case clinically significant error becomes acceptable in the statistical evaluation. The main benefit of visual evaluation is to help you determine whether the deviation is clinically significant.

Nonlinearity carryover and drift are calculated from a multiple linear regression analysis. Each component is represented by a parameter from the multiple linear regression equation. Doing these calculations is the most difficult part of EP10, but following thet data sheets and the extensive example given in the protocol facilitate ejalysis. You may prefer not to bother following the directions for calculating a multiple linear regression coefficient for slope, which does not add any meaningful information and will probably be eliminated from the next edition.

The example on the adjoining page illustrates the system.

Would you like to be able to test the initial performance of a new method or instrument quickly and inexpensively? Or get quick insight into an analytical problem? If so, the National Committee for Clinical Laboratory Standards has a protocol that can really help you.

Guideline EP10-T, "Preliminary Evaluation of Clinical Chemistry Methods," from the Evaluation Protocols Area Committee of the NCCLS provides a written procedure that uses only 55 assays over five days. The protocol allows you not only to evaluate precision, linearity, and bias without statistical manipulation (unless the data suggest a problem) but also to evaluate carryover and short-term drift. Like all NCCLS standards and guidelines, EP10 is open to public comments as part of the consensus process. Comments may be sent directly to the NCCLS.

While EP10 does not answer all evaluation questions and its statistical "power" is limited because of the small number of assays, it provides useful information with a minimal expenditure of time, money, and materials. The protocol can be used to provide documentation when you put a new method or instrument into service. One of the main advantages of the various NCCLS protocols is that each provides a procedure to follow for evaluating certain types of analytical problems.

If you find a problem, you can compensate for EP10's limited statistical power by using additional protocols prepared by the NCCLS's Evaluation Protocols Area Committee (Table I). You may also increase the power by using more specimens, as described in the appendix to EP10.

Because data are evaluated visually, most users will not have to calculate the statistics. If you prefer to proceed, you can easily obtain statistical verification of analytical problems by using the convenient data sheets provided.

Since few clinical or analytical criteria, analytical goals, or allowab le errors are universally accepted, users must determine their own. These allowable errors are used to test the significance of the observed analytical problems. This means that you must set your own values for allowable imprecision, bias, nonlinearity, drift, and carryover. Or you may use the manufacturer's stated performance claims to test calculated statistics that appear significant.

If the calculated statistics exceed the allowable goals or you suspect a problem, you can evaluate statistical significance at that time. The optional and more complicated calculation and interpretation of statistical significance is simplified by the written instructions and data sheets.

* Protocol summary. The secret to using EP10 efficiently is to follow the recommended analytical sequence of specimens precisely. You begin by preparing two well-mixed stable pools representing the high and low concentrations to be tested. The high pool may be wliked with appropriate materials. A third sample is prepared by intermixing the high and low pools in an even ration of 1:1. The resulting concentration, called the mid-pool, must be exactly midway between those of the two extreme pools.

Take care to insure matrix compatibility with your method. This usually means that human-based material, representing the desired state of health or disease to be tested, is best. If the materials for the high and low pool have known concentrations that are traceable to reference methodology, then bias can be estimated. Following the analytical sequence as specified in the protocol--mid, mid, hi, low, mid, mid, low, low, hi, hi, mid--is crucial, since it is from this sequence that statistical estimations will be made.

As in all statistical models, outliers are very destructive in this model. An outlier is a detached point that represents an error. Unless recognized and eliminated from the data, outliers, which seriously skew calculated statistics, will invalidate your interpretation.

To simplify this protocol, the user determines by visual inspection whether a data point is an outlier. This step is easier after the data have been plotted as shown in the example accompanying this article. If you suspect a point to be an outlier but aren't sure, allow the data it represents to remain in the study. If you do identify an outlier, try to determine the reason for the error so that you can document it.

All outliers, including the other assays in the run (11 in all), are eliminated from the study. The run involved is then repeated. Finding more than one unexplained outlier provides enough evidence of analytical problems to consult the manufacturer. to follow up, you may wish to run one of the other Evaluation Protocols from Table I

Most of the statistical tests for identifying an outlier require that a point be detached by a distance equal to or greater than multiple within-run standard deviations (3 or more) from the rest of the data. If your within-run standard deviation was 2 mg/dl, for example, each outlier would have to be detached by more than 6 mg/dl. Visual evaluation is a simple yet highly sensitive method for eliminating data outside the legitimate population.

The 55 data points gathered over five days represent two primes and three replicates of each pool per day. Recording the data on summary and calculation sheers and ploting points facilitate analysis. Commercial software packages to assist you are available, but many users complete this step with a hand calculator.

The plotted data make it easy to evaluate precision, linearity, bias, and outliers visually. Only if you suspect an unacceptable variation do you continue with the calculations. You may want to determine an uncorrected total standard deviation with a calculator, using all the data points in each pool (n = 15). If the uncorrected total imprecision is acceptable, you won't have to calculate total corrected imprecision or evaluate such components of total imprecision as within run or day to day.

EP10 calculates total imprecision with corrections for the effect of within-run variance by reducing its contribution to one-third. A common result is to end up with lower estimates of total imprecision than when the uncorrected calculation is used, especially if within-run variance is high.

While linearity is also evaluated visually, statistical verification and significance can be calculated. The current edition of EP10 contains an error in the calculation of statistical significance; this will be corrected in the next edition. If you suspect nonlinearity from your visual inspection, investigate it with EP6-P, "Evaluation of the Linearity of Quantitative Analytical Methods."

Bias is calculated as the difference between the grand mean (n = 15) of each of the three pools and their known values. If the values of your original pools are in doubt, you won't be able to calculate valid bias values. Take care not to have matrix interferences, changes in the analysis values caused by constituents in the specimens that behave differently from the specimens you normally analyze.

For example, some methods provide different values if different proteins are found in the specimen or if the electrolyte concentration is inconsistent. Compare your calculated bias with the bias that you will allow or that the manufacturer claims to be acceptable.

Be aware of the danger of over-interpreting statistical significance when precision is very high (SD is small) and deviation or error wgp jically insignificant. The opposite danger exists when imprecision is large and deviations are significant, in which case clinically significant error becomes acceptable in the statistical evaluation. The main benefit of visual evaluation is to help you determine whether the deviation is clinically significant.

Nonlinearity carryover and drift are calculated from a multiple linear regression analysis. Each component is represented by a parameter from the multiple linear regression equation. Doing these calculations is the most difficult part of EP10, but following thet data sheets and the extensive example given in the protocol facilitate ejalysis. You may prefer not to bother following the directions for calculating a multiple linear regression coefficient for slope, which does not add any meaningful information and will probably be eliminated from the next edition.

The example on the adjoining page illustrates the system.

Printer friendly Cite/link Email Feedback | |

Author: | Passey, Richard B. |
---|---|

Publication: | Medical Laboratory Observer |

Date: | Dec 1, 1990 |

Words: | 1308 |

Previous Article: | Developing a laboratory chemical hygiene plan. |

Next Article: | The university's role in helping MTs feel like professionals. |

Topics: |