# Taking the 'byte' out of method evaluation number crunching.

Taking the |byte' out of method evaluation number crunching

I created the template described here out of frustration with the calculator mode of our laboratory's mainframe, which I had been using for statistical calculations. Not only were its editing capabilities very limited, but graphs of the data had to be done by hand (and therefore were rarely done at all).

In the clinical laboratory we are often confronted with evaluation data from method comparison studies. Time and cost constraints make it important for the study data to provide information toward making practical decisions. My program accomplishes this.

Using our lab's method evaluation protocol, based in part on NCCLS guideline EP9-P,(1) we developed a spreadsheet template in Excel (Microsoft Corp., Redmond, Wash.). Unlike some spreadsheets created for business applications, Excel's produces graphs with two value axes.

Data for entry are derived from precision, interference, linearity, and patient comparison studies in the laboratory. We also enter the manufacturer's claims for reproducibility, recovery, and other performance characteristics and the allowable error (Ea) at the medical decision level.(2) The program calculates within-run, between-run, and between-day precision and compares generated data with manufacturer's claims for those areas, when given.

The t test (a statistical test to measure the significance of difference between means of paired data), F test (a statistical test to measure the significance of variability between standard deviations of the paired data), and least squares linear regression are calculated from the patient comparison data. The results are evaluated for statistical significance at a 95% confidence level. Scatter, bias, and linearity plots are generated. From the computed statistics, total error at the medical decision level is evaluated and tested for acceptability.

* Data entry and calculations. There are seven macros. The first three facilitate entering the manufacturer's specifications for the three concentration levels spanning the range of results to be reported. For each macro, you can enter the mean and the standard deviation (SD) for within-run, between-run, and between-day values at a particular concentration level. The fourth macro is used to prepare linearity, scatter, and bias graphs. The last three macros assist entry of the medical decision level (MDL), the allowable error at the MDL for the analyte being tested, and the QC parameters for the reference method used.

It is usually best to start by running the macros for entering the manufacturer's specifications, followed by the macro for entering reference values. Study data are then entered directly into the template.

The template is set up somewhat like a tax form, with boxed cells indicating the places to insert data. Precision data can be evaluated for up to three levels of control, five replicates per run, and two runs per day for five days. The spreadsheet will accept up to 40 patient comparisons, done in duplicate on both the reference method and the evaluation method. If linearity and interference studies have been done, the resulting data are entered as well. (Because the automatic calculate slows down data entry, I have set up the spreadsheet to calculate only upon request.) Once all data have been entered, selecting "Calculate now" from the option menu will run the macros to prepare the graphs.

* Manufacturer's claims. The precision data are printed on the same page as the manufacturer's claims. Using "if tests" helps me evaluate the manufacturer's claims. (These are logical function tests, with true-false replies.) Next, I generate a printout stating that the study data meet or do not meet the manufacturer's claims.

*Interference. Interferent in varying amounts is added to a baseline pool. The spiked pools are measured in duplicate. Data are entered in the spreadsheet. Absolute bias and percentage bias are calculated for each level and printed with the manufacturer's claim. I evaluate the data by inspecting the printout.

*Linearity. Data from a set of dilutions run on both the reference and the evaluation methods are used to generate the linearity graph, which is assessed by visual inspection. Using the patient comparison data, the template will calculate the means, SDs, t test, and F test.

I compare the date from the precision study, displayed next to the patient comparison data, with the reference values previously entered via the macro (Figure I). The statistical parameters for the patient comparison study are displayed with the values for critical t and critical F, obtained from a lookup table within the template.

Statistical significance is evaluated by an "if test" comparing the observed value for t or F with the critical value at a 95% confidence level. Boxed cells on the screen represent the values typically entered via the macro. It is necessary to enter the control serum reference means in order to calculate the average difference of means. This last calculation, when compared with average difference of the means of patient samples, is useful for hinting at any matrix effects present. The reference value SDs are necessary for calculating critical F. Least squares linear regression, calculated from the patient comparison data, is displayed in Figure I.

One advantage of analyzing method evaluations on a computer is the ease with which graphs may be prepared. It is considered good practice to look at a scatterplot of linear regression data to see whether visual representation of the data reveals something that isn't apparent from statistical calculations alone. Bias plots (Figure II) are an increasingly popular means for examining method evaluation data.

In a bias plot, the arithmetic difference of an individual value of the evaluation method from the corresponding reference method value is plotted against the total range of values obtained on the reference method. I think of the bias plot as a dot chart displaying variation over the range of the method rather than against time. The positive bias evident in the sample bias plot in Figure II is more pronounced above the value 150 on the reference method.

* Analysis of total error. To analyze total error at the medical decision level (Figure III), one must perform several steps conceptually. This is true even though the calculations are accomplished by the template simultaneously, provided that all data and reference values have been entered.

The first step is to select the MDL and the allowable error at that level for the analyte being evaluated. The Ea for some analytes is listed in tables in reference books.(3) If you can't find such a table for the analyte used in your study, you will have to rely on your own judgment.

Random error (RE), defined as 1.96 multiplied by the SD of the control, includes 95% of the variations that will occur by chance. The magnitude of the RE is compared with the allowable error. If RE < Ea, the amount of error due to random variation is considered clinically acceptable.

Similarly, the equation for systemic error (SE), SE = y - Xc, is calculated by solving y = b{Xc} + a. This represents the equation of the line generated by performing the regression analysis for y and subtracting Xc, where b is the slope, a is the intercept, and Xc is the analyte concentration at the medical decision level.

In the example shown in Figure III, y = 101.5, giving an SE of 1.5. Again, the magnitude of the SE is compared with the Ea. When SE < Ea, the amount of error due to bias is considered acceptable. If bias is detected, inspecting the bias plot will help determine whether the bias is constant or proportional.

Total error (TE) represents the sum of random error and systemic error. Like SE, TE is compared with allowable error. If RE + SE < Ea, the total error is considered acceptable. If TE is acceptable at all three levels at which the control is run, the entire method evaluation is judged to be acceptable.

* Practicality. I set up the template so that it would be easy to use, even for medical technologists with limited PC expertise. The data input and calculations of a complete method evaluation can be accomplished in about 30 minutes. The template has been handy for assessing method evaluation data as well as for quick studies when troubleshooting instruments.

A prime advantage is the objectivity rendered possible by the template calculations and graphs. Although part of my goal was to make the template readily accessible to others, I often enter the data myself, since I find myself evaluating the summary statistics and error analysis as I go along.

Being able to generate graphs quickly and easily is a tremendous benefit of the program. When a problem arises during an evaluation, I come up with new studies for further evaluation as I do the data entry and calculations.

My spreadsheet template enables our laboratory to evaluate new procedures and methodologies completely and appropriately. It is particularly helpful in evaluating test kits. The ability to perform rapid yet thorough method evaluation analyses has improved our evaluation process and documentation of new methods. [Figures 1 to 3 Omitted] (1)National Committee for Clinical Laboratory Standards. User comparison of quantitative clinical laboratory methods using patient samples. Proposed guideline EP9-P. Villanova, Pa., NCCLS, 1986. (2)Statland, B.E. "Clinical Decision Levels for Lab Tests," 2nd ed. Oradell, N.J., Medical Economics Books, 1987. (3)Kaplan, L.A., and Pesce, A.J., eds. Evaluation of methods, chap. 19, and Laboratory statistics, chap. 16, in "Clinical Chemistry: Theory, Analysis, and Correlation." St. Louis, C.V. Mosby, 1989.

Donna J. Walsh, M.S., MT(ASCP) The author is a laboratory scientist in chemistry at New England Deaconess Hospital, Boston.

I created the template described here out of frustration with the calculator mode of our laboratory's mainframe, which I had been using for statistical calculations. Not only were its editing capabilities very limited, but graphs of the data had to be done by hand (and therefore were rarely done at all).

In the clinical laboratory we are often confronted with evaluation data from method comparison studies. Time and cost constraints make it important for the study data to provide information toward making practical decisions. My program accomplishes this.

Using our lab's method evaluation protocol, based in part on NCCLS guideline EP9-P,(1) we developed a spreadsheet template in Excel (Microsoft Corp., Redmond, Wash.). Unlike some spreadsheets created for business applications, Excel's produces graphs with two value axes.

Data for entry are derived from precision, interference, linearity, and patient comparison studies in the laboratory. We also enter the manufacturer's claims for reproducibility, recovery, and other performance characteristics and the allowable error (Ea) at the medical decision level.(2) The program calculates within-run, between-run, and between-day precision and compares generated data with manufacturer's claims for those areas, when given.

The t test (a statistical test to measure the significance of difference between means of paired data), F test (a statistical test to measure the significance of variability between standard deviations of the paired data), and least squares linear regression are calculated from the patient comparison data. The results are evaluated for statistical significance at a 95% confidence level. Scatter, bias, and linearity plots are generated. From the computed statistics, total error at the medical decision level is evaluated and tested for acceptability.

* Data entry and calculations. There are seven macros. The first three facilitate entering the manufacturer's specifications for the three concentration levels spanning the range of results to be reported. For each macro, you can enter the mean and the standard deviation (SD) for within-run, between-run, and between-day values at a particular concentration level. The fourth macro is used to prepare linearity, scatter, and bias graphs. The last three macros assist entry of the medical decision level (MDL), the allowable error at the MDL for the analyte being tested, and the QC parameters for the reference method used.

It is usually best to start by running the macros for entering the manufacturer's specifications, followed by the macro for entering reference values. Study data are then entered directly into the template.

The template is set up somewhat like a tax form, with boxed cells indicating the places to insert data. Precision data can be evaluated for up to three levels of control, five replicates per run, and two runs per day for five days. The spreadsheet will accept up to 40 patient comparisons, done in duplicate on both the reference method and the evaluation method. If linearity and interference studies have been done, the resulting data are entered as well. (Because the automatic calculate slows down data entry, I have set up the spreadsheet to calculate only upon request.) Once all data have been entered, selecting "Calculate now" from the option menu will run the macros to prepare the graphs.

* Manufacturer's claims. The precision data are printed on the same page as the manufacturer's claims. Using "if tests" helps me evaluate the manufacturer's claims. (These are logical function tests, with true-false replies.) Next, I generate a printout stating that the study data meet or do not meet the manufacturer's claims.

*Interference. Interferent in varying amounts is added to a baseline pool. The spiked pools are measured in duplicate. Data are entered in the spreadsheet. Absolute bias and percentage bias are calculated for each level and printed with the manufacturer's claim. I evaluate the data by inspecting the printout.

*Linearity. Data from a set of dilutions run on both the reference and the evaluation methods are used to generate the linearity graph, which is assessed by visual inspection. Using the patient comparison data, the template will calculate the means, SDs, t test, and F test.

I compare the date from the precision study, displayed next to the patient comparison data, with the reference values previously entered via the macro (Figure I). The statistical parameters for the patient comparison study are displayed with the values for critical t and critical F, obtained from a lookup table within the template.

Statistical significance is evaluated by an "if test" comparing the observed value for t or F with the critical value at a 95% confidence level. Boxed cells on the screen represent the values typically entered via the macro. It is necessary to enter the control serum reference means in order to calculate the average difference of means. This last calculation, when compared with average difference of the means of patient samples, is useful for hinting at any matrix effects present. The reference value SDs are necessary for calculating critical F. Least squares linear regression, calculated from the patient comparison data, is displayed in Figure I.

One advantage of analyzing method evaluations on a computer is the ease with which graphs may be prepared. It is considered good practice to look at a scatterplot of linear regression data to see whether visual representation of the data reveals something that isn't apparent from statistical calculations alone. Bias plots (Figure II) are an increasingly popular means for examining method evaluation data.

In a bias plot, the arithmetic difference of an individual value of the evaluation method from the corresponding reference method value is plotted against the total range of values obtained on the reference method. I think of the bias plot as a dot chart displaying variation over the range of the method rather than against time. The positive bias evident in the sample bias plot in Figure II is more pronounced above the value 150 on the reference method.

* Analysis of total error. To analyze total error at the medical decision level (Figure III), one must perform several steps conceptually. This is true even though the calculations are accomplished by the template simultaneously, provided that all data and reference values have been entered.

The first step is to select the MDL and the allowable error at that level for the analyte being evaluated. The Ea for some analytes is listed in tables in reference books.(3) If you can't find such a table for the analyte used in your study, you will have to rely on your own judgment.

Random error (RE), defined as 1.96 multiplied by the SD of the control, includes 95% of the variations that will occur by chance. The magnitude of the RE is compared with the allowable error. If RE < Ea, the amount of error due to random variation is considered clinically acceptable.

Similarly, the equation for systemic error (SE), SE = y - Xc, is calculated by solving y = b{Xc} + a. This represents the equation of the line generated by performing the regression analysis for y and subtracting Xc, where b is the slope, a is the intercept, and Xc is the analyte concentration at the medical decision level.

In the example shown in Figure III, y = 101.5, giving an SE of 1.5. Again, the magnitude of the SE is compared with the Ea. When SE < Ea, the amount of error due to bias is considered acceptable. If bias is detected, inspecting the bias plot will help determine whether the bias is constant or proportional.

Total error (TE) represents the sum of random error and systemic error. Like SE, TE is compared with allowable error. If RE + SE < Ea, the total error is considered acceptable. If TE is acceptable at all three levels at which the control is run, the entire method evaluation is judged to be acceptable.

* Practicality. I set up the template so that it would be easy to use, even for medical technologists with limited PC expertise. The data input and calculations of a complete method evaluation can be accomplished in about 30 minutes. The template has been handy for assessing method evaluation data as well as for quick studies when troubleshooting instruments.

A prime advantage is the objectivity rendered possible by the template calculations and graphs. Although part of my goal was to make the template readily accessible to others, I often enter the data myself, since I find myself evaluating the summary statistics and error analysis as I go along.

Being able to generate graphs quickly and easily is a tremendous benefit of the program. When a problem arises during an evaluation, I come up with new studies for further evaluation as I do the data entry and calculations.

My spreadsheet template enables our laboratory to evaluate new procedures and methodologies completely and appropriately. It is particularly helpful in evaluating test kits. The ability to perform rapid yet thorough method evaluation analyses has improved our evaluation process and documentation of new methods. [Figures 1 to 3 Omitted] (1)National Committee for Clinical Laboratory Standards. User comparison of quantitative clinical laboratory methods using patient samples. Proposed guideline EP9-P. Villanova, Pa., NCCLS, 1986. (2)Statland, B.E. "Clinical Decision Levels for Lab Tests," 2nd ed. Oradell, N.J., Medical Economics Books, 1987. (3)Kaplan, L.A., and Pesce, A.J., eds. Evaluation of methods, chap. 19, and Laboratory statistics, chap. 16, in "Clinical Chemistry: Theory, Analysis, and Correlation." St. Louis, C.V. Mosby, 1989.

Donna J. Walsh, M.S., MT(ASCP) The author is a laboratory scientist in chemistry at New England Deaconess Hospital, Boston.

Printer friendly Cite/link Email Feedback | |

Author: | Walsh, Donna J. |
---|---|

Publication: | Medical Laboratory Observer |

Date: | Apr 1, 1991 |

Words: | 1578 |

Previous Article: | Preparing the small lab for a first state inspection. |

Next Article: | Equal pay for phlebotomists. |

Topics: |