Blood gas Method Validation: the basics.

The February 28, 1992 update to CLIA allowed testing sites that were using unmodified, FDA approved, moderate complexity testing, to accept the manufacturer's performance data and were not required to perform Method Validation studies. The January 24, 2003 Final Rules for CLIA now combines Moderate Complexity and High complexity testing into a single category called Non-waived Testing. The Non-waived testing category includes Blood Gas analysis and now requires specific QC procedures including Method Validation studies. The requirements are outlined in Section [section]493.1253(b): (1) Verification of performance specifications. Each laboratory that introduces an unmodified, FDA-cleared or approved test system must do the following before reporting test results:

(i) Demonstrate that it can obtain performance specifications comparable to those established by the manufacturer for the following performance characteristics: (A) Accuracy (B) Precision and (C) Reportable Range of test results for the test system.

(ii) Verify the manufacturer's reference intervals (normal values) are appropriate for the laboratory's patient population.

I am going to outline the methods that need to be followed to satisfy sections (A) and (B) for accuracy and precision. This will require the performance of two different types of studies; a Comparison of Methods Study to estimate inaccuracy (bias) and a Replication Study to estimate imprecision. These two methods will allow you to estimate the Random Error (imprecision) and Systematic Error (inaccuracy). These two together give the total error involved.

[ILLUSTRATION OMITTED]

Comparison of Methods Study

This study involves testing 40 patient samples on both the New Device (Test method) and an established method (Comparison method). For example, the Test method might be a newly purchased POCT device and the Comparison method is the bench top analyzer that has been in use in a central location for a period of time that has allowed an assessment of its quality to be completed. The POCT device will need to be evaluated before it is placed into service as stated in the CLIA regulation mentioned above. Blood samples are placed into both machines and measured. The two machines need to be sitting right next to each other to prevent any extraneous errors from occurring. The POCT device can be carried to the central location for this method comparison. The patient specimens should cover a wide range of possible clinical values and not all within normal ranges. The testing time period should be between 5 and 20 days to assure that different environmental factors are accounted for. Also, it goes without saying that since blood gas values change rapidly after being collected, the blood should be introduced into both machines at the same time without a long waiting period between them.

The first step in data analysis is to plot the difference between the methods (Test minus Comparison; y-axis) against the value for the Comparison method (x-axis). The data points should cluster around a zero difference across the whole range of values. Visual inspection of the graph can be used to see of there are any outliers that may need to be repeated or if the points cluster differently at low values compared to high values. The latter result may indicate that there is some type of proportional error occurring--i.e. the systematic error amount depends on the comparison value on the x-axis. For example, the systematic error may increase as the x-axis value increases. Since blood gas values normally fall within a narrow range, the correct statistics to calculate include the average difference as an estimate of the systematic error. This average difference (Mean of the differences = bias) should be close to zero if the two methods are measuring the same values.

Replication Study

The replication study is easier since it involves data that you should already be collecting. Using three different levels of external controls over a 20 day period is the suggested method of performing a replication study. An estimate of short-term imprecision can be estimated by running 20 different control vials within a single day and a long-term imprecision can be estimated by running 3 different levels daily over a 20 day period. For each level used you then calculate the Mean, Standard Deviation (SD), and the Coefficient of Variation (CV) from the 20 data points. The SD and/or the CV are estimates of the observed random error of the device involved.

Total Error

The Total Error (TE) is the sum of the systematic error (SE) and the Random Error (RE): TE = SE + RE. From the two method comparison studies you can estimate the Total Error (TE) as TE = bias + 3*SD. Where, the bias is the inaccuracy from the Method Comparison study and the SD is from the Replication Study. This total error then needs to be less than the acceptable performance error limits for US CLIA proficiency testing. For example, the US CLIA limits for Blood pH are: Target Value +/- 0.04 pH unit

Using an example from Dr. Westgard: We have a pH Bias of 0.01 and an average SD of 0.005. TE = 0.01 +/- 3(0.005) = 0.01 +/- 0.015 = (-0.005 to 0.025). Expected TE is 0 +/- 0.04 = (-0.04 to 0.04). Our results from our two studies fall within this range so we have acceptable accuracy and precision. I would like to recommend that you check out www.westgard.com for additional information and a method for using these results to gauge your Quality Control procedures.

by Wesley Granger PhD, RRT