Evaluation of assigned-value uncertainty for complex calibrator value assignment processes: a prealbumin example.
To support global quality improvements in clinical diagnostics, the European Union has created the In Vitro Diagnostics Medical Devices Directive, which requires that calibrator products offered in Europe have defined traceability and calibrator uncertainty (2). In addition, International Organization of Standardization (ISO) document 17511 provides guidance to manufacturers on how to establish assigned-value traceability (3). For determining calibrator-assigned value uncertainty, the ISO document refers to the Guide to the Expression of Uncertainty in Measurement (GUM) (4). Unfortunately, the GUM does not describe uncertainty analysis methods for complex value-transfer processes that involve multilevel calibrators with correlated errors. One approach to estimating the uncertainty of value assignment in such situations is by simultaneous building of multiple lots of calibrators and calculating SDs of the assigned value for each calibrator level. This approach, although it is conceptually simple and enables unbiased estimation, is prohibitively expensive. A more feasible approach is Monte Carlo simulation of the process.
Aside from fulfilling In Vitro Diagnostics Medical Devices compliance requirements, the clinical utility of estimating and reporting uncertainty in diagnostic assays has been debated (5,6). We found Monte Carlo simulations to be useful in the design and optimization of the value-assignment process, because this method allows for changing simulated process parameters such as testing replication and number of instruments. In essence the process of PAB calibrator value assignment consists of transferring the value from the reference material CRM470 (7,8) to a multilevel set of internal working calibrators (IWC), and then from the IWCs to the end-use calibrators during their production. The value-assignment process transfers the value from CRM470 to the production calibrator with additional uncertainty that is much smaller than that of the published uncertainty of CRM470 itself.
Materials and Methods
The Beckman Coulter reagent used to measure PAB concentration is a turbidimetric method. In the analytical reaction, serum PAB combines with a specific antibody to form light-scattering antigen-antibody complexes. The system monitors the changes in absorbances at several wavelengths and uses a nonlinear calibration curve obtained with the multilevel calibrator to calculate the concentrations of PAB in patient samples.
FORMULATION OF CALIBRATORS AND STANDARDS
Delipidized human serum (9) containing endogenous PAB was used to formulate the IWC and production calibrators. High PAB concentrations were achieved by ultrafiltration concentration, and lower concentrations were created through dilutions of the concentrated ultrafiltrate with phosphate buffered saline (PBS). For the IWC, the dilution rations were quantified gravimetrically. Sodium azide at 1 mL/L is added as an antimicrobial. The upper limit of the analytical range for the PAB assay is 600 mg/L, which determines the requirement for the highest concentration of the production calibrator. To transfer reference values to the production calibrator from the IWC, the latter spans the range from 0 to 700 mg/L.
IWC VALUE ASSIGNMENT
The IWC are directly traceable to CRM470 through a value-assignment process. The highest concentration calibrator is directly value assigned using CRM470, and then the values of the lower concentrations are determined from dilution ratios.
We initially considered a stochastic approximation approach to value assignment (10). However, because of the complexity of this method and accumulation of dilution errors with each iteration, we developed a method that is a modification of the stochastic approach. Both the Beckman Coulter and stochastic approximation methods assume that the logarithm of the signal vs the reciprocal dilution ratio (for narrow signal ranges) is approximately linear. In our modified method, however, rather than doing a number of calculations over a number of iterations, the value-assignment testing and calculations are performed in 1 step.
To compare the 2 value-assignment methods, we used the same dilutions and data from the stochastic approximation evaluation. For the modified approach, to determine the PAB concentration for the highest-level (level 8) calibrator, (1/dilution) vs ln(signal), a linear equation was fitted to data (Fig. 1) in a narrow interval around the In (signal) obtained with CRM470. The fitted line was used to estimate dilution of the highest-level calibrator needed to obtain the same signal as the one obtained with CRM470. The assigned value of the highest-level calibrator is equal to the reciprocal of the dilution of the highest-level calibrator multiplied by the assigned value of CRM470 (243 mg/L). This analysis is done for each of the 3 instruments used for data collection. The final value assigned to the highest level PAB IWC was the average across the instruments. The assigned values of the lower concentrations are determined by use of the respective dilution ratios applied to the assigned value of the highest concentration calibrator.
PRODUCTION CALIBRATOR VALUE ASSIGNMENT
The production calibrators form a 6-level calibrator with PAB concentrations ranging from 0 mg/L to ~600 mg/L. For the value assignment of the production calibrators, 18 measurements per concentration of the production calibrator and the IWC were performed on each of 4 SYNCHRON CX instruments with each of 2 reagent lots (144 measurements per test sample). Within a reagent lot, the data were averaged over all the instruments, and an IWC standard curve was determined by fitting a 5-parameter logit function to the averages of instrument responses vs assigned concentrations:
R = [R.sub.0] + [[K.sub.c]/[(1 + [e.sup.-a-bln(conc)]).sup.c]]. (1)
[FIGURE 1 OMITTED]
[FIGURE 2 OMITTED]
This standard curve was then used to calculate production calibrator PAB concentrations. The assigned values of the production calibrators are the average concentrations across 2 reagent lots, 4 instruments, and 18 replicates per reagent lot per instrument.
ESTIMATION OF UNCERTAINTY
In estimating the uncertainty of the values assigned to the calibrators, it is useful to consider the traceability chain (See Fig. 1 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vo153/issue4). Each process step in the traceability chain is considered when determining the total uncertainty of the production calibrator. At the beginning of the chain is the international protein reference material CRM470, with an uncertainty, [U.sub.std], characterized by the SD of 9.0 mg/L, as published in the CRM470 product insert. [U.sub.iwc], is the uncertainty in the assigned values for the IWC. Lastly, [U.sub.pc] denotes the uncertainties of the production calibrator assigned values If these were single calibrator level process steps with uncorrelated additive error variance components, then the uncertainty of each step could be combined into 1 value representing the total uncertainty (4):
[U.sub.tot] = [square root of (J[U.sup.2.sub.std] + U[U.sup.2.sub.iwc] + U[U.sup.2.sub.pc])]. (2)
However, for multilevel value transfers with cross-correlated errors of assigned values of calibrator levels, deriving closed form equations for total uncertainty for each calibrator level is practically impossible. Therefore, we evaluated the uncertainties of the IWC, as well as of the production calibrators, by Monte Carlo simulation of the value transfer process. To be performed correctly, the simulation process must be well described, including accurate estimation of error contributions and the correlations of the errors in various stages of the process. In addition, all the variance/covariance components used in the simulation are evaluated from the real value assignment process data. The concepts of Monte Carlo simulation are described in (11). The program for the simulation for this study was developed in S-Plus on a PC platform (12); it is included in the Supplemental Appendix. A distribution of the assigned values is generated through simulations of 1000 value assignment events. The SD of these simulated assigned values is the assigned value uncertainty. The 95% confidence intervals of the uncertainty were determined as 2.5th and 97.5th percentiles of the SDs estimated from 40 runs of 1000 simulated value assignment events. This simulation procedure augments the method applicable to simple value assignment processes described in GUM (4).
IWC AND PRODUCTION CALIBRATOR UNCERTAINTIES
The entire uncertainty evaluation process is flow-charted in Fig. 2. Items 1.1 through 1.7 refer to the actual IWC and production calibrator formulation and value-assignment processes previously described. During the preliminary assessment of the value-assignment events for the IWC and the production calibrator, it was determined that instrument-to-instrument variation, as well as inter-sample covariation of the errors, were significant and thus must be accounted for in the Monte Carlo simulations. Items 2.3 and 2.6 in Fig. 2 refer to the extraction of covariance structure and variance components of the instrument signals from the actual value-assignment data. Here instrument signal averages obtained with the CRM470 dilutions, IWC, and the production calibrator samples were calculated. Also, the instrument-to-instrument and within-instrument variance components of instrument signals were estimated with variance components analysis (13) along with the covariance matrix of instrument signals for various calibrator levels. To impose the above covariation structure on the simulated instrument responses for various calibrator levels, the Cholesky decomposition of the (P x P) covariance matrix is multiplied with the (P x N) matrix of random numbers with independent gaussian distribution with zero means and unit variances, then divided by the diagonal of the same covariance matrix (11). Above, P is the number of calibrator levels and N is the number of instruments. Items 3.3 to 3.9 (Fig. 2) refer to the Monte Carlo simulation of the value-assignment events. The value-assignment calculations, using the simulated instrument signals, are performed as they were in the actual calibrator value assignments. For the IWC value-assignment calculations the natural log of the signal data for the IWC dilutions is regressed with the reciprocal dilution factors. The IWC set point estimate is based on the dilution required to match the CRM470 signal. For the PC value assignment, a concentration vs signal standard curve is created from the IWC data by curve fitting. By inputting the production calibrator signal data into this standard curve, the calibrator concentrations are calculated.
The gravimetric error contributions during formulation of the IWC and value assignment were much less than 0.01%, a value that was insignificant compared to the 3.72% total error of the assigned values. Therefore we ignored the contribution of errors of gravimetric dilution. The uncertainty of the CRM470 standard, which was included in the simulations, was the major source of IWC uncertainty because with the described process the error of transferring the reference value to the IWC is very small. The uncertainty of IWC was included in the simulations of the value assignment of the production calibrators. Simulation of 1000 value assignment events was repeated 40 times. The pooled SDs and the nonparametric 95% confidence intervals obtained from these simulation results characterize the uncertainties. The 95% confidence intervals of the uncertainties were set as the nonparametrically estimated 2.5th and 97.5th percentiles of the 40 groups of 1000 simulated assigned values for each level of the calibrator.
[FIGURE 3 OMITTED]
This method enables estimation of the calibrator uncertainties; however, because of the multilevel nonlinear calibration, it is not obvious how these uncertainties propagate to patient result uncertainty. To evaluate these effects, recoveries of patient sample concentrations were simulated. In this simulation the calibrator-assigned values varied according to the previously determined uncertainties. Then, using target responses for various concentrations, patient results were calculated with various calibrations using those variable assigned values. Calculated patient result SDs showed the effect of the calibrator uncertainty. This method allows for establishing requirements to the calibrator uncertainty.
The IWC value-assignment results are shown in Table 1. During the value-assignment evaluations, multiple dilutions of the highest level IWC were made such that their signals were near the signal of CRM470, and a linear fit was made to find the dilution necessary to obtain the same signal as the reference material (Fig. 1). Once the dilution was determined, the assigned value of the highest-level calibrator was calculated by dividing the assigned value of the reference material by the dilution factor used to match the highest-level calibrator. Table 1 shows the 3 assigned values calculated for the 3 different instruments, and the average of those values as the final assigned value for the highest level IWC. For comparison, the results using the stochastic approximation method (10) are also included. Our approach, because it uses all the data from many dilutions to estimate the dilution of the level 8 IWC, might provide for smaller uncertainty of the assigned values than does stochastic approximation (SD of 7.7 vs 11.2 mg/L), in which the signal data from dilutions that do not match those of the reference material are discarded. Here, however, the difference is not statistically significant based on F-test with 2 degrees of freedom. The assigned values of the lower levels of the IWC were calculated using the dilution ratios of the highest-level calibrators.
Simulation of the IWC value assignment event was repeated; 1000 value assignments were simulated 40 times yielding 40 sets of 1000 assigned values. A histogram of 1 of the sets of 1000 assigned values is shown in Fig. 3. The pooled SD of these assigned-value sets, SD = 26.9 mg/L, is the estimate of the assigned-value uncertainty, [U.sub.iwc], for the highest-level IWC. The uncertainties, based on dilution ratio, for all 8 levels of the IWC are summarized in Table 2.
For the production calibrator value assignment, measurements of the IWC were taken to create a standard curve, from which the assigned values of the production calibrators were calculated (See Fig. 2 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue4). The pooled SD of the 40 sets of 1000 simulated value-assignment events is the uncertainty. By using this approach, the production calibrator uncertainty includes the IWC and CRM470 components and the error of value transfer from IWC. The final results of the value assignment, as well as the uncertainty analysis, are summarized in Table 2. The relative uncertainties over all levels of the production calibrator are ~3.75%.
Because the uncertainty component associated with the production calibrator value assignment was rather small, we assessed the effect of reducing the amount of testing. Rather than making 144 measurements (4 instruments, 36 replicates per sample), we made 48 measurements (4 instruments, 12 replicates per sample) were made. The results of those simulations are summarized in Table 3. Here the production calibrator value assignment was isolated from the previous events such that the error contribution from this event alone may be considered. Lastly, the effect of calibrator uncertainty on patient result uncertainty is provided in Table 3.
Our results demonstrate that that the process of transferring PAB values from the reference material to the production calibrators contributes little to the total error. The assigned-value uncertainty of CRM470 is 243 (9.0) mg/L, with a relative error of 3.70%. Through the value transfer process from this reference material to the IWC, the highest-level IWC had an assigned value of 722 (26.9) mg/L with a relative uncertainty of 3.72%. The component of uncertainty of the IWC assigned value, contributed by the process of transferring the value from reference material, is negligible. So, the described value-assignment process allows for a precise transfer of values from a single-level international standard CRM470 to a multiple level IWC. Unlike the stochastic approximation approach (10), which relies on determining the assigned values by an iterative process involving many tests and calculations, the described approach is simpler and provides for at least as good calibrator uncertainty. Multiple dilutions of the highest-level IWC were made such that the signals were near that of CRM470 when assayed. The relationship between the dilutions and the signals allows for the determination of the highest-level IWC assigned value, while gravimetric dilutions introduce no significant errors to lower levels of the IWC.
The final relative total uncertainties of the PAB production calibrators were all -3.75%, insignificantly higher than the relative uncertainty of the reference material. The relative uncertainties of the assigned values of all levels of the production calibrator, excluding the uncertainty of CRM470, were all <0.8%. So, the major part of the uncertainty of the BCI PAB production calibrators is comprised of the uncertainty of the reference material, because the value-transfer process adds very little to the total uncertainty. The exceedingly small contribution to the total error of the IWC and production calibrator value-transfer steps, although counterintuitive, is partially of the result of taking the square root of the sum of squared individual contributions per equation (2). In addition, the relatively large number of measurements reduces the uncertainty. In this instance the reduction is ~10-fold, the square root of one hundred. As shown in Table 3, reducing the amount of testing of the production calibrator by 33% reduces the cost of the calibrator but has little effect on the total calibrator uncertainty or on the assay precision.
The question of clinical usefulness can be addressed in relation to patient result uncertainties due to calibrator uncertainties (Table 3). To be clinically useful, the errors of the patient results caused by the calibrator uncertainties must be small relative to the PAB reference range of 180-380 mg/L (14). At the lower and upper limits of the reference range, uncertainties of the patient results caused by the uncertainties of the calibrator are 1.6 and 2.8 mg/L, respectively, with 96 measurements per sample. These values as percentages of the width of the reference range are 0.9% and 1.6%, respectively, small relative to the reference range. Therefore, the risk of patient misdiagnosis due to calibrator uncertainty is small.
The contributions to this work by Phebe Shaw, Phil Martz, Art Kessner, and Steve Alter were invaluable.
Received October 3, 2006 ; accepted January 15, 2007. Previously published online at DOI: 10.1373/clinchem.2006.081174
(1.) Beck FK, Rosenthal TC. Prealbumin: A Marker for Nutritional Evaluation. Am Fam Physician 2002;65(8):1575-8.
(2.) "Council Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on in Vitro Diagnostic Medical Devices", Official Journal of the European Union L331 (December 12, 1998).
(3.) International Organization for Standardization. In vitro diagnostic medical devices-measurement of quantities in biological samples-metrological traceability of values assigned to calibrators and control materials (17511). Geneva: ISO, 2003:23 pp.
(4.) International Organization for Standardization. Guide to the Expression of Uncertainty in Measurement, Geneva: ISO, 1995;: 101pp.
(5.) Krouwer J. Critique of the Guide to the Expression of Uncertainty in Measurement Method of Estimating and Reporting Uncertainty in Diagnostic Assays. Clin Chem 2003;49(11):1818-21.
(6.) Kristiansen J. The Guide to Expression of Uncertainty in Measurement Approach for Estimating Uncertainty: An Appraisal. Clin Chem 2003;49(11):1822-9.
(7.) Whicher JT, Ritchie RF, Johnson AM, Baudner S, Bienvenu J, Blirup-Jensen S, et al. New International Reference Preparation for Proteins in Human Serum (RPPHS). Clin Chem 1994;40(6): 934-8.
(8.) Johnson AM, Sampson EJ, Blirup-Jensen S, Svendsen PJ. Recommendations for the Selection and Use of Protocols for Assignment of Values to Reference Materials. Eur J Clin Chem Clin Biochem 1996;34:279-85.
(9.) Vance DE, Weinstein DB, Steinberg D. Isolation and Analysis of Lipoproteins Secreted by Rat Liver Hepatocytes. Biochim Biophys Acta 1984;792:39-47.
(10.) Schlain B. A Stochastic Approximation Method for Assigning Values to Calibrators. Clin Chem 1998;44(4):839-48.
(11.) Rubenstein RY. Simulation and the Monte Carlo Method. New York: John Wiley, 1981.
(12.) Venables WN, Ripley BD. Statistics and Computing. Modern Applied Statistics with S-Plus. New York: Springer-Verlag, 1994.
(13.) Corbeil RR and Searle SR. Restricted maximum likelihood (REML) estimation of variance components in the mixed model. Technometrics 1976;18:31-38.
(14.) Synchron CX Chemistry Information Manual. Beckman Coulter reorder no. 249595.
 Nonstandard abbreviations: ISO, International Organization of Standardization; PAB, prealbumin; IWC, internal working calibrator; PC, production calibrator; VA, value assignment; GUM, Guide to the Expression of Uncertainty in Measurement.
JOHN MIDDLETON  * and JEFFREY E. VAKS 
 Department of Clinical Chemistry Development, Beckman Coulter Inc., Fullerton, CA.
 Math/Stats Group, Beckman Coulter, Inc., Fullerton, CA.
* Address correspondence to this author at: Department of Clinical Chemistry Development, Beckman Coulter Inc., Fullerton, CA. 92821. Fax 714-961-3740; email firstname.lastname@example.org.
Table 1. PAB IWC assigned values (mg/L). A. IWC level 8 results. (a) Instrument serial no. BCI method Stochastic method (10) CX S/N 611 714.4 713.3 CX S/N 1114 720.5 725.3 CX S/N 3044 729.7 735.6 Average 721.5 724.7 SD 7.7 11.2 B. IWC assigned values based on level 8 assigned value and dilution ratio (b) Mass IWC, Mass diluent, Assigned LVL [M.sub.i] [M.sub.d] Values, C 1 n/a n/a 0.0 2 18.67 242.92 50.1 3 37.33 224.26 100.4 4 74.70 188.25 200.7 5 112.01 151.28 301.7 6 149.31 115.07 402.2 7 197.87 67.93 533.0 8 279.00 0.00 721.5 IWC level 8 density ([D.sub.s]) 1.03 Diluent density ([D.sub.d]) 1.00 (a) Prealbumin level 8 (highest-level) Internal Working Calibrator assigned-value estimates by instrument for the BCI value-assignment method and the stochastic approximation method (10). Of course the 2 methods give the same values within the uncertainty of the estimates. (b) The mass values (grams) of the level 8 IWC and the diluent were converted to volumes by using their respective densities (g/mL). Assigned values, C, were calculated from the level 8 assigned value and the dilution ratios by using the formula below. [M.sub.i] is the mass of IWC level 8,[M.sub.d] is the mass of diluent,[D.sub.s] is the density of serum, and [D.sub.d] is the density of diluent. [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] Table 2. PAB IWC and production calibrator uncertainties (mg/L). A. IWC1 IWC2 IWC3 IWC4 Assigned Value 0 50.1 100.5 200.8 Uncertainty 0 [+ or -]1.87 [+ or -]3.74 [+ or -]7.48 Uncertainty, % 0 [+ or -]3.725 [+ or -]3.725 [+ or -]3.725 95% Confidence interval (a) 0 [+ or -]0.17 [+ or -]0.35 [+ or -]0.69 IWC5 IWC6 Assigned Value 301.9 402.5 Uncertainty [+ or -]1.12 [+ or -]1.50 Uncertainty, % [+ or -]3.725 [+ or -]3.725 95% Confidence interval (a) [+ or -]1.04 [+ or -]1.39 IWC7 IWC8 Assigned Value 533.3 722.0 Uncertainty [+ or -]1.99 [+ or -]2.69 Uncertainty, % [+ or -]3.725 [+ or -]3.725 95% Confidence interval (a) [+ or -]1.84 [+ or -]2.49 B. Production calibrator total uncertainty (b) CAL CAL2 CAL3 Assigned value 0 74.4 154.0 Uncertainty 0 [+ or -]2.78 [+ or -]5.75 Uncertainty, % 0 [+ or -]3.743 [+ or -]3.735 95% Confidence interval 0 [+ or -]0.23 [+ or -]0.54 CAL4 CAL5 CAL6 Assigned value 290.6 429.4 573.5 Uncertainty [+ or -]10.89 [+ or -]16.14 [+ or -]21.65 Uncertainty, % [+ or -]3.746 [+ or -]3.758 [+ or -]3.774 95% Confidence interval [+ or -]1.04 [+ or -]1.53 [+ or -]1.86 IWC assigned values with uncertainties. (a) The 95% confidence half-interval in the uncertainty estimates is determined nonparametrically from the minimum and maximum of the 40 estimates (excluding the highest and lowest estimates accounting for 5% of the distribution). The percent uncertainty is constant across the levels as the lower levels are calculated from the highest level and the appropriate dilution factor, with dilution errors being negligible. An assigned value of zero implies zero uncertainty because the sample is not human serum based and is PAB free. The confidence interval is expressed in absolute terms (mg/L). (b) Tabulated are results from 40 sets of 1000 simulated value-assignment events. The 95% confidence half-interval (in mg/L) of the uncertainties is determined from the minimum and maximum of the 40 estimates (excluding the highest and lowest estimates accounting for 5% of the distribution). An assigned value of zero implies zero uncertainty because the sample is not human serum based and is PAB free. The confidence interval is expressed in absolute terms (mg/L). Table 3. Uncertainty (in mg/L) as a function of the number of measurements. Production calibrator value assignment uncertainty (a) A. Testing Protocol of 144 Measurements Calibrator level 2 3 4 5 6 Assigned value, mg/L 74.3 153.9 290.4 429.3 573.2 Uncertainty, mg/L 0.43 0.66 1.99 3.65 5.72 Relative uncertainty, % 0.574 0.428 0.684 0.850 0.997 B. Testing protocol of 96 measurements Calibrator level 2 3 4 5 6 Assigned value, mg/L 74.3 153.9 290.4 429.2 573.5 Uncertainty, mg/L 0.49 0.78 2.39 4.29 7.07 Relative uncertainty, % 0.660 0.507 0.822 1.000 1.234 Component of patient result uncertainty due to calibrator value assignment (b) C. Testing Protocol of 144 Measurements PAB concentration, mg/L 50 180 (a) 280 380 (a) 500 Result uncertainty, mg/L 0.97 1.55 2.44 2.83 2.62 Reference range, % 4.83 7.74 12.21 14.17 13.12 D. Testing Protocol of 96 Measurements PAB concentration, mg/L 50 180 (a) 280 380 (a) 500 Result uncertainty, mg/L 1.11 1.8 2.80 3.30 3.23 Reference range, % 5.55 9.00 14.02 16.50 16.17 (a) Tabulated are results from 1000 simulated value-assignment events. The number of measurements per sample was reduced from 144 to 96. Comparison of the 2 testing configurations showed a small increase of the uncertainty of transferring the reference material value to the production calibrators. (b) Results for 5 different PAB concentrations from 1000 simulated value-assignment events. The percentage of reference range entries is based on width of 20 mg/L of the reference range 18 to 38 mg/L. Because the uncertainties resulting from calibrator value assignment for both testing protocols are small compared with the width of the reference range, the risk of patient misdiagnosis because of value assignment error is low.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||Automation and Analytical Techniques|
|Author:||Middleton, John; Vaks, Jeffrey E.|
|Date:||Apr 1, 2007|
|Previous Article:||Spectrophotometric assay for complex I of the respiratory chain in tissue samples and cultured fibroblasts.|
|Next Article:||HPLC for simultaneous quantification of total ceramide, glucosylceramide, and ceramide trihexoside concentrations in plasma.|