Printer Friendly

Improved Diagnostic Performance of High-Sensitivity Cardiac Troponin Assays Is an Artifact of Censored Data.

The diagnostic paradigm of acute coronary syndrome has changed considerably over the past decade and the quest for improved analytical performance at low concentrations led to the development of assays for "high-sensitivity" cardiac troponin (hs-cTn). [3] The touted advantages of hs-cTn include earlier detection of acute myocardial infarction, albeit with a warning that this is accompanied by a lamentable loss of specificity (1, 2). This is somewhat surprising since imprecision, which is a major determinant of "analytical sensitivity," is known to have a negligible effect on diagnostic accuracy under most conditions (3, 4). In extreme circumstances in which measurement error is severe and the error variance overwhelms the combined variances of healthy and diseased subjects, the diagnostic accuracy, however, may be negatively impacted (5).

The diagnostic accuracy of a test refers to its ability to discriminate between diseased and nondiseased states. Diagnostic sensitivity, or the true positive rate, is defined as the ability of a procedure to return a positive outcome in an individual that has the condition, while specificity or true negative rate is the ability to return a negative outcome in someone without the condition (3). ROC curves graphically illustrate the variable binary diagnostic performance of a test procedure as a function of a change in the decision limit. "Analytical sensitivity" in contrast, is a detection capability attribute of a measuring system and can be defined as the quotient of the change in instrument signal and the corresponding change in a value of the quantity being measured (6). To avoid confusion the use of the term "analytical sensitivity" is discouraged and it is recommended rather to use limit of the blank (LoB) and limit of detection (LoD) when discussing the detection capability of a measurement system (6, 7). The LoB and LoD are objective statistical constructs which characterize performance at the low end of the measuring range, and both are functions of the precision and bias of the measurement procedure (7).

We investigated the impact of detection capability on diagnostic accuracy with a software model where the LoB and LoD of a hypothetical cardiac troponin assay was varied independently of the reference clinical diagnosis. We also examined the effect of censoring cardiac troponin data below a hypothetical LoD concentration on diagnostic accuracy.

Materials and Methods

CLINICAL DATA AND SOFTWARE MODELING

The patient data were previously presented as part of a comparison of z-scores, actual and relative 8 troponins (8). In a cohort of 762 patients who presented with chest pain, 52 patients (6.8%) had a reference clinical diagnosis of acute myocardial infarction, which was an adjudicated diagnosis based on all the available clinical and laboratory information. For the purpose of this investigation, we used the baseline troponin value measured with the Abbott hs-cTnI as the designated "true" value. The median (interquartile range) ofcTnI results in the nonmyocardial infarction group was 2.9 ng/L (1.9 - 5.3) and in the myocardial infarction group was 114.2 ng/L (46.0 - 284.1). Forty-eight (92.3%) individuals in the myocardial infarction group had a result above 30 ng/L compared to 34 (4.8%) in the nonmyocardial infarction group.

To investigate the effect of analytical sensitivity on diagnostic performance, we generated 4 sets of results with decreasing detection capability (Table 1). The precision profile for each model was based on a linear relationship between the respective SDs and the concentration of cardiac troponin. The 99th percentiles of all the assays were taken as 30 ng/L, while the LoBs and LoDs were calculated according to CLSI guidelines (7). Four normally distributed random values were generated for every patient in Excel 2010 (Microsoft) based on the SD of the respective model and the hypothetical "true" value with the function NORMINV (probability, "true" value, SD). The probability was randomly varied between 0 and 1 with the RAND () function. To test if the findings were generalizable to situations with lower diagnostic accuracy we repeated the procedures above after modeling the results to achieve a lower area under the curve (AUC).

STATISTICAL ANALYSIS

ROC curves were constructed with Sigmaplot version 13 (Systat Software Inc.) and a paired analysis was used to test for differences in the respective AUCs. To test for the influence of data censoring on diagnostic performance, the data below the hypothetical LoD values were set to zero before repeating the ROC curve analysis. We also censored the original "true" results in increments from 10-30 ng/L before comparing the AUCs to that obtained with uncensored data. Statistical significance was tested for at the 5% level (P < 0.05).

Results

The diagnostic performances of the 4 hypothetical assays with decreasing detection capabilities are compared to the original ("true") set of results in Table 2. There were no significant differences in AUCs of the ROC curves constructed from the uncensored data. We confirmed this finding after manipulating the original data to decrease the AUC to 0.88 before repeating scenarios A to D (data not shown). With a 30 ng/L cutoff as a binary decision point, 41 individuals from the nonmyocardial group in model D had an increased cTnI concentration. The specificity of 94.2%, however, was not significantly decreased compared to the 95.2% of the original data ([chi square] = 0.51; P = 0.48). The change in diagnostic sensitivity was also not significant (P = 1.0) with one fewer myocardial infarction patient classified with an increased cTnI.

When data below the LoD of the hypothetical assays were censored, the ROC analysis indicated a progressive decrease in AUC, which reached statistical significance for model D (Table 2). The impact of censoring data was confirmed with the original data set when results were censored in 10 ng/L increments. The AUC decreased progressively and reached statistical significance when results below 20 ng/L were censored (P = 0.021). The progressive degrading effect of censoring data on the AUCs is illustrated in Fig. 1. A hallmark of ROC curves constructed with censored data is the straight line between the point on the curve corresponding to the threshold concentration for censoring and the top right hand corner.

Discussion

We demonstrated no adverse effect on the diagnostic performance with a progressive deterioration in detection capability (analytical sensitivity) of cardiac troponin. The detection capabilities of model D would be regarded as unacceptable by conventional wisdom (9), but nonetheless judging by the AUC, it performed similarly to the hs-cTn of model A. This outcome is not surprising, as detection capability is largely a function of imprecision at low concentrations and it is well known that imprecision per se does not significantly moderate diagnostic accuracy in general (3, 4, 9). When we censored the data with results deemed as unmeasurable below the LoD, the AUCs decreased progressively and this reached statistical significance at an approximate 20-ng/L cutoff in the model. This progressive degrading effect on the diagnostic performance was confirmed when the original data was incrementally censored as illustrated in Fig. 1. The negative impact of data censoring not a novel finding and is recognized in the literature (10).

The lack of a direct relationship between the detection capability of a cardiac troponin assay and the diagnostic accuracy is also consistent with other reports in the literature. The Vidas cTnI assay, which was considered to be unacceptable for clinical practice owing to its poor detection capability, was found to be clinically equivalent or superior to other cardiac troponin assays that did meet the acceptance criteria (11). In a comparison of the fourth generation cTnT assay against the hs-cTnT assay, there was no significant difference in the diagnostic performance with respect to acute myocardial infarction, although the latter assay did detect more cases with cardiac disease (12).

It is reasonable to pose the question why this association between diagnostic performance and detection capability is so entrenched in the literature with an almost universal acceptance that this phenomenon will be accentuated with the introduction of hs-cTn. In a sentinel paper Reichlin and colleagues compared the diagnostic performance of a selection of so-called "sensitive" cTnI assays and a hs-cTnT assay against a "standard assay" (13). The areas under the respective curves for the sensitive and hs-cTn assays were indistinguishable from each other and all had a significantly greater AUC than that of the "standard" assay. Inspection of the ROC curve however demonstrates a flat line for the "standard assay," similar to the effect seen in Fig. 1. This flatline phenomenon presumably arose due to data censoring and this is also a notable feature in a selection of other articles that reached similar conclusions about the relationship between diagnostic performance and detection capability (14, 15, 16).

The practice of censoring data below the LoD as "unmeasurable" is common practice in the diagnostic industry (7) and is so pervasive and generally accepted that none of the authors deemed it worthy to mention directly (13, 14, 15, 16). The rationale behind this practice is unclear and not universally supported as information that can be used to describe, among others, the location and distribution characteristics of a population is needlessly discarded (17, 18, 19). As we have demonstrated, the censoring of data also produces a predictable artifactual deterioration of diagnostic accuracy. In our opinion, it is probable that the assumption of a relationship between the detection capability and diagnostic accuracy of sensitive and hs-cTn assays is based on an erroneous interpretation of ROC curves constructed from censored data. It is not surprising that the predicted changes from hs-cTn assays in acute coronary syndrome patients have yet to materialize with some authors commenting on the diminishing return from further increases in detection capability (16).

A more plausible explanation for the apparent rise in diagnostic sensitivity and the decrease in diagnostic specificity that accompanied the introduction of "sensitive" and "highly sensitive" cardiac troponin assays is the concurrent introduction of the 99th percentile as the decision level. This was often significantly lower than the previous cutoff defined by an arbitrary concentration where a 10% imprecision target was achieved and led to a predictable increase in the rate of myocardial infarction diagnosis (20). It is also now recognized that physiological and pathological causes other than acute myocardial infarction may lead to increased cardiac troponin concentrations. The definition of myocardial infarction evolved to include nonthrombotic myocardial infarction and the boundary between unstable angina and non-STsegment elevation myocardial infarction tilted toward the latter (21). The regrettable choice of "sensitive" and "highly sensitive" nomenclature may have exacerbated the impression that the diagnostic accuracy was altered as a consequence of improved detection capability.

In summary, there is no direct relationship between the diagnostic accuracy and the detection capability of cardiac troponin assays. The artifactual decrease in diagnostic performance can be added to the list of reasons why data should not be censored. As a minimum requirement, this practice should be disclosed in studies on diagnostic accuracy.

Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 3 requirements: (a) significant contribution to the conception and design, acquisition ofdata, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; and (c) final approval of the published article.

Authors' Disclosures or Potential Conflicts of Interest: No authors declared any potential conflicts of interest.

Role of Sponsor: No sponsor was declared.

References

(1.) Januzzi JL. What to expect when measuring high-sensitivity troponin. Practical advice for clinicians. J Am Coll Cardiol 2015; 65:1665-7.

(2.) De Lemos JA. Increasingly sensitive assays for cardiac troponins. JAMA 2013; 309:2262-9.

(3.) CLSI. Assessment of the diagnostic accuracy of laboratory tests using receiver operating characteristic curves; approved guideline-second edition. Wayne (PA): CLSI; 2011. CLSI document EP24-A2.

(4.) Kupchak P, Wu AHB, Ghani F, Newby LK, Ohman EM, Christenson RH. Influence of imprecision on ROC curve analysis for cardiac markers. Clin Chem 2006; 52: 752-3.

(5.) Perkins NJ, Schisterman EF. The Youden-index and the optimal cut-point corrected for measurement error. BiomJ 2005; 47:428-41.

(6.) International vocabulary of metrology--Basic and general concepts and associated terms (VIM). 3rd ed. 2008 version with minor corrections. Joint Committee for Guides in Metrology; 2012. http://www.bipm.org/en/ publications/guides/vim.html (Accessed October 2016).

(7.) CLSI. Evaluation of detection capability for clinical laboratory procedures; approved guideline-second edition. Wayne (PA): CLSI; 2012. CLSI document EP17-A2.

(8.) Pretorius CJ, Cullen L, Parsonage WA, Greenslade JH, Tate JR, Wilgen U, Ungerer JPJ. Towards a consistent definition of a significant delta troponin with z-scores: a way out of chaos? Eur Heart J: Acute Cardiovasc Care 2014; 3:149-57.

(9.) Jaffe AS, Apple FS, Morrow DA, Lindahl B, Katus HA. Being rational about (im)precision: a statement from the Biochemistry Subcommittee of the Joint European Society of Cardiology/American College of Cardiology Foundation/American Heart Association/World Heart Federation task force for the definition of myocardial infarction. Clin Chem 2010; 56:941-3.

(10.) Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden index and optimal cut-point estimated from observations affected by a lower limit of detection. BiomJ 2008; 50:419-30.

(11.) Venge P, Lindahl B. Cardiac troponin assay classification by both clinical and analytical performance characteristics: a study on outcome prediction. Clin Chem2013; 59:976-81.

(12.) Hammerer-Lercher A, PionerT, NeurerS, Schratzberger P, Griesmacher A, Pachinger O, Mair J. High-sensitivity cardiac troponin T compared with standard troponin T testing on emergency department admission: how much does it add in everyday clinical practice? J Am HeartAssoc2013; 2:e000204.

(13.) Reichlin T, Hochholzer W, Bassetti S, Steuer S, Stelzig C, Hartwiger S, et al. Early diagnosis of myocardial infarction with sensitive cardiac troponin assays. N Engl J Med 2009; 361:858-67.

(14.) Keller T, ZellerT, Peetz D, Tzikas S, Roth A, CzyzE, et al. Sensitive troponin I assay in early diagnosis of acute myocardial infarction. N Engl J Med 2009; 361:86877.

(15.) Aldous SJ, Florkowski CM, Crozier IG, Elliot J, George P, Lainchbury JG, et al. Comparison of high sensitivity and contemporary troponin assays for the early detection of acute myocardial infarction in the emergency department. Ann Clin Biochem 2011; 48:241-8.

(16.) Bonaca MP, Ruff CT, Kosowsky J, Conrad MJ, Murphy SA, Sabatine MS, et al. Evaluation of the diagnostic performance of current and next-generation assays for cardiactroponin in the BWH-TIMIED chest pain study. Eur Heart J: Acute Cardiovasc Care 2013; 2:195-202.

(17.) Currie LA. Nomenclature in evaluation of analytical methods including detection and quantification capabilities (IUPAC recommendation 1995). Pure Appl Chem 1995; 67:1699-723.

(18.) Cressie N. Limits of detection [Editorial]. Chemometr Intell Lab Syst 1994; 22:161-3.

(19.) Ungerer JPJ, Pretorius CJ. A fit-for-purpose approach to analytical sensitivity applied to a cardiac troponin assay: time to escape the "highly sensitive" trap. Clin Chem Lab Med 2014; 52:553-6.

(20.) Mills NL, Lee KK, McAllister DA, Churchhouse, MacLeod M, Stoddart M, et al. Implications of lowering threshold of plasma troponin concentration in diagnosis of myocardial infarction: cohort study. BMJ 2012; 344:e1533.

(21.) Reichlin T, Twerenbold R, Reiter M, Steuer S, Bassetti S, Balmelli C, et al. Introduction of high-sensitivity troponin assays: impact on myocardial infarction incidence and prognosis. Am J Med 2012; 125:1205-13.

Carel J. Pretorius [1, 2] * and Jacobus P.J. Ungerer [1]

[1] Department of Chemical Pathology, Pathology Queensland, Queensland Health, Brisbane, Australia; [2] University of Queensland, School of Medicine, Brisbane, Australia.

* Address correspondence to this author at: Level 3, Block 7, Royal Brisbane Hospital, Herston 4029, Queensland, Australia. Fax +61-7-3646-1392; e-mail carel.pretorius@ health.qld.gov.au.

Received June 20, 2016; accepted July 27, 2016.

Previously published online at DOI: 10.1373/clinchem.2016.262634

[3] Nonstandard abbreviations: hs-cTn, high-sensitivity cardiac troponin; LoB, limit of the blank; LoD, limit of detection; AUC, area under the curve.

Caption: Fig. 1. ROC curves constructed with progressive censoring of data in 10-ng/L increments. The AUCs with data censored below 20 and 30 ng/L were significantly smaller than that of the uncensored data (P = 0.021 and 0.005). The crosses indicate the position on the curves that correspond to cardiac troponin concentrations of 30,20, and 10 ng/L, respectively, from left to right.
Table l. Precision profilesand detection capabilities of 4
hypothetical cardiac troponin assays. (a, b)

Model    SD       SD     LoB    LoD        CV        CV at
        slope   offset                  at 99th     100 ng/L
                                       percentile

A       0.05     0.1      0.2    0.3       5.3        5.1
B       0.05     1.5      2.5    5.1      10.0        6.5
C       0.05     4.5      7.4   15.4      20.0        9.5
D       0.05     9       14.8   30.8      35.0       14.0

(a) SD, LoB, and LoD are presented in ng/L and CVas a percentage.

(b) The precision profile for each model was calculated as linear
function wherethe SD varied with a specified slope and offset relative
to the cardiac troponin concentration.

Table 2. The diagnostic performance of 4 cardiac troponin models with
decreasing detection capabilities.

         Model A      Model B      Model C      Model D
        Uncensored   Uncensored   Uncensored   Uncensored

AUC      0.976        0.977        0.975        0.958
SE       0.006        0.005        0.006        0.017
P (a)    0.432        0.187        0.749        0.189

         Censored     Censored     Censored     Censored
        <0.3 ng/L    <5.1 ng/L    <15.4 ng/L   <30.8 ng/L

AUC     0.976        0.977        0.955        0.904
SE      0.006        0.005        0.017        0.026
P (a)   0.432        0.187        0.092        0.001

(a) P values were calculated in a paired analysis relative to the
curve constructed from the original values.
COPYRIGHT 2016 American Association for Clinical Chemistry, Inc.
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Informatics and Statistics
Author:Pretorius, Carel J.; Ungerer, Jacobus P.J.
Publication:Clinical Chemistry
Date:Dec 1, 2016
Words:2922
Previous Article:Short-term Variability of Vitamin D-Related Biomarkers.
Next Article:Dried Blood Spot Reference Intervals for Steroids and Amino Acids in a Neonatal Cohort of the National Children's Study.
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters