Printer Friendly

Sensitivity, specificity, and receiver operating characteristics: a primer for neuroscience nurses.

ABSTRACT

It is Important for neuroscience nurses to have a solid understanding of the instruments they use in clinical practice. Specifically, when reviewing reports of research instruments, nurses should be knowledgeable of analytical terms when determining the applicability of instruments for use in clinical practice. The purpose of this article is to review 3 such analytical terms: sensitivity, specificity, and receiver operating characteristic curves. Examples of how these terms are used in the neuroscience literature highlight the relevance of these terms to neuroscience nursing practice. As the role of the nurse continues to expand, it is important not to simply accept all instruments as valid but to be able to critically evaluate their properties for applicability to nursing practice and evidence-based care of our patients.

Keywords: instrument, sensitivity, specificity

**********

Nurses need a solid understanding of key terms and analytical approaches that are reported in various articles. Specifically, when reviewing reports of research instruments, nurses should be knowledgeable of analytical terms when determining the applicability of instruments for use in clinical practice. Selection of an inappropriate instrument for routine use in clinical practice may result in care decisions that are not aligned with patient needs or characteristics. For example, integrating a screening instrument for delirium commonly used in the critical care setting may not be appropriate for routine use in the longterm care setting and may cause inaccuracies in delirium detection and coordinated therapies. Therefore, the purpose of this article is to review 3 key analytical terms that neuroscience nurses should be familiar with when evaluating instruments: sensitivity, specificity, and receiver operating characteristic (ROC) curves. Examples of how these terms are used in the neuroscience literature will be used to highlight the relevance of these terms to neuroscience nursing practice.

Sensitivity

The sensitivity of a measurement instrument is the ability to detect differences or changes in a characteristic that are being measured. (1) In statistical terms, this is the probability that a diagnostic test or instrument will be positive in persons who have a disease or condition. (2) Sensitivity is also referred to as a true positive rate. Diagnostic tests or instruments that have high sensitivity are more likely to rule in, or accurately confirm, the disease or condition when the disease or condition exists. (2)

Specificity

In contrast, specificity is the ability of a measurement instrument to correctly identify persons without a disease or condition. In statistical terms, this is the probability that a diagnostic test or instrument will be negative in persons who do not have the disease or condition. (2) Tests or instruments that have high specificity are able to more accurately rule out a disease or condition. Specificity is often referred to as the true negative--meaning that a test is negative in persons without the disease or condition. (1)

Sensitivity and specificity are often reported together when describing the characteristics of a measurement instrument. As clinicians, we must be cognizant of the fact that there is no perfect instrument to accurately measure every parameter, every time. Instead, there is a delicate balance between being able to accurately rule in (sensitivity) and the ability of that same instrument to also accurately rule out (specificity) a disease or condition. Oftentimes, as sensitivity increases (ie, the instrument becomes better at detecting true positives), there is a corresponding decrease in specificity (the instrument begins to identify more false positives). (3)

ROC Curve

The ROC curve sounds confusing, but is very easy to interpret. In rough mathematical terms, the ROC curve is simply a graph that illustrates the relationship between the true positive rate and the false positive rate. (4) In other words, the ROC is a visual and mathematical model to express the combination of sensitivity and specificity on a scale from 0 to 1.

Receiver operating characteristic curves are a plot of the true positive rate against the false positive rate or sensitivity against 1 -specificity. The y axis is the true positive or sensitivity rate, whereas the x axis is the false positive rate (measured as 1-specificity) (4) (Fig 1). The diagonal line represents essentially a worthless test, where the ability of the tool to discern between true and false positives would be equivalent to tossing a coin. Therefore, the closer the ROC curve is to the upper left border, the more accurate is the instrument. Conversely, the closer the curve is to the diagonal line, the less accurate the instrument is in discerning between true positives and false positives. (5) Figure 1 also displays ROC curves for 2 different diagnostic tests. It is clear that curve A is much closer to the upper left border and thus represents a larger area under the curve (AUC). In contrast, curve B lies closer to the diagonal line and occupies a smaller AUC. Therefore, curve A represents an instrument with higher sensitivity (high true positive) and higher specificity (low false positives) when compared with the tool represented by curve B. In this example, the instrument represented by curve A is the better diagnostic predictor or measure.

Oftentimes, the ROC curve is accompanied by a report of the AUC. This parameter provides additional information about the accuracy or predictive ability of the tool. Whereas the ROC curve is the graphic representation, the AUC is the actual numerical value that identifies how well the tool can decipher between true positives (sensitivity) and false positives (specificity) (4) An AUC of 1 represents a perfectly accurate test, which would be depicted by an ROC curve that lies in the upper left corner. In contrast, an AUC of 0.5 represents a test equivalent to the toss of a coin and is essentially worthless, which would be reflected by an ROC curve lying adjacent to the diagonal line. (4) The following scale is commonly used for classifying accuracy using AUC: 0.90 to 1, excellent accuracy; 0.80 to 0.90, good accuracy; 0.70 to 0.80, fair accuracy; 0.60 to 0.70, poor accuracy; 0.50 to 0.60, no accuracy. (3) Therefore, an ROC curve that has a high AUC of 0.9, for example, reflects an instrument that is 90% accurate in deciphering between true positive and false positive and is a strong instrument to use when measuring a particular quality. (3)

Examples of Sensitivity, Specificity, and ROC Curves in Neuroscience

The neuroscience nursing literature is robust with examples of sensitivity and specificity. First, a completely fictional example is presented. The Out of Bed on Purpose Scale (OOPS) is created by St Falls Hospital to determine whether a patient is stable enough to get out of bed. The OOPS ranges from 0 to 10; scores of 8 or higher indicate that the patient is high risk for falls. After testing the OOPS, the authors report that the sensitivity was 0.90 and the specificity was 0.40. These values indicate that almost all the patients who fall will have a high (8 or greater) OOPS score. The OOPS is really good at identifying patients who will fall (sensitivity, 0.90). However, OOPS is not very good at identifying the patients who will not fall (specificity, 0.40).

There is some nuance to the language that is important. The high sensitivity indicates that nearly every patient who fell had an OOPS of 8 or higher. However, the specificity indicates that there were a lot of patients who had high (8 or greater) OOPS scores who did not fall. Table 1 presents these data visually so we can see that, in this study of 110 patients, 69 patients had high OOPS but only 9 of these patients experienced a fall. Further understanding of this data is provided by an AUC of 0.50.

To incorporate a recently published example of sensitivity, specificity, and ROC/AUC analyses, Czaikowski et al (6) published a report describing the performance measures for a Pediatric FOUR Score Scale (PFSS). The authors examined whether PFSS scores were predictive of poor outcome and compared these scores with the predictive ability of Glasgow Coma Scale scores. In the Results section, the authors report the sensitivity of PFSS scores to be 0.9333, with a specificity of 0.7903. From this, we can conclude that most of the patients who have a poor outcome will have a low PFSS (sensitivity, 0.93). We can also conclude that many of the patients who do well (not a poor outcome) will not have low PFSS scores. The authors include an AUC value of 0.9043 for this relationship. This AUC indicates that, in this sample, the PFSS is a very good instrument for predicting outcomes. The ROC curve in this example supports this conclusion because it lies very close to the upper left border, reflecting a large AUC.5 When comparing the predictive ability of PFSS scores with that of GCS scores, the ROC curves show similar AUC for both instruments, with curves for both PFSS and GCS lying close together and near the upper left border. Thus, both tools are similar in their predictive ability and possess high sensitivity (true positive) and high specificity (low false positives).

Implications for Neuroscience Nurses

It is important for neuroscience nurses to have a solid understanding of the instruments they use in clinical practice. There are numerous instalments used throughout the continuum of care to evaluate different characteristics of neuroscience patients, such as level of consciousness, severity of neurological injury, functional status, quality of life, and ability to return to work. When determining the applicability of these tools for routine use in clinical practice, nurses should have knowledge of the accuracy of these tools to effectively diagnose or predict outcomes. Having a solid understanding of sensitivity and specificity and being able to interpret ROC curves and AUC of these tools are important to determine the usefulness of the instrument for routine clinical care. Nurses must be sure to also review the applicability of the tool for their patient population and other psychometric properties, such as reliability and validity. As the role of the nurse continues to expand, it is important not to simply accept all instruments as valid but to be able to critically evaluate their properties for applicability to nursing practice and evidence-based care of our patients.

DOI: 10.1097/JNN.0000000000000267

REFERENCES

(1.) Hulley SB, Cummings SR. Browner WS, et al. (2007). Designing Clinical Research. Philadelphia, PA: Lippincott Williams & Wilkins.

(2.) Munro, BH. (2005). Statistical Methods for Health Care Research. Philadelphia, PA: Lippincott Williams & Wilkins.

(3.) Florkowski, CM. Sensitivity, specificity, receiver-operating characteristic (ROC) curves and likelihood ratios: communicating the performance of diagnostic tests. Clin Biochem Rev. 2008;29:S83-S87.

(4.) Zou KH, O'Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic test and predictive models. Circulation. 2007;115:654-657.

(5.) Ma X, Nie L, Cole SR, et al. Statistical methods for multivariate meta-analysis of diagnostic tests: an overview and tutorial. Stat Methods Med Res. 2016;25:1596-1619.

(6.) Czaikowski BL, Liang H, Stewart CT. A pediatric FOUR score coma scale: interrater reliability and predictive validity. J Neurosci Nurs. 2014;46(2):79-87.

Questions or comments about this article may be directed to Molly McNett, PhD RN CNRN, at mmcnett@metrohealth.org. She is the Director, Nursing Research, MetroHealth System, Cleveland, OH.

Shelly Amato, CNS RN CNRN, is Clinical Nurse Specialist, MetroHealth Center for Brain Injury Rehabilitation, Cleveland, OH.

DaiWai M. Olson, PhD RN CCRN FNCS, is Associate Professor of Neurology & Neurotherapeutics, Associate Professor of Neurosurgery, University of Texas Southwestern, Dallas, TX.

The authors declare no conflicts of interest.

Caption: FIGURE 1 ROC Curves Comparing 2 Different Diagnostic Tests
TABLE 1. Fictitious Data From the
OOPS Study

                                Condition
                                 Patient
                      Patient    Did not
                       Fell       Fall      Total

Test    OOPS = 8-10      9         60        69
        OOPS = 1-7       1         40        41
Total                   10         100       110

Note. OOPS = Out of Bed on Purpose Scale.
COPYRIGHT 2017 American Association of Neuroscience Nurses
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:McNett, Molly; Amato, Shelly; Olson, DaiWai M.
Publication:Journal of Neuroscience Nursing
Article Type:Report
Date:Apr 1, 2017
Words:1963
Previous Article:Intracranial pressure values are highly variable after cerebral spinal fluid drainage.
Next Article:Implementation of an early mobility pathway in neurointensive care unit patients with external ventricular devices.
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters