National Institutes of Health Stroke Scale reliable and valid in plain English.
The National Institutes of Health Stroke Scale (NIHSS) is commonly used in the assessment of stroke severity. Nurses, who use the tool infrequently, find it difficult to use due to the neurologic terminology embedded in the scale. For this project, we modified the NIHSS by replacing the neurologic terminology for each component of the original scale with plain English. No components were deleted or changed; the language was merely simplified. Testing showed the modified tool to be reliable (0.96) and valid (0.977) when compared with the NIHSS.
The National Institutes of Health Stroke Scale (NIHSS) was originally designed as a research tool to measure stroke severity (Brott et al., 1989). Over time, it has developed into the gold standard for stroke assessment and measurement (Richardson, Murray, House, & Lowenkopf, 2006). The NIHSS is currently used to determine treatment options, anticipate discharge planning, and measure patient outcomes (Adams et al., 1999; Schlegel et al., 2003; Weimar, Konig, Kraywinkel, Ziegler, & Diener, 2004). Specialized training is required to use the scale, in part because of the specialized neurology terminology used in the body of the tool (Andre, 2002). Online training in the NIHSS, offered by the American Stroke Association, can take, on average, 2 to 3 hours per trainee (American Heart Association, 2004).
The NIHSS is used for both initial assessments of stroke severity and monitoring changes in a patient's condition. The NIHSS gives a numerical value to stroke severity, allows caregivers to compare scores over time, and gives a common language to the stroke community. The American Heart Association's Get With the Guidelines collects data on indicators of quality stroke care, including whether the initial NIHSS was done on each stroke patient (American Heart Association, 2008).
Providence Portland Medical Center (PPMC) is a 483-bed hospital in the Providence Health System, located in Portland, OR. PPMC admits approximately 400 stroke patients per year. PPMC is nationally recognized by the Joint Commission as a certified Primary Stroke Center by the American Stroke Association for exceeding national benchmarks using "Get With the Guidelines--Stroke" having received the 2007 Gold Level Award for sustained performance and by the American Nurses Credentialing Center as a Magnet Hospital for Excellence in Nursing. This study was done with the support of the Providence Brain Institute.
The NIHSS was first used at PPMC in patient care as an assessment tool with the opening of a new neuroscience unit in fall 2003. Floor nurses on the general neuroscience unit were taught to use the NIHSS as part of a formalized stroke treatment and recovery program. NIHSS education was further expanded shortly thereafter to include the intensive care unit (ICU) and Emergency Department (ED) nursing staff. Training originally consisted of reviewing the NIHSS training tapes in formal education sessions (Stroke Group, 1998), and PowerPoint presentations. With availability of the online American Stroke Association NIHSS program (American Heart Association, 2004), training has gradually moved to this standardized venue.
Over time, the neuroscience nursing staff has become very comfortable, confident, and efficient in completing the NIHSS because they use this tool daily in their patient assessment. The ED and ICU use the tool less frequently, which, we believe, accounts for their continued complaints that the tool is difficult to use, time-consuming, and intimidating. Their lack of confidence in their use of the tool was highlighted by NIHSS scores that often differed significantly from those obtained by the stroke team physicians and nurses.
In spring 2005, at the request of the ED and ICU nursing staff, we modified the NIHSS language by replacing neurologic terminology with plain, common-use English for each component of the original scale. No components were deleted and no changes were made in scoring; only the language was simplified. Both the stroke neurologist and stroke nurse had formal training and testing in the NIHSS through the American Stroke Association and years of experience using the NIHSS before rewording the NIHSS. The reworded scale was then reviewed by the Providence Stroke Program staff, including a second stroke neurologist and two additional stroke nurses, all formally trained in the use of the NIHSS, with further simplifications made. The ICU and ED nurse educators then reviewed the wording, making suggestions for further improvement that were incorporated. A final tool, which we called the Stroke Scale in Plain English (SSPE) was finalized in fall 2005.
To test for reliability and validity, the study was designed to compare scores obtained using the NIHSS versus those using the SSPE. Six study sessions were held, including a total of 46 nurse participants. Using Benner's novice-to-expert concept (Benner, 1984), nurses with varying levels of expertise and experience in using the NIHSS were enlisted to participate. Sixteen nurses were novice users and were trained in the use of the NIHSS and tested on the same day. Fifteen nurses worked in the ICU or ED and were considered competent users, using the tool two to four times per month. Fifteen nurses worked on the general neuroscience unit and were considered expert users, using the scale daily
In the first study session, nurse participants watched five videotaped stroke patients (American Academy of Neurology, 1999) using the NIHSS to score the patients. This same group of nurses then watched the same five videotaped stroke patients again using the SSPE. In subsequent study sessions, we alternated which scale was used first to score patients to reduce the chance of bias.
To test the reliability of the SSPE, we calculated interrater reliability and internal consistency. We used the intraclass correlation coefficient (ICC) as a measure of interrater reliability. Using terminology of Shrout and Fleiss (1979), the form of intraclass correlation coefficient chosen was ICC (2,1); that is, we operated as if we selected raters randomly after which each rater rated each patient (target).
For internal consistency, we used two measures: Cronbach's [alpha] (Cronbach, 1951) and omega (Heise & Bohrnstedt, 1970). Cronbach's [alpha] is the most commonly used method of estimating reliability. It has, however, one drawback: unless the scale is unidimensional, [alpha] will underestimate reliability. When used with a multidimensional scale, [omega] gives higher reliability estimates, and factor analysis of the NIHSS showed that it was multidimensional (Lyden et al., 1999). To calculate [omega], the rating scales must be factor analyzed, for which we used principal axis factoring. To determine concurrent validity, defined as the relationship between a new instrument and existing, valid measures, we calculated the correlation between the SSPE and the NIHSS. A high correlation between the two would show concurrent validity
In addition, we used the validity and invalidity formulas of Heise and Bohrnstedt (1970). Theirs is a unique method because it demonstrates validity through factor analysis. Their validity formula gives the correlation between the composite score of the measure being tested and each factor derived from factor analysis. The invalidity formula measures the correlation between the composite score and a factor that the composite score is not supposed to measure. With a tool like the NIHSS or SSPE, one expects that the total (composite) score would have a high validity score and low invalidity score on whichever factor best represents the total score. Usually, this factor is the first, unrotated, extracted factor. Thus, to the extent that the first factor represents stroke severity, a high validity score and low invalidity score would contribute evidence of the validity of the scale. For our analyses, we used SPSS Version15 (SPSS, Inc., Chicago, IL) and Excel 2000 (Microsoft Corp., Redmond, WA).
The data set had a handful of missing values and a legitimate answer "untestable," which carried no rating score. In these cases, we decided to impute scores for the missing values and for the valid but nonscored ratings. The imputation was completed by regressing an item with missing values on the other scale items. We used the resulting regression equation to impute a value, rounded to the nearest legitimate score (Table 1).
Total scores were calculated for all raters, regardless of missing data. We did not change these original, total scores despite value imputation, and, as described above, we used the total score for estimating interrater reliability. We used imputation to facilitate factor analysis and [omega], [alpha], validity, and invalidity calculations.
As Table 2 shows, the reliability estimates of the NIHSS and SSPE are high. Because of the multidimensional nature of the two rating scales, the reliability estimate of co exceeds that of Cronbach's [alpha]. Moreover, both scales had high interrater reliabilities that were comparable to [omega]. Finally, the reliability estimates between the scales were comparable, showing that the SSPE had the same reliability as the NIHSS.
As Table 3 shows, the correlation between the SSPE and NIHSS--concurrent validity--was very high. Moreover, the Heise and Bohrnstedt (1970) validity and invalidity scores for both measures strongly demonstrate the validity of these scales as measures of stroke severity.
From study results and using several different testing measures, the SSPE is a highly reliable tool and comparable to the NIHSS. The SSPE also has high concurrent validity with the NIHSS, given that that NIHSS measures stroke severity.
The original objective, to maintain the integrity and meaning of the NIHSS while removing the neurologic terminology, was met according to these study findings. However, further work will need to be done to make a stand-alone tool that any caregiver could use with minimal to no training in the use of the scale.
Although the SSPE may become an acceptable tool for the bedside nurse in the ED, ICU, and stroke units, patients being enrolled in clinical trials would still require the traditional NIHSS to be completed by a person trained and tested for intrarater reliability in the scale. The NIHSS remains the established and proven gold standard for measurement of stroke scale severity.
In reviewing data from both the NIHSS and the SSPE, several components of the current scales, including best language, dysarthria, and extinction/ inattention, need refinement. These findings are reflected in the current literature as also being problem areas with the NIHSS (Lyden, Lu, Levine, Brott, & Borderick, 2001). A future challenge would be to rewrite these items so that nurses could obtain more consistent scores with little to no training in the use of the stroke scale. It may be possible to rewrite these components of the NIHSS to the point where scores are more consistent compared with scores of expert users, actually improving intrarater reliability scores to be more consistent in these three problem-prone areas of the traditional NIHSS.
Additional benefits of redesigning the SSPE so that minimal training is needed would be the cost savings from reduced training time, as well as potentially expanding its use to smaller centers. A stand-alone tool requiring no training with proven validity and reliability would allow stroke care providers to more easily and accurately communicate a patient's condition, make treatment decisions, and improve the overall quality of patient care.
As a footnote, nurses in the ED and ICU preferred the SSPE, whereas nurses on the stroke unit were more comfortable with the standard NIHSS because it was more familiar to them.
We thank all our volunteers from the Emergency Department/Intensive Care Unit/Float Pool/Orthopedic Unit/Neurology Unit at Providence Portland Medical Center; Emergency Department at Providence St. Vincent's Medical Center; students at University of Portland; and Susie Fisher, Lisa Shields, and Lisa Brooks for their support.
Adams, H. P, Davis, P H., Leira, E. C., Chang, K. C., Bendixen, B. H., Clarke, W. R., et al. (1999). Baseline NIH Stroke Scale score strongly predicts outcome after stroke: A report of the Trial of Org 10172 in Acute Stroke Treatment (TOAST). Neurology, 53, 126-131.
American Academy of Neurology (1999). First certification session. National Institutes of Health Stroke Scale (Tape 2). Saint Paul, MN: Author.
American Heart Association/American Stroke Association Professional Education Center. (2004). NIH Stroke Scale (NIHSS) training online. Retrieved January 2008 from http://www.professionaleducationcenter.americanheart.org/ ihtml/application/student/interface.heart/
American Heart Association. (2008). Get with the guidelines--Stroke 2008. Retrieved May 2008 from http://www.americanheart.org/ downloadable/heart/1203521452783StrokeFactSheet.pdf
Andre, C. (2002). The NIH Stroke Scale is unreliable in untrained hands. Journal of Stroke and Cerebrovascular Diseases, 11, 43-46.
Benner, P. (1984). From novice to expert: Excellence and power in clinical nursing practice (pp. 13-34). Menlo Park, CA: Addison-Wesley
Brott, T., Adams, H. P, Olinger, C. P, Marler, J. R., Barsan, W. G., Biller, J., et al. (1989). Measurements of acute cerebral infarction: A clinical examination scale. Stroke, 20, 864-870.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
Goldstein, L. B., Bartels, C., & Davis, J. N. (1989). Interrater reliability of the NIH stroke scale. Archives of Neurology, 46, 660-662.
Heise, D. R., & Bohmstedt, G. W. (1970). Validity, invalidity, and reliability. In: E. F. Borgatta, & G. W. Bohrnstedt (Eds.), Sociological methodology (pp. 104-128). San Francisco: Jossey-Bass.
Lyden, P., Lu, M., Jackson, C., Marler, J., Kothari, R., Brott, T., et al. (1999). Underlying structure of the National Institutes of Health Stroke Scale: Results of a factor analysis. Stroke, 30, 2347-2354.
Lyden, P. D., Lu, M., Levine, S. R., Brott, T. G., & Broderick, J. (2001). A modified National Institutes of Health Stroke Scale for use in stroke clinical trials: Preliminary reliability and validity. Stroke, 32, 1310-1317.
Richardson, J., Murray, D., House, K., & Lowenkopf, T. (2006). Journal of Neuroscience Nursing, 38, 309-314.
Schlegel, D., Kolb, S. J., Luciano, J. M., Tovar, J. M., Cucchiara, B. L., Liebeskind, D. S., et al. (2003). Utility of the NIH Stroke Scale as a predictor of hospital disposition. Stroke, 34, 134-137.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
Stroke Group. (1998). The 'Quick & Easy' NIH Stroke Scale and tutorial program [Videotape]. Littleton, CO: SNR Resources.
Weimar, C., Konig, I. R., Kraywinkel, K., Ziegler, A., & Diener, H. C. (2004). Age and National Institutes of Health Stroke Scale score within 6 hours after onset are accurate predictors of outcome after cerebral ischemia. Stroke, 35, 158-162.
Allen J. Brown is a Senior Research Analyst at the Center for Outcomes Research and Education at Providence Portland Medical Center, Portland, OR.
Lisa Rietz Yanase, MD, is a medical director for the Stroke Team at Providence Portland Medical Center, Portland, OR.
Questions or comments about this article may be directed to Sandy Dancer, RN MS ANP-C, at Sandy.Dancer@providence. org. She is a stroke nurse practitioner at Providence Portland Medical Center, Portland, OR.
TABLE 1. Missing and Imputed Values No. missing or No. of Imputed Scale Category "untestable" imputations score NIHSS 1b. LOC questions 3 3 2 1c. LOC commands 1 1 0 4. Facial paresis 1 1 0 10. Dysarthria 27 7 1 20 2 SSPE 1 b. Questions 4 2 1 2 2 10. Dysarthria 30 13 1 17 2 Note. NIHSS = National Institutes of Health Stroke Scale; SSPE = Stroke Scale in Plain English. TABLE 2. Reliability Statistic NIHSS SSPE [omega] .964 .974 Cronbach's [alpha] .854 .849 Intraclass correlation coefficient .950 .959 (interrater reliability) Note. NIHSS = National Institutes of Health Stroke Scale; SSPE = Stroke Scale in Plain English. TABLE 3. Validity Validity NIHSS SSPE Concurrent validity -- .977 (total score correlation of SSPE to NIHSS) Heise & Bohrnstedt (1970) .979 .977 validity (correlation with first factor) Heise & Bohrnstedt (1970) .005 .020 invalidity (first factor) Note. NIHSS = National Institutes of Health Stroke Scale; SSPE = Stroke Scale in Plain English.
|Printer friendly Cite/link Email Feedback|
|Author:||Dancer, Sandy; Brown, Allen J.; Yanase, Lisa Rietz|
|Publication:||Journal of Neuroscience Nursing|
|Date:||Feb 1, 2009|
|Next Article:||The domains of stroke recovery: a synopsis of the literature.|