Defining the minimum level of detectable change for the Roland-Morris Questionnaire.

Physical therapists regularly use measurements (eg, of range of motion, pain, or disability) to determine whether a patient's status has changed over time. Often, when the results differ from one assessment to the next, therapists assume patients have undergone true change. Unfortunately, some or all of the difference between assessments can occur as a result of measurement error, including random fluctuation in patients who may or may not have truly changed. A patient who at the initial assessment scores 14 out of a possible 24 points on a particular disability questionnaire and 4 weeks later scores 10 points may appear to have undergone change. The 4-point difference may represent true change, or it could fall within the limits of measurement error and inherent variability in a truly unchanged patient. The importance of this issue is underscored when the management options available to therapists are considered. For example, therapists who view the difference in scores as representing true change may elect to continue with an intervention. Therapists who consider the 4-point change to be within the limits of measurement error, however, may choose to alter the intervention in hopes of selecting a treatment that is more effective. The goal of this report is to provide clinicians with guidelines for assessing change over time when they use the Roland-Morris Questionnaire (RMQ)1,2 to assess disability in patients with low back pain (LBP).

The RMQ was selected because its measurement properties have been shown to be equal to or better than those of similar measures used to assess change in disability in patients with LBP.[1-15] In Table 1, we provide a brief summary of the more frequently used and researched measures. The RMQ is a self-administered questionnaire consisting of 24 items chosen from the Sickness Impact Profile (SIP).[16] Items were chosen to reflect a variety of activities of daily living. To improve the specificity of the response, Roland and Morris[1] added the phrase "because of my back" to each item. An item receives a score of 1 if it is checked as applicable by the respondent and a score of 0 if it is not marked. Accordingly, total scores can vary from 0 (no disability) to 24 (severe disability). Research of the RMQ's measurement properties has provided consistent estimates of internal consistency, test-retest reliability (accounting for the interval between assessments), construct validity, and sensitivity-to-change coefficients. The term "sensitivity to change" describes a measure's ability to assess change over time.

One strategy for assessing and reporting change over time, reported in Table 1, is the receiver operating characteristic (ROC) curve.[15] With this technique, sensitivity (y-axis) is plotted against 1-specificity (x-axis). When assessing change over time, sensitivity is defined as the number of patients correctly identified (by a given questionnaire) as having undergone a clinically important change divided by all patients who truly underwent a clinically important change. Specialty refers to the number of patients who were correctly identified (by a given questionnaire) as, not undergoing a clinically important change divided by all patients who truly did not undergo a clinically important change. The greater the area under the curve, the greater a questionnaire's ability to distinguish patients who did and did not undergo a clinically important change. The area under the curve can be interpreted as the probability of correctly identifying a patient who has undergone a clinically important change from randomly selected pairs of patients who have and have not undergone an important change.

[TABULAR DATA 1 OMITTED]

Numerous studies[4,6,9,10,12,14,15] have assessed measures of sensitivity to change in patients with LBP. In only one of the many investigations reported, however, was information presented in a format that is suitable for making decisions on individual patients.[9] Using a test-retest reliability design (the interval between assessments was 3-6 weeks), Stratford et al[9] calculated the standard error of measurement (SEM) for RMQ scores in 36 stable patients with LBP to be 1.79 RMQ points. The SEM expresses measurement error in the same units as those of the original measurement, in this case RMQ points. The SEM is a measure of within-patient variability and is calculated by taking the square root of the mean square error term from the usual reliability study analysis-of-variance table. In addition to reporting the SEM, these authors calculated the minimal level of detectable change at the 95% confidence level to be 5 RMQ points.[9] This value defines the smallest difference that can be detected between two measurements. It is also referred to as the "reliability change index."[17] The interpretation of the minimal level of detectable change is that an observed change in a patient that is less than this value is deemed to be indistinguishable from measurement error. Accordingly, a patient who demonstrates a change score that is less than this value is viewed as not having undergone change. The principal limitation of early work reporting the SEM[9] is that this statistic assumes measurement error is constant across the range of possible scores. In this report, the conditional standard error of measurement (CSEM) will be used. This measure is defined in the "Method" section, and an illustration is provided in the Appendix.

In this study, we attempted to improve on previous research by providing clinicians with estimates of minimal detectable change using a process that takes into account that the level will change for different combinations of initial and follow-up RMQ score comparisons. These estimates can be used to determine whether the disability of an individual patient is likely to have actually changed. The research question was: What are the minimum levels of detectable change for all possible score comparisons on the RMQ when it is applied to patients with LBP?

Method

Subjects

The sample consisted of 60 patients with LBP (37 male, 23 female) who were referred by their physicians to the outpatient physical therapy departments of two hospitals. Patients werc eligible for this study if they (1) were diagnosed as having LBP of apparent musculoskeletal origin, (2) could read English, and (3) provided written consent on a form approved by the centers' research review boards. The patients were aged 18 to 72 years (X[bar]=41, SD=12). Forty-eight patients were employed at the time of onset of the episode of back pain associated with these referrals, and the referrals of 35 of these patients involved insurance claims. Thirty-eight patients experienced sudden onset of discomfort, 20 patients experienced a gradual onset of discomfort, and 2 patients were uncertain as to the nature of the onset of discomfort. Nineteen patients had a limited straight leg raise (estimated at less than 60[degrees], and 12 patients had episodes of discomfort distal to the knee at the time of initial assessment. This episode of LBP was less than 6 weeks for all patients. The sample size of 60 patients was based on an expected internal consistency coefficient of .90[6-8] and a lower 95% confidence interval width of .05.[23]

Design

A before-after study design was used to obtain two RMQ scores for each patient. Patients completed the RMQ, prior to beginning physical therapy and following 4 to 6 weeks of treatment. This duration was chosen for two reasons: (1) The natural history of acute LBP is such that over 60% of patients show significant improvement within this interval,[2] and (2) intervention studies on patients with acute LBP often report outcomes between 4 and 6 weeks.[18,19] It is important to note that the physical therapy interventions applied to patients were neither of interest nor under investigation. These interventions, like the interval between assessments, served as a construct for achieving a change. Patients were asked to complete the RMQ in accordance with the instructions provided by Roland and Morris.[1] This process allowed estimates of measurement error to be assessed for both points in time.

Data Analysis

Conditional standard errors of measurement were used to estimate the minimum levels of detectable change.20 The method is based on the binomial theory of measurement error[21] and the correction approach described by Keats.[22] All possible initial and follow-up score combinations were compared using the Z statistic. A 90% confidence level was chosen, and the corresponding Z value is 1.65. An illustration of the analysis is provided in the Appendix. Actual patient data were required only to

estimate the reliability coefficients used for this correction factor.

Results

The means and 90% confidence intervals for the initial and follow-up RMQ scores were 11.5 (9.9-13.1) and 6.6 (5.1- 8.1) RMQ points, respectively. The [KR.sub.20] reliability coefficient was .92 for both the initial and follow-up visits, whereas the [KR.sub.21] coefficients for the initial and follow-up visits were .89 and .90, respectively. Table 2 provides a summary of the initial and follow-up conditional error variances and CSEMs for all possible RMQ scores. For example, the CSEM for initial and follow-up RMQ scores of 14 are 2.13 and 2.24, respectively. The small difference in CSEM scores between initial and follow-up visits of thc same score is due to the slight difference in the magnitude of the KR21 coefficient for these two points in time. The Figure provides a summary of minimum detectable change values for improvement and deterioration. The data points for this figure were calculated in accordance with the procedure outlined in step 6 of the Appendix. In order to be confident at the 90% level that a change has occurred, the intersection of the initial and follow-up scores must be outside the shaded area. For example, a patient who had an initial score of 14 must achieve a score of 9 or lower for the clinician to be confident that improvement has occurred, or a score of 18 or higher to be convinced that deterioration has taken place. Finally, the Figure shows that improvement cannot be detected for patients who have initial RMQ scores lower than 4 and that deterioration cannot be ascertained for patients who have initial RMQ scores greater than or equal to 20.,

[TABULAR DATA 2 OMITTED]

Discussion

Researchers using the SEM have estimated the minimum level of detectable change to be approximately 5 RMQ points at the 95% confidence level.[9] A limitation of using the SEM is that it assumes that the magnitude of measurement error is uniform across the entire scale (ie, equal for all scores). Moreover, a shortcoming of our previous study[9] was that most of the patients' initial RMQ scores were in the central portion of the scale. Accordingly, a clinician cannot be confident that a change of 5 RMQ points accurately reflects the minimum level of detectable change for values more distant than those located near the central portion of the scale. One strategy for estimating the score-specific level of minimum detectable change would be to conduct a number of reliability studies in which patients are stratified on the basis of their initial scores. To obtain a reasonable confidence interval on the reliability coefficient, approximately 30 stable patients per group would be required. Given that patients can have 25 possible initial RMQ scores (ie, 0-24), 25 studies would be required. The In our study, we attempted to address the deficiency of previous work by calculating the CSEM and the minimum level of detectable change for various initial and follow-up score combinations. Rather than performing a stratified study, we estimated the CSEM using the binomial theory of measurement error. Using this approach, the magnitude of measurement error is dependent on the actual scores being compared. This approach is appropriate when item scoring is dichotomous, as in the RMQ. Our results are consistent with those of previous work to the extent that initial scores located near the central portion of the scale require a change of 5 RMQ points for a clinician to be confident at the 95% level that a change has really occurred. The results, however, add to existing knowledge by suggesting that a change of only 4 RMQ points is required to detect improvement in patients with initial scores of 4 to 11 RMQ points and in patients with scores greater than 16 RMQ points. Similarly, a change of only 4 RMQ points is needed to detect deterioration in patients with initial scores lower than 7 RMQ points and in patients with scores of 13 to 20 RMQ points. Improvement in patients with initial RMQ scores lower than 4 RMQ points and deterioration in patients with initial scores greater than 20 RMQ points cannot be We believe our findings can be generalized to various clinical settings. The ages, gender distribution, and initial and follow-up RMQ scores of our sample are consistent with those of other researchers reporting on patients with acute or subacute LBP.[9,14,15,18,19,24,25] Furthermore, the magnitudes of the KR20 and KR2, coefficients display a remarkable similarity to internal consistency coefficients for the RMQ reported by other authors.[6-8] For these reasons, we believe that the reported levels of minimum detectable change are generalizable to patients with acute and subacute LBP attending outpatient physical therapy.

Our study was a reliability study, and the values for minimum detectable change represent estimates of measurement error. These values are not to be confused with the minimal clinically important difference (MCID).[26] Minimal clinically important difference has been defined as the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient's management."[26(p408)] Clinical decision making is impeded when minimum detectable change exceeds the MCID. There are currently no reports that identify the MCID for the RMQ or for any of the disability measures applied to patients with LBP. We believe that future inquiry should attempt to estimate the MCID and determine the extent to which it is dependent on patients' initial scores.

Conclusion

This work calculated CSEM to estimate minimum levels of detectable change in RMQ points for patients with LBP. The magnitude of minimum detectable change, 4 to 5 RMQ points determined at the 90% confidence level, is dependent on the scores being compared. The results of our study indicate that improvement in patients with initial scores lower than 4 RMQ points and deterioration in patients with initial scores greater than 20 RMQ points cannot be detected with a high degree of confidence. Ongoing challenges include defining the MCID and identifying strategies for detecting improvement in patients with low levels of disability and deterioration in patients with high levels of disability.

RELATED ARTICLE: Invited Commentary

My comments on the article of Stratford and colleagues deal with two major issues. The first issue relates to the statistical approach used by the authors to describe the error associated with Roland-Morris Questionnaire (RMQ) change scores. The second and most important issue relates to the application of the current report and related work of Stratford and colleagues to clinical practice.

When attempting to document whether a patient's level of disability (or any other attribute) has changed, the therapist has to be concerned about the error present in both the initial and follow-up measurements. The initial and follow-up measurements are compared to derive a change score. It is this change score that is important for clinical decision making. Many of our clinical decisions are based on comparisons of measurements of an attribute taken during a patient's care. This report is one of the few in our literature that establishes the magnitude of error associated with change scores.

Stratford and colleagues referenced their earlier work that demonstrated that the standard error of the measurement (SEM) at the 95% confidence level is 5 RMQ points. The SEM is used in the calculation of the Reliability Change Index (RCI).[1] The RCI is a statistical procedure for estimating the error associated with change scores. This earlier work is closely related to the current report, but unfortunately, the earlier work was not yet published at the time this commentary was written. The authors reported that they used the RCI as defined by Ottenbacher et al.[2] Ottenbacher and colleagues, however, also described a revised Reliability Change Index (RCI') that requires the use of the standard error of the difference in the calculation of the index. Ottenbacher et al argued that the standard error of the difference is thought to be more indicative of the error present in change scores because it takes into account the error in both the initial and follow-up scores. The RCI accounts for error in only one of the two measurements used to determine the change score. It is not clear whether the RCI or the RCI' was used by Stratford and colleagues in their earlier work.

I used the raw data reported by Roland and Morris,[3] to calculate the standard error of the difference at the 95% confidence level for repeated measurements of the RMQ on 20 patients with low back pain. I found the standard error of the difference at the 95% confidence level to be rounded off to 5 points. Estimates of the error associated with change scores are very similar, not only for the current work and the earlier work of Stratford et al but also for the original data reported by Roland and Morris. In the current article, Stratford and colleagues also have demonstrated that measurement error varies depending on where the measurements fall on the scale. Error is "conditional" on where the measurement falls on the scale. Measurements that fall nearer to the ends of the scale will generally have less error than measurements that fall near the middle of the scale.[4] Less variability is essentially "built-in" at the ends of the scale; therefore, the error theoretically should be less as compared with measurements that fall near the middle of the scale. As Stratford and colleagues report, the conditional SEM is one method used to account for this variability.[5]

The fact that measurement error varies along a scale is recognized as being important by many groups. The American Psychological Association describes the following in their measurement standards, The Standards for Educational and Psychological Testing.

Standard 2.10. Standard errors of measurement should be reported at critical score levels. Where cut scores are specified for selection or classification, the standard errors of measurement should be reported for score levels at or near the cut score. Comment: Reporting standard errors of measurement at every score level may not be feasible in some circumstances, but they should be reported at appropriate, well-separated levels or intervals.6(p22)

The American Physical Therapy Association does not directly address the issue of conditional standard error of measurement, but the Association's Standards for Tests and Measurements in Physical Therapy Practice state the following:

S13.5. Research reports on reliability written by secondary purveyors [researchers] must include a description of the statistics used to dcrive reliability estimates. The rationale for the use of these statistics must be provided. When methodologically appropriate, reports of confidence intervals and standard errors of measurement should be included. Examples of how the reliability estimates are to be used as part of data interpretation should be included.[7(p28)]

U44.2. Test users must consider the error associated with their measurements when they interpret their test results. Reliability and validity estimates should be considered when the test user makes interpretations of measurements.[7(p42)]

Stratford and colleagues have taken reliability assessment to a much more sophisticated but, paradoxically, a much more user-friendly level. Specifically, the authors have highlighted the need for assessing the reliability of change scores taken on a patient. The authors then used a statistical approach that allows clinicians to simply view a graph (see Figure in their article) to determine when true change in disability has occurred in their patients.

(*) In brief, the binomial theory of measurement error dictates that when item scoring is dichotomotis, as it is on the RMQ the error variance ([MATH]) for anv given score is equal to

[(n - [X.sub.p])([X.sub.])/n-1]

where n equals the number of items on the test and [X.sub.p] is the patient's RMQ score. The Keats correction factor takes into account that the variance formula tends to overestimate MATH when the forms are similar." To apply the Keats correction factor, MATH is multiplied by

[1 - [KR.sub.20] / 1 - [KR.sub.21]]

where [KR.sub.20] and [KR.sub.21] are the Kuder-Richerson reliability, coefficients.21 The CSEMs were determined for all possible RMQ scores.

References

[1] Roland M, Morris R. A study of the natural history of back pain, part I: development of a reliable and sensitive measure of disability in low-back pain. Spine. 1983;8:141-144. [2] Roland M, Morris R. A study of the natural history of back pain, part 11: development of guidelines for trials of treatment in primary care. Spine. 1983;8:14,5-150. [3] Fairbank JC, Couper J, Davies JB, O'Brien JP. The Oswestry low back pain disability questionnaire. Physiotherapy. 1980;66:271-273. [4] Waddell G, Main CJ. Assessment of severity in low-back disorders. Spine. 1984;9:204-208. [5] Ware JE, Sherbourne CD. The MOS 36-item short-form health survey (SF-36), 1: conceptual framework and item selection. Med Care. 1993; 31:247-263. [6] Kopec JA, Esdaile JM, Abrahamowicz M, et al. The Quebec back pain disability scale: measurement properties. Spine. 1995;20:341-352. [7] Jarvikoski A, Mellin G, Estlandre AM, et al. Outcome of two multimodal back treatment programs with and without intensive physical training. J Spinal Disord. 1993;6:93-98. [8] Hsieh CJ, Phillips RB, Adams AH, et al. Functional outcomes of low back pain: comparison of four treatment groups in a randomized controlled trial. J Manipulative Physiol Ther. 1992; 15:4-9. [9] Stratford PW, Finch E, Solomon P, et al. Using the Roland-Morris Questionnaire to make decisions about individual patients. Physiotherapy Canada. In press. [10] Deyo RA. Comparative validity of the sickness impact profile and shorter scales for functional assessment in low-back pain. Spine. 1986; 11:951-954. [11] Gronbald M, Jupli M, Wennerstrand P, et al. Intercorrelation and test-retest reliability of the pain disability index (PDI) and the Oswestry disability questionnaire (ODQ) and their correlation with pain intensity in low back pain patients. Clin J Pain. 1993;9:189-195. [12] Ruta DA, Garratt AM, Russell IT. Developing a valid and reliable measure of health outcome for patients with low back pain. Spine. 1994;19:1887-1896. [13] Ruta DA, Garratt AM, Leng M, et al. A new approach to the measurement of quality of life: the patient-generated index. Med Care. 1994;32:1109-1126. [14] Stratford PW, Binklev J, Solomon P, et al. Assessing valid change over time in patients with low back pain. Phys Ther. 1994;74:528-533. [15] Deyo RA, Centor RM. Assessing the responsiveness of functional scales to clinical change: an analogy to diagnostic test performance. J Chronic Dis. 1986; 11:897-906. [16] Bergner M, Bobbitt RA, Carter WB, et al. The Sickness Impact Profile: development and final revision of a health status measure. Med Care. 1981;19:787-805. [17] Ottenbacher KJ, Johnson MB, Hojem M. The significance of clinical change and clinical change of significance: issues and methods. Am J Occup Ther. 1988;42:156-163. [18] Weber H, Holme I, Amlie E. The natural course of acute sciatica with nerve root symptoms in a double-blind placebo-controlled trial evaluating the effect of piroxicam. Spine. 1993;18:1433-1438. [19] Herman E, Williams R, Stratford PW, et al. A randomized controlled trial of transculaneous electrical nerve stimulation (Codetron) to determine its benefits in a rehabilitation program for acute occupational low back pain. Spine. 1994;19:561-568. [20] Feldt LS, Brennan RL. Reliability. In: Linn RL, ed. Educational Measurement. New York, NY: Macmillan Publishing Co; 1989:108-127. [21] Lord FM. Standard errors of measurement at different score levels. Journal of Educational Measurement. 1984;21:239-243. [22] Keats JA. Estimation of error variances of test scores. Psychometrika. 1962;27:59-72. [23] Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford, England: Oxford University Press; 1989. [24] Bowman SJ, Wedderburn L, Whaley A, et al. Outcome assessment after epidural corticosteroid injection for low back pain and sciatica. Spine. 1993;10:1345-1350. [25] Hadler NM, Curtis P, Gillings DB, et al. A benefit of spinal manipulation as adjunctive therapy for acute low-back pain: a stratified controlled trial. Spine. 1987;12:703-706. [26] Jaeschke R, Singer J, Guyatt GH. Measurement of health status ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10:407-415.

PW Stratford, MSc, PT, is Assitant Professor, Faculty of Health Sciences, School of Rehabilitation Science, McMaster University, Bldg T16, 1280 Main St W, Hamilton, Ontario, Canada L8S 4K1 ([e-mail]). Address all correspondence to Mr Stratford.

J Binkley, MCISc, PT, COMP, is Director of Research, Rehab Management Systems, Dahlonega, GA 30597. She was an orthorpedic clinical specialist and Assistant Professor, Department of Physical Therapy, North Georgia College, Dahlonega, GA 30597, at the time of this study.

P Solomon, PhD, PT, is Assistant Professor Faculty of Health Sciences, School of Rehabilitation Science, McMaster University.

E Finch, MHSc, PT, is Assistant Professor, Faculty of Health Sciences, School of Rehabilitation Science, McMaster University.

C Gill, PT, is Senior Physiotherapist-Orthopaedics, St Joseph s Hospital, Ontario, Canada.

J Moreland, MSc, PT, is Research Therapist, St Joseph's Hospital and St Peter's Hospital, and Assistant Clinical Professor, Faculty of Health Sciences, School of Rehabilitation Science, McMaster University.

This study was approved by the Ethics Committee of St Joseph's Hospital.

This article was submitted February 9, 1995, and was accepted November 28, 1995.