Reliability, validity, and responsiveness of functional tests in patients with total joint replacement.

We developed a physical therapy computerized clinical practice database to study physical therapy outcomes. The information retrieved from any database, however, is only as good as the quality of the measurements that are entered into the database.[1,2] Consequently, issues such as reliability, validity, and responsiveness should be addressed for any measurement in a database.

We focused our data-quality studies on patients who were receiving acute care following total hip or knee replacements (THKRs) for several reasons. First, it has been estimated that more than 300,000 THKRs are performed annually in the United States (National Center for Health Statistics, Hyattsville, Md; personal communications; 1991). Second, patients with THKRs represent a large percentage of the patients treated in acute care settings by physical therapists. Third, through a multicentered modified Delphi study, we found that measures relating to function (supine to sit, sit to stand, standing, ambulation, and stair climbing) were the most important criteria used by physical therapists to decide when patients with THKRs could be discharged from the hospital.(3) Fourth, we were concurrently developing measures to be used in a randomized clinical trial to assess the dose effect of acute physical therapy interventions on patients who receive THKRs.

Assessments of functionally related activities such as the ability of a patient with a THKR to move from a supine to a sitting position or from a sitting to a standing position, stand, ambulate, and climb stairs are commonly performed by physical therapists.[3] Grading of the assistance required to complete these activities, however, was highly variable within our institution. Terms such as "independence," "standby assistance," "minimal assistance," "moderate assistance," and "maximal assistance" were routinely used, although these terms had no standard operational definitions.

Definitions for levels of assistance used in other scales such as the Functional Independence Measure (FIM),[4-9] Katz ADL Index,[9] Pulses Profile,[9] and Barthel Index,[9] were reviewed but were not ideally suited for patients in acute care settings. The well-known FIM defines, as a percentage, the level of assistance required to complete a battery of functionally related tests. Some of the tests comprising the FIM appeared most suited for patients undergoing long-term rehabilitation programs (eg, sphincter control, selfcare, communications, cognitive skills). We also could not gain staff consensus on what constituted 25%, 500%, 75%, or 100% effort as required in the FIM. Furthermore, a report of the intertester reliability of FIM scores has been presented only in abstract form[8] or briefly alluded to in a separate report.[10] The Acute Care Index[11] was found to be reliable and predictive of discharge status for a heterogenous sample of patients with neurological impairment in acute care settings, but the level of assistance scale in that study consisted of only three points, which appeared to reduce its overall sensitivity.

Consequently, we developed, via consensus among our clinical staff, the Iowa Level of Assistance Scale.[12] Preliminary studies[13, 14] showed that measurements obtained with this scale were reliable, but a systematic assessment of reliability, validity, and responsiveness (sensitivity) has not been undertaken in patients with THKRs.

The purposes of our study were (1) to determine the intertester and intratester reliability for therapists assessing five functional tests on patients with total joint replacements using the Iowa Level of Assistance Scale, (2) to determine the validity of the total functional score by relating this measure to the Harris Hip Rating Scale,(15) and (3) to determine the responsiveness of the total functional score in patients with THKRs during their acute hospital stay.

Method

Subjects

Eighty-six patients with either a total knee replacement or a total hip replacement participated in the study. Thirty-seven subjects had a total knee replacement, and 49 subjects had a total hip replacement. Sixty-five of the subjects had single joint replacements for the first time (primary joint replacement), and 21 subjects had a single joint revision of their joint replacement. The group ranged in age from 34 to 88 years (X = 63, SD = 14.7), in weight from 52 to 126 kg (X = 77.8, SD = 15.8), and in height from 147 to 185 cm (X = 168.5, SD = 9.34). Forty-seven subjects were female, and 39 subjects were male. The female subjects ranged in age from 48 to 76 years (X = 60, SD = 11.2), in height from 52 to 84 kg (X = 61.4, SD = 9.6), and in height from 147 to 169 cm (X = 162, SD = 8.2). The male subjects ranged in age from 34 to 88 years (X = 66, SD = 15.2), in weight from 58 to 126 kg (X = 84, SD = 16.4), and in height from 161 to 185 cm (X = 174, SD = 10.2). All subjects reviewed and signed informed formed before participating in the study.

Fifty-five subjects (27 with hip replacements and 28 with knee replacements) were evaluated (by each therapist pair) at two time periods during their hospitalization. The initial assessment was done approximately 2 days postoperatively (X = 2.5, SD = 0.94), and the second assessment was done approximately 6 days postoperatively (X = 5.9, SD = 1.4). This procedure served as a mechanism to assess the responsiveness of these measures during the acute phase of therapy following total joint replacement.[16]

Functional Tests

The four functionally related activities assessed were the subjects' ability to get out of bed, stand from the bed, ambulate 4.57 m (15 ft), and climb up and down three steps. Consensus among 19 centers with expertise in treating patients with total joint replacements indicated that these activities were the most important measures when establishing discharge criteria for these patients.[3] The level of assistance scale used to grade these activities has been described in detail.[12] In general terms, independence indicates that the therapist could leave the room and the patient could safely perform the activity being assessed. Standby assistance indicates that the therapist would not feel comfortable leaving the patient; the therapist, however, provides no physical assistance. Minimal assistance indicates that the therapist provides one point of contact with the patient, moderate assistance indicates that the therapist applies two points of contact, and maximal assistance indicates that the therapist or therapists are applying a total of three or more points of contact. An activity that is attempted but is not completed with maximal assistance indicates that the patient has faded maximal assistance. A patient who is not tested for reasons of safety, as determined by the therapist's judgment, is graded as not tested. Each activity is graded on an ordinal scale from 0 to 6 for the level of assistance required. A five-point ordinal scale provides a weighting factor for the assistive device used during the assessment of individual activities.

An additional test was completed to measure the speed that a patient with a total joint replacement could ambulate over a 13.4-m (44-ft) distance. This distance was selected for this test because of the extensive oxygen uptake data with established metabolic equivalent levels available for ambulating this distance.[17] In addition, a close association has been found between walking speed and functional abilities.[18] An ordinal ambulation velocity scale based on the time taken to walk 13.4 m is shown in Table 1. Ratio-scaled measurements of speed were converted to an ordinal scale to assist therapists in recording data into our computerized medical record. This conversion also allowed ambulation velocity to be included in the total functional score.

A total functional score was developed, which represented the sum of the ordinal level of assistance values for all five functionally related activities tested. This score was calculated for a subset of patients with THKRs (n = 55) and assessed for reliability.

Table 1. The University of Iowa Ambulation
Velocity Scale With Associated Ordinal Values

0 - ambulates 13.4 m (44ft.) in [less than or equal to] 20 s
1 - ambulates 13.4 m in 21-30 s
2 - ambulates 13.4 m in 31-40 s
3 - ambulates 13.4 m in 41-50 s
4 - ambulates 13.4 m in 51-60 s
5 - ambulates 13.4 m in 61-70 s
6 - ambulates 13.4 m in [greater than] 70 s

Testers

Four staff physical therapists with a range of 3 to 11 years of experience were testers in this study. All testers completed 1 hour of training a day for 5 weeks. Initially, training consisted of testers becoming acquainted with the definitions of levels of assistance. Later, each tester applied the scale to three subjects and openly discussed discrepancies found in utilization of the scale. All testers met minimal competencies in scale use as determined by a written test on the operational definitions at the end of 5 weeks. This test verified that all testers knew the definitions for level of assistance well enough to write them down. A tester training manual* was developed so that similar levels of training could be duplicated and thus equal competencies could be expected in new staff members.

Procedure

All subjects were assessed within 2 to 3 days postoperatively. Fifty-five of die subjects were reevaluated between day 5 and day 7 postoperatively. Each subject was assessed for the assistance required to get out of bed, stand from the bed, ambulate 4.57 m, and climb stairs. Finally, the subjects were asked to ambulate 13.4 m as quickly as they could without compromising safety. For ease of admnistering the battery of tests, each subject was assessed in the order described. Prior to the subject attempting each test, the therapist demonstrated the correct method of performing die activity. Following each activity, the subject was allotted 5 minutes for rest. The total time required for the entire test battery was typically between 20 and 30 minutes. Each subject was allotted at least 15 minutes for rest between repeated therapist evaluations.

Each tester was matched with the other tester in the therapist pair 24 times. Testers evaluating the subject second were selected at random. All tests were videotaped by an assistant. Videotaped assessments were viewed by therapists 3 to 6 months after the assessments to determine intratherapist reliability. In addition, therapists graded the videotape of each of the other therapists' assessments in an attempt to gain an estimate of the error attributed to patient change (learning or fatigue) between repeated trials.

A subset of subjects (n = 65) completed a self-administered modified Harris Hip Rating Scale[15] and were also tested with the five functional assessment tests performed by the therapists. The scale was modified to accommodate patients with total knee replacements[19] well as total hip replacements. The Harris scale ranges from 1 to 100, with 100 allocating the highest level of function. The Harris scale is calculated on the basis of pain, ambulation, stair-climbing ability, activities of daily living, muscle strength, and range of motion of the affected joint. The scale has been used for patients with hip replacements'5 and for patients with knee replacements.(19) The self-administered Harris scale scores were correlated with our total functional assessment index to gain insight into the validity of the physical therapists' functional assessments using the level of assistance scale.

Data Analysis

All ordinal data were analyzed using a weighted Kappa statistic and a weighted percentage of agreement, as recommended by Kramer and Feinstein[20] for ordinal-scale data. The Kappa statistic and percentage of agreement were weighted to provide credit for responses that were off by only one level. For example, if tester 1 graded an activity level with minimnal assistance and tester 2 graded the activity with standby assistance, the level of agreement would be considered better than if tester 2 graded the activity with maximal assistance. A standard error was calculated for each weighted Kappa.[20] An intraclass correlation coefficient (ICC[1,1]) was used to estimate the degree of reliability between repeated total functional scores.[21] The responsiveness of the measures was determined by calculating the mean square error associated with the 55 subjects who were assessed approximately 2 days postoperatively and then reassessed approximately 6 days postoperatively. We derived an estimate of what we considered important clinical change in the total functional score by calculating the change in total functional score between 2 and 6 days post-operatively. The responsiveness index is calculated by taking the ratio of the minimal estimated clinical change divided by the square root of twice the mean square error.[16] This responsiveness index has been described by Guyatt et al[16] and has a range of 0 to 2. The higher the responsiveness index, the fewer number of subjects needed to demonstrate die clinically important change. In essence, the index is higher if the variability of the response is small relative to the magnitude of the change m the test s between days tested.

Results

The weighted percentage of agreement for each therapist pair for each functional test is shown in Table 2. The weighted Kappa statistic and standard error calculated for all functional activities combined except ambulation velocity was .58 (SE = .089) for the actual measurements and .80 (SE = .086) for the video-based measurement. The ambulation velocity test could not be adequately recorded by video; consequently, that test was not included in this initial analysis. The actual weighted Kappa statistic was derived from the paired responses across afl four therapists. The weighted Kappa statistic for the observed video) measures was derived by correlating each therapist's actual functional assessment with the paired therapists' observation of that assessment from a video. All paired responses were pooled for the analysis. The video data correlations may serve to reduce a source of error attributed to patient change between two re peated trials. This additional assessment appeared important because patients with total joint replacements in acute care setting known to progress quickly, often after only one test. This progress would lead to differences between repeated tests that could not be attributed to therapist error.

Table 2. Weighted Percentage of Agreement for Each Therapist Pair for All
Five of the Functional Activities Tested

                       Therapist Pair

Activity               1-2   1-3   1-4   2-3   2-4   3-4

Supine to sit         .92    .95   .88   .96   .87   .83
Sit to stand          .92    .91   .93   .88   .87   .92
Ambulation            .93    .94   .89   .85   .92   .94
Ambulation velocity   .93    .94   .89   .90   .91   .94
Stair climbing        .86    .92   .96   .82   .87   .89

Table 3 shows the weighted Kappa statistic and standard error for all paired data across four therapists for each functional activity tested. Representative paired data for attribulation velocity are shown in Figure 1. Note that perfect agreement would be present if all count data fell on a diagonal in the contingency table. Table 4 shows the intratherapist weighted Kappa statistics and their standard errors for afl five functional tests combined.

Table 3. Weighted Kappa Statistic and Standard Error (in Parentheses) for All
Paired Observations Across Therapists for Each Activity Tested

Supine       Sit to                    Stair         Ambulation
to Sit       Stand        Ambulation   Climbing     Velocity

.66 (.072)   .53 (0.68)   .48 (.071)   .76 (.081)   .78 (.086)

 Table 4. Weighted Kappa Statistic
and Standard Error (in Parentheses)
Depicting the Intratherapist Reliability
for the Battery of Functional Tests

Therapist No.

1            2            3            4

.90 (.113)   .80 (.124)   .87 (.137)   .79 (.128)

A total functional score was calculated by taking the sum of the ordinal grades for each functionally related activity assessed (Fig. 2). The ICC (1,1) and the Pearson Product-moment Correlation Coefficient were .82 and .89, respectively. The standard error of the measurement for the total functional score was 2.15. The sum of the ordinal grades was determined so that our scale reliability could be compared with that reported by other researchers.[8] The use of arithmetic functions on original data, however, may require test item scaling and calibration using methods of test development theory.[10]

The responsiveness index of the total functional score was approximately .75, according to the criteria described by Guyatt et al.[16] There was a mean change of 7 points from approximately day 2 to approximately day 6 postoperatively, with a mean square error of 25. If the minimal clinical change to be detected in a randomized clinical trial was 7 points, approximately 20 subjects, making up two independent groups, would be necessary to detect this change.

The criterion-related validity was established by correlating scores obtained with a modified Harris Hip Rating Scale[22] With the scores of the functional measures. A Pearson correlation of -.86 was found, indicating a significant relationship between the Harris Hip Rating Scale scores and the total functional score (P [less than] .0001). The correlation was negative because a lower total functional score indicates greater function, whereas a lower Harris Hip Rating Scale score indicates less function. A separate correlational analysis between the Harris Hip Rating Scale scores and the total functional score for each of the 64 subjects assessed revealed Pearson correlations that ranged from -.78 to -.98. A plot graphically depicting the overall relationship between these two scales is presented in Figure 3.

Discussion

The analysis of the level of assistance definitions as applied to the five functional activities completed in this study represents one phase of a multiphasic approach to the development of standardized measures. Ultimately, these measures may represent meaningful data summarized from our acute care physical therapy database. The correlations in our study do not directly reflect the reliability of measurements that would be obtained during routine clinical practice. How rigorously a therapist applies clinical tests during routine clinical practice may be highly variable as compared with those tests performed during a reliability study. This notion was recently supported when we found differences in outcome data between the subjects in this study and patients with THKRs who qualified for the study but did not participate. On average, the patients who participated in this study were discharged 3 days earlier than those patients who did not participate in the study. This finding suggests that therapists provide less assistance sooner for patients who are being tested in a study (see Shields et al[12] for further discussion).

We studied patients with total joint replacements because they represent a major proportion of patients treated with physical therapy in acute care institutions and because there is some consensus among physical therapists as to the ideal treatment or assessment of these patients.[3]In addition, this patient group, within our institution, is targeted for specific randomized clinical trials evaluating the effect of physical therapy intervention frequency on function and disability. Therefore, the development of high-quality physical therapy outcome measures is of major importance.

We agreed to focus our initial functional measures on tests that therapists reported were routinely completed to make clinical decisions regarding patients with THKRs. Survey data indicated these functional activities (sit to stand, ambulation, supine to sit, stair climbing, ambulation velocity) were the criteria used for discharging patients with THKRs from physical therapy.[3] Given this degree of consensus, we opted to develop an ordinal level of assistance scale that could systematically be applied to patients. To this end, a training program (manual) was developed so that therapists with varied degrees of experience could gain scholar competencies prior to using this functional scale.

The range of weighted Kappa statistics for all activities combined and for each of the activities (Tab. 3) was considered to show moderate agreement, according to the interpretations of Landis and Koch.[23] The weighted Kappa statistic takes into account chance agreement for ordinal measures.[19] One factor that may have influenced our individual activity Kappa values was that the distribution was skewed toward the lower end (independence) of the level of assistance scale for activities such as ambulation and moving from sitting to a standing position. This skewed distribution would reduce the variability of the level of assistance scores, which would decrease the calculated reliability coefficients. In addition, we believe that one source of error contributing to lower weighted Kappa values between repeated tests may be that the patients' status actually changed following the initial evaluation. The repeated mean total functional index was consistently lower than the initial mean total functional index, suggesting that some change did occur. Furthermore, weighted Kappa values generated when therapists viewed a video of a therapist performing a test on a patient they had previously evaluated were considerably higher than when correlating the actual repeated test scores, suggesting that patient change between repeated trials may have been a source of the error contributing to lower weighted Kappa values. Clinical reliability studies of this type routinely make the assumption that the phenomenon under study is a constant, but if die phenomenon does change, an overestimation of the error associated with the test may be present. This important issue cannot be overlooked, especially when assessing patients who are acutely disabled. Despite this limitation, all weighted Kappa values showed moderate to good agreement.[23] These data suggest that die five functional measures could be applied reliably among four therapists.

We considered the intratherapist reliability to be excellent, with a range for the weighted Kappa values of .79 to .90. These high weighted Kappa values suggest that the therapists had a clear understanding of the definitions they invidually used. When viewing the video of their previous assessments, they appeared to be able to reproduce the level of assistance necessary for each task even 6 months following the initial assessment.

To make our functional scale comparable to others, we chose to sum the ordinal grades for each functional item to derive a total functional scale index. The ICC (1,1) was .82 and the Pearson correlation was .88 when correlating die repeated tests for the total functional score, indicating a high level of agreement. In addition, the standard error of the measurement for the total functional score was 2.15, indicating die extent of expected error in different raters' scores. As recently reported in abstract form,[8] the FIM was found to have ICCs of .94 and .93 for mobility and locomotion scores, respectively. It would appear, however, that die targeted population for Hamilton and colleagues' FIM assessment was general patients with mixed diagnoses undergoing rehabilitation[8] and not patients with a common diagnosis as in our study. The multiple diagnoses studied in a rehabilitation center may have increased the variability of the functional abilities of Hamilton and colleagues' sample, which could have improved the calculated reliability coefficients. In addition, patients undergoing long-term rehabilitation may be more stable between repeated trials than patients with total joint replacements in acute care settings. Based on our study, it appears that the level of assistance definitions recently developed for grading patients with THKRs have interrater reliability comparable to that of preliminary reports for the FIM. The FIM, however, has not been assessed for reliability in patients in acute care settings, and the FIM is composed of items that are not routinely assessed in such patient groups. Consequently, we believe the Iowa Level of Assistance Scale provides physical therapists with a straightforward method of obtaining standardized measurements of clinical function in patients in acute care hospital settings.

As previously mentioned, a subset of the patients completed a modified Harris Hip Rating Scale in an effort to establish criterion-related validity. This scale is recognized among orthopedic surgeons as the gold standard used to assess physical status in patients with total joint replacements.[15] The functional index score that we calculated was reasonably correlated (Pearson r = -.86) with the Harris Hip Rating Scale scores. The Harris Hip Rating Scale was recently found to be closely associated with the short-form questionnaire (SF-36), which is a widely accepted generic self-administered test of health-related quality of life.[22] The high correlation between our physical therapist-generated total functional score and the Harris Hip Rating Scale suggests that changes in functional status as routinely measured by physical therapists may provide valid and reliable information.

A measurement that is responsive reflects real change. That is, the measurement changes when the patient's status changes, and it remains stable when the patient's status does not change. Guyatt et al[16] devised an index of responsiveness defined as the ratio of the minimum clinically important difference to the square root of twice the mean square error. The higher the responsiveness index for a given scale, the smaller the size of the sample that is needed to detect the important change in a clinical trial. What constitutes an important clinical change, however, may be arguable. We considered a change of 7 points on the total funijnctional scale as clinically important because that was the mean change between day 2 and day 6 postoperatively in our subjects. If a smaller clinical change was considered important, more subjects would be required and the measure would be considered less responsive. A feasible sample size to detect clinically important changes is crucial if a clinical measurement tool is to be useful.[16]

Many studies of clinical tests fail to address the issue of responsiveness. Frequently, it is assumed that reliable measurement tools are also responsive. The amount of between-subject variance to the total variance (ICC) that can be tolerated in a measurement tool, however, depends on the sensitivity of the measurement tool. Consequently, in evaluations of clinical measurement tools, therapists should always strive to include assessments of not only reliability and validity, but also the responsiveness of the measurement under true clinical conditions.

Summary and Conclusions

The five activities graded with The University of Iowa Level of Assistance Scale appear to have moderate to good between-tester reliability and good to excellent within-tester reliability when applied to patients with THKRs. Some error associated with repeated measures in this population may be attributed to true changes in patient status (learning, fatigue), as demonstrated by paired comparisons between real and video data. During the acute phase of rehabilitation, the assessment of the five functional tests appears to be valid, as demonstrated by the close association between the total functional score and the Harris Hip Rating Scale scores.[15] Finally, the total functional score represented by the sum of the Iowa Level of Assistance Scale scores for five functionally related activities appears to be responsive to changes in patients' functional status between 2 and 6 days postoperatively during the acute phase of rehabilitation following THKR.

(*) University of Iowa Hospitals and Clinics Functional Testing Orientation Manual, University of Iowa Hospitals and Clinics, Iowa City, IA 52242.

References

[1] Byar DP. Problems with using observational databases to compare treatment. Stat Med. 1991;10:663-666. [2] Moses LE. Innovative methodologies for research using databases. Stat Med. 1991;10:629-633. [3] Enloe LM, Shields RK. Standardization of physical therapy for patients with total hip and knee arthroplasty. Phys Ther. 1992;72:58. Abstract. [4] Keith RA, Granger CV. The functional independence measure: a new tool for rehabilitation. In: Eisenberg MG, Gryesiak RC, eds. Advances in Clinical Rehabilitation. New York, NY: Springer Publishing Co Inc; 1987:6-18. [5] Granger CV, Hamilton BB. The uniform data system for medical rehabilitation: report of first admissions for 1991. Am J Phys Med Rehabil. 1993;72:33-38. [6] Granger CV, Cotter AC, Hamilton RB, Fiedler RC. Functional assessment scales: a study of persons after stroke. Arch Phys Med Rehabil. 1993;74:133-138. [7] Granger CV, Hamilton BB. Measurement of stroke rehabilitation outcome in the 1980s. Stroke. 1990;21:1146-1147. [8] Hamilton BB, Laughlinja, Granger CV. Interrater agreement of the seven-level functional independence measure (FIM). Arch Phys Med Rehabil. 1991;72:790. Abstract. [9] Gresham GE, Labi ML, Functional assessment instruments currently available for documenting outcomes in rehabilitation medicine. In: Granger CV, Gresham GE, eds. Functional Assessment in Rehabilitation Medicine. Baltimore, Md: Williams & Wilkins; 1984:65-85. [10] Heinemann AW, Linacre JM, Wright BD, et al. Relationship between impairment and physical disability as measured by the Functional Independence Measure. Arch Phys Med Rehabil. 1993;74:566-573. [11] Roach KE, Van Dillen LR. Development of an acute care index of functional status for patients with neurologic impairment. Phys Ther. 1988;68:1102-1108. [12] Shields RK, Leo KC, Miller B, et al. An acute care physical therapy clinical practice database for outcomes research. Phys Ther. 1994;74:463-470. [13] Shields RK, Dostal WF, Drake KK, Saehler PS. Interrater reliability of the standing test in neurologically impaired patients. In: Abstracts of Platform and Poster Presentations, 65th Annual Conference of the American Physical Therapy Association, June 24-28, 1990, Anaheim, Calif. Alexandria, Va: American Physical Therapy Association; 1990:99. Abstract. [14] Shields RK, Enloe L, Evans R, et al. Analysis of the reliability of clinical functional tests in total hip replacement patients. Phys Ther. 1992;72 (suppl): S113. Abstract. [15] Harris WH. Traumatic arthritis of the hip after dislocation and acetabular fractures: treatment by mold arthroplasty. J Bone Joint Surg [Am]. 1969;51:737-755. [16] Guyatt GH, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Disease. 1987;40:171-178. [17] Nielsen DH, Gerleman DG, Amundsen LR, et al. Clinical determination of energy cost and walking velocity via stopwatch or speedometer cane and conversion graphs. Phys Ther. 1982;62:591-596. [18] Friedman PJ Richmond DE, Bashott JJ. A prospective trial of serial gait speed as a measure of rehabilitation in the elderly. Age Ageing. 1988;17:227-235. [19] Maloney WJ, Schurman DJ, Hanzen D, et al. The influence of continuous passive motion on outcome in total knee arthroplasty. Clin Orthop. 1990;256:162-168. [20] Kramer MS, Feinstein AR. Clinical biostatistics: the biostatistics of concordance. Clin Pharmacol Ther. 1981;28:111-123. [21] Shrout PE, Fleiss JL. Intraclass correlation: uses in assessing rate reliability. Psychol Bull. 1970;86:420-428. [22] Lansky D, Butier JBV, Waller Fr. Using health status measures in the hospital setting: from acute care to outcome management. Med Care. 1992;30:MS57-MS73. [23] Landis JR, Koch GG. The measurement of observer agreement for categorized data. Biometrics. 1977;33:159-174.