Performance measures provide assessments of pain and function in people with advanced osteoarthritis of the hip or knee.

Patients with osteoarthritis (OA) and those proto arthroplasty often present with pain limitations of physical function. The rate, pattern, and direction of change may differ for pain and function depending on the period in the natural or clinical history over which a patient is assessed. (1,2) For example, at 2 months after total joint arthroplasty of the hip or knee, patients' pain ratings were shown to be comparable to or lower than their preoperative values; however, the time to complete performance tasks was substantially increased. (2,3) Because pain and physical function represent different but related health concepts and interventions targeting them frequently differ, separate assessments of these attributes were recommended at the Outcome Measures in Arthritis Clinical Trials conference (OMERACT) III. (4) An intriguing aspect of the OMERACT III outcome model was that although the assessment of physical function was essential, the application of performance tests was optional. (4) Self-report measures of physical function also were favored over performance measures in an authoritative review of outcome measures for people with OA. (5) This recommendation apparently was based on the lower cost and ease of administration associated with self-report measures; however, it assumes that self-report measures and performance measures of physical function assess the same attribute and nothing else.

Although many self-report measures profess to assess physical function, few provide an operational definition of its intended meaning. A noted exception is the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) physical function subscale, which provides the following statement: "By this we mean your ability to move around and to look after yourself." (6) We suspect that this statement captures the intended meaning of lower-extremity physical functional status left undeclared by many researchers, and it is representative of our view of lower-extremity functional status.

Contrary to the belief that self-report measures and performance measures of physical function provide comparable information is a body of work refuting this idea. (2,3,7) Parent and Moffet, (2) in a study of patients after total knee arthroplasty, noted improvement in self-reported physical function as measured by the WOMAC and the Medical Outcomes Study 36-Item Health Survey Questionnaire (SF-36) physical function subscales but a significant reduction in the 6-minute walking distance when assessed at 2 months after arthroplasty. Maly et al, (7) in an investigation of patients with OA of the knee, reported higher correlations between pain and WOMAC and SF-36 physical function scores than between pain and 3 performance measures (Six-Minute Walk Test, Timed "Up & Go" Test, and a stair test). Using a stepwise linear regression analysis that included pain and thigh muscle strength as independent variables and WOMAC and SF-36 physical function subscales as dependent variables, these investigators also found that pain was more predictive of self-reported function than muscle strength. (7) Stratford and Kennedy, (3) in a study of patients after hip or knee arthroplasty, reported higher standardized regression coefficients between pain and WOMAC physical function scores and change scores than between pain and the time or distance associated with several performance tasks. Also reported in this article was the finding that self-reported Lower-Extremity Functional Scale (LEFS) scores were most strongly associated with pain preoperatively, exertion when assessed within 2 weeks of arthroplasty, and the time or distance associated with performance measures when evaluated approximately 2 months after arthroplasty. (3)

A further insight is provided in a study that examined the relationship between performance-rated components of pain, exertion, and function (time or distance) and LEFS scores. (8) Patients with end-stage OA of the hip or knee and awaiting arthroplasty performed 3 performance tasks--40-m self-paced walk, stair test, and Timed "Up & Go" Test--and completed the LEFS. Immediately following each performance task, patients reported the amount of pain and exertion that they experienced. (8) An exploratory factor analysis identified 3 factors, with pain responses loading on 1 factor, exertion loading on the second factor, and time loading on the third factor. The LEFS loaded on all 3 factors (pain=.44; exertion=.41; and time=.35). (8) Recently, Terwee et al (9) examined the relationship between the WOMAC and SF-36 pain and function subscales with the performance-based DynaPort Knee Test * for patients with OA of the knee before and after arthroplasty. Applying an exploratory factor analysis, these investigators found that the self-report measures of pain and function loaded on 1 factor and that the performance measure loaded on a second factor. (9) The 8F-36 function score loaded on both factors, with the higher loading on the factor composed of the self-report measures (.78 and .69). (9) Collectively, these findings support the premise that self-report measures of physical function assess more than a patient's ability to move around. (2,3,7-9) It appears that, in addition to providing patients' perceptions of their ability to move around, self-report measures of physical function also are influenced by what patients experience when moving around (eg, pain and exertion).

A further understanding of the relationship between self-report assessments of pain and physical function is offered by a number of studies that examined the factorial validity of the WOMAC. Factorial validity exists to the extent that items cluster in accordance with the specified domains to which they have been assigned by the measure's developer. The WOMAC was conceived to assess 3 domains: pain, stiffness, and physical function. (10) Accordingly, factorial validity would exist if the 5 pain items loaded on 1 factor, the 2 stiffness items loaded on a second factor, and the 17 physical function items loaded on a third factor. However, there is consistent evidence demonstrating that the WOMAC pain and physical function items group more by activity than by the hypothesized domains of pain and physical function. (11-14)

There is no doubt that pain and physical function are related health concepts. Yet to the extent that during assessments, clinicians routinely inquire about pain and physical function separately, outcome measures have separate scales to assess pain and function, and due to the fact that authoritative groups such as OMERACT III have identified pain and physical function as 2 core outcome measures rather than 1, investigators are challenged to develop assessment methods that maximize valid information concerning the attributes of interest. It was with these challenges in mind that we undertook the present study.

Our intent was to determine whether performance test assessments of pain and physical function provided responses consistent with these 2 domains. Specifically, our goal was to evaluate the factorial validity of performance assessments of pain and physical function. Our specific hypotheses were as follows: (1) responses to the performance assessments could be explained by 2 factors, 1 consisting of pain items and the other consisting of time (distance) items; (2) each pain or performance item would be related only to the health concept that it was perceived to be assessing (each item would have a nonzero loading on the factor that it was conceived to measure and a zero loading on the other factor); (3) the factors pain and physical function would be correlated; and (4) the measurement error terms associated with the items would be uncorrelated.

Method

Subjects

Patients were eligible for this study if they were able to speak and comprehend written English; were diagnosed with end-stage OA of the hip or knee, as labeled by the surgeon and confirmed by radiographs; were scheduled to undergo primary total hip arthroplasty or total knee arthroplasty; were able to complete the performance tests; and provided written informed consent. Patients undergoing revision or bilateral arthroplasty or additional operative procedures or those demonstrating comorbidities associated with cognitive impairment were excluded. Of the 188 patients reviewed, 177 (94%) met the eligibility criteria. The study sample consisted of 81 participants with hip OA and 96 participants with knee OA. Eighty-five of the participants were women, 36 of whom received total hip arthroplasty. The mean age and body mass index of the sample were 65 years (first and third quartiles: 58.0 and 72.0) and 29.1 kg/[m.sup.2] (first and third quartiles: 26.4 and 33.0), respectively. A breakdown of the participants' characteristics by site of OA is shown in Table 1. The study took place at a tertiary-care orthopedic hospital in Toronto, Ontario, Canada, and data were collected from November 2001 to February 2003.

Design

We applied a cross-sectional study design. Participants completed the performance tests at a median interval of 20 days (first and third quartiles: 12 and 22) before surgery.

Measures

Participants completed 4 performance measures in the following order: self-paced walk, Timed "Up & Go" Test, stair test, and Six-Minute Walk Test. Several minutes were provided between the self-paced walk, Timed "Up & Go" Test, and stair test. A 10-minute rest interval was provided between the stair test and the Six-Minute Walk Test. With the exception of the Six-Minute Walk Test, the outcome was the time to complete the task. Time was measured on a stopwatch to the nearest one-hundredth of a second, and distance was measured to the nearest meter.

Self-paced walk. Participants walked 2 lengths of a 20-m indoor course in response to the instructions, "Walk as quickly as you can without overexerting yourself." (15) The turnaround time was excluded. An intraclass correlation coefficient (ICC) for test-retest reliability of .91 and a standard error of measurement of 1.73 seconds have been reported for this measure for patients similar to the participants in the present study. (15)

Timed "Up & Go" Test. Participants were instructed to rise from a standard arm chair, walk at a safe and comfortable pace to a line 3 m away, cross the line, turn, and return to a sitting position in the chair. (16) An ICC for test-retest reliability of .75 and a standard error of measurement of 1.07 seconds have been reported for this measure for patients with OA and those undergoing arthroplasty of the hip or knee. (15)

Stair test. Participants ascended and descended 9 stairs (step height, 20 cm; step depth, 27 cm) in their usual manner at a safe and comfortable pace. (15) A handrail was available. An ICC for test-retest reliability of .90 and a standard error of measurement of 2.35 seconds have been reported for this measure for patients similar to the participants in the present study. (15)

Six-Minute Walk Test. Participants were instructed to cover as much distance as possible during the 6-minute time frame. Standardized encouragement--"You are doing well, keep up the good work"--was provided at 60-second intervals. The test was conducted on a premeasured, 46-m, unobstructed, uncarpeted, rectangular circuit. The outcome was the distance walked in 6 minutes. (15,17) An ICC for test-retest reliability of .94 and a standard error of measurement of 26.29 m have been reported for this measure for patients similar to the participants in the present study. (15)

Activity-specific pain rating. Participants marked the pain that they experienced on an 11-point (0-10) numeric rating scale immediately following each performance test. (15) We are not aware of test-retest reliability values for patients similar to the participants in the present study; however, a reliability estimate (ICC) of .86 and a standard error of measurement of 1.04 have been reported for people with a spectrum of lower-extremity problems. (18)

Data Analysis

We applied confirmatory factor analysis with a maximum-likelihood estimation method (AMOS 4.0 ([dagger])) to assess the factorial validity of the performance tests. (19-22) Unlike exploratory factor analysis, which provides all possible factor loadings, confirmatory factor analysis provides factor loadings for the specified model only. We conceptualized a measurement model with 2 factors, which we labeled pain and physical function (Fig. 1). (22) We applied the following indexes to assess model fit: comparative fit index (CFI), relative fit (RF), Tucker-Lewis Index (TLI), root-mean-square error of approximation (RMSEA), and the model fit chi-square test and associated P value. (22) Although no single standard exists for defining acceptable model fit, the following values are generally accepted: CFI, RF, and TLI values exceeding .95 indicate good fit; RMSEA values of less than .05 indicate good fit; and RMSEA values of less than .08 indicate reasonable fit. (22,23) A significant chi-square value (eg, P<.05) indicates that the data do not fit the model. Prior to conducting the analyses, we assessed the data and found several of the underlying distributions to be nonnormal. Accordingly, we applied the bootstrap feature of AMOS 4.0 for 1,000 samples with replacement to estimate the parameter values and model fit indexes. (22)

[FIGURE 1 OMITTED]

To enhance the validity and generalizability of our final model, we performed 2 cross-validation procedures. First, we stratified by site (hip or knee) and used a random-number generator to create 2 samples with approximately equal representation of hips and knees.

One group was used to generate the initial model and modifications (n=88:48 knees, 40 hips), and the second group was used to cross-validate the model (n=89: 48 knees, 41 hips). The second cross-validation procedure repeated the steps described above; however, this time 1 group was composed of participants with knee OA and the other group was composed of participants with hip OA.

Results

Results for the first cross-validation analysis were similar for the combined samples, which included participants with knee OA and hip OA in each group (initial sample: [[chi square].sub.8]=7.7, P=.473; cross-validation sample: [[chi square].sub.8]=9.6, P=.269; simultaneous test for a difference between model structures: [[chi square].sub.16]=17.6, P=.351). Similar results also were obtained for the second cross-validation analysis (knee sample: [[chi square].sub.8] 12.1, P=.148; hip sample: [[chi square].sub.8] = 13.3, P=. 108; simultaneous test for a difference between model structures: [[chi square].sub.16]=25.3, P=.064). Given that the cross-validation analyses supported the model for various independent subgroups of participants, we present the results for the entire sample of 177 participants.

Descriptive statistics for the performance measures are shown in Table 2, Figure 1 shows the standardized factor loadings for the initial measurement model (model 1), and Table 3 shows the fit statistics. The observed or measured variables in Figure 1 are shown in rectangles, and the latent variables are shown in circles. The larger circles labeled "pain" and "function" designate the factors, and the smaller circles with numbered "e" values signify the measurement error terms associated with each observed variable. The numbers between the factors and observed variables connected by single-headed arrows represent the standardized factor loadings. The negative value associated with the function component of the 6-minute walk test occurs because higher functional levels are associated with greater distances, whereas shorter times reflect higher functional levels for the other 3 performance tests. The curved double-headed arrow showing a value of .48 represents the correlation between the factors pain and function. Although the CFI, RF, and TLI exceeded .90 (Tab. 3), the root-mean-square coefficient indicated a less-than-desirable fit. The modification index for this model (not shown) suggested that the model could be improved by adding a correlation between the stair pain and time error terms, and we elected to address this association with 2 revised models. To ascertain the magnitude of the correlated error terms, the first revised model (model 2a) specified a correlation between the stair pain and time error terms (Fig. 2a: curved double headed arrow showing a correlation of .41). The second revised model (model 2b) removed the stair pain and time terms (Fig. 2b). The fit statistics for both models are shown in Table 3. Both modified models improved the fit over that of the initial model. However, of the 2 modified models, only the 1 that removed the stair terms achieved a good fit for all indexes and was consistent with all of our initial hypotheses.

[FIGURE 2 OMITTED]

Discussion

Although OMERACT III identified pain and physical function as 2 core outcome measures requiring separate assessments, (4) we suspect that most investigators would agree that these health concepts are related in patients with OA and those progressing to arthroplasty. The question unanswered to this point is whether a more distinct assessment of these health concepts can be obtained than has been reported for self-report measures such as the WOMAC. Repeatedly, studies have failed to support the factorial validity of the WOMAC pain and physical function subscale (the principal self-report measure for patients with OA of the hip or knee), (11-14) and these data have led some investigators to abandon the notion of separate assessments of these concepts in favor of grouping WOMAC items by activity regardless of their health domain. For example, Angst et a124 suggested combining pain and physical function items to form 4 subscales, which they labeled lying/sitting, standing/walking, bending, and ascending/descending. This approach ignores the recommendation of OMERACT III that pain and physical function should represent 2 core outcome measures. Moreover, from a clinical perspective, combining pain and physical function items into a single domain does not assist clinicians in identifying treatment goals. Is the patient's principal problem pain, physical function, or a combination of these 2 concepts? In addition, after a course of treatment targeting pain is provided, is the patient's poor score on the standing/walking domain a result of ineffective pain management or a consequence of the patient's reduced ability to move around for reasons other than pain (eg, poor balance, muscle weakness, or restricted range of motion)?

Rather than adhering to the notion that self-report measures represent the preferred method of assessing physical function, we examined whether performance-specific evaluations of pain and physical function provide a viable method for obtaining a more distinct assessment of these 2 related health concepts than has been reported for self-report measures, such as the WOMAC. Our initial model yielded a correlation of .48 between pain and function, providing support for hypotheses 1 and 3, which conceptualized 2 correlated health concepts. Moreover, our second hypothesis was sustained in that significant correlations were obtained for the specified health concepts, and no evidence of loading on the nonspecified health concept was evident. However, our fourth hypothesis was not supported in that the error terms for stair pain and function were correlated. This finding led to the exploration of 2 revised models: 1 allowed a correlation between stair pain and time error terms, and the other removed the stair terms from the model. The intent of the model that allowed a correlation between the stair pain and time terms was to examine the extent to which these components were correlated beyond the correlation between the factors pain and function. The second revised model excluded the stair test from the analysis, and this model provided results consistent with our 4 hypotheses. The correlation between the factors pain and function was .43 for the final model. This correlation is lower than that typically reported between the pain and function subscales of the WOMAC (.74-.84) (7,9,25) and SF-36 (.57) (9) for patients reasonably similar to the participants in the present study.

Inclusion of the stair test makes the distinction between the health concepts of pain and function less discernable. This finding is reflected in the lower correlation between pain and function noted in model 2b (r=.43) than in models 1 and 2a (r =.48). Accordingly, when the clinical goal is to obtain as distinct an assessment as possible between the health concepts of pain and function, our results suggest that the stair test not be included in a composite score. However, we are not suggesting that the stair test be excluded from a patient's assessment. It is clear that 1 of the physical therapist's responsibilities for patients similar to the participants in the present study is to ascertain their ability to safely ascend and descend stairs and to intervene when appropriate. We simply stress that if the results from the stair test are combined with the results from the other performance measures, then the impressions of pain and function will be less distinct.

Assessments of pain and function are important both to identify patients' problems at a point in time and to assess change over time. Information from these assessments is applied by clinicians to guide decisions concerning individual patients, by researchers to ascertain the relative effectiveness of competing interventions in clinical trials, and by health care policy makers to set benchmarks regarding the maximum number of patient visits and corresponding payment plans. Previous work demonstrated that self-reports of physical function after arthroplasty are strongly influenced by pain and change in pain. (3) The consequences are that patients report their physical function to be higher than is demonstrated by performance tests and that health care professionals who rely on self-reports alone overestimate patients' functional status levels. (2,3) The results of the confirmatory factor analysis of the present study indicate that performance-rated pain and function represent 2 factors that have not emerged in previous factor analyses of self-report measures. Accordingly, complementing existing self-report assessments of physical function with performance-rated pain and function tests may provide clinicians with a more valid assessment of these health concepts.

There are several potential limitations of the present study. First, the study sample was patients awaiting hip or knee arthroplasty. Presumably, these patients have more severe OA than the typical patient seen in general physical therapist practice. However, in considering this point, it should be remembered that the study participants were able to complete all of the performance tests. A second limitation relates to the sample size for the cross-validation portion of the present study. Although there is no standard method for estimating sample size, it is generally agreed that the sample size should be at least 10 subjects per observed variable or a minimum of 100 subjects. (21,26) Although our overall sample size of 177 participants exceeded the recommended minimum sample size, the number of participants in each of the cross-validation samples was slightly smaller than the recommended sample size.

Conclusion

Our goal was to determine whether performance-specific assessments of pain and physical function could provide a more distinct evaluation of these attributes than has been found for self-report measures. We conceived a 2-factor model consisting of pain and physical function and tested the model with 4 activities (self-paced walk test, Timed "Up & Go" Test, stair test, and Six-Minute Walk Test) by use of a confirmatory factor analysis. Although the initial model appeared promising, the stair test pain and function error terms were correlated. Dropping the stair test from the analysis provided results that supported the application of performance-specific assessments of pain and function as a method of i obtaining reasonably distinct assessments of these attributes. We believe that performance-specific assessments of pain and function offer a more distinct method of assessing these attributes than can be obtained by self-reports alone and that performance measures should be viewed as core measures for people with OA of the hip or knee and those progressing to arthroplasty.

This article was received January 4, 2006, and was accepted July 5, 2006.

References

(1) Kennedy DM. Modelling Early Recovery Following Total Hip and Knee Replacement: The Case for Self-report and Physical Performance Measures [thesis]. Hamilton, Ontario, Canada: McMaster University; 2003.

(2) Parent E, Moffet H. Comparative responsiveness of locomotor tests and questionnaires used to follow early recovery after total knee arthroplasty. Arch Phys Med Rehabil. 2002;83:70-80.

(3) Stratford PW, Kennedy DM. Performance measures were necessary to obtain a complete picture of osteoarthritic patients. J Clin Epidemiol. 2006;59:160-167.

(4) Bellamy N, Kirwan J, Boers M, et al. Recommendations for a core set of outcome measures for future phase III clinical trials in knee, hip, and hand osteoarthritis: consensus development at OMERACT III. J Rheumatol. 1997;24:799-802.

(5) Bellamy N. Osteoarthritis clinical trials: candidate variables and clinimetric properties. J Rheumatol. 1997;24:768-778.

(6) Bellamy N. WOMAC Osteoarthritis Index User Guide IV. Herston, Queensland, Australia: University of Queensland; 2000.

(7) Maly MR, Costigan PA, Olney SJ. Determinants of self-report outcome measures in people with knee osteoarthritis. Arch Phys Med Rehabil. 2006;87:96-104.

(8) Stratford PW, Kennedy D, Pagura SMC, et al. The relationship between self-report and performance-related measures: questioning the content validity of timed tests. Arthritis Rheum. 2003;49:535-540.

(9) Terwee CB, van der Slikke RMA, van Lummel RC, et al. Self-reported physical functioning was more influenced by pain than performance-based physical functioning in knee-osteoarthritis patients. J Clin Epidemiol. 2006;59:724-731.

(10) Bellamy N, Buchanan WW, Goldsmith CH, et al. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. J Rheumatol. 1988;15:1833-1840.

(11) Guermazi M, Poiraudean S, Yahia M, et al. Translation, adaptation and validation of the Western Ontario and McMaster Universities osteoarthritis index (WOMAC) for an Arab population: the Sfax modified WOMAC. Osteoarthritis Cartilage. 2004;12:459-468.

(12) Thumboo J, Chew LH, Soh CH. Validation of the Western Ontario and McMaster University osteoarthritis index in Asians with osteoarthritis in Singapore. Osteoarthritis Cartilage. 2001;9:440-446.

(13) Faucher M, Poiraudeau S, Lefevre-Colau MM, et al. Algo-functional assessment of knee osteoarthritis: comparison of the test-retest reliability and construct validity of the WOMAC and Lequesne indexes. Osteoarthritis Cartilage. 2002;10:602-610.

(14) Kennedy D, Stratford PW, Pagura SMC, et al. Exploring the factorial validity and clinical interpretability of the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC). Physiother Can. 2003;55:160-168.

(15) Kennedy DM, Stratford PW, Wessel J, et al. Assessing stability and change of four performance measures: a longitudinal study evaluating outcome following total hip and knee arthroplasty. BMC Musculoskelet Disord. 2005;6:3.

(16) Podsiadlo D, Richardson S. The timed "Up & Go": a test of basic functional mobility for frail elderly persons. J Am Geriatr Soc. 1991;39: 142-148.

(17) Guyatt GH, Sullivan MJ, Thompson PJ, et al. The 6-minute walk: a new measure of exercise capacity in patients with chronic heart failure. Can Med Assoc J. 1985;132:919-923.

(18) Stratford PW, Spadoni G. The reliability, consistency, and clinical application of a numeric pain rating scale. Physiother Can. 2001;53: 88-91, 114.

(19) Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 3rd ed. New York, NY: Oxford University Press; 2003.

(20) Swisher LL, Beckstead JW, Bebeau MJ. Factor analysis as a tool for survey analysis using a professional role orientation inventory as an example. Phys Ther. 2004;84:784-799.

(21) de Vet HC, Ader HJ, Terwee CB, et al. Are factor analytical techniques used appropriately in the validation of health status questionnaires? A systematic review on the quality of factor analysis of the SF-36. Qual Life Res. 2005;14:1203-1218.

(22) Byrne BM. Structural Equation Modeling With AMOS: Basic Concepts, Applications, and Programming. Mahwah, NJ: Lawrence Erlbaum Associates; 2001.

(23) Schumacker RE, Lomax RG. A Beginner's Guide to Structural Equation Modeling. Mahwah, NJ: Lawrence Erlbaum Associates; 1996.

(24) Angst F, Ewert T, Lehmann S, et al. The factor subdimensions of the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) help to specify hip and knee osteoarthritis: a prospective evaluation and validation study. J Rheumatol. 2005;32:1324-1330.

(25) Stratford PW, Kennedy DM, Hanna SE. Condition-specific Western Ontario McMaster Osteoarthritis Index was not superior to region-specific Lower Extremity Functional Scale at detecting change. J Clin Epidemiol. 2004;57:1025-1032.

(26) Norman GR, Streiner DL. Biostatistics: The Bare Essentials. 2nd ed. Hamilton, Ontario, Canada: BC Decker; 2000.

* McRoberts BV, The Hague, the Netherlands.

([dagger]) SmallWaters Corp, 1507 E 53rd St, Suite 452, Chicago, IL 60615.

PW Stratford, PT, MSc, is Professor, School of Rehabilitation Science, and Associate Member, Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, and a Scientific Affiliate in the Department of Surgery, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada. Address all correspondence to Mr Stratford at: [e-mail].

DM Kennedy, BScPT, MSc, is the Manager of Program Development for Hip and Knee Replacement, Holland Orthopaedic & Arthritic Centre of Sunnybrook Health Sciences Centre, and Part-time Assistant Clinical Professor, School of Rehabilitation Science, McMaster University.

LJ Woodhouse, PT, PhD, is Assistant Professor, School of Rehabilitation Science, McMaster University, and a Scientific Affiliate in the Department of Surgery, Sunnybrook Health Sciences Centre.

All authors provided concept/idea/research design and writing. Ms Kennedy provided data collection, subjects, and institutional liaisons. Mr Stratford and Dr Woodhouse provided data analysis and fund procurement.

The Research and Ethics Committee of the Holland Orthopaedic & Arthritic Centre of Sunnybrook Health Sciences Centre approved this study.

Table 1.
Descriptive Statistics of Study Sample

                                Patients           Patients
                                With Hip           With Knee
                                Osteoarthritis     Osteoarthritis
Characteristic                  (n = 81)           (n = 96)

Sex (no. of women/men)          36/45              49/47
Age, y, median                  60.5 (53.0, 69.0)  69.0 (63.0, 74.0)
  (first, third quartiles)
Body mass index, kg/[m.sup.2],  27.3 (25.0, 31.6)  30.2 (27.6, 34.4)
  median (first, third
  quartiles)

Table 2.
Descriptive Statistics for Performance Measures

                                      Function, Time/
Test                       Pain (a)   Distance (a)

Self-paced walk (s)        3 (2, 5)   30.4 (25.6, 36.3)
Stair test (s)             4 (2, 6)   15.2 (11.1, 24.6)
Timed "Up & Go" Test (s)   3 (1, 5)   9.4 (7.7, 11.7)
Six-Minute Walk Test (m)   5 (3, 7)   407 (325, 499)

(a) Median (first, third quartiles).

Table 3.
Fit Statistics for Initial and Modified Measurement Models

Statistic                    Model 1       Model 2a (a)  Model 2b (b)

Comparative fit index         .957           .975           .991
Relative fit index            .914           .939           .964
Tucker-Lewis Index            .936           .961           .983
Root-mean-square error of     .124           .097           .071
  approximation
Model fit chi-square (P)   70.71 (<.001)  48.11 (<.001)  15.14 (.056)

(a) Stair test error terms were correlated.

(b) Stair test--both pain and time--was removed from the model.

Invited Commentary

The article by Stratford et al provides an excellent model for analyzing the performance of outcome measures. In the true context of construct validity, they evaluated whether the scales are measuring what they are intended to measure and they brought modern analytic techniques into the analysis to do so. They analyzed 4 performance-based measures of lower-extremity pain and function with the goal of seeing whether performance-based measures are better able to separate out the concepts of pain and function than previous experience with patient self-report measures. The work by Stratford et al is an excellent example of the need to push our measurement work and test all of the assumptions under which we believe we are measuring a certain construct.

There are 2 main points regarding the art of statistical modeling and conceptual frameworks that I would like to raise in this commentary, which the readers might consider as they review this work.

The Art of Statistical Modeling

Structural equation modeling is a powerful and attractive tool for this type of work. Indeed, it offers a route into exactly what the authors wanted to explore. However, there is always a bit of art to the modeling process, and guidelines as to how to make it work best for your situation. (1,2) Structural equation modeling is a large-sample method. Stratford et al used a time-honored method of splitting their data in order to have a test and validation sample to confirm their methods; however, in so doing, they were working at or below the very lowest limits of the sample size requirements. Kline (1) suggested a minimum of 100 to 150 people, and approximately 10 to 20 people per parameter estimate. With 8 attributes loading onto 2 factors, the authors were estimating 17 parameters. The small sample size in general and the number of parameters being estimated could lead to a misestimation of the model and an inability to converge. The latter was not the case in this study as the authors reached convergence. Two other things also support the conclusion that the model did not suffer from the small sample. First, the authors had similar findings on the second half of the data. Second, the variables in the model were highly correlated with the latent trait, which means they were less vulnerable to misestimation. The authors made the choice to split the data, taking the risks associated with small samples. Another choice would have been to only run the model on the full sample and take advantage of being closer to the recommended sample sizes. This is an important consideration for the application of analytic techniques such as structural equation modeling--they take large numbers of observations.

Analysts using the standard error of the mean also make a series of decisions about how to model. Stratford et al began with a 2-factor model to see whether there were 2 concepts being measured: pain and function. Their description seems to suggest that they loaded the items on the respective factors and tested the model. I wonder whether we would have had even more information had they allowed the items to load on both factors or if they had tried just 1 factor first--both of which would have allowed them to demonstrate that their measures did not fit a model with either of the cross loadings, or only one latent trait. This would add strength to their findings that they observed a 2-factor model, and it would have offered more information and better data fit than if they had tried to load all indicators onto 1 factor (latent trait). Indeed, had they done this, they could have tested for the improved explanatory power and fit of their model compared with one with just 1 factor with a chi-square difference test. (1) The authors were clear in their decisions as to how they modeled, and these comments highlight the interpretive part of any form of path or structural equation modeling.

Conceptual Frameworks

Measurement requires 2 conceptual frameworks to be considered: (1) that of the target construct and (2) that of the instrument selected to define that construct. We have come a long way in rehabilitation toward an understanding of the former conceptual framework. Stratford et al stated that they were measuring physical function, and they used the definition based on the work of Bellamy (3,4) to define their target: "the ability to move around and to look after yourself." Pain was not specifically defined by the authors. They identified these as 2 distinct concepts using the Outcome Measures in Arthritis Clinical Trials conference (OMERACT) III recommendations. What they have not provided is an overall framework that would help "guide our communication, clinical research, and patient care" as Jette noted in this journal. (5) Such a framework helps to elaborate on the anticipated relationships between factors such as pain and function, and in this case may have helped in offering an explanation for the differences between self-report and performance-based measurement.

Stratford et al discussed the difference between their findings--where performance-based indicators of function and self-reported pain fell into separate factors. They contrasted this with past experiences of being unable to separate self-reported function and self-reported pain. A broader conceptual framework might have aided in this. If placed within Verbrugge and Jette's framework, (6) Stratford et al would have been measuring a functional limitation when using a time test of function. Several of the self-report measures might be measuring more at the level of the whole task in an unrestricted context--the disability level--and more likely to be influenced by personal and environmental factors than the structured timed tests. This supports Stratford and colleagues' findings, but from a position of difference in concept rather than difference in quality of either type of scale (timed versus self-report measures of physical function). The same would be true if put within the International Classification of Functioning, Disability and Health (ICF) model, (7) where the timed test would be considered an activity limitation and appraisal of functioning as a whole might shift that to participation. In both situations, the timed test and self-report both tap physical functioning, but at different levels of the conceptual frameworks we use. Interestingly, the final revision of the model provided by Stratford et al removed one task due to an undesirable correlation between the timed outcome for this test and its pain rating. The task was performing the stair test, which was distinct from the other tasks in that it was moving toward a higher level of contextualized complexity, much like serf-report of physical function at the level of disability. (5) Just as with the self-report measures, perhaps the distinction between pain and disability in this severely disabled population is lost when you move toward more applied complex appraisals of activities in daily life.

The second framework that must be considered is that of the measure that has been chosen. Hopefully, the developers have provided their definition and have described how they operationalized it in the development of their outcome measure; however, many do not. (8) The user then must appraise this and make sure it matches with the intended target. Stratford et al have clearly articulated an a priori definition of their target--physical function--that includes both mobility and taking care of oneself. In the end, they have 3 of the 4 measures in the model. All 3 measures focus on the timed performance of mobility tasks: Timed "Up & Go" Test, serf-paced walk, and Six-Minute Walk Test. The second domain is the pain experience immediately after each of these tasks. As mentioned above, the most functional one--the stair test--was dropped. Were the 3 remaining tasks measuring something congruent with Stratford and colleagues' definition of physical function, or were they measuring timed performance of mobility tasks alone? Similarly, Stratford and colleagues' pain scale focused on the pain associated with the timed tasks, not pain as we typically might measure it. Is this a broad enough measure of pain related to osteoarthritis, or do we need to consider a broader scale (pain visual analog scale or numeric rating scale without attribution to a task as suggested by OMERACT, or other arthritis-related pain scales)?

Stratford et al have pushed us to demand more from our measures, and specifically to make sure that our measures are measuring what they are supposed to be measuring in the way that we expect. The points raised above offer some possible explanations, all of which are testable in a study that fields both self-report and performance-based measures with attributed and non-attributed pain items and in a large enough sample to allow for even more confidence in large-sample modeling techniques, perhaps in people along the course of their experience with osteoarthritis rather than just at the end stage. The door has been opened, and the rehabilitation community could easily collaborate on such a venture.

Measurement is the application of a set of rules to get numeric quantification of attributes--in Stratford and colleagues' case, the pain and physical function concepts. Once we believe we have a good way of measuring these concepts, we keep pushing the boundaries to make sure we can interpret the scores in ways that we should. Stratford and colleagues have done that in their study.

References

(1) Kline RB. Principles and Practice of Structural Equation Modeling. 2nd ed. New York, NY: The Guilford Press; 2005.

(2) Fayers PM, Hand DJ. Factor analysis, causal indicators and quality of life. Qual Life Res. 1997;6:139-150.

(3) Bellamy N. WOMAC Osteoarthritis Index User Guide IV. Herston, Queensland, Australia: University of Queensland; 2000.

(4) Bellamy N, Kirwan J, Boers M, et al. Recommendations for a core set of outcome measures for future phase III clinical trials in knee, hip, and hand osteoarthritis: consensus development at OMERACT III. J Rheumatol. 1997;24:799-802.

(5) Jette AM. Toward a common language for function, disability, and health. Phys Ther. 2006;86:726-734.

(6) Verbrugge LM, Jette AM. The disablement process. Soc Sci Med. 1994;38:1-14.

(7) International Classification of Functioning, Disabilty and Health: ICF. Geneva, Switzerland: World Health Organization; 2001.

(8) Lohr KN, Aaronson NK, Alonso J, et al. Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clin Ther. 1996;18:979-992.

Dorcas Beaton, BScOT, PhD

Scientist and Director

Mobility Program Clinical Research Unit

St Michael's Hospital

Assistant Professor

Department of Occupational Sciences

Graduate Departments of Rehabilitation Sciences and Health Policy Management and Evaluation

University of Toronto

Toronto, Ontario, Canada

Author Response

We thank Dr Beaton for her stimulating commentary, which conveys important information concerning statistical modeling and conceptual frameworks relevant to the assessment of health outcomes. Her commentary includes general insights and issues specific to our article. Beaton has organized her review under 2 headings, and we will apply the same 2 headings in response to her comments.

The Art of Statistical Modeling

We agree with Beaton's position that art complements the science of statistical modeling in general and structural equation model specifically. Beaton wonders how information concerning a single-factor model or a model that includes the cross-loading of items on both factors would affect our conclusions. Our rationale for starting with a 2-factor model without cross-loading was that we conceptualized 2 factors with the relevant items loading uniquely on their respective factors. Although we did not desire cross-loading at the item-factor level, we imagined the factors would be related and allowed for a correlation between the factors. However, in response to Beaton's query regarding additional factor loading models, we will briefly summarize the results from 2 additional analyses. The first was a confirmatory factor analysis that specified a single factor with all pain and time items loading on this factor. The chi-square value indicated a poorly fitting mode ([chi square]=419.4, df=20, P<.001) with the following factor loadings: self-paced walk-pain=.49, stair test-pain=.44, Timed "Up & Go" Test-pain=.49, Six-Minute Walk Test-pain=.34, self-paced walk-time =.94, stair test-time=.88, Timed "Up & Go" Test-time=.92, and 6-Minute Walk Test-distance= -.84. The negative loading with distance occurs because greater distances are associated with more desirable health status levels, whereas faster times and lower pain levels are related to more desirable health status levels. Our initial 2-factor model, which specified pain and time (distance) items loading on their respective factors, yielded a chi-square value of 70.7 on 19 degrees of freedom. The difference in chi-square values between the single-factor model and our initial 2-factor model is highly significant in favor of the 2-factor model ([chi square]= 348.7, df=1, P<.001).

The second supplementary analysis did not assume an a priori factor structure and allowed for all possible cross-loadings. We accomplished this by applying exploratory factor analysis using maximum likelihood estimation and oblique rotation. Two factors had eigenvalues greater than 1. Time/distance items loaded highly on one factor, and pain items loaded highly on the other factor. The correlation between factors was .46. The Table reports a summary of the pattern matrix factor loadings. It is evident from the Table that cross-loadings on the no-congruent factor were negligible. In summary, these supplementary analyses further support a 2-factor model that does not include a cross-loading of items.

Conceptual Frameworks

We have divided Beaton's comments into the following 3 topics and respond to each in turn: (1) our application of Bellamy's definition of lower-extremity physical function, (2) the proposition that the distinction between pain and function may diminish as the contextual complexity of an item increases (framed in the International Classification of Functioning, Disability and Health [ICF] lexicon, participation items are more contextually complex than activity items), and (3) the proposition that pain may be better assessed by applying nonattributed measures compared with attributed (eg, task-specific) measures of pain.

Bellamy is one of the few measure developers to offer a definition or clarifying phrase for the concept of lower-extremity functional status: "by this we mean your ability to move around and to look after yourself." (1,2) We applied Bellamy's definition; however, we agree with Beaton's point that, in the context of our assessment process, the performance measures focus on "the ability to move around" and not on "the ability to look after yourself." Thus, within the ICF classification scheme, our performance measures assess aspects of activity limitation and not participation restriction.

The second point offered by Beaton is the speculation that the distinction between pain and function may become less distinct as the contextual complexity of an item or performance task increases. We are not aware of a study that has explored this exciting hypothesis prospectively. However, in a previous study that attempted to explain why the WOMAC physical function subscale could not detect deterioration identified by performance measures in patients 2 weeks after hip or knee arthroplasty, physical function items were divided into 2 sets. (3) One set contained items similar to those on the WOMAC pain subscale, and the second set consisted of items not specific to the items on the WOMAC pain subscale. Item scores within each set were summed to yield total scores for the similar and not-specific item sets. The total item score for the not-specific item set detected deterioration consistent with the times from the performance tests. The item structure of the 2 sets of items is pertinent to our current discussion. All items in the set that did not detect deterioration tapped activity limitation; all but one item in the set that detected deterioration assessed participation restrictions. Although the results of this study are at odds with Beaton's hypothesis, the study was not conceived to investigate Beaton's hypothesis. We strongly support the need to investigate prospectively the hypothesis raised by Beaton. As a minor point, Beaton has suggested that our stair climbing test may be more contextually complex; however, given the way this activity was framed in our investigation--without any more relevance to self-care, occupation, and recreation than the other 3 performance tests--we are uncomfortable accepting this specific example.

Beaton's third point addresses the assessment of pain. Given the body of evidence suggesting that self-report measures of physical function are strongly influenced by pain, (4-6) we examined whether a more distinct assessment of these 2 attributes could be achieved by performance measures. Our goal was to determine whether performance measures could bring into focus 2 health concepts that to date have been blurred by self-report measures. We believe that complementing, not replacing, self-report functional status and pain measures with performance measures will increase the validity of impressions of pain and physical function formed by clinicians. We concur with Beaton's point that pain is a complex health concept and that further investigation concerning attributed and nonattributed pain ratings is warranted in a larger sample of patients.

References

(1) Bellamy N. An Evaluative Index for Clinical Trials. Hamilton, Ontario, Canada: McMaster University; 1982.

(2) Bellamy N. WOMAC Osteoarthritis Index User Guide IV. Herston, Queensland, Australia: University of Queensland; 2000.

(3) Stratford PW, Kennedy DM. Does parallel item content on WOMAC's Pain and Function Subscales limits its ability to detect change in functional status? BMC Musculoskelet Disord. 2004;5:17.

(4) Maly MR, Costigan PA, Olney SJ. Determinants of self-report outcome measures in people with knee osteoarthritis. Arch Phys Med Rehabil. 2006;87:96-104.

(5) Terwee CB, van der Slikke RM, van Lummel RC, et al. Self-reported physical functioning was more influenced by pain than performance-based physical functioning in knee-osteoarthritis patients. J Clin Epidemiol. 2006;59:724-731.

(6) Stratford PW, Kennedy DM. Performance measures were necessary to obtain a complete picture of osteoarthritic patients. J Clin Epidemiol. 2006;59:160-167.

Paul W Stratford, PT, MSc

Deborah M Kennedy, BScPT, MSc

Linda J Woodhouse, PT, PhD

Table.
Pattern Matrix Factor Loading

                                  Factor

                                 1       2

Self-paced walk-time           .992   -.063
Timed "Up & Go" Test-time      .898    .029
Stair test-time                .841    .057
6-Minute Walk Test-distance   -.841    .004
Self-paced walk-pain           .044    .869
Timed "Up & Go" Test-pain      .051    .852
6-Minute Walk Test-pain       -.077    .792
Stair test-pain                .024    .789