Reliability of safe maximum lifting determinations of a functional capacity evaluation. (Research Report).

One challenge faced by clinicians when treating individuals who are off work due to low back pain (LBP) is balancing recommendations for early return to work with concerns of delayed recovery or pain exacerbation that could result from premature spinal loading. (1,2) Functional capacity evaluations (FCEs) are measurement tools created to assist in determining safe, tolerable levels of function and for predicting when an individual is ready to return to work duties. (3) In an FCE, a trained clinician attempts to measure an injured worker's maximum physical abilities for job-related tasks. Tasks assessed may include lifting, carrying, trunk flexion or rotation, and activities requiring walking or hand coordination. Manual handling, including lifting and carrying, has been described as the primary determinant for rating a job's physical demands. (4) If the worker does not have a job to return to, the information gained is used during vocational rehabilitation or job placement services by comparing results with known demands of other occupations. The determinations of performance levels during an FCE, therefore, have far-reaching implications with respect to return to work and employability.

Various types of FCEs exist. Two common approaches have been described as psychophysical and kinesio-physical evaluations. (5) Psychophysical FCEs place the worker in control, and performance is stopped when the worker believes maximal function has been reached. (5) The kinesiophysical approach places the administering therapist in control, and tasks are stopped when biomechanical signs of maximal effort are observed, such as accessory muscle usage and counterbalancing (altered biomechanics judged as being unsafe). (5) A set of standardized criteria for judging increased effort and maximal levels are outlined for the kinesiophysical method. (5) Theoretically, this ensures the safety of the injured worker, as assessment is to be stopped prior to overexertion. (5)

If the FCE is to be considered a useful tool, reliability and validity must be demonstrated. (6-9) As determinations require judgments regarding safety, some variance is expected with repeated measures within individual therapists and between therapists. In addition, variations in subject performance due to wellness on the day of the evaluation, motivation, pain levels, or interactions between the client and therapist conducting the evaluation may influence results. With these considerations in mind, interrater and test-retest reliability have been viewed as the most important forms of test reliability. (6,10,11)

Some work has been done to estimate the reliability of measurements obtained for various aspects of kinesio-physical testing. (4,12-14) A limitation of previous studies was the utilization of videotaped subject performance, resulting in a loss of some clinical information such as cardiovascular responses to testing used in maximal effort determination, which is gained during real-life observation. Studies that were done using real-life observation did not overcome the potential bias resulting from one rater influencing the judgment of the other rater when stopping the test. Lastly, all previous studies used a categorical outcome variable, rather than the interval-level outcome of amount of weight handled, as is determined in routine FCE testing.

Our goal was to determine the interrater and test-retest reliability of lifting determinations of maximal safe manual handling levels during kinesiophysical FCE using the Isernhagen Work Systems' * protocol in patients with LBP who were medically stable and receiving workers' compensation.

Method

Subjects

The sample was one of convenience and drawn from a rehabilitation center of the Workers' Compensation Board of Alberta. Subject inclusion criteria were selected to ensure the safety of participating subjects and to enroll subjects at a point in recovery when FCE testing is routinely performed. Inclusion criteria were: off work and receiving compensation for LBP; participation in an occupational rehabilitation program (subjects had plateaued with treatment and were in the process of being discharged); medical stability as determined by a physician (15); absence of metastatic disease, nonstable musculoskeletal conditions, or uncontrolled medical disorders; and a physician's determination of suitability for FCE following review of an electrocardiogram for subjects over 45 years of age. Written informed consent was obtained from all subjects prior to enrollment. Subjects were free to stop testing or withdraw at any time.

Subjects were recruited through consultation with treating rehabilitation teams to identify eligible clients nearing the end of their treatment program. All prospective subjects were scheduled for FCE testing at discharge whether or not they participated in the study. Twenty-eight subjects with LBP were enrolled in the study from April to July 2000. At an alpha level of .05, using chi-square tests for categorical variables and independent-sample t tests for continuous variables, no significant differences were observed between our subjects and the entire group of clients with low back injuries discharged from the center during the data collection period. Variables compared were: age, sex, National Occupation Classification (NOC) code, job attachment status, duration of injury, and length of time off work, as determined from the center's clinical database for all subjects discharged (Tab. 1).

Basic client characteristics and medical history data were collected at the time of enrollment, and subjects were asked 3 proposed core outcome measure questions advocated by Deyo et al. (16) From the core outcome questions asked of subjects, the modal bothersomeness of pain and interference with work due to pain were both moderate. Subjects most frequently reported being very dissatisfied with their symptoms, however, despite having nearly completed their rehabilitation program.

Five occupational therapists (3 male, 2 female) were enrolled to perform testing and act as raters. All raters had previously been trained by representatives of Isernhagen Work Systems, were conducting FCEs in clinical practice, and had at least 5 years of experience using kinesiophysical observation techniques. Raters reported an average length of time being trained in and performing kinesiophysical FCEs of 7.4 years (range=5-9 years). All raters were full-time employees and reported an average completion of 4,4 evaluations per week using kinesiophysical observation methods. Their average length of time spent in professional practice was 15.4 years.

Prior to the study, kinesiophysical principles and an operational definition of maximal effort were reviewed with the raters. Raters were asked to observe the following signs of increased effort in judging when subjects had reached maximal, safe levels:

1. Muscle bulging of prime movers

2. Involuntary use of accessory muscles

3. Altered body mechanics, including counterbalancing or use of momentum

4. Loss of equilibrium

5. Increased base of support

6. Decreased efficiency and smoothness of movement

7. Cardiovascular signs, including heart rate and breathing patterns

8. Peripheralization of radicular or referred symptoms

Study Protocol

A repeated-measures design was used with the goal of independent, yet simultaneous observation of each subject by 2 raters. Observations occurred on 2 separate occasions separated by 2 to 4 treatment days, a time period during which no significant change was expected in subject performance while allowing some time to lessen recall of the previous performance. Between occasions, raters continued to perform regular work duties, including other FCEs. Time of day and place of testing were held constant. Testing took place within the subject's last week of a rehabilitation program.

The FCE tasks of floor-to-waist, waist-to-crown, and horizontal lifting and front, right, and left side carrying were completed. The specific protocol for each lift and carry was followed as outlined in the Isernhagen Work System's Functional Capacity Evaluation Manual, (17) with sets of 5 repetitions being completed for each subtest at each successive weight level.

To obtain independent, yet simultaneous observation by the raters, 3 raters were selected randomly from the group of 5 raters for each enrolled subject. The first rater selected was referred to as the "primary rater." The primary rater's responsibility was to converse with the subject, guide the subject through testing, and upgrade weight in the lifting unit. Weight upgrades were possible in 1.1-, 2.2-, or 4.5-kg increments or any combination of these weights. The primary rater was the only individual with exact knowledge of the weight lifted or carried; the other raters were not able to see into the lifting unit and did not observe weight upgrades. The primary rater documented the amount of weight lifted or carried during each set, and other raters did not have access to this documentation. The primary rater also had the major responsibility for ensuring subject safety and was to stop testing if he or she judged safety to be obviously compromised.

The next 2 raters selected were referred to as "secondary raters." They observed performance and prompted the primary rater throughout testing, but they were instructed not to interact with the subjects. Secondary raters were instructed not to observe or talk to each other, but they were allowed to walk around the testing area for observation angle of choice.

Secondary raters were masked to each other's prompts and determinations in the following manner to avoid any potential bias. For each subject and subtest, the primary rater progressed testing from low to higher weight levels. Sets for each subtest were sequentially numbered on both the primary and secondary rater documentation forms. The primary rater documented the weight level, and secondary raters documented their observations for each set. After observing subject performance on an individual set, secondary raters documented their observations, then were allowed to prompt the primary rater nonverbally as to whether the weight in the lifting unit should be upgraded or testing stopped because maximal levels had been determined. They did this by pointing to one of 2 closely placed boxes with the words "Stop" and "Upgrade" on the bottom of their documentation forms.. Documentation stations were placed far enough apart for secondary raters not to see their companion's prompt. Primary raters walked between documentation stations to receive feedback. When a particular set was judged as maximal, the secondary rater pointed to the box stating "Stop," documented the observations, and circled the corresponding set number. All further prompting by this secondary rater was made by indicating "Stop." Testing continued with the primary rater upgrading weight until both secondary raters indicated "Stop." At the end of testing, all raters sealed their documentation forms in envelopes and delivered them to a secure location.

Maximal weight levels (in kilograms), as judged by the secondary raters, were determined through comparison of the primary rater's documentation with the corresponding set circled by each secondary rater. The factor leading to test termination for each lifting subtest also was recorded by the secondary raters. Limiting factors were categorized as physical maximum, cardiovascular limitation, nonfunctional time, or subject desire or pain.

Data Analysis

Intraclass correlation coefficients (ICCs [Shrout and Fleiss model 1,1 (18)]) with 95% confidence intervals (CIs) were calculated for interrater and test-retest reliability of secondary raters' judgments of maximal weight levels measured in kilograms. Two comparisons per subject were available for both forms of reliability. Because ICC values diminish when variance in a sample decreases, which would be the case if duplicate or repeat measures for both raters were used in analysis of test-retest data, calculations were performed separately for the 2 secondary raters' determinations. (18) In addition, interrater ICCs were calculated using the first session, with values from the second session used to judge stability of results.

Paired t tests with alpha level set at .05 were used to compare mean differences between occasions on each subtest to determine whether a testing effect existed between days of testing. Kappa values and percentages of agreement were calculated for agreement on factors limiting subject performance. The statistical software package SPSS [dagger] was used for ICC, t-test, and Kappa calculations.

The ICC is currently the statistic of choice for reliability analyses of interval data; however, classical test theory may not provide a complete understanding of this issue. Generalizability theory may provide a more effective conceptual approach, and comprehensive reviews have been published. (19-21) Generalizability coefficients and estimated variance components for the factors controlled for were calculated. Generalizability coefficients represent the relative generalizability of a measurement to the total range of possible scores for that measurement, with results ranging from 0 to 1, similar to the ICC. Estimated variance components show the contribution made to total variance by each controlled factor. These statistics were calculated using formulas discussed elsewhere. (20)

Results

Of the 28 subjects enrolled, 75% participated in both testing sessions. Three subjects did not attend on day 2, and 3 others attended but stated they did not feel capable of any manual handling due to LBP. Partial data sets were obtained from 6 subjects due to rater reporting error, subject desire, primary rater overruling a decision to upgrade (1 subject each), and lack of time to complete testing (3 subjects). The partial data are reflected in the various numbers of subjects per subtest in Tables 2 and 3.

The ICC values for interrater reliability on session 1 ranged from .95 to .98 (Tab. 2). Results were equally high for the second session. Test-retest ICC values ranged from .78 to .94 when calculated using the first secondary rater's scores and from .81 to .91 when using the second secondary rater's scores (Tab. 3). The high degree of similarity between the ICC values and CIs for the duplicate measures provides an indication of the stability of the test-retest values.

Mean scores of weight lifted on the 2 days were compared for all subjects who completed testing. Consistently, subjects lifted more on day 2, but these differences were statistically significant only for low-level lifting (21.8 kg for day 1, 25.7 kg for day 2; P=.01) and front carrying (32.2 kg for day 1, 34.7 kg for day 2; P=.02).

Findings from analysis of agreement for factors limiting test performance are summarized in Table 4. Kappa values ranged from .47 to 1.00, and overall percentage of agreement was 86.4% (235/272). Raters both judged a particular subject's performance as physical maximum on 68.8% of the comparisons. Of the 37 incidents where the raters disagreed, the same weight level was judged as maximum in 30 cases, with 26 of these cases being judged as physical maximum versus subject desire.

Estimated variance components and generalizability coefficients were also calculated and are shown in Table 5. Estimated variance components showed the highest portion of variance consistently resulted from between-subject variability (80.3%-91.4%), as expected. With respect to sources of measurement inconsistencies, however, the greatest portion of variance was explained by the subject-occasion interaction (4.5%-16.8%). Generalizability coefficients ranged from .90 to .96.

Discussion

Interrater reliability was excellent, with all subtest ICC values above .90. Results were similar when values from either day of testing were used in analyses. The ICC results were similar on similar subtests (ie, right and left side carrying), possibly reflecting internal consistency. When ratings of subjects who completed testing in both test sessions were analyzed, ICC values for test-retest reliability were lower (.78-.94) than those for interrater reliability. Test-retest reliability results were stable between secondary raters. Good generalizability was also seen, as all generalizability coefficients were equal to or greater than .90.

Three subjects returned for day 2 of testing but stated they did not feel capable of participating in manual handling activities due to reported pain exacerbation. The ease with which subjects could withdraw or terminate testing may have led to more subjects declining testing during the second session than would have occurred under normal FCE test conditions. However, the subjects' beliefs and perceptions of pain, disability, and physical capacity that led them to decline testing may represent valid influences on FCE results. The first test session was not cited as the reason for increased pain by any of the subjects who declined testing.

The testing interval was selected to minimize functional change. Return to work was imminent in this group of subjects deemed medically stable, yet the performance of some subjects varied between occasions. This was especially true of those subjects who were unwilling to participate on the second occasion. Variations in subjects' performance between days may have been due to the reasons discussed previously such as wellness, motivation, or pain level. Another potential contribution to the observed variability is a testing effect in subjects participating in both days. Comparison of means between days, with significant increases on the second occasion for low-level lifting and front carry, indicates that a testing effect likely did exist. It was not great enough, however, to diminish test-retest ICC values below acceptable levels.

Estimated variance components for subjects participating on both days clarify what factors were responsible for the variance observed. Consistently, subjects were responsible for the greatest variance, a desirable finding supporting the acceptable ICCs. The subject-occasion interaction, defined by Shavelson and Webb (20) as variance arising due to inconsistencies between occasions in particular subjects' performance, was consistently the second leading source of variance. The minimal residual variance in maximal ratings was made of various combinations of other factors, depending on the subtest, but these factors contributed little to the total variance.

Due to the variability observed between days and the fact 3 subjects felt they could not participate on the second occasion, manual handling is recommended over a 2-day period. The Isernhagen Work System's FCE protocol acknowledges client performance may vary between days and recommends a 2-day session of manual handling ability.

Raters agreed substantially or perfectly on the performance-limiting factor for test termination on most subtests according to the Landis and Koch categorization for Kappa values. (22) Agreement on front and left side carrying was moderate.

No previous study has looked specifically at the reliability of determinations of maximal levels using actual weight lifted, but other aspects of reliability of the kinesiophysical approach have been examined. When Isernhagen et al (4) studied interrater reliability of gross judgments of lifting effort, raters were able to accurately discriminate between "light" and "heavy" lifting efforts (Kappa=.81). Their study used videotapes of the subjects' performance; therefore, some clinical detail would have been lost. Smith (14) studied the ability of trained and experienced therapists to reliably judge whether patients with low back injuries can lift from the floor to waist with "safe body mechanics," as operationally defined by the author. Interrater Kappa values ranged from .62 to .64. In Smith's study, as in the study by Isernhagen et al (4) and a study by Gardener and McKenna, (12) videotape was used for viewing subject performance. Our study's design allowed clinically realistic observation and gave access to all information gained during a typical FCE, while allowing simultaneous observation of subjects. The slightly higher reliability we found may be due to added information available to our raters such as subject cardiovascular responses, symptoms, and three-dimensional viewing.

In a study by Lechner et al, (13) interrater reliability of measurements of maximal effort during another FCE protocol was examined. In this assessment, maximal effort was determined through observation of body mechanics and lifting technique. Interrater Kappa values found for manual handling determinations within Dictionary of Occupational Titles categories ranged from .62 to .88. These findings of substantial to almost perfect reliability are similar to, but slightly lower than, our findings. As the FCE under study was newly developed, raters had minimal experience, with total training time being approximately 20 to 24 hours. Conversely, raters in our study had at least 5 years of experience. The study protocol used by Lechner et al did not achieve independent observation between raters, resulting in a potential bias of one rater by the primary rater responsible for test termination.

One limitation of the present study affecting evaluation of test-retest reliability, in particular, was subject mortality. As noted previously, 3 subjects felt incapable of participating on day 2 of testing. In addition, only partial data sets were obtained from 6 subjects due to rater reporting error, subject lack of desire to perform all subtests, primary rater overruling a decision to upgrade, or lack of time to complete testing. A diminished sample size resulted and may have altered reliability calculations had all subjects been tested on all subtests. Yet, the consistency seen when alternate rater or occasion ICC values were calculated indicate the stability of the findings in the subjects tested. Although our design allowed us to overcome limitations of previous studies, the effect of multiple raters within the test setting as opposed to only one rater as in regular FCE practice is unknown. The effect on reliability when altering factors such as therapist discipline, level of therapist experience, and setting remains unknown.

Conclusions

Interrater reliability of kinesiophysical lifting and carrying determinations as conducted by experienced raters on a sample of workers' compensation claimants with low back injuries was excellent. Test-retest reliability, although lower, was generally good in subjects who completed testing. A subgroup of subjects was unwilling to participate on the second day of maximal testing due to a reported increase in symptoms unrelated to FCE testing. Assessment of manual handling over more than one occasion, therefore, is recommended to capture variability in function between occasions.

Table 1.
Characteristics of Workers' Compensation Claimants With Low Back
Pain

                         Subjects   Eligible Clients
Characteristic           (n=28)     (n=172)

Sex (% male)              71         71

Age (y)
  [bar]X                  41         41
  Range                  23-62      19-65

Occupation (%)
  Truck drivers           21         14
  Laborers                18          5

Job attached (%)          71         61

Duration of injury (d)
  Median                 123        136
  [bar]X                 165        213
  Range                  71-584     52-2,921

Time off work (d)
  Median                 112        114
  [bar]X                 125        152
  Range                  54-255     24-579
Table 2.
Interrater Reliability for Session 1 (a)

Task                     ICC   95% CI    N

Floor-to-waist lift      .98   .96-.99   27
Waist-to-overhead lift   .96   .92-.98   27
Horizontal lift          .96   .91-.98   27
Front carry              .96   .90-.98   25
Right side carry         .96   .91-.98   24
Left side carry          .95   .90-.98   23

(a) ICC=intraclass correlation coefficient, CI=confidence interval.
Table 3.
Test-Retest Reliability: Intraclass Correlation Coefficients (a)

                         Secondary Rater      Secondary Rater
                         1                    2
Task                     ICC   95%CI     N    ICC   95% Cl    N

Floor-to-waist lift      .78   .51-.91   18   .83   .60-.93   18
Waist-to-overhead lift   .84   .63-.93   18   .81   .56-.92   18
Horizontal lift          .86   .67-.95   18   .88   .71-.95   18
Front carry              .90   .75-.96   17   .87   .68-.95   17
Right side carry         .94   .85-.98   16   .91   .76-.97   16
Left side carry          .86   .65-.95   15   .83   .57-.94   15

(a) ICC=intraclass correlation coefficient, CI=confidence interval.
Table 4.
Rater Agreement on Performance-Limiting Factors

                                 Percentage of
Task                     Kappa   Agreement       Comparisons

Floor-to-waist lift       .64     79.2           48
Waist-to-overhead lift    .62     83.0           47
Horizontal lift           .77     97.5           48
Front carry               .47     82.2           45
Right side carry         1.00    100             43
Left side carry           .56     87.8           41
Table 5.
Generalizability Calculations (a)

                                           Percentage
                                Estimated  of Total    Generalizability
Task             Factor         Variance   Variance    Coefficient

Floor-to-waist
  lift           Subject (S)    435.8      83.6        .95
                 Rater (R)        0.0       0.0
                 Occasion (O)    24.7       4.7
                 SxR (b)          1.8       0.3
                 SxO (c)         34.5       6.6
                 RxO (d)          0.0       0.0
                 SxRxO (e)       24.2       4.7
Waist-to
  -overhead
  lift           Subject        127.2      80.3        .90
                 Rater            0.0       0.0
                 Occasion         0.2       0.1
                 SxR              1.6       1.0
                 SxO             26.6      16.8
                 RxO              0.5       0.3
                 SxRxO            2.3       1.5

(a) Only 2 tasks are shown. Variance components from tasks not shown
were similar.

(b) SxR=subject-rater interaction.

(c) SxO=subject-occasion interaction.

(d) RxO=rater-occasion interaction.

(e) SxRxO=residual, error.

This study was approved by the University of Alberta Health Research Ethics Board-Panel B and supported by the Clinical Research Partnership Fund, jointly sponsored by the Alberta Physiotherapy Association and the University of Alberta's Department of Physical Therapy.

This article was submitted May 2, 2001, and was accepted October 24, 2001.

* Isernhagen Work Systems, 1015 E Superior St, Duluth, MN 55802.

[dagger] SPSS Inc, 233 S Wacker Dr, Chicago, IL 60606.

References

(1) Abenhaim L, Rossignol M, Valat JP, et al. The role of activity in the therapeutic management of back pain: report of the International Paris Task Force on Back Pain. Spine. 2000;25(4 suppl):1S-33S.

(2) Waddell G, Burton AK. Occupational Health Guidelines for the Management of Low Back Pain at Work: Evidence Review. London, England: Faculty of Occupational Medicine; 2000.

(3) Gibson L, Strong J. A review of functional capacity evaluation practice. Work. 1997;9:3-11.

(4) Isernhagen SJ, Hart DL, Matheson LM. Reliability of independent observer judgments of level of lift in kinesiophysical Functional Capacity Evaluation. Work. 1999;12:145-150.

(5) Isernhagen SJ. Functional capacity evaluation: rationale, procedure, utility of the kinesiophysical approach. J Occup Rehabil. 1992;2:157-168.

(6) Innes E, Straker L. Reliability of work-related assessments. Work. 1999;13:107-124.

(7) Innes E, Straker L. Validity of work-related assessments. Work. 1999; 13:125-152.

(8) Sheikh K. Disability scales: assessment of reliability. Arch Phys Med Rehabil. 1986;67:245-249.

(9) Velozo CA. Work evaluations: critique of the state of the art of functional assessment of work. Am J Occup Ther. 1993;47:203-209.

(10) King PM, Tuckwell N, Barrett TE. A critical review of functional capacity evaluations. Phys Ther. 1998;78:852-866.

(11) Lechner D, Roth D, Straaton K. Functional capacity evaluation in work disability. Work. 1991;1:37-47.

(12) Gardener L, McKenna K. Reliability of occupational therapists in determining safe, maximal lifting capacity. Australian Occupational Therapy Journal. 1999;46:110-119.

(13) Lechner DE, Jackson JR, Roth DL, Straaton KV. Reliability and validity of a newly developed test of physical work performance. J Occup Med. 1994;36:997-1004.

(14) Smith RL. Therapist's ability to identify safe maximum lifting in low back pain patients during functional capacity evaluation. J Orthop Sports Phys Ther. 1994;19:277-281.

(15) Hart DL, Isernhagen SJ, Matheson LN. Guidelines for functional capacity evaluations of people with medical conditions. J Orthop Sports Phys Ther. 1993;18:682-686.

(16) Deyo RA, Battie MC, Beurskens AJHN, et al. Outcome measures for low back pain research: a proposal for standardized use. Spine. 1998; 23:2003-2013.

(17) Functional Capacity Evaluation Manual. Duluth, Minn: Isernhagen Work Systems; 1997.

(18) Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. 2nd ed. Englewood Cliffs, NJ: Prentice Hall; 2000.

(19) Roebroeck ME, Harlaar J, Lankhorst GJ. The application of generalizability theory to reliability assessment: an illustration using isometric force measurements. Phys Ther. 1993;73:386-395.

(20) Shavelson RJ, Webb NM. Generalizability Theory: A Primer. London, England: Sage Publications; 1991.

(21) Stratford PW, Norman GR, McIntosh JM. Generalizability of grip strength measurements in patients with tennis elbow. Phys Ther. 1989;69:276-281.

(22) Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.

DP Gross, PT, BScPT, is a doctoral student, Faculty of Rehabilitation Medicine, University of Alberta, 348 Corbett Hall, Edmonton, Alberta, Canada T6G 2G4 ([e-mail]). Address all correspondence to Mr Gross.

MC Battie, PT, PhD, is Professor, Department of Physical Therapy, University of Alberta.

Both authors provided concept/research design, writing, and fund procurement. Mr Gross provided data collection and analysis and project management. The authors thank the staff of Millard Centre for assistance with data collection and the Rehabilitation Research Centre at the University of Alberta for valuable input related to the study methods.