Intraobserver and interobserver reliability of assessments of impairments and disabilities.

Key Words: Assessment; Functional health status; International Classification of Impairments, Disabilities, and handicaps; Intraobserver and interobserver agreement; Kappa.

In primary care, the physical therapist relies on the patient's history and examination and other information (diagnosis and referral data) to make an assessment of the patient's functional status. The physical therapist's ability to reliably assess patient status is important because medical information (diagnosis and referral data) by itself is often an inadequate starting point for physical therapy intervention.[1-15] The treatment goals in physical therapy are often aimed at the restoration or preservation of a patient's functional status, thereby contributing to the patient's quality of life.[8,3] Analysis and evaluation of a patient's functional status are especially important when the relationship between a disorder or disease and perceived health status is not evident.[16-18] For example, research has shown that there is no longitudinal relationship between the experienced health status and the severity of obstruction of the airways (forced expiratory volume in 1 second) in patients with chronic obstructive pulmonary disease.[17]

In the Dutch health care system, about two thirds of the physical therapists work in primary health care.[19,20] The majority of the patients in primary health care (about 80%) are referred by general practitioners after being seen for complaints of the musculoskeletal system and the nervous system,[19,21,22] with these complaints most often resulting in problems with locomotion. The assessment form that we used to examine reliability in our study was designed as a generic tool for assessing impairments and disabilities. Pertinent findings, such as impairments, location of impairment, and disabilities, are recorded following the history taking and physical examination. A manual has been developed for the use of the assessment form.

The methodological prerequisites that have to be met by an assessment form are standardization, reliability, validity, and sensitivity to change (responsiveness).[23,24] Reliability is a measure of agreement (consistency) and refers to the reproducibility of measurement results or precision of measurements or outcomes. We examined the intraobserver and interobserver reliability of these assessments. Each impairment and disability was assessed as "present" (observed or identified), "not present," or "not applicable." To determine the reliability of the assessments, the following questions must be answered:

1. How reproducible is the assessment of the separate impairments and disabilities recorded by one therapist for the same patient (intraobserver agreement)?

2. To what extent do two therapists agree when they independently assess the same patient in terms of presence of impairments and disabilities (interobserver agreement)?

Method

Assessment Form

The assessment form used in this study has a two-level structure. The first level deals with 36 impairments based on the chapters of the Classification of Impairments from the Proposal for Adjustment of the International Classification of Impairments, Disabilities, and Handicaps (ICIDH).[1,25] The therapist assesses whether an impairment is "present" and indicates its severity (on a three-point Likert scale). In addition, the location (body region) and affected organ or tissue of each impairment are recorded. In the Proposal for Adjustment of the ICIDH, an impairment is defined as "a loss or deviation of an anatomical structure or a physiological or psychological function, taking into consideration the age of a person."[1,25]

The second level of the assessment tool we used consists of 27 disabilities, which are taken from the chapters of the Classification of Disabilities from the Proposal for Adjustment of the ICIDH.[1,25] The presence and severity are recorded in the same way as for the impairments. In the Proposal for Adjustment of the ICIDH, disability is defined as "a loss or deviation, in both a qualitative and quantitative way, of activity performance or behavior of a person, taking into consideration age, gender, and the physical, social, and cultural environment."[1, 25]

Design

The reliability study was carried out as part of the field study, "Physical Therapist's Consultation in Primary Health Care," in which a random sample of 62 physical therapists from 62 private practices in four different regions of the Netherlands participated (target sample of patients=1,533).[26] For the intraobserver reliability part of the study, one physical therapist (in practice A) was randomly selected from the study sample. For the interobserver reliability part of the study, two pairs of physical therapists (in practices B and C) were randomly selected from the study sample.

Intraobserver reliability. The characteristics of the physical therapist (PT A) in practice A are shown in Table 1. A total of 33 patients were assessed twice by this physical therapist. In an effort to minimize recall bias, the patients were separated into three groups of 11 patients. Following the initial evaluation, all patients were reevaluated 2 days later by the same therapist. The patients were seen in a different order than during the initial evaluation, and the therapist was blinded to the results of the previous evaluation. The patients did not receive any treatment between the two evaluations.

Table 1.
Characteristics of the Physical Therapists (PTs)

                                      Practice A   Practice B
                                      (PT A)       PT B1

Gender                                Female       Male
Year graduated                        1990         1979
Experience (y)                        1            >10
Postgraduate education(a)             Movement     Manual therapy
                                       sciences    Sports physical
                                                    therapy
Training in use of assessment form    Yes          Yes

                                      PT B2

Gender                                Female
Year graduated                        1975
Experience (y)                        >10
Postgraduate education(a)
                                      Manual therapy
                                      Sports physical
                                       therapy
Training in use of assessment form    Yes

                                      Practice C
                                      PT C1                 PT C2

Gender                                Female                Female
Year graduated                        1974                  1986
Experience (y)                        2                     7
Postgraduate education(a)             Manual therapy
                                      Neurodevelopmental
                                       treatment
                                      Cardiac
                                       rehabilitation
Training in use of assessment form    Yes                   No

(a) Recognized by the Royal Dutch Physical Therapy Association
(KNGF).

Interobserver reliability. In each private physical therapy practice (practices B and C), a pair of therapists participated (PTs B1 and B2 and PTs C1 and C2). To assess interobserver reliability, each pair of physical therapists assessed a series of patients. Three of the therapists (PTs B1, B2, and C1) were trained in the use of the assessment form, the theoretical background of the terms of the ICIDH, and the definitions for selected items used in the assessment form. The training also consisted of oral explanations (by authors EJMH and JWB and research associates Trudy Bekkering, PT, and Karin van Pelt, PT) and written explanations of the items (impairments and disabilities) that were included in the assessment form.

Practical experience in the use of the assessment form was gained during the 2 months prior to the study. During this time, the therapists completed the assessment form for at least five patients, and they had the opportunity to comment on and discuss the problems that they experienced. In practice B, the therapists discussed with each other the patient assessment performed during this training period. Oral feedback was provided (by authors EJMH and JWB and the research associates) after the first month and at the end of the training period. The fourth therapist (PT C2) was not included in the training in order to allow us to investigate the influence of training on reliability. The characteristics of the therapists are given in Table 1.

In the interobserver reliability part of the study, patients were evaluated by one of the two therapists within a practice. Findings were recorded after the therapist had taken the patient's history and had performed the physical examination. Within 3 days, the other therapist examined the same patient. Forty patients were assessed in practice B, and 41 patients were assessed in practice C. The patients did not receive any treatment between the sessions.

Patients

All new patients referred to the private practices were candidates for the study. Prospective subjects received written and verbal information about the study. Patients were excluded from the study if they did not want to participate, if the nature of their complaints did not allow a second examination (eg, severe acute complaints or disorders such as reflex sympathetic dystrophy), or when treatment (from the physical therapist's point of view) could not be postponed (eg, acute ankle sprain, severe acute tension headache). All selected subjects gave their written consent prior to participation in the study. The medical "diagnosis" was classified using the International Classification of Primary Care (ICPC).[27]

Data Analysis

The data analysis consisted of a description of the demographic and referral data of the patients and the analysis of the reliability of the categorization of impairments and disabilities as being present or not present by one therapist in the intraobserver reliability part of the study and by both pairs of therapists in the interobserver reliability part of the study. The prevalence of the observed or identified impairments and disabilities was described by frequency distributions. Agreement was determined by percentage of agreement (de, the number of patients on which the observers agreed as a proportion of all patients studied) and Cohen's kappa (K). The agreement within and between therapists was calculated separately for each impairment and disability (2x2 table). An impairment was included in the analysis only when the first and second assessments determined the same anatomical location for the impairment. When more than one location of impairment was determined, only the first-mentioned (primary) location of the impairment was included in the analysis.

Cohen's kappa, as opposed to percentage of agreement, is a chance-corrected estimate of agreement. Kappa indicates the actual agreement as a proportion of the potential agreement following correction for chance agreement.[23,24,28,29] The value of Cohen's kappa may vary between 1 (perfect agreement) and -- 1 (less agreement than can be expected on the basis of chance alone). For the interpretation of kappa, the classifications of Fleiss[29] and van Triet et al[30] have been used. In their opinions, a value greater than .75 indicates "excellent" agreement, a value between .40 and .75 indicates "fair to good" agreement (in our terminology, "satisfactory" agreement), and a value less than .40 indicates "poor" agreement. For the purpose of our study, a percentage of agreement greater than 75% was considered sufficient.[30] For instance, a high percentage of agreement can be present while the corresponding value of kappa is very low, or kappa can increase unexpectedly for identical percentages of agreement when there is not a spread in the distribution of judgments.[31,32] In the absence of criteria relating kappa for the skewness of observations, we adopted the guidelines proposed by van Triet et al.[30] Kappa was not calculated if each of the impairments or disabilities was identified in less than 10% or more than 90% of the patients. In that case, only the percentage of agreement is given, but these values may be inflated by chance agreement and need to be interpreted cautiously. The kappa values should be interpreted with caution if each of the impairments or disabilities is identified in 10% to 20% or 80% to 90% of the patients because without a widespread distribution of possible judgments, kappa values tend to be very low and thus not necessarily reflecting the agreement that can be seen in a more diverse group.

Results

Patients

Thirty-three patients (16 men, 17 women), with a mean age of 38.3 years (SD= 10.6, range= 18-56), participated in the intraobserver reliability part of the study. The most frequent reasons for their referral for physical therapy were low back complaints or disorders and complaints involving the thigh, hip, or leg.

Eighty-one patients participated in the interobserver reliability part of the study. Forty patients (22 women, 18 men), with a mean age of 41.5 years (SD=5.4, range=22-45), participated in practice B. Forty-one patients (22 women, 19 men), with a mean age of 41.8 years (SD=9.2, range=22-55), participated in practice C. In practice B, most patients were referred with diagnoses of low back complaints and osteoarthritis of the spine. In practice C, most patients were referred with diagnoses of low back complaints and neck and shoulder complaints. Table 2 presents descriptive data for the patients and their medical "diagnoses," as classified with the ICPC.[27]

Table 2.
Characteristics of Patients

Patient Characteristic                       ICPC(a) Code

Gender
  Male
  Female

Age (y)
[bar]X
SD
Range

Medical diagnosis (ICPC)
  Neck complaints                            L01
  Back complaints                            L02
  Low back complaints (no radiating pain)    L03
  Low back complaints (radiating pain)       L86
  Shoulder complaints/syndromes              L08/L92
  Arms/elbow complaints                      L09
  Wrist/hand/finger complaints               L11/L12
  Hip/leg/thigh complaints                   L13/L14/L75
  Knee complaints                            L15/L96
  Ankle/foot/toe complaints                  L16/L17
  Arthrosis/spondylosis/spine                L84
  Coxorthrosis                               L89
  Other                                      N02/N94/S93/L04/L81/
                                              L94

                                             Intraobserver
                                             Reliability
                                             (Practice A)

Gender
  Male                                        16
  Female                                      17

Age (y)
[bar]X                                        38.3
SD                                            18-56
Range                                         10.6

Medical diagnosis (ICPC)
  Neck complaints                             2
  Back complaints                             2
  Low back complaints (no radiating pain)     7
  Low back complaints (radiating pain)        3
  Shoulder complaints/syndromes               3
  Arms/elbow complaints                       2
  Wrist/hand/finger complaints                2
  Hip/leg/thigh complaints                    4
  Knee complaints                             2
  Ankle/foot/toe complaints                   2
  Arthrosis/spondylosis/spine                 2
  Coxorthrosis
  Other                                       2
Total                                         33

                                             Interobserver
                                             Reliability
                                             Practice B

Gender
  Male                                        18
  Female                                      22

Age (y)
[bar]X                                        41.5
SD                                            22-45
Range                                          5.4

Medical diagnosis (ICPC)
  Neck complaints                              8
  Back complaints
  Low back complaints (no radiating pain)     10
  Low back complaints (radiating pain)         2
  Shoulder complaints/syndromes                4
  Arms/elbow complaints                        2
  Wrist/hand/finger complaints                 2
  Hip/leg/thigh complaints                     1
  Knee complaints                              4
  Ankle/foot/toe complaints
  Arthrosis/spondylosis/spine                  6
  Coxorthrosis
  Other                                        1
Total                                         40

                                             Practice C
Gender
  Male                                        19
  Female                                      22

Age (y)
[bar]X                                        41.8
SD                                            22-55
Range                                          9.2

Medical diagnosis (ICPC)
  Neck complaints                              7
  Back complaints                              2
  Low back complaints (no radiating pain)     11
  Low back complaints (radiating pain)         1
  Shoulder complaints/syndromes                7
  Arms/elbow complaints                        2
  Wrist/hand/finger complaints
  Hip/leg/thigh complaints
  Knee complaints                              2
  Ankle/foot/toe complaints                    2
  Arthrosis/spondylosis/spine                  2
  Coxorthrosis                                 2
  Other                                        3
Total                                         41

(a) ICPC=International Classification of Primary Care [27]

Intraobserver Reliability

The percentages of agreement and kappa values for intraobserver reliability are presented in Tables 3 and 4. Table 3 shows that 22 of the 36 impairments were assessed; 13 items were assessed for more than 20% of the patients, and 5 items were recorded in less than 10% or more than 90% of the patients. For 18 of the 22 impairments, the percentage of agreement was greater than 75% (range=76%-97%). Fifteen of the 17 impairments for which kappa values were calculated showed "fair to good" or "excellent" agreement (range=.40.91). Four impairments were assessed between 10% and 20% of the patients or between 80% and 90% of the patients, which means that the results should be interpreted with caution.

Table 3.
Percentage of-Agreement and Kappa Values for Assessments of
Impairments(a)
                                          Intraobserver Reliability
                                          (Practice A)
                                          Percentage of
Impairments                               Agreement    Kappa

1. Pain                                      94        .64
2. Pain following movement                   88        .41
3. Pain during movement                      76        .34
4. Pain at rest                              70        .18
5. Pain during static stress                 73        .46
6. Pain sensation                            79        .45
7. Pain behavior                             88        .65!
8. Decreased range of motion                 82        .63
9. Increased range of motion                 *         *
10. Impairment in gait                       91        .79
11. Decreased active stability               88        .71
12. Decreased passive stability              *         *
13. Impairment in coordination               82        .41!
14. Increased resting tone (muscle)          70        .40
15. Increased resting tone                   85        -
16. Impairment in proprioception             *         *
17. Impairment in kinesthesia                *         *
18. Impairment in muscle strength            73        .46
19. Trophic impairment                       82        .54
20. Impairment in body posture               85        .46
21. Impairment in joint position             94        .80!
22. Impairment in pliability of the skin     *         *
23. Impairment in sensibility                82
24. Impairment in circulation                91        -
25. Edema                                    97        .91!
26. Impairment in mucus production/
    transport                                *         *
27. Impairment in respiration                *         *
28. Impairment in respiratory movement       *         *
29. Impairment in continence                 *         *
30. Intellectual and psychological
    impairments                              97        -
31. Eye and visual impairments               *         *
32. Ear and hearing impairments              *         *
33. Impairments in balance                   *         *
34. Dizziness                                *         *
35. Tiredness                                76        -
36. Other impairments                        *         *

                                          Interobserver Reliability
                                          Practice B
                                          Percentage of
                                          Agreement    Kappa

1. Pain                                      *         *
2. Pain following movement                   85        .67
3. Pain during movement                      95        .80!
4. Pain at rest                              83        .65
5. Pain during static stress                 83        .48!
6. Pain sensation                            88        .48!
7. Pain behavior                             93        .78!
8. Decreased range of motion                 92        .75!
9. Increased range of motion                 *         *
10. Impairment in gait                       85        .60
11. Decreased active stability               93        .78!
12. Decreased passive stability              *         *
13. Impairment in coordination               *         *
14. Increased resting tone (muscle)          83        .57
15. Increased resting tone                   *         *
16. Impairment in proprioception             88        .41!
17. Impairment in kinesthesia                *         *
18. Impairment in muscle strength            85        .41!
19. Trophic impairment                       85        .47!
20. Impairment in body posture               83        .59
21. Impairment in joint position             78        .26!
22. Impairment in pliability of the skin     85        -
23. Impairment in sensibility                *         *
24. Impairment in circulation                *         *
25. Edema                                    *         *
26. Impairment in mucus production/
    transport
27. Impairment in respiration                *         *
28. Impairment in respiratory movement       *         *
29. Impairment in continence                 *         *
30. Intellectual and psychological
    impairments                              *         *
31. Eye and visual impairments               *         *
32. Ear and hearing impairments              *         *
33. Impairments in balance                   *         *
34. Dizziness                                95        -
35. Tiredness                                *         *
36. Other impairments                        *         *

                                          Interobserver Reliability
                                          Practice C
                                          Percentage of
                                          Agreement    Kappa

1. Pain                                      *          *
2. Pain following movement                   66         .32
3. Pain during movement                      80         .41!
4. Pain at rest                              66         .33
5. Pain during static stress                 63         .18
6. Pain sensation                            39        -.23
7. Pain behavior                             83         -
8. Decreased range of motion                 73         .10!
9. Increased range of motion                 98         -
10. Impairment in gait                       83         .52
11. Decreased active stability               44        -.15
12. Decreased passive stability              68        -.19
13. Impairment in coordination               51        -.24
14. Increased resting tone (muscle)          61         .20
15. Increased resting tone                   *          *
16. Impairment in proprioception             80         -
17. Impairment in kinesthesia                56        -.19
18. Impairment in muscle strength            53        -.47
19. Trophic impairment                       56        -.09!
20. Impairment in body posture               73         .30
21. Impairment in joint position             56        -.01
22. Impairment in pliability of the skin     63         .27
23. Impairment in sensibility                *          *
24. Impairment in circulation                *          *
25. Edema                                    95         .84!
26. Impairment in mucus production/
    transport
27. Impairment in respiration                *           *
28. Impairment in respiratory movement       *           *
29. Impairment in continence                 *           *
30. Intellectual and psychological
    impairments                              80          -
31. Eye and visual impairments               *           *
32. Ear and hearing impairments              *           *
33. Impairments in balance                   *           *
34. Dizziness
35. Tiredness                                68          -
36. Other impairments                        83          .17

(a) Asterisk (*) indicates not applicable. Dash (-) indicates kappa cannot be calculated (impairment of [is less than or equal to] 10% or [is greater than] 90% of patients). Kappa of [is less than] .40=poor agreement; kappa between .40 and .75=fair to good agreement; kappa of [is greater than] .75=excellent agreement. Exclamation point (!) indicates contents of cells between 10% and 20% or between 80% and 90%.

Table 4.
Percentage-of-Agreement and Kappa Values for Assessments
of Disabilities(a)
                                          Intraobserver Reliability
                                          (Practice A)
                                          Percentage of
Disabilities                              Agreement    Kappa

Disability in sensorimotor skills
1. Disability in maintaining postures      90            .71!
2. Disability in changing postures         *             *
3. Disability in carrying                  77            .55
4. Disability in lifting                   77            .55
5. Disability in picking up                90            .67
6. Disability in pulling                   84            -
7. Disability in pushing                   90            -
8. Disability in reaching                  94            -
9. Disability in moving objects            77            .55
10. Disability in handling objects         *             *
11. Disability in grasping                 94            .71!
12. Disability in squatting              4 86            .68
13. Disability in kneeling               2 89            .67!
14. Disability in bending over           9 94            -
15. Disability in crawling                 93            .76!
16. Other disabilities                     *             *

Disability in locomotion
17. Disability in walking                  90            .81
18. Disability in climbing stairs          81            .54
19. Disability in cycling                  86            .52!
20. Other disabilities                     87            .52!

Disability in complex skills
21. Disabilities in personal care          *             *
22. Disabilities in domestic activities    87            .75
23. Disabilities related to work           84            .57
24. Disabilities related to recreation     85            .70
25. Disabilities in psychosocial skills    *             *
26. Disabilities in communication          *             *
27. Other disabilities                     *             *

                                          Interobserver Reliability
                                          Practice B
                                          Percentage of
                                          Agreement    Kappa

Disability in sensorimotor skills
1. Disability in maintaining postures      90            .73
2. Disability in changing postures         73            .45
3. Disability in carrying                  76            .46
4. Disability in lifting                   72            .43
5. Disability in picking up                70            .21!
6. Disability in pulling                   76            .41
7. Disability in pushing                   76            .39!
8. Disability in reaching                  76            .15!
9. Disability in moving objects            85            .57!
10. Disability in handling objects         88            .59!
11. Disability in grasping                 *             *
12. Disability in squatting                93            .72!
13. Disability in kneeling                 95            .77!
14. Disability in bending over             75            .48
15. Disability in crawling                 *             *
16. Other disabilities                     *             *

Disability in locomotion
17. Disability in walking                  90            .79
18. Disability in climbing stairs          90            .69!
19. Disability in cycling                  89            .68!
20. Other disabilities                     93            -

Disability in complex skills
21. Disabilities in personal care          *             *
22. Disabilities in domestic activities    90            .78
23. Disabilities related to work           90            .80
24. Disabilities related to recreation     90            .79
25. Disabilities in psychosocial skills     *            *
26. Disabilities in communication           *            *
27. Other disabilities                      *            *

                                           Practice C
                                           Percentage of
                                          Agreement   Kappa

Disability in sensorimotor skills
1. Disability in maintaining postures       73           .26!
2. Disability in changing postures          68           .11
3. Disability in carrying                   78           .57
4. Disability in lifting                    76           .52
5. Disability in picking up                 66           .31
6. Disability in pulling                    63           .11
7. Disability in pushing                    61           .01
8. Disability in reaching                   71           .38
9. Disability in moving objects             63          -.25
10. Disability in handling objects          54          -.22
11. Disability in grasping                  63           .17
12. Disability in squatting                 73           .34
13. Disability in kneeling                  83           .52
14. Disability in bending over              80           .59
15. Disability in crawling                  88           -
16. Other disabilities                      *            *

Disability in locomotion
17. Disability in walking                   85           .59!
18. Disability in climbing stairs           90           .72!
19. Disability in cycling                   85           .57
20. Other disabilities                      78           .18!

Disability in complex skills
21. Disabilities in personal care           90           -
22. Disabilities in domestic activities     76           .47
23. Disabilities related to work            60           .20
24. Disabilities related to recreation      51           .03
25. Disabilities in psychosocial skills     *            *
26. Disabilities in communication           *            *
27. Other disabilities                      *            *

(a) Asterisk (*) indicates not applicable. Dash (-) indicates kappa cannot be calculated (disability of < 10% or >90% of patients). Kappa of <.40=poor agreement; kappa between .40 and .75=lair to good agreement; kappa of >75=excellent agreement. Exclamation point (!) indicates contents of cells between 10% and 20% Or between 80% and 90%.

The results for disabilities were more consistent. Twenty of the 27 disabilities were assessed; 10 items were assessed for more than 20% of the patients, and 4 items were recorded for less than 10% or more than 90% of the patients. They all showed greater than 75% agreement (range=77%-94%). Kappa values could not be calculated for four disabilities (prevalence of [is less than] 10% or [is greater than] 90%). The remaining 16 disabilities showed "fair to good" or "excellent" agreement (range=.52-.81).

In summary, 42 (67%) of the 63 items (impairments and disabilities) on the assessment form were assessed. Four of the items assessed did not meet the criterion value of greater than 75% agreement. Kappa values could be calculated for 33 items. For 2 (6%) of these 33 items, there was "poor" agreement.

Interobserver Reliability

Interobserver reliability differed between the two practices (Tabs. 3 and 4). Seventeen of the 36 impairments were assessed in patients in practice B, and 24 of the impairments were assessed in patients in practice C.

The 17 impairments that were assessed in practice B all showed relatively high levels of agreement ([is greater than] 75%, range=78%-95%). For 2 impairments, kappa values could not be calculated. All impairments that were assessed, except "impairment in joint position," had kappa values in the "fair to good" or "excellent" range (.41-.80). "Impairment in joint position" showed "poor" agreement (.26), but the percentage of agreement was sufficient (78%).

In practice C, 8 (33%) of the 24 impairments assessed showed a percentage of agreement greater than 75% (range=80%-93%). Kappa values could be calculated for 19 of these 24 impairments. Three impairments for which kappa values could be calculated showed "satisfactory" or "excellent" agreement (range=.41-.84; "pain during movement," "impairment in gait," and "edema"). Sixteen of the impairments for which kappa values could be calculated showed "poor" agreement (range= -.47 to .33). Eight impairments had a negative kappa value, indicating that agreement was lower than could be expected on the basis of chance.

The results for findings of disabilities showed the same differences between practices B and C (Tab. 4). In practice B, 20 of the 27 disabilities were assessed. The percentages of agreement varied between 70% and 95%. Four disabilities ("disability in changing body posture," "disability in lifting," "disability in picking up," and "disability in bending over") did not meet the criterion value. Kappa values could not be calculated for one disability. For 3 disabilities, there was "poor" agreement. For 3 disabilities with less than 75% agreement, there was "fair to good" agreement when the percentage-of-agreement values were corrected for chance (satisfactory kappa, low percentage of agreement). The reverse happened for "disability in pushing" and "disability in reaching."

In practice C, 23 of the 27 disabilities were assessed. For 11 of these 23 disabilities, percentage of agreement was greater than 75%. Kappa values could not be calculated for 2 disabilities. The remaining disabilities had kappa values between -.25 and .72. Thirteen disabilities showed "poor" agreement. Eight disabilities had "fair to good" kappa values (range=.47-.72).

In summary, 37 (59%) of the 63 items (impairments and disabilities) in practice B and 47 (75%) of the 63 items in practice C were assessed. What we considered an acceptable percentage of agreement ([is greater than] 75%) was not found for 4 (11%) of the 37 items assessed in practice B and for 28 (60%) of the 47 items assessed in practice C. Kappa values could be calculated for 34 items in practice B or for 40 items in practice C. "Poor" agreement, in our view, was found for 4 items (12%) in practice B and for 29 items (74%) in practice C when the percentage-of-agreement values were corrected for chance.

Discussion

Based on the kappa values, the intraobserver reliability part of the study showed that impairments and disabilities could be reliably assessed with exception of three impairments ("pain at rest," "pain during movement," and "increased resting tone of a muscle"). Without exception, however, disabilities were recorded, in our view, with an acceptable level of reliability. Our results compare well with the results of similar studies in which patients' functional status was assessed for various health care professions.[30,33-39]

In our study, the different aspects of pain showed generally "poor" agreement. Epidemiological studies[19,21,22] show that there is widespread prevalence of pain and suggest that pain is one of the most important impairments at which physical therapy interventions are directed. Operationally defining pain remains difficult in part because different dimensions of pain need to be assessed. Bakx et al[33] have shown that pain can be assessed reliably when it is related to specific dimensions of disability. Furthermore, van der Kloot et al[40] have shown that pain can be assessed reliably when different dimensions of pain (eg, sensory, emotional-affective, cognitive-evaluative) are assessed. We believe, therefore, that expansion of the assessment form is needed to include other categories and dimensions of pain.

The results of the interobserver reliability part of the study are difficult to interpret and appear to be contradictory. Based on the kappa values in practice B, there was "fair to good" or "excellent" agreement at the level of impairments, with the exception of "impairment in joint position." At the level of disabilities, it appears that most disabilities can be reliably recorded, although some items showed a high percentage of agreement and poor kappa values and vice versa ("disability in pushing" and "disability in reaching"). We conclude that on the basis of the results in practice B, the interobserver reliability of the analysis of the patients' functional status is satisfactory.

In practice C, agreement was generally "poor." The differences in results between the two pairs of therapists, in our opinion, can largely be explained by the differences in characteristics of the physical therapists. First, the therapists in practice C showed a difference in areas of interest, based on their postgraduate education. Second, there was a great difference in years of experience in working in a primary health care setting. Third, and in our opinion probably the most important difference, one of the therapists in practice C had not received any formal instruction in the use of the assessment form. This finding seems to be supported by the large number of negative kappa values.

A negative kappa value can be interpreted as a negative correlation coefficient. In our study, this interpretation would mean that the first physical therapist assessed an impairment or disability as "present" and the second therapist assessed the same impairment or disability as "not present." These extreme disagreements (less agreement than can be expected on the basis of chance alone) between the two observers could be caused by a therapist's lack of experience or unfamiliarity with the form, as was the case with one of the therapists. Extreme disagreement may also be caused by an insufficient "description" of the item in the manual for the assessment form. Besides instruction in the use of the form, definition of terms and proper description of the impairments and disabilities in the Proposal for Adjustment of the ICIDH therefore seem to be important. In view of the limited number of participating therapists, our conclusions must be viewed with some caution until more data are available.

Van Triet et al[30] observed that there was greater agreement for the assessment of disabilities than for the assessment of impairments. This observation could be explained by the fact that the disabilities listed in their "assessment schedule" were more general or less specific than those included in our assessment form. A disease-specific form (eg, for assessing the functional status of patients following a stroke) would include other items. Our assessment form was developed as a generic instrument to be used by physical therapists for patients in primary health care. More than 90% of the patients referred for primary health care, however, are patients with complaints involving the musculoskeletal system.[19,21,22]

The detail of information most likely influences the reliability of a functional assessment.[1] The assessment form used in our study is based on the Proposal for Adjustment of the ICIDH, in which abilities and disabilities have been hierarchically classified in great detail.[1,25]

In our hierarchical classification, there is a step-by-step ordering of concepts into general (less-detailed) classes that are further subdivided into more detailed subclasses. Assessments at a lower hierarchical level (subclasses with a higher level of detail) may influence the reliability. An increase in detail, therefore, could be accompanied by a decrease in reliability. This possible disadvantage should be weighed against the advantage of having more specific information. The only conclusive way of determining the effect of greater detail will be further research. There are indications, however, that this may not be the case when the form is constructed on the basis of the Proposal for Adjustment of the ICIDH.[1,25]

Our assessment form was developed as a generic instrument for physical therapists in primary health care settings. For patient populations with more specific problems, depending on the purpose of the assessment form, more specific information on impairments (including anatomic location and involved tissues) and disabilities is probably needed. We believe that this information is especially important at the level of impairments. In our view, it would be advisable to operationally define the terms "impairment" and "disability" in the Proposal for Adjustment of the ICIDH to stimulate and enhance the uniformity of professional language.

With regard to the design and overall outcomes of our study, it should be noted that there might be some real changes in impairments and disabilities during the time intervals between the first and second observations (intraoberver reliability) or between the first and second observers (interobserver reliability). For practical reasons, there was a maximum duration of 2 and 3 days for the intraobserver and interoberserver reliability parts of the study, respectively. Theoretically, it could be that the severity of the impairments and disabilities might have changed within 3 days. Because the study was carried out with new patients without any treatment between the two observations and the data were analyzed on a lessdetailed dichotomous level ("not present" versus "present"), however, it is very unlikely that the time interval could have greatly affected the outcomes.

Kappa values are affected by prevalence of the identified (observed) impairments and disabilities.[30,41] In those instances with a prevalence of less than 10% or more than 90%, kappa may not be calculated.[30,41] In our study, some contradictory findings were observed between percentages of agreement and kappa values because of problems with the distributions of findings. Cicchetti and Feinstein[31,32] discuss two paradoxes of kappa based on a 2 x 2 table. The first paradox concerns a high agreement with poor kappa. In relation to the first paradox, kappa could increase with the same percentage of agreement (paradox 2). This seemingly apparent contradiction occurred several times in our study. For example, if one physical therapist assesses an impairment as "present" in 3 of 50 patients and as "not present" in the other 47 patients while another physical therapist assesses the impairment as "present" in 1 patient and as "not present" in 49 patients, the kappa value would be .49. If both therapists, however, assess the same impairment as "present" in 3 patients and as "not present" in 47 patients, the kappa value would be 1.00. For further discussion of this matter, see Cicchetti and Feinstein.[31,32]

In our study, the therapists were asked to indicate the severity of an impairment or disability on a three-point Likert scale. By using the "weighted" kappa, the degree of agreement can be calculated with more precision. Use of the weighted kappa with four ordered categories (eg, "not present" and "not applicable" versus "present" on a three-point Likert scale), however, requires a prevalence per item of 20 patients.[42] In our study, many items did not meet this criterion; consequently, weighted kappas were not calculated for these items.

Conclusion

Our results show that the intraobserver and interobserver reliability of the assessment form are within a range that many consider to be satisfactory for scientific investigation and survey research and for physical therapy practice, provided that necessary training is given. For some items (eg, pain), however, the assessment form may need to be improved with the addition of more specific dimensions. Before definite statements on reliability can be made, however, data from more therapists in more practices and with more patients are needed. Although the conceptual framework of the ICIDH offers physical therapists opportunities to describe a patient's functional status, it should be emphasized that the assessment form we examined can be used only to approximate the severity of a patient's health problem (indirectly) based on the judgment of the physical therapist. Nevertheless, in our opinion, an inventory of impairments and disabilities can be used by therapists to formulate priorities and treatment goals. It can also be used for determining the prognosis of the outcome and for determining the total number and frequency of treatment sessions (practice patterns). Although the interventions by physical therapists are usually not directed at the handicap level (social participation), the effects of the interventions at the impairment and disability levels may also result in changes at the handicap level.

Acknowledgments

We recognize the assistance provided by Mrs Trudy Bekkering, PT, and Mrs Karin van Pelt, PT, for their contribution in this study and the preparation of this manuscript.

References

[1] Heerkens YF, Brandsma JW, Lakerveld-Heyl K, van Ravensberg CD. Impairments and disabilities--the difference: proposal for adjustment of the International Classification of Impairments, Disabilities, and Handicaps. Phys Ther. 1994;74:430-442.

[2] Dekker J, van Baar ME, Curfs EC, Kerssens JJ. Diagnosis and treatment in physical therapy: an investigation of their relationship. Phys Ther. 1993;73:568-580.

[3] Enderby P, Kew E. Outcome measurement in physiotherapy using the World Health Organisation's Classification of Impairment, Disability and Handicap: a pilot study. Physiotherapy. 1995;81:177-180.

[4] Mischner-van Ravensberg CD, Paauw HJM, Gestel JLM. De fysiotherapeutische werkdiagnose in relatie tot de medische diagnose. Ned Tijdschr Fysiotherapie. 1988;5: 104-107.

[5] Hendriks HJM, Wagner C, Brandsma JW, et al. Experiences with physiotherapist's consultation: results of a feasibility study. Physiotherapy Theory and Practice. 1996;12:211-220.

[6] Jette AM. Outcomes research: shifting the dominant research paradigm in physical therapy. Phys Ther. 1995;75:965-970.

[7] Hulme JB, Wackernagel B, Lewis JW. Communication between physicians and physical therapists. Phys Ther. 1988;68:26-31.

[8] Brandsma JW, Heerkens YF, van Ravensberg CD. The International Classification of Impairments, Disabilities, and Handicaps (ICIDH): a manual of classification relating to the consequences of disease? Journal of Rehabilitation Sciences. 1995;8:2-7.

[9] Diagnosis in physical therapy: a roundtable discussion. PT Magazine. 1993;6(1):58-65.

[10] Sahrmann SA. Diagnosis and classification by the physical therapist--a prerequisite for treatment: a special communication. Phys Ther. 1988;68:1703-1706.

[11] Rose SJ. Physical therapy diagnosis: role and function. Phys Ther. 1989;69:535-537.

[12] Delitto A. Are measures of function and disability important in low back pain? Phys Ther. 1994;74:452-462.

[13] Guccione AA. Physical therapy diagnosis and the relationship between impairments and function. Phys Ther. 1991;71:499-504.

[14] Jette AM. Diagnosis and classification by physical therapists. Phys Ther. 1989;69:967-969.

[15] Jette AM. Physical disablement concepts for physical therapy research and practice. Phys Ther. 1994;74:380-386.

[16] Bindman AB, Keane D, Lurie N. Measuring health changes among severely ill patients. Med Care. 1990;28:1142-1152.

[17] van Schayk CP, Rutten MPMH, van den Boom G, et al. Longfunctie en ervaren gezondheid bij patinten met astma en chronisch obstructive longziekten in longitudinaal onderzoek. Huisarts Wet. 1995;38: 129-131.

[18] Bo K. Stress urinary incontinence, physical activity, and pelvic floor muscle strength training. Scand J Med Sci Sports. 1992;2:197-206.

[19] Kerssens JJ, Curfs EC. Extramurale Fysiotherapie. Utrecht, the Netherlands: Utrecht l University; 1993. Dissertation.

[20] Hingstman L, Harmsen J. Beroepen in de Extramurale Gezondheidszorg 1994: Aanbod van Beroepsbeoefenaren en Samenwerkingsverbanden in de Extramurale Gezondheidszorg in de Periode 1980-1993. Utrecht, the Netherlands: Nivel; 1994.

[21] van der Valk RWA, Dekker J, Boschman M. Basisgegevens Extramurale Fysiotherapie 1989-1992: Gegevens uit het Project "Beleidsgericht Evaluatie.--en Effectonderzoek Fysiothecapie (BEEF)." Utrecht, the Netherlands: Nivel; 1995.

[22] Klaveren AAJ, Hendriks HJM, Oostendorp RAB, et al. Basisgegevens van het Fysiotherapiesiotherapie Ontwikkelings Netwerk: Een Inventarisatie van het Fysiotherapeutisch Handelen in de Eerste Lijn m. b. v. het Basis Registratieformulier Fysiotherapie. Amersfoort, the Netherlands: SWSF; 1995.

[23] Sackett DL, Hayes RB, Guyatt GH, Tugwell P. Clincal Epidemiology: A Basic Science for Clinical Medicine. 2nd ed. Boston, Mass: Little, Brown and Co Inc; 1991.

[24] Bouter LM, van Dongen MCJM. Epidemiologisch Onderzoek: Opzet en Interpretatie. Utrecht, the Netherlands: Bohn, Scheltema en Holkema; 1988.

[25] Heerkens YF, Brandsma JW, van Ravensberg CD. Proposals for Adaptation of the Classification cation of Impairments and the Classification ration of Disabilities of the ICIDH From the Perspective of Five Dutch Health Professions. Zoetermeer, the Netherlands: Nationale Raad voor de Volksgezondheid/WHO Collaborating Centre for the ICIDH; 1993.

[26] Hendriks HJM, Wagner C, Brandsma JW, Dekker J. Fvaluatie van het Consultatief Fysiotherapeutisch Onderzoek in de Eertse Lijn: Resultaten van een Veldexperiment. Utrecht, the Netherlands: Nivel; 1994.

[27] Lamberts H, Wood M. International Classification of Primary Care. Oxford, England: Oxford University Press; 1987.

[28] Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37-46.

[29] Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY John Wiley & Sons Inc; 1981.

[30] van Triet EF, Dekker J, Kerssens JJ, Curfs EC. Reliability of the assessment of impairments and disabilities in survey research in the field of physical therapy. Int Disabil Stud. 1990;12:61-65.

[31] Cicchetti DV, Feinstein AR. High agreement but low kappa, II: resolving the paradoxes. J Clin Epidemiol. 1990;43:551-558.

[32] Feinstein AR, Cicchetti DV. High agreement but low kappa, I: the problem of two paradoxes. J Clin Epidemiol. 1990;43:543-549.

[33] Bakx VEM, Oostendorp RAB, Elvers JWH, et al. Pijnmeting in de fysiotherapie: een etrouwbaarheidonderzoek. Ned Tijschur Fysiotherapie. 1993;2:60-67

[34] Jelles F, van Bennekom CAM, Lankhorst GJ, et al. Inter- and intra-rater agreement of the rehabilitation activities profile. J Clin Epidemiol. 1995;48:407-416.

[35] van Gisbergen MJWM, Dekker J. Reliability of the diagnosis of impairments and disabilities by excercise therapists. Journal of Rehabilitation Sciences. 1992:5;67-73.

[36] van Gisbergen MJWM, Dekker J, Zuijderduin W. Reliability of the diagnosis of impairments in survey research in the field of chiropody. Disabil Rehabil. 1993; 15:76-82.

[37] Driessen MJ, Dekker J, Lankhorst GJ, van der Zee J. Inter-rater and intra-rater reliability of the occupational therapy diagnosis. Occupational Therapy Journal of Research. 1995;15:259-274.

[38] van de Berg J, Lankhorst GJ. Inter-rater and intra-rater reliability of disability ratings based on the modified D code of the ICIDH. Int Disabil Stud. 1990;12:20-21.

[39] Dekker J. Application of the ICIDH in survey research on rehabilitation: the emergence of the functional diagnosis. Disabil Rehabil. 1995;17:195-201.

[40] van der Kloot WA, Oostendorp RAB, van der Meij J, van den Heuvel J. De Nederlandse versie van "McGill pain questionnaire": een betrouwbare pijnvragenlijst. Ned Tijschur Geneeskd. 1995; 139: 669-673.

[41] Schoutsen HJA. Statistical Measurement of Inter-observer Agreement: Analysis of Agreements and Disagreements Between Observers. Utrecht, the Netherlands: Utrecht University; 1985. Dissertation.

[42] Cichetti DV. Testing the normal approximation and minimal size requirements of weighted kappa when the number of categories is large. Applied Psychological Measurement. 1981;5:101-104.

EJM Hendriks, PT, is Physical Therapist and Epidemiologist, Dutch National Institute of Allied Health Professions, PO Box 1161, 3800 BD Amersfoot, the Netherlands ([e-mail]). Address all correspondence to Mr Hendriks.

JW Brandsma, PhD, PT, is Physical Therapist, Dutch National Institute of Allied Health Professions and Academic Medical Centre, Amsterdam, the Netherlands.

YF Heerkens, PhD, is Movement Scientist, Dutch National Institute of Allied Health Professions.

RAB Oostendrop, PnD, PT, MT, is Physical Therapist, Manual Therapist, and Professor of Physical Therapy, Dutch National Institute if Allied Health Professions and Faculty of Medicine and Pharmacology, Postgraduate Education in Manual Therapy, Free University, Brussels, Belgium.

RM Nelson, PhD, PT, is Physical Therapist and Professor of Physical Therapy, Department of Physical Therapy, College of Allied Health Sciences, Thomas Jefterson University, Philadelphia, Pa.

This article was submitted April 8, 1996, and was accepted January 7, 1997.