Reliability of the sources for the finger-to-nose test in adults with traumatic brain injury.

Coordination disturbances are only one of the clinical manifestations of the complex motor disorders affecting persons with traumatic brain injury (TBI). Disturbances in coordination among this population are frequent[1-4] and can severely compromise motor function. The documentation of coordination disturbances (eg, the inability to execute smooth, accurate, and controlled movements[5]) forms an integral part of the assessment of sensorimotor function in the population of patients with TBI. Clinicians usually assess coordination disturbances by observation, noting the quality of movements performed by the patient. Instrumented, quantitative methods have recently been used to measure coordination in patient populations with varied neurologic disorders. These methods include tracking tasks[6,7] reciprocal tapping tasks,[8] and kinematic analysis.[9] A traditional and long-standing method of the measurement of upper-extremity (UE) coordination, however, consists of the clinician observing the patient's performance during the "finger-to-nose" test[10] and noting certain characteristics of the movement such as time of execution and presence or absence of dysmetria and tremor. The time of execution of this test can be recorded using a stopwatch. Dysmetria is characterized as an impaired ability to judge the force and range of a movement; tremor refers to an involuntary oscillatory movement.[5] information regarding the degree of dysmetria and tremor is usually scored using ordinal rating scales and is heavily dependent on the clinician's judgment.

Although the protocol for recording results from the finger-to-nose test may vary among institutions and individual clinicians, the literature indicates that it is used routinely in the clinical setting and can perhaps be identified as the "gold standard" measure of UE coordination.[5,10,11] The use of ordinal rating scales such as those used in the finger-to-nose test has recently been questioned, however, because of their reliability and sensitivity limitations.[12]

Recently, Mayo et al[13] examined the interrater reliability of several clinical measures (including the finger-to-nose test) from a standard neurophysical evaluation. Four experienced physical therapists rated the finger-to-nose test performances of 18 adults with neurological impairment (including 16 patients with TBI). The degree of agreement among raters was found to be 49.7% for dysmetria and 76% on the scoring of tremor; each variable was evaluated using a three-point ordinal rating scale. Kappa statistics calculated for dysmetria and tremor were .17 and .08, respectively, indicating "poor" interobserver agreement. The authors did not, however, examine the reliability of the timed measure of this test, nor did they examine intrarater reliability. We believe, therefore, that further research must be conducted to fully examine both the intrarater and interrater reliability of this clinical measure of coordination in persons with TBI.

The purpose of this study was to determine the intrarater and interrater reliability of the scoring system used by experienced physical therapists who independently rated the videotaped performances of the finger-to-nose test among persons with TBI.

Method

Subjects

Thirty-seven persons with TBI (26 male, 11 female) participated in the study. The sample was one of convenience, with subjects recruited from three different rehabilitation centers(*) on the basis of the diversity of the levels of severity of sensorimotor deficits. Inclusion criteria were (1) an adequate UE motor function (sufficient active range of motion and voluntary movement) to perform the finger-to-nose test with at least one UE and (2) sufficient comprehension of simple verbal commands. The protocol for this study was approved by each institution from which the subjects were recruited (and the data were collected). Informed consent to be filmed was obtained from all subjects.

The data analyzed in this study represent a subset of data (n=37) from a previously published study [14] (n=40). The present study included only those subjects who consented to be filmed. Subjects ranged in age from 17 to 64 years (X=29.1, SD=9.9). The duration of coma varied considerably among subjects, ranging from 1 to 135 days (X=38.2, SD=32.7), as did the time elapsed since the subjects' injury, ranging from 3 to 117 months (X=34.0, SD=30.9).

Raters

Five physical therapists (including the original investigator [BRS]) participated as raters in this study. All raters, except the original investigator, were recruited from a neurology team of a physical therapy department within a Montreal rehabilitation center. The raters, each with between 4 and 15 years (X=10.8, SD=4.5) of clinical experience in the treatment of patients who have neurological impairment, were all very familiar with the application of the finger-to-nose test in persons with TBI.

Equipment and Procedure

The finger-to-nose test was administered to all subjects during a previous study conducted approximately 1 year before the current study.[14] At that time, subjects were instructed to perform the finger-to-nose test in the typical clinical manner. With his or her eyes open, each subject began the test with an arm extended in front of the body at shoulder level. The subject then flexed the elbow, touched his or her nose as accurately as possible, and returned the arm to the fully extended position. The need for both speed and accuracy of performance was emphasized to the subjects. Subjects performed the task with each UE whenever possible (two trials with each arm). The total time for the completion of five complete cycles of movement was recorded using a stopwatch. In addition, the degree of both dysmetria and tremor observed during the movement was recorded for each UE. These variables were rated on separate four-point ordinal scales using the following criteria: 3=normal performance or the absence of a deficit, 2 = slight deficit, 1 = moderate deficit, and 0 = severe deficit.

Each subject's performance (two trials for each UE) was videotaped while the investigator administered and scored the test. The subject was videotaped from a lateral view so that the rater could clearly observe whether subjects accurately touched their finger to their nose. The videotaped performances were then analyzed 1 year later by the original investigator and by four other physical therapists.

Rater training. No specific training of the therapists was conducted. Prior to the viewing of the videotape, however, the raters participated in a brief orientation session consisting of the presentation of the evaluation form, the scoring protocol, and the procedure for the rating session. The scoring system (ordinal rating scales) was provided for all raters on a printed sheet. The rating scales were undefined for dysmetria and tremor because they are typically not defined when using this clinical test. The variable time of execution (for each trial), however, was defined for the raters as the time for the completion of five complete cycles of movement. A cycle began with the arm extended in front of the body at shoulder level, included the flexion of the elbow so as to bring the index finger to touch the nose, and finished with the return of the arm to the fully extended position. During this session, the therapists were also given time to practice and become comfortable with the manipulation of their stopwatches.

This study sought to determine the reliability of measurements obtained by the "typical" user of the test. It was assumed that the typical therapist in terms of years of clinical experience) knows how to skillfully score this test; therefore, this session did not include a training demonstration with examples of a performance characterizing each of the four ratings. Likewise, therapists were not given the opportunity to view any videotaped performances prior to beginning the rating procedure.

Intrarater scoring procedure. The original investigator scored the subjects' performances (two trials) on the finger-to-nose test using the previously described procedure on two different occasions: (1) during the original data collection and (2) during the viewing of the videotaped performances 1 year later. This rater did not review any of the previous scores collected I year earlier, prior to the viewing of the videotape a year later. The videotaped performances were presented in such a way that the order of subjects tested differed from that in the original data collection. Therefore, for the analysis of intrarater reliability with 1 year between tests, there was one examiner.

Interrater scoring procedure. The videotape was viewed at the Institut de Readaptation de Montreal during two 1-hour sessions. Each subject's videotaped performance was presented to the group of five raters (including the original investigator). The raters scored each subject's performance (two trials for each UE) simultaneously and independently and did not consult with each other during or after the evaluations. The raters were given the opportunity to review a particular segment of the videotaped performance only if necessary. The total time for the completion of five complete cycles of movement and the degree of both dysmetria and tremor observed during the movement performed by each UE (two trials for each UE) were recorded by each therapist using the scoring system described earlier.

Data Analysis

The generalized Kappa statistic was chosen to determine the intrarater and interrater reliability of the measurements of dysmetria and tremor. This statistic, developed by Cohen,[15] is a coefficient of agreement (among two or more raters) for categorical data that corrects for chance agreement. Kappa values less than .00 have been characterized arbitrarily as indicating "poor" agreement, values between .00 and .20 have been characterized as indicating "slight" agreement, and those between .21 and .40 are said to represent "fair" agreement. Kappa values between .41 and .60 have been characterized as "moderate," those between .61 and .80 have been characterized as "substantial," and values greater than .81 are said to indicate "almost perfect" agreement.[16]

Although Kappa was the preferred statistic, agreement was also described as the total percentage of subjects in which both of a pair of raters agreed on the score. The agreement among rater pairs, however, does not indicate the degree of reliability that can be attained.

In order to establish the intrarater and interrater agreement for the measurement of "time of execution" (interval data), intraclass correlation coefficients (ICC[3,1]) were calculated. This particular model was used with the assumption that judges are considered fixed effects[17] as the selection procedure of raters did not meet the requirements of randomness. Tests of significance using an F test were performed at the .01 level.

Intrarater agreement. Intrarater agreement for this study was determined by having the same person (BRS) measure the same three variables on two different occasions. The patients' scores obtained by the original investigator were compared with those recorded I year later from the videotaped performances of the finger-to-nose test.

Interrater agreement in order to assess the interrater agreement among the pairs of raters for the scoring of dysmetria and tremor, generalized Kappa coefficients were calculated for each of the 10 possible pairs of raters (for trials 1 and 2) and are reported as average values across all possible pairs.

Results

The distributions of the data from the finger-to-nose test for the 37 subjects are shown in Tables 1 and 2. Only data for right-sided performances (trials 1 and 2) are presented because the distributions for right- and left- sided performances were very similar.

Reported in Table 1 are the distributions of the combined data obtained when the same rater (BRS) measured the same three variables on two different occasions for each of the 37 subjects (maximum of 74 observations). For example, the scores for all subjects from trial I on the first testing occasion (test) have been combined with the scores from trial 1 on the second testing occasion (retest) I year later, and likewise for trial 2. Not all subjects could perform the finger-to-nose test with their right UE; therefore, only data from 67 performances are resented.

Table 1. Descriptive characteristics and distributions of data obtained with the
Finger-to-Nose Test When the Same Rater Evaluated Each Subject (N=37) on Two
Different Occasions(a)
                         Number of Observations(b)
Variable                  Trial 1          Trial 2
Dysmetria
  0=severe                1 (1.5)          1 (1.5)
  1=moderate              3 (4.5)          3 (4.5)
  2=slight              26 (38.8)         22(32.8)
  3=normal              37 (55.2)         41(61.2)
Tremor
  0=severe                 0 (0)            0 (0)
  1=moderate              1 (1.5)           (1.5)
  2=slight                4 (6.0)          4(6.0)
  3=normal              62 (92.5)        62(92.5)
Time of execution (s)
  X                        4.98              4.58
  SD                       1.6               1.7
  Range                .7-10.0          2.2-10.7
(a) Right-sided performances only.
(b) Percentage of evaluations shown in parentheses.

In general, scores for dysmetria and tremor did not vary greatly among the subjects. The majority of subjects tested were scored as having normal performances when dysmetria ( >55% of the subjects) and tremor (92.5% of the subjects) were evaluated. This scoring, however, creates a distribution problem that makes our use of the Kappa statistic problematic, because the Kappa corrects for chance agreement. Mean times of execution of the finger-to-nose test were 4.98 and 4.58 seconds for trials 1 and 2, respectively, with the range of times being similar for both trials. Furthermore, the distribution of the scores did not vary over the two trials. This finding may indicate that this particular rater consistently judged subjects as having the same degree of dysmetria or tremor while performing each trial, or that the subjects' performance was indeed consistent over the two trials.

The distribution of scores obtained when the five raters evaluated each of the 37 subjects, for a maximum of .185 possible observations, is shown in Table 2. Different numbers of observations were recorded for individual variables and trials. These differences were due to either a subject's inability to perform both trials of the finger-to-nose test with the right UE or a therapist's inability to rate a particular subject. Although the variability of scores exceeds that found for the data obtained by the same rater (Tab. 1), scores for dysmetria and tremor still did not vary greatly among the subjects. This finding again makes interpretation of values obtained with the Kappa, a statistic that corrects for chance agreement, problematic, with an uneven distribution likely as chance agreement increases. A greater degree of variability was found for the scoring of the presence of dysmetria, with only 45.9% (trial 1) and 48.8% (trial 2) of the subjects being scored as having normal performances. The majority of subjects (74.6% and 81.3% for trials 1 and 2, respectively) were scored as having normal performances when tremor was evaluated. Consistency between trials was again very high.

Table 2. Descriptive Characteristics and Distributions of Data Obtained with tbe
Finger-to-Nose Test When Five Raters Evaluated Each Subject (N=37)(a)
                    Number of Observations(b)
Variable            Trial 1           Trial 2
Dysmetria           n=170             n=168
  0 = severe         3 (1.8)           3 (1.8)
  1 =moderate        9 (5.3)          10 (6.0)
  2=slight         80 (47.1)         73 (43.5)
  3=normal         78 (45.9)         82 (48.8)
Tremor              n=170             n=169
  0 = severe         2 (1.1)           2 (1.2)
  1 =moderate        4 (2.2)           4 (2.4)
  2=slight         26 (14.1)         26 (15.4)
  3=normal        138 (74.6)        137 (81.1)
Time of execution (s)
  X                     5.13              4.73
  SD                    1.6               1.6
  Range            2.7-10.7          2.1-10.7
(a) Right-sided performances only.
(b) Percentage of evaluations shown in parentheses.

Tables 3 and 4 present the results of three indexes of agreement - percentage of agreement, Kappa statistic, and ICC - for the three variables (ie, degree of dysmetria, degree of tremor, and time of execution). The values for these indexes of agreement were originally calculated with the data from each trial; however, the values reported here represent the mean of the values from both trials for each UE. For simplicity, data are presented separately for the intrarater and interrater reliabilities.

Table 3. Intrarater Reliability Coefficients
                  Percentage of         Mean
Variable            Agreement         Kappa(a)    ICC(3,1)(b)
Dysmetria
  R(c)                 72               .54
  L(d)                 78               .54
Tremor
  R                   90.5              .18
  L                   89.5              .31
Time of execution (s)
  R                                                 .971
  L                                                 .986
(a) Mean Kappa represents the mean of the Kappa coefficients calculated for tria
ls 1 and 2.
(b) P<.005.
(c) R=right upper extremity.
(d) L=left upper extremity.
Table 4. Interrater Reliability Coefficients
                 Percentage of        Mean
Variable           Agreement         Kappa(a)   ICC(3,1)(b)
Dysmetria
  R(c)               62.9             .36
  L(d)               68.2             .40
Tremor
  R                  78.6             .27
  L                  80.8             .26
Time of execution (s)
  R                                               .920
  L                                               .913
(a) Mean Kappa represents the mean of the Kappa coefficients calculated for tria
ls 1 and 2.
(b) P<.005.
(c) R=right upper extremity.
(d) L=left upper extremity.

Intrarater Agreement

Percentage of agreement was 72% or higher for the scoring of dysmetria and tremor (Tab. 3). In general, Kappa values for the scoring of dysmetria were higher than those for tremor, indicating moderate agreement on the scoring of dysmetria but only slight to fair agreement on the scoring of tremor. This result, however, may also be a factor of the differences in distribution for the scores on the two variables. Both ICCs were above .95 for time of execution and were determined to be significant (P<.005).

Interrater Agreement

Percentage of agreement among the five raters exceeded 62% (Tab. 4). Kappa statistics for the scoring of dysmetria were again slightly higher than those for tremor but indicated only fair agreement for both variables. All ICCs were above .89 for time of execution and were significant (P<.005).

Discussion

Generally, the Kappa coefficients (agreement statistic) indicated that the five therapists did not agree well on the scoring of dysmetria and tremor. In contrast, agreement was somewhat higher when the same therapist (intrarater) evaluated these variables on two different occasions. The greatest agreement among raters was observed in the assessment of the time of execution of the finger-to-nose test, with both intrarater and interrater reliability being very high. As expected, the scoring of the qualitative aspects of the performance was more difficult than the recording of the quantitative measurement of time. It should be noted, however, that there was not an equal representation of all observations and, because Kappa corrects for chance agreement, our reliability coefficients may be low, reflecting not poor reliability but rather the poor distribution of observations.

These results are similar to those reported elsewhere.[13] Interrater agreement (Kappa) for the assessment of dysmetria and tremor remained low. Because most observations of tremor were normal, however, the Kappa values really do not accurately reflect reliability. The agreement of these measures improved slightly (by .20 for tremor) when the team of raters consisted of therapists practicing within the same rehabilitation setting. In the previous study, both the raters and the subjects were drawn from two different rehabilitation centers and one acute care setting. Another difference, perhaps also contributing to the improved reliability, is the nature of the sample. In the present study, the sample was restricted to only subjects with TBI and in particular to those who were in the later stages of the physical rehabilitation process. It is perhaps noteworthy that in this study, in which a slightly higher agreement was obtained, a four-point rating scale was used (as opposed to a three-point scale). The choice of the four-point scale was predicated by the clinical protocol used by the therapists participating in the study.

There may be several reasons for the slight to moderate reliability (Kappa) reported for the variables of dysmetria and tremor. A number of causes of low reliability have been discussed by Rothstein[18] and include those attributable to (1) the person administering the test, (2) the instrument itself, or (3) the characteristics of the subject(s) being tested. We will address each of these causes as they pertain to this study.

First, in this study, the same person administered the finger-to-nose test to all 37 subjects, thereby introducing a constant source of error. Five different raters, however, scored (one aspect of the administration of the test) each performance. The purpose of this study was not to determine the reliability of scores on the finger-to-nose test among physical therapists who had specialized training in its application and to conclude that expert users of the test can obtain reliable measurements. Instead, its purpose was to determine the reliability of scores on the finger-to-nose test for assessing the ability of the "typical" user of the test. The typical therapist (in terms of years of experience) knows how to skillfully perform this test and frequency uses this measure of coordination in his or her clinical practice.

We assumed that the raters from this study, by virtue of their training and clinical experience, had the necessary skills to score the finger-to-nose test. Perhaps the raters had difficulty noting the deficits from the videotaped performances (a method that they are not familiar with), or they misinterpreted the scoring criteria because the criteria were not sufficiently defined. Furthermore, the interrater agreement may perhaps be different among therapists with less clinical experience in the treatment and evaluation of patients who have TBI. likewise, the reliability reported in this article may have been overestimated, because the five therapists were recruited from the same physical therapy department (sample of convenience). The results might perhaps have been quite different if the raters had been selected from a number of rehabilitation centers.

A second possible cause of the low reliability relates to the instrument itself. The finger-to-nose test uses ordinal rating scales to evaluate the presence of dysmetria and tremor, thereby providing an appreciation of the qualitative nature of the movement performed by the patient. Descriptors such as "moderate deficit," however, lack precision and consequently may not be sensitive to small and perhaps meaningful changes in patient status.[5,19] These descriptors are vague and could, in theory, be based on more definitive criteria that might improve the reliability of the scores on the finger-to-nose test (eg, number of times the subject misses touching his or her nose).

A probable cause of the low reliability in this study (and a limitation) was restrictions in observed scale variability. The values of the Kappa statistic depend on the prevalence of the scores over all subjects.[20] When the proportion of observed values is heavily skewed toward one of the scale values (Tabs. 1 and 2), Kappa values can be quite low, even with little rater error. If the percentage of agreement is high and Kappa is relatively low, then a restriction in the distribution of judgments is a likely explanation for the low reliability. For example, for the rating of tremor (Tabs. 3 and 4), low Kappa values were associated with a high percentage of agreement, indicating that the raters were observing subjects with little or no presence of tremor consistently and were in fact agreeing on that score.

Attempts were made to obtain a sample of subjects with a diversity of levels of severity of sensorimotor deficits. We expected our sample to be similar to those reported in the literature,[21,22] in which at least 33% of the subjects presented "cerebellar syndromes" and coordination deficits. As measured by the finger-to-nose test, however, only a small percentage (maximum of 6%) of our sample was judged by the five therapists as having moderate and severe coordination deficits. in actuality, the sample may have been composed of a greater percentage of subjects with coordination deficits, but the deficits were not identified using the finger-to-nose test.

The use of video recordings has provided an alternative method for the examination of interrater reliability issues.[23,24] Furthermore, by eliminating the interest interval (when establishing intrarater reliability), no change in the variables to be measured can occur, and the measurement obtained on the second occasion is not influenced by the first measurement. Such an approach has allowed the examination of the reliability of the scoring protocol of the finger-to-nose test - the focus of this report. This approach, however, does not necessarily allow generalizations to the traditional test - retest situation in which the test is administered and scored on two occasions. In addition, the use of videotaping permits a number of therapists to observe and score a performance without disturbing the patient. Patients with TBI may not perform optimally while being observed by a team of raters.

Clinical Implications

The results of this study provide some important information regarding the amount of error associated with the measurement of coordination using the finger-to-nose test. Therapists can now better interpret the measurements associated with the finger-to-nose test in view of the reliability associated with it. In turn, therapists may then be able to make more appropriate inferences or judgments regarding patient progress in coordination performances or treatment intervention. Therapists should realize, however, that the Kappas reported in this study may be underestimating reliability because of the distribution problems we discussed.

Therapists should be encouraged in learning that they can reliably measure the time of execution of the finger-to-nose test using a stopwatch. The reliable data obtained with this method offer therapists, and possibly third-party payers, important information regarding a particular dimension of performance (eg, speed of movement) in coordination deficits and its resolution in patients with TBI. Improvement in this aspect of performance, however, may not necessarily reflect an improvement in the overall coordination abilities of a patient, because the accuracy of performance may be sacrificed for the increased speed of movement.

Knowing the limitations of the finger-to-nose test should prompt therapists to seek more objective and quantitative methods for evaluating coordination performances. Computerized methods for the measurement of coordination are currently available to therapists,[25,26] and the relationship between this method and the standard clinical finger-to-nose test has recently been established.[14]

Suggestions for Further

Research

We suggest that if therapists continue to use the finger-to-nose test to measure UE coordination among patients with TBI, efforts should be made to establish clearer scoring criteria for the rating of dysmetria and tremor. The need for accuracy criteria for the timed measure, for example, is clearly evident.

The results of this study do not allow generalizations to other patient populations, and therapists cannot assume a similar level of reliability in other patient groups.[18] If therapists wish to know the reliability of data obtained with the finger-to-nose test among other patient populations, further research must be conducted. Additional research with patients who have TBI is also needed. Before the reliability of scores in patients with TBI can be understood, a study must be conducted in which the Kappa values are not influenced by the types of distribution problems we encountered.

Conclusions

Physical therapists appeared to demonstrate low reliability (intrarater and interrater) for scoring the presence of dysmetria and tremor during the videotaped performances of the finger-to-nose test. The therapists, however, reliably measured the time of execution of this test (using a stopwatch). These results suggest a need to establish more definitive-scoring criteria in order to improve the reliability of these measures. Therapists should be aware of the amount of error associated with the measurements from the finger-to-nose test and should be very cautious with interpretations of this test in the population of patients with TBI.

References

[1] Jennett B. Some aspects of prognosis after severe head injury. Scand J Rehabil Med. 1972; 4:16-20. [2] Rinehart MA. Considerations for functional training in adults after head injury. Phys Ther. 1983;63:1975-1977. [3] Caronna JJ. The neurologic evaluation. In: Rosenthal M, Bond MR, Griffith ER, Miller JD, eds. Rehabilitation of the Adult and Child with Traumatic Brain Injury. Philadelphia, Pa: FA Davis Co; 1984:59-73. [4] Whyte J, Rosenthal M. Rehabilitation of the patient with head injury. In: DeLisa JA, ed. Rehabilitation Medicine: Principles and Practice. Philadelphia, Pa: JB Lippincott Co; 1988:585-611. [5] Schmitz TJ. Coordination assessment. In: O'Sullivan SB, Schmitz T), eds. Physical Rehabilitation: Assement and Treatment. 2nd ed. Philadelphia, Pa: FA Davis Co; 1988:121-133. [6] Behbehani K, Kondraske GV, Tintner R, et al. Evaluation of quantitative measures of upper extremity speed and coordination in healthy persons and in three patient populations. Arch Phys Med Rehabil. 1990;71:106-111. [7] Jones RD, Donaldson IM. Measurement of integrated sensory-motor function following brain damage by a computerized preview tracking task. International Journal of Rehabilitation Medicine. 1981;3:71-83. [8] Turton A, Fraser C. The use of a simple aiming task to measure recovery following stroke. Physiotherapy Practice. 1987;3:117-125. [9] Rosecrance JC, Giuliani CA. Kinematic analysis of lower-limb movement during ergometer pedaling in hemiplegic and nonhemiplegic subjects. Phys Ther, 1991;71:334-343. [10] Grinker RR, Sahs AL. Neurology. 6th ed. Springfield, Ill: Charles C Thomas, Publisher; 1966:40-41. [11] Adams RD, Victor M. Principles of Neurology. 4th ed. New York, NY: McGraw-Hill Inc; 1989. [12] Merbitz C, Morris J, Grip JC. Ordinal scales and foundations of misinference. Arch Phys Med Rehabil. 1989;70:308-312. [13] Mayo NE, Sullivan SJ, Swaine BR. Observer variation in assessing neurophysical signs among patients with head injuries. Am J Phys Med Rehabil. 1991;70:118-123. [14] Swaine BR, Sullivan SJ. The relation between clinical and instrumented measures of motor coordination in traumatically brain injured persons. Arch Phys Med Rehabil. 1992;73:55-59. [15] Cohen JA. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37-46. [16] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174. [17] Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-427. [18] Rothstein JM. Measurement in Pbysical Therapy, New York, NY: Churchill Livingstone Inc; 1985:8-14. [19] Sullivan SJ, Swaine BR. Objective measures in physiotherapy: utility and advantages. In: Proceedings L'Evaluation Mesurable en Medecine de Readaptation. Montreal, Quebec, Canada; 1989. [20] Feinstein AR, Cicchetti DV. High agreement but low kappa, I: the problems of two paradoxes. J Clin Epidemiol. 1990;43:543-549. [21] Held JP, Mazeau M, Rodineau J. Les troubles de la motilite et du language chez les traumatises craniens severes. Annales de la Medecine Physique. 1975;18:337-354. [22] Cohadon F, Richer E. Evolution et devenir des comas traumatiques graves. Neurochirurgie. 1983;29:303-325. [23] Eastlack ME, Arvidson J, Snyder-Mackler L, et al. Interrater reliability of videotaped observational gait-analysis assessments. Phys Ther. 1991;71:465-472. [24] Henderson L, Kennard C, Crawford TJ, et al. Scales for rating motor impairment in Parkinson's disease: studies of reliability and convergent validity. J Neurol Neurosurg Psychiatry 1991;54:18-24. [25] Kondraske GV, Potvin AR, Tourtellotte WW, Syndulko K. A computer-based system for automated quantification of neurologic function. IEEE Trans Biomed Eng. 1984;31:401-414. [26] Smith SS, Kondraske GV. Computerized system for quantitative measurement of sensorimotor aspects of human performance. Phys Ther. 1987;67:1860-1866.