Accuracy and reliability of observational gait analysis data: judgments of push-off in gait after stroke. (Research Report).

Although instrumented gait analysis systems are frequently used in research, therapists use observation on a daily basis to evaluate gait because ready access to such measurement systems is currently impractical. (1) Observational gait analysis (OGA) refers to the qualitative approaches to gait analysis used by clinicians whereby gait deviations are identified in patients from visual observations. (2) As with all clinical measures, establishing the accuracy and reliability of OGA measurements is an essential consideration to ensure optimal clinical practice.

Despite its widespread use, there is a scarcity of evidence to support the accuracy (validity) of clinical observational analysis (Tab. 1). In 3 studies, (3-5) observational accuracy was compared with various criterion measures and revealed mixed results. In a study of gait following stroke, Pearson product moment correlations (r) between observation of selected gait components and waveform indexes from a foot force measurement device varied from .32 to -.72, with a mean correlation of .55. (3) Clinicians observing prosthetic alignment detected only 22% of the deviations predicted by biomechanical analysis of gait. (4) Observations of a mixed subject group were compared with kinematic angles obtained from simple protractor measurements of videotaped gait, with variable results. (5) Although the accuracy of determining foot placement was high, observers were generally inaccurate at judging joint angles, with an average score of 1.2 (1-2) out of a maximum of 5. Results from these studies contrast with those of a more recent study of gait after stroke, (6) which determined high Pearson correlations (r=.88) between physical therapists' judgments of timing symmetry and those concurrently obtained with an instrumented footswitch measurement device. Physical therapists also were highly accurate in judgments of other movements, including single-leg stance steadiness (7) and upper-limb tasks after stroke. (8,9) These are encouraging findings and suggest that therapists have the capacity to make accurate judgments under certain conditions.

Studies of the reliability of OGA measurements are more common and have included a wide range of gait variables in varied populations, using different methods. The majority of these studies have been reviewed comprehensively by Malouin (2) and are further described in Table 1. (3-6,10-18) Despite the growing number of studies and the diversity of results, the majority of OGA studies have demonstrated poor to only moderate reliability. (12,14,16) The widespread use of OGA in everyday clinical management of gait dysfunction highlights the need for further investigation of this clinical tool. Prior to further evaluation of OGA, we believe it is important to carefully evaluate the methods used in previous studies.

A range of factors have been identified and described as potential influences contributing to the generally poor to moderate reliability of OGA measurements reported. (2) These factors include the potential error associated with observation of live gait performances and the lack of both operational definitions and uniform observer instruction and training. Furthermore, the majority of studies of OGA have encompassed a wide variety of gait variables, thus requiring multiple and complex judgments. The low reliability found may relate to these design factors, rather than accurately reflecting the underlying agreement of the observers. Systematic studies that minimize variability due to live gait performances and reduce the number of observed variables provide insight into the capacity of therapists to reliably observe gait. (8) Although the current study deviates from the role of OGA in clinical practice, it represents an initial step in the exploration of the potential accuracy and reliability of observation under optimized circumstances.

The selection of gait variables in observational studies also warrants careful attention. (2,8) Variables selected should have clinical significance and be representative of gait capacity in the desired population. (2) Examination of the variables previously studied reveals wide diversity in both the type and number of selected variables. Almost all of the variables were spatiotemporal or kinematic in nature and reflect those variables typically considered in OGA and appearing on common gait evaluation forms. (18-20) Although therapists have traditionally focused OGA on these gait features, a rapid expansion in measurement technology has allowed insight into other important aspects of gait. A review of current biomechanical knowledge of gait suggests strongly that kinetic variables also should be considered in any form of gait analysis. (21)

A comprehensive biomechanical description of the kinetic forces underlying walking in subjects without known impairments or pathology (21-23) and in subjects with hemiplegia (24,25) has been provided by instrumented gait analysis. In walking, concentric and eccentric muscle contractions create moment forces across the lower-limb joints. Joint mechanical power is the product of the moment of force and the angular velocity across the joint. (21) The visible kinematic gait pattern observed by the clinician is simply an outcome of these invisible kinetic forces. Knowledge of these underlying patterns of power generation and absorption, therefore, provides an explanation of any kinematic deviations that can be seen. Herbert et al (26) have argued that clinical analysis of movement dysfunction requires more than a simple description of kinematic deviations. Rather, they suggested that therapists may be able to observe the visible features of gait and combine these observations with biomechanical knowledge to make inferences about the muscle forces that occur. This approach can provide insight into the nature of gait deficits and direct guidance toward appropriate intervention strategies.

A review of kinetic features of gait suggests that ankle power generation in late stance is an important gait variable that is impaired in subjects with gait dysfunction following a stroke. (21,24,25) Ankle plantar-flexor muscles contract rapidly in late stance phase to provide the single largest burst of power generation in the gait cycle of adults without impairments. (21) The magnitude of ankle power generated by individuals after stroke also is highly correlated with gait speed, a measure of gait performance for this population. (24,27) Reduced ankle power also is associated with reduced peak knee flexion in the swing phase, which is assumed to be related to reduced walking efficiency and increased risk of tripping. (25) This action in the late stance phase is commonly described as "push-off" and has been considered 1 of 6 critical features of human gait. (28)

To date, only 3 studies (3,5,10) have included push-off as a component of OGA, and they had different methods and highly variable results. Goodkin and Diller (10) reported 23 of 30 possible agreements across 3 therapists when observing subjects with hemiparesis. However, the absence of statistical analysis accounting for chance agreement limits conclusions from this study. Miyazaki and Kubota (3) reported interobserver agreement of .63 among 4 observers who viewed 48 subjects with hemiparetic gait. A Pearson correlation of .59 also was obtained between the observations of push-off and the data from a foot-force measurement device. Although accuracy and reliability were low, the use of live rating sessions, multiple rating variables, and a limited ordinal scale may have adversely influenced the results. In marked contrast, kinesiology students achieved high reliability on a nominal scale (average agreement: 4.5 out of 5) when judging push-off in videotaped performances by 8 subjects with varied pathologies. (5) However, because 2 subjects had amputations, it is not surprising that the observers were consistent in judging push-off as either normal or abnormal. No decisive evidence has emerged that indicates whether therapists can reliably or accurately use OGA to evaluate kinetic aspects of gait such as push-off. Further investigation with optimal methods, therefore, is warranted.

Some authors (1,29) have proposed that OGA requires much practice, combined with an understanding of biomechanics. This reflects what we believe is a common assumption by physical therapists that superior OGA skills may be associated with experience and practice. The few researchers (30-33) who have examined the relationship between experience and reliability of data obtained with assessment techniques have not provided convincing evidence to support the assumption that experience has a positive influence on reliability. The only study (14) of the relationship between therapist experience and reliability of OGA data demonstrated no consistent difference between experienced and less experienced therapists. In our research, therefore, we examined the relationship between the accuracy and reliability of therapists' observations and factors such as length and type of clinical experience.

In summary, the primary aim of this study was to investigate, after eliminating error due to variability in subject performance through the use of videotapes: (1) the accuracy, or criterion-related validity, of observations of push-off in videotaped gait performances of individuals after stroke, (2) the intraobserver and interobserver reliability of such observations, and (3) the relationship between clinical experience and the accuracy and reliability of OGA measurements.

Development of a New Observational Rating Scale

Two factors were considered in selecting and developing a new rating scale for study. These factors included a review of previously used scales and consideration of the role of OGA in clinical settings. A review of existing scales showed that almost all scales previously used to investigate OGA were nominal or ordinal scales of no more than 4 points (see, for example, studies by Potter et al, (13) Hughes and Bell, (15) and Keenan and Bach (16) in Tab. 1). The limited precision of these scales does not necessarily reflect the role of OGA in clinical practice. We believe therapists use OGA in conjunction with other techniques to make refined judgments about movement quality. Furthermore, OGA may be used to monitor change and evaluate the effects of intervention, rather than to simply identify the presence of gait abnormalities. (14) A therapist may use OGA to determine whether improvement has occurred in a patient's gait, although the gait may remain abnormal. A 3-point scale may not adequately reflect the small changes that can occur over a short period of gait training. More discriminative scales have been used in studies that have obtained high accuracy and reliability. (7-9) Accordingly, we chose an 11-point (0-10) scale to allow observers to form what we would consider more precise discriminations in their judgments of the degree of abnormality of the observed gait performance.

We believe the literature describing ankle power generation in adults without impairment also supports the inclusion of an 11-point (0-10) range for grading "normal" push-off. This range contrasts with the single category typically found in existing studies of OGA. (3,5,10,12,14,15) A wider scale enables the full range of normal abilities to be incorporated, recognizing that it may be desirable to detect clinical change in an individual throughout the normal range of ability. Scales that allow discrimination within the normal range of performance values permit continued measurements beyond the lower values of the normal range. For example, normal gait speed for adults may be described as a range of values between 1.2 and 1.5 m/s. (34) We believe that clinicians continue to measure gait speed beyond the lower threshold limit when evaluating change in an individual during gait rehabilitation after a stroke. Older subjects ([+ or -] 60 years of age) without impairments also have demonstrated a wide range of values for push-off power. (21) Accordingly, a range of 0 to 10 was chosen for the normal scale to reflect the variability of ankle power generation within the population of people without impairments. Together, these two 11-point scales provide a model of ankle power generation as a continuum of human performance values (Fig. 1).

Method

Participants

Observers. Eighteen physical therapists were recruited as observers as a sample of convenience from 4 rehabilitation centers. Their clinical experience varied between 2.1 and 11.2 years ([bar]X=6.1, SD=3.0), with "neurology-specific" experience ranging from 0.8 to 11.0 years ([bar]X-4.8, SD=3.1). All therapists reported always using OGA when assessing gait after stroke. These therapists frequently assessed stroke gait, with 16 therapists assessing stroke gait daily.

Subjects after stroke. Eleven subjects with hemiplegia following a single stroke were recruited using the subject database of the Human Motion Laboratory, Queens University, Kingston, Ontario, Canada. All subjects walked independently or with an assistive device or an orthosis. Eleven walking trials were selected from a wider database of 32 subjects with hemiparesis, each of whom completed 3 walks. The 11 walking trials were selected to provide a sample with the widest range and most even spread of ankle power generation values. Characteristics of the individuals and their gait variables are described in Table 2. All participants provided informed consent prior to the commencement of the study.

Apparatus and Procedure

Obtaining the measurements of ankle power generation. The criterion measure data were obtained as the subjects walked along a 9-m walkway at a self-selected, comfortable speed. Reflective markers were attached at the following anatomical landmarks: the fifth metatarsal joint, the heel, the ankle lateral malleolus, the lateral femoral condyle, the greater trochanter at hip joint level, and the acromioclavicular joint. Gait was filmed with a single LOCAM 51 cine camera * (50 frames per second) that was guided manually on a tracking cart beside the walkway. These data were synchronized with force data obtained simultaneously from a single forceplate. ([dagger]) A digitizer (GTCO Datatizer ([double dagger]) was used to extract the coordinates of the body and background markers from each cine film frame for each stride. After scaling and correction for parallax error, the coordinate data were digitally filtered using a second-order low-pass Butterworth recursive filter. These processed coordinates then were combined with anthropometric and forceplate data using biomechanical software adapted from Winter (21) to determine the kinematic and kinetic variables. Anthropometric data in the biomechanical modeling were obtained by combining individual subject measurements with standard constants (35) to estimate segment masses and inertias. The data collection process is further described in detail in Olney et al. (24)

Ankle joint power ([P.sub.a]) was calculated as the product of the equation:

(1) [P.sub.a] = [M.sub.a] x [[omega].sub.a]

where [M.sub.a] is the net moment across the ankle joint (in newton-meters) and [[omega].sub.a] is the ankle joint angular velocity (in radians per second). The data were then normalized by dividing each value by the subject's body weight. The criterion measure selected was the peak value of ankle power generation.

Construction of the gait performance videotape. Videotapes for each of the 2 testing occasions were produced, with each videotape containing 11 randomly ordered images of subjects walking. Gait was viewed from a sagittal perspective closest to the affected side, as the subjects walked at their self-selected speed. Each walking trial was edited to include 4 repetitions of a single gait stride, from foot contact of the affected lower extremity through to early stance of the affected lower extremity in the subsequent stride. An audible tone was inserted at foot contact to alert the observers to watch the subsequent stance and push-off action. The total duration of the videotape was approximately 7 minutes, with individual subject performances varying from 12 to 40 seconds.

Testing procedure. All observers participated in a 15-minute standardized instruction/familiarization session at the beginning of each testing occasion. This session included a brief review of normal ankle joint kinematics and kinetics (21) and an operational definition of push-off. Push-off was defined as "a component of gait late in stance phase where the plantarflexors generate a concentric `explosive' burst of energy, causing the foot to rapidly plantarflex" (adapted from Winter (21)). The rating scale was introduced, and the anchor descriptors were defined (Fig. 1). Observers practiced rating 2 example gait performances to become familiar with both the rating scale and the observational task. No specific training of OGA was provided, and no feedback was given regarding practice ratings, as this study aimed to investigate the observers' baseline ability.

Observers were tested in small groups at the rehabilitation centers on 2 occasions. The entire videotape was first shown to allow observation of the total range of subject variability. This aimed to minimize potential contraction bias effect. (36) This form of bias occurs when observers use a range of responses smaller than the range of stimuli, such as clustering scores into the center of a scale. The videotape was then replayed and rated by the observers, who recorded their scores for each walk by simply circling a number on either the abnormal or normal scale (test 1). The videotape was played at normal speed to reflect the self-selected gait speed of each subject. Stopping, repeating, or slow-speed viewing of the videotape did not occur.

Observers attended an identical second testing session (test 2) approximately 4 weeks after the first testing session and viewed a second test videotape with a randomly altered order of image presentation. The 4-week time delay was used in an effort to maximize the possibility that the observers had forgotten their previous allocation of ratings. The altered order of subjects aimed to minimize the influence of a rating order effect. Observers also completed questionnaires about their clinical practice and experience.

Data Reduction and Analysis

Data available for analysis included both the observational rating scale data and the corresponding criterion measure data. The observational data comprised both categorical data (abnormal/normal) and ordinal data (the ratings). First, the ratings on the 2 scales were considered as parts of a single scale measuring push-off. This continuous scale extended from a "zero" point on the abnormal scale to 10 on the normal scale, with a total range of 22 points. Observers' ratings were converted to numbers on a single scale, with potential values of 0 to 21. In this manner, the values of any ratings of the 11-point (0-10) abnormal scale were unchanged, with a rating of 2 on the abnormal scale represented in the analysis as a score of 2. Ratings of 0 to 10 on the normal scale were converted to numbers 11 to 21, with a rating of 0 on the normal scale converted to a score of 11, a rating of 1 converted to a score of 12, and so on. In view of the interval construction and high number (22) of categories used to measure this continuous variable, parametric statistical methods were selected for data analysis involving the rating scale numbers. Second, data from the categorical judgments of normality were available to enable comparison with other studies of OGA that used categories of normal or abnormal in their scales These data were analyzed using nonparametric techniques.

Criterion-related validity: individual judgment models and accuracy. The relationships between the recorded ratings and the associated criterion measure data were plotted for each observer, and we visually examined the plots to confirm the linearity of the relationships. Each scatterplot and regression equation, therefore, reflected a unique model of judgment. (8) To investigate accuracy, individual Pearson product moment correlations were calculated using each observer's scores of push-off and the measurements of ankle power generation. These correlations were then transformed to Z scores using the Fisher r to Z transformation, averaged, then converted back to Pearson r indexes to provide a group mean correlation value for test 1.

The precision of the rating scale was investigated by calculating the standard error of estimating the peak ankle power generation ([S.sub.est]). This was determined for each of the observers, using the test 1 results. The [S.sub.est] was calculated, according to the following formula. (37)

(2) [S.sub.est] = SD [square root of 1 - [r.sub.xy.sup.2]]

where:

SD = the standard deviation of the 11 subjects' measurements of ankle power generation and [r.sub.xy] = the Pearson product moment correlation value for test 1 for each observer

These calculations are in the same units (watts per kilogram) as the criterion measure, thus providing a direct method of interpreting the accuracy of the observers' use of the 22-point rating scale. The [S.sub.est] quantifies the amount of judgment error (in watts per kilogram) for each observer using the rating scale.

The discrimination between normal and abnormal push-off. The categorization of each walk as abnormal or normal was inspected in relation to the associated measurements of ankle power generation. This inspection provided us with insight into the judgment patterns of the observer group. Each of the 18 judgment models was explored to determine the range of measurements of ankle power generation at which observers divided the gait performances into each of the abnormal or normal categories. Regression equations from each judgment model enabled prediction of this division or cutoff point, which reflected the intersection point of the 2 scales. This point was determined in the following manner. A value of 10.5 was selected to substitute into each regression equation as the "x" rating value. This value represented a midline point that may arbitrarily separate the abnormal scale range of points (0-10) and the normal scale range of points (11-21). From this "x" midline point, a corresponding "y" criterion measure value could be predicted, representing the threshold value of the criterion measure at which each observer's judgement model divided the gait performances as normal or abnormal. An example of this prediction process is illustrated in Figure 2. Values for these predicted thresholds of normality can be compared with the range of ankle power generation values attained by older subjects without impairments.

[FIGURE 2 OMITTED]

Interobserver reliability. Intraclass correlation coefficients (ICC [2,1]) were selected to investigate the agreement among observers on both test occasions. (38) Average ICCs across the 2 tests were determined by converting the ICCs of each test occasion to Z scores using the Fisher r to Z transformation, averaging the Z scores, and converting the average back to ICCs. The standard error of measurement (SEM) for the ratings of push-off among the observers also was determined for each test occasion according to the following formula. (39) In this instance, an ICC (2,1) value was used in the calculation.

(3) SEM = SD [square root of 1 - [r.sub.xx]]

where SD = standard deviation of the rating scale scores recorded by observers on a test occasion and [r.sub.xx] = the ICC value (among observers).

The SEM is provided in the same units as the 22-point observational rating scale, thus providing a direct method of interpreting the reliability of the scale. In this instance, the SEM reflects how much error there was across the observer group when rating the gait performances.

Interobserver reliability of the categorical data was investigated using the Cohen Kappa statistic. Kappa is a corrected measure of agreement between observers that takes into account the effect of chance agreement. (40) A value of 1 represents perfect agreement, and a value of 0 indicates agreement that is no better than would be expected by chance. Kappa was calculated for each pair of observers for test 1. Kappa values were transformed to Z scores, averaged, and converted back to Kappa values to provide an average value.

Introobserver reliability. An ICC (2,1) also was calculated to determine the agreement consistency of each observer across the 2 test occasions. Individual ICC (2,1) values were determined for each observer and then averaged by the previously described method. The SEM was calculated for each clinician to reflect the error involved when a single observer repeated the rating on 2 occasions. (39) The SEM for each observer then was calculated and averaged by the method described previously, using the standard deviation of the ratings of individual observers (averaged over test 1 and test 2) and the individual observers' ICC (2,1) values. Percentage of agreement and the Cohen Kappa statistic also were determined to evaluate the agreement of the categorical data.

Experience. The relationship between the observers' length of experience and individual values of accuracy and intraobserver reliability was investigated by calculation of Pearson product moment correlation coefficients. Prior to calculating the correlation, the individual ICC (2,1) or Pearson product moment values were again transformed to Z scores, and the data were inspected to confirm a linear relationship. The relationship between experience and interobserver reliability also was investigated by selecting the 5 most experienced therapists (mean experience=8.3 years) and the 5 least experienced therapists (mean experience=2.8 years) and then comparing these 2 groups. Intraclass correlation coefficients (2,1) were obtained for the "most experienced" and "least experienced" groups by the method previously described, and confidence intervals around these values were determined.

Results

Use of the Observational Rating Scale

In view of the development of the new 2-part observational rating scale, the continuity of the underlying single scale was examined by investigating the relationship between the observers' ratings and the associated criterion measure values. The observer group used all numbers of the 22-point scale to assign ratings to the 11 gait performances. Because the actual ankle power generation values of the subjects allocated to each scale number were known, the mean ankle power generation of subjects ranked at each rating scale number could be determined. Further examination of this relationship by simple regression analysis (Fig. 3) revealed a linear relationship with a Pearson product moment correlation coefficient of .94, suggesting a strong association between the rating scale and the mean ankle power generation of the subjects rated at each number. Greater variability in this relationship was evident at the higher rating scale numbers; however, these data were based on fewer subjects, and no extreme deviations were apparent. The interval between numbers 10 and 11 did not appear to be different from the other intervals across the scale. This finding suggests that dividing the rating scale into the two 11-point (0-10) scales did not influence the continuity of the underlying single variable.

[FIGURE 3 OMITTED]

Therapist Accuracy: Criterion-Related Validity

Inspection of the 18 judgment scatterplots revealed linear relationships, with Pearson product moment correlation coefficients varying from .69 to .91 and an average correlation of .84 for both test occasions (Tab. 3). These correlation coefficients represented fair to high values. Figure 4 illustrates the relationship between the rating scale scores and the criterion measure values for 2 observers: (1) the least accurate and (2) the most accurate. The [S.sub.est] varied from 0.38 to 0.67 W/kg, with a group mean of 0.51 W/kg.

[FIGURE 4 OMITTED]

Discrimination of Normality: Regression Analysis

Figure 5 illustrates the pattern of discrimination used by the observers to categorize the gait performances as abnormal or normal. Inspection of the raw scores shows a discrimination of normality between the values of 1.00 W/kg and 1.23 W/kg. Performances of subjects with criterion measure values of [less than or equal to] 1.0 W/kg were rated by all observers as abnormal. At higher values, there was a progressive trend to rate performances as normal. This trend indicates a systematic scoring process relative to power generation.

[FIGURE 5 OMITTED]

Further insight into this discriminative process can be gained by closer inspection of the individual judgment models. Table 3 contains the individual predicted threshold of normality values for each therapist during test 1, as predicted by the regression analysis. These values indicate that the therapists varied in the point of the criterion range at which they separated normal performance from abnormal performance, from 1.21 W/kg to 3.02 W/kg, with a mean value of 1.80 W/kg.

Interobserver Reliability

Interobserver agreement was moderately high for both test occasions, with ICC (2,1) values ranging between .75 and .77 (Tab. 4). The average SEM (between all observers) was reasonably consistent across observers, at around 2.7 rating scale units.

Prior to determining Kappa values, the categorical data were inspected to ensure that variability was present in each observer's ratings, as a requirement for Kappa calculations. Because observers 1 and 5 consistently rated all subjects' performances as abnormal, data from these subjects were excluded from this analysis. Kappa coefficients varied from .23 to 1.00, with an average of .66, representing substantial agreement among the 16 observers, according to the classification of Landis and Koch. (41)

Intraobserver Reliability

Observers were consistent in their individual use of the rating scale, with a mean ICC (2,1) of .89 obtained (Tab. 5). Variability was apparent, with ICC (2,1) values ranging from .64 to .96. The SEM also varied across observers, with a relatively low average of less than 2 rating scale units (Tab. 5). Consistency of the categorical judgment of normal versus abnormal across the 2 tests was similarly high, with average agreement of 89.5%. An averaged Kappa value of .79 for the observers (Tab. 5) indicated substantial to almost perfect agreement of categorical judgment. (41)

Experience

No relationships existed between the observers' general or neurology-specific experience and their accuracy or intraobserver reliability indexes. Experience was poorly associated with interobserver reliability, with the least experienced group showing higher agreement than the most experienced group (ICCs [2,1] of .88 and .82, respectively). Inspection of confidence intervals (CIs) around each value (least experienced group's CIs=.59-.97; most experienced group's CIs=.49-.95) confirmed that no difference occurred between these 2 groups.

Discussion

Accuracy of Therapists' Observational Judgments

The high correlations between the observational ratings and the criterion measure values indicate a strong relationship between the subjects' ankle power generation, as measured by the motion analysis system, and push-off, as observed by the therapists. This positive linear relationship (Fig. 3) suggests that therapists view both abnormal and normal push-off along a continuum. This finding is consistent with data describing therapists' observations of upper-limb movement after stroke. (8)

The indexes of accuracy of the observational ratings in this study were high. We believe that this is an important result because clinicians report including push-off as a component of OGA when observing gait. (42) The importance of the therapists' individual accuracy coefficients in relation to clinical practice is difficult to define. It cannot be stated with certainty what values of criterion-related validity are acceptable for such a clinical measure to be used with confidence. This study, therefore, provides evidence that therapists are able to accurately discriminate between levels of ankle power generation in gait after stroke, when observing videotaped gait performances.

These high correlations between the observational ratings and the criterion measure values are encouraging. We contend that further insight into observational accuracy can be achieved by examining the metric estimates of the accuracy of the therapists' ratings. These estimates provide an indication of whether therapists are able to detect what we would consider clinically meaningful change. The mean standard error of estimating the criterion measure values was found to be 0.51 W/kg, compared with the range of 3.16 W/kg in the sample. This means that in using the rating scale, therapists could be 68% confident that an individual rating score would fall plus or minus 0.51 W/kg from the true criterion value. A 95% CI would result from approximately 2 times this range, or plus or minus 1.02 W/kg.

The mean error range of 1.02 W/kg is large, covering approximately one third of the range of the criterion measure. This error range implies that therapists would only be able to accurately discriminate changes of approximately 1 W/kg. Therapists frequently use measures to quantify changes in push-off across time, thereby evaluating a series of small changes at intervals during rehabilitation. Unfortunately, there are no group data for sequential measurements of ankle power generation after stroke, and the amount of change in ankle power generation that occurs during the rehabilitation phase is unknown. It is impossible, therefore, to know how precise or accurate therapists' observational measures need to be. Further research is needed to quantify the timing and amount of change in ankle power generation after stroke.

This typical error size of 0.51 W/kg can be considered in relation to the ability to discriminate between older people without known impairments and people following stroke. Olney et al (24) reported the mean maximum ankle power generation of a group of 30 subjects following stroke as 0.60 W/kg (SD=0.51). This value is substantially different from the reported mean of 2.48 W/kg (SD=0.46) of elderly subjects without known impairments. (21) This typical error of estimation of ankle power generation, in our opinion, would allow the use of observational judgments to differentiate among subjects with performance values near to the means of these groups.

Further insight into clinical judgment can be gained by consideration of the predicted normality values. Although therapists commonly decide whether a movement is normal, it is not known how this judgment is made. In the literature, it is common to find individual subject performance measures compared with an appropriate normative reference groups using a statistical framework. For example, gait speed may be considered abnormal when it is beyond 1.65 standard deviations of the mean of a reference group. (43) Similarly, the measurements of ankle power generation following stroke in our study can be compared with those obtained from older subjects without impairments. The most appropriate reference group is that described by Winter, (21) who collected gait data from 18 older subjects without impairments in an identical manner to that of the collection of gait data in our study. This provides what we consider the most valid comparable normative reference group.

An estimate of the fifth percentile of this group's performance indicates that 95% of these subjects obtained ankle power generation values of 1.72 W/kg or greater. These values lie within the range of predicted normality values determined from the therapist judgment models and are remarkably close to the average of 1.8 W/kg. This finding indicates that, on average, the group of 18 therapists were placing the cutoff defining normality at around the fifth percentile mark.

Reliability of the Therapists' Observational Judgments

The indexes of agreement achieved in this study indicate that the therapists were able to make highly stable individual agreements about observation of a single kinetic gait variable on 2 occasions when viewing videotaped gait performances. These observers also demonstrated moderately consistent agreement with each other about the push-off ability of the subjects during gait.

In addition to measures of agreement, metric indexes of error are useful to provide a meaningful indication of the size of measurement error in units of the rating scale. The SEM (within a single observer) was relatively low, at around 1.8 units of the 22-point scale. This value allows estimation of the error size that could be expected when an observation is repeated by the same observer on another occasion. In this instance, a change of plus or minus 3.6 units would be required to be 95% confident that change had occurred beyond that potentially due to measurement error. The average SEM (among the group of observers) was larger, at around 2.5 units. This finding means that change of around 5 units would be necessary to demonstrate change beyond measurement error if an observation were repeated by a group of observers. These error values provide a framework based on measurement error against which true change can be evaluated. Because the magnitude of typical change in ankle power generation in recovery after stroke is not known, the clinical significance of these error sizes cannot easily be determined. However, relative to the 22-point scale, these error sizes appear to be small.

The higher reliability values for individual intraobserver reliability compared with interobserver reliability were expected. It is more likely that an individual observer will agree more consistently with himself or herself than with other therapists. This finding is consistent with previous investigations of OGA data reliability (15,16) and other studies of movement observation. (7,8) In current clinical practice, patients are commonly assigned to an individual therapist, who is responsible for their treatment for the duration of intervention. The findings of our study confirm that this OGA measurement is more consistent and has lower error when repeated by the same therapist, as compared with OGA measurements obtained by multiple therapists. This finding suggests that in clinical practice an individual therapist should endeavor to be responsible for OGA measures in a single patient across multiple sessions whenever possible. This practice should enhance reliability and minimize between-observer error.

Experience

The poor relationship between therapist experience and intraobserver reliability we demonstrated is comparable to that reported by Eastlack et al. (14) Furthermore, we showed poor correlations between the obtained accuracy indexes and the experience of the therapists. More experienced therapists were not more accurate and did not obtain more reliable measurements than therapists with less experience. In addition, specialized neurology-specific experience did not enhance either reliability or accuracy. This finding contrasts with the belief that clinicians with more specialized experience are likely to be more accurate and to obtain more reliable measurements. These poor correlations strongly suggest that experience is not a factor that influences accuracy or reliability of measurements obtained for this rating task. We examined single observations only, and we did not examine the way in which these observations are interpreted or how the information is used. It is possible that expert-novice differences may lie more in the complex decisions arising from observations, rather than the observations themselves.

Methodological Considerations

The reliability and accuracy indexes obtained in this study were higher than the majority of those previously reported in the OGA literature (Tab. 1). Our higher values may relate to the method used and the design chosen. Provision of an operational definition, the selection of a single variable, and the simple scoring system allowing discriminating judgments are likely to have enhanced the accuracy of the observers and the reliability of their measurements. Our inclusion of these design features and the favorable results achieved support the premise that observers can be accurate and can obtain reliable measurements if the task is structured with specific guidelines.

The selection of a single videotaped stride as a rating stimulus may have enhanced the achieved reliability by reducing error due to patient variability. Live gait performance raises a strong question of performance inconsistency between gait trials, which is potentially accentuated by fatigue. The poor reliability reported by authors such as Miyazaki and Kubota (3) and Goodkin and Diller (10) thus may have been influenced by the inclusion of live rating sessions. Designs including videotaped performances have attempted to control performance variability, but reliability has not necessarily improved. (5,12,14,16) Such studies, however, have included subjects who walked a number of strides across varied distances over a number of trials. In such instances, videotapes are likely to have controlled the variation in gait performance due to subject fatigue. However, the videotapes, in our opinion, may not have controlled variability due to stride-to-stride or trial-to-trial gait fluctuations, which may have contributed to increased error in rating scores when combined with inadequately defined rating instructions. For example, one observer may observe the best, worst, or a random stride performance for a single variable and then rate the stride performance accordingly. A second observer may seek to rate an average stride after viewing a number of strides. Thus, unless instructions are very specific, error in rating due to stimulus variability is likely and may result in lower reliability. All previous OGA studies have included a number of strides and thus may have been subject to this type of increased rating error.

The higher accuracy and reliability achieved with a wider 22-point scale in our study counters the contention that discriminative grading of OGA is very demanding and that use of broader scales may adversely influence reliability. (2,5) Other studies of movement observation have described successful use of similarly discriminating scales, with comparable reliability outcomes. (7,8) An expanded scale with clear descriptors may be easier to use than a narrow scale with descriptors relating to ill-defined categories of abnormality, such as "noticeably," "moderate," or "severe." In our study, observers were able to demonstrate discrimination of rating that is more comparable to the refined judgments of movement quality that may be necessary in clinical settings.

In nearly all studies of OGA, multiple variables were rated, with up to 50 variables considered by the observer. (5) Furthermore, frequently cited OGA tools, such as the Ranchos Los Amigos Medical Center form, (20) demand that multiple joints be assessed over the many phases of the gait cycle, with in excess of 100 decisions to be made. We contend that with a large number of variables to be judged, the greater are the attentional demands on the observer. As the number of demands, or attention required, increases, the potential for error and inconsistency may also increase. It is not known how observers rated multiple variables in previous reliability studies. In clinical practice, it is suggested that a systematic approach to observation should be adopted, with observers attending to single variables at a time. (29,44) When observation was structured in this manner in Bernhardt and colleagues' study of 3 kinematic tasks, high accuracy and reliability were demonstrated. (8) The results of our study and those of the study by Bernhardt et al (8) suggest that it may be desirable for observers to attend to single variables sequentially when observing gait. Use of a videotape for analysis may augment the reliability of data obtained with this decision process. Indeed, videotaping of performance has frequently been advocated as a method to optimize movement analysis. (1,29,44)

Our study represents an initial step in the exploration of the accuracy of observations of a single variable in OGA by use of videotape. We believe the method we used enhanced observer accuracy, but also limits the immediate generalizability. The findings indicate that therapists are able to make accurate and reliable judgments of a single kinetic gait variable when viewing videotaped gait performances in a quiet environment. This finding confirms that therapists may be able to provide reliable and accurate decisions about observed movement when the task is clearly defined and focused on a single judgment and their attention is focused on preselected videotaped segments. The findings are limited to the context of this study and are not able to be immediately translated into clinical practice, where patients are observed directly and multiple variables are considered. These favorable results, however, establish that further research directed into clinical observation is appropriate. In view of the fundamental role of OGA within clinical decision making, it is important that further research define and direct the clinical utility of such observations.

Considerations for Future Research

The widespread continuing use of OGA to assess gait after stroke suggests that clinicians value information gained from these observations. Despite the emergence and emphasis of other measures such as gait speed, clinicians continue to observe their patients walking and focus on these "qualitative" aspects of gait. We therefore believe that research should focus on exploring and defining the potential value of such observations. Key questions for clinical practice have emerged from this and other studies of movement observation after stroke. (8)

Conclusion

Observational gait analysis is currently used on a daily basis by therapists because of its easy application and the lack of easily accessible instrumented measurement tools. (1,8,29,44) We believe OGA has an important role in clinical decision making about gait training and in evaluation of progress after stroke. The therapists in this study, under specific standardized conditions, were able to accurately infer the impaired ankle power generation after stroke from videotaped gait performances. The high accuracy and moderate to high reliability indexes obtained in this study indicate that further investigation of the measurement properties of OGA is warranted. Despite these favorable findings, further work is needed to generalize these results directly to the clinical situation. Further research is warranted to explore the ability of therapists to judge further kinetic aspects of gait and to define the circumstances that enhance the accuracy of this judgment.

Table 1.
Characteristics of Studies of Validity (Accuracy) or Reliability of
Measurements Obtained With Observational Gait Analysis (OGA) (a)

Study                  Study Type                     Observers

Goodkin and            Reliability (interobserver)    PT (n =7)
  Diller, (10)
  1973

De Bruin et al, (11)   Reliability (interobserver,    Medical (n=6)
  1982                   intraobserver)
                       Validity

Miyazaki and           Reliability (interobserver)    PT (n=1)
  Kubota, (3)          Validity                       PT student (n= 1)
  1984                                                Medical (n=2)

Krebs et al, (12)      Reliability (interobserver,    PT (n=3)
  1985                   intraobserver)

Saleh and              Validity                       Varied (medical,
  Murdoch, (4)                                          bioengineers,
  1985                                                  prosthetists,
                                                        PT) (n=not
                                                        stated)

Patla and              Reliability (interobserver)    Kinesiology
  Clouse, (5)          Validity                         students
  1988                                                  (n=5)

Eastlack et al, (14)   Reliability (interobserver)    PT (n=54)
  1991

Spencer et al, (6)     Validity                       PT (n=20)
  1992

Potter et al, (13)     Reliability (interobserver)    PT (n=8)
  1993

Hughes and             Reliability (interobserver,    PT (n=3)
  Bell, (15) 1994        intraobserver)
                       Validity of total OGA
                         change scores

Keenan and             Reliability (interobserver,    Podiatrists (n=5)
  Bach, (16) 1996        intraobserver)

Riley et al (17)       Validity                       PT (n=not
  1996                                                  stated)

Lord et al, (18)       Reliability (interobserver,    PT (n= 1-7)
  1998                   intraobserver)
                       Validity of total OGA
                         scores

                       Experience
                       of
Study                  Observers      Subject Type

Goodkin and            Not stated     Hemiplegia (n= 10)
  Diller, (10)
  1973

De Bruin et al, (11)   Not stated     Children with cerebral
  1982                                  palsy (n= 18)

Miyazaki and           0-20 y         Hemiplegia following
  Kubota, (3)                           stroke (n=48)
  1984

Krebs et al, (12)      >5 y highly    Children with orthoses
  1985                   trained        (n = 15)

Saleh and              Not stated     Amputations (n=5)
  Murdoch, (4)
  1985

Patla and              Varied         Mixed disorders
  Clouse, (5)                           (n=8)
  1988

Eastlack et al, (14)   Varied:        Rheumatoid arthritis
  1991                   1-20 y         (n = 3)

Spencer et al, (6)     Not reported   Hemiplegia following
  1992                                  stroke (n = 14)

Potter et al, (13)     >1 y           Hemiplegia (n= 15)
  1993

Hughes and             Not stated     Hemiplegia following
  Bell, (15) 1994                       stroke (n=6)

Keenan and             4-10 y         Adults without
  Bach, (16) 1996                       impairments
                                        (n=14)
Riley et al (17)       Not stated     Parkinson disease
  1996                                  (n= 10)
                                      Hemiplegia (n=10)
Lord et al, (18)       1-15 y         Neurological,
  1998                                  including multiple
                                        sclerosis and stroke
                                        (n=8-27)

                                                        Type of Gait
                                      Rating Scale      Variables
Study                  Observations   Construction      Rated

Goodkin and            Live           Ordinal 3-point   Kinematic
  Diller, (10)                                          Spatiotemporal
  1973                                                  Kinetic

De Bruin et al, (11)   Videotape      Ordinal 3-point   Kinematic
  1982

Miyazaki and           Live           Ordinal 4-point   Kinematic
  Kubota, (3)                                           Spatiotemporal
  1984                                                  Kinetic

Krebs et al, (12)      Videotape      Multicomponent    Kinematic
  1985                                  OGA form        Spatiotemporal
                                      Ordinal 3-point

Saleh and              Live           Nominal 2-point   Kinematic
  Murdoch, (4)                                          Spatiotemporal
  1985

Patla and              Videotape      Nominal 2-point   Kinematic
  Clouse, (5)                                           Spatiotemporal
  1988                                                  Kinetic
                                                        Others

Eastlack et al, (14)   Videotape      Ordinal 3-point   Kinematic
  1991                                                  Spatiotemporal

Spencer et al, (6)     Videotape      Ordinal 5-point   Temporal
  1992

Potter et al, (13)     Videotape      Ordinal 4-point   Kinematic
  1993

Hughes and             Videotape      Multicomponent    Kinematic
  Bell, (15) 1994                       hemiplegic      Spatiotemporal
                                        gait OGA
                                        form
                                      Ordinal 3-point

Keenan and             Videotape      Ordinal 3-point   Kinematic
  Bach, (16) 1996                                       Temporal

Riley et al (17)       Live           Ordinal 3-point   Spatiotemporal
  1996

Lord et al, (18)       Live           Multi component   Kinematic
  1998                                  OGA form
                                      Ordinal 4-to
                                        7-point

Study                  Statistical Analysis

Goodkin and            Agreement
  Diller, (10)
  1973

De Bruin et al, (11)   Reliability: not reported
  1982                 Validity: regression analysis

Miyazaki and           Pearson r
  Kubota, (3)
  1984

Krebs et al, (12)      Percentage of agreement,
  1985                   ICC, Pearson r

Saleh and              Percentage of agreement
  Murdoch, (4)
  1985

Patla and              Occurrence/nonoccurrence
  Clouse, (5)            reliability coefficient
  1988

Eastlack et al, (14)   Kappa, ICC
  1991

Spencer et al, (6)     Pearson r
  1992

Potter et al, (13)     ICC
  1993

Hughes and             Reliability: Kendall
  Bell, (15) 1994        coefficient of
                         concordance
                       Validity: Spearman rho

Keenan and             Kappa
  Bach, (16) 1996

Riley et al (17)       Simple agreement
  1996

Lord et al, (18)       Reliability: percentage of
  1998                   agreement, Kendall
                         coefficient of
                         concordance
                       Validity: Spearman rho

Study                  Results

Goodkin and            Average: 42 of maximum of
  Diller, (10)           51 agreements
  1973

De Bruin et al, (11)   Incomplete detail
  1982                 Conclusion: unreliable

Miyazaki and           Reliability: r=.28-.63
  Kubota, (3)          Validity: r=.32 to .-72
  1984

Krebs et al, (12)      Interobserver: average
  1985                   67.5% agreement,
                         average ICC=.73
                       Intraobserver: average: 69%
                         agreement, average r=.60

Saleh and              22% agreement with
  Murdoch, (4)           predicted gait deviations
  1985

Patla and              Reliability: variable across
  Clouse, (5)            characteristics
  1988                 Validity: variable across
                         characteristics
                       Average: 1-2 of possible
                         agreement score of 5

Eastlack et al, (14)   Kappa=.11-.52.
  1991                 ICC =. 19-.69

Spencer et al, (6)     Mean r=.88
  1992

Potter et al, (13)     Mean ICC=.63
  1993

Hughes and             Intraobserver: variable across
  Bell, (15) 1994        form sections: W=39-.97
                       Interobserver: variable across
                         form sections: W=.71-.90
                       Validity: variable across form
                         section and criterion:
                         rho=-.37 to .94

Keenan and             Interobserver: Kappa=.19
  Bach, (16) 1996      Intraobserver: Kappa=-.12
                         to .59

Riley et al (17)       Agreement range=2-9 of
  1996                   possible 10

Lord et al, (18)       Interobserver: 63.8%
  1998                   agreement, W=.84
                       Validity (total scores):
                         variable across pathology
                         and criterion: rho=-.24
                         to -.77

(a) PT = physical therapist, ICC: intraclass correlation coefficient.

Table 2.
Characteristics of Subjects After Stroke and Gait Characteristics (a)

                                               Gait Aid/
Subject   Age         Years Since   Affected   Assistive
No.       (y)   Sex   Stroke        Side       Devices (b)

1         68    M     0.50          L          SPS, AFO
2         65    M     0.25          R          SPS, AFO
3         61    F     0.75          R          SPS
4         66    M     N/A           R
5         57    M     N/A           L          SPS
6         24    F     0.25          R
7         73    M     0.75          L
8         63    M     0.75          L
9         66    M     0.50          R
10        55    F     1.50          R
11        66    F     0.75          R

                       Maximum Ankle
Subject   Gait Speed   Power Generation
No.       (m/s)        (W/kg)

1         0.30         0.01
2         0.21         0.19
3         0.31         0.40
4         0.43         0.59
5         0.56         0.82
6         0.51         1.00
7         0.70         1.23
8         0.65         1.43
9         0.52         1.75
10        0.66         1.95
11        1.02         3.17

(a) N/A=data not available, M=male, F=female, L=left, R=right.

(b) SPS=single-point stick, AFO=ankle-foot orthosis.

Table 3.
Accuracy Findings From Test 1: Pearson Product Moment Correlations
Between Observations of Push-Off and Ankle Power Generation, Standard
Error of the Estimate of the Criterion Measure, and the Predicted
Threshold of Normality Values

                      Standard Error   Predicted Threshold
Therapist   Test 1    of Estimate of   of Normality
No.         r         Measure (W/kg)   Scares (W/kg)

1           0.69      0.67             3.02
2           0.89      0.42             1.58
3           0.85      0.48             1.71
4           0.83      0.51             1.58
5           0.84      0.50             2.45
6           0.83      0.51             1.44
7           0.69      0.67             1.95
8           0.89      0.42             1.67
9           0.82      0.53             1.61
10          0.81      0.54             1.21
11          0.88      0.44             2.24
12          0.87      0.45             2.13
11          0.83      0.51             1.27
14          0.74      0.62             1.57
15          0.91      0.38             1.98
16          0.85      0.48             1.78
17          0.78      0.58             1.48
18          0.89      0.42             1.64
Average     0.84      0.51             1.80
Range       .69-.91   0.38-0.67        1.21-3.02

Table 4.
Mean Intraclass Correlation Coefficients (ICC [2,1]) for Interobserver
Reliability of Observations of Push-off and Mean Standard Error of
Measurement (SEM) for Tests 1 and 2

                         Test 1   Test 2   Average

Mean ICC                  .75      .77      .76
Mean SEM
  (rating scale units)   2.70     2.65     2.68

Table 5.
Intraobserver Reliability of Observations of Push-Off (a)

                            [bar]X     Range

ICC (2,1)                     .89      .64-.96
Agreement (%)               89.5        64-100
Kappa                         .79      .21-1.00
SEM (rating scale units)    1.79      1.10-3.20

(a) ICC=intraclass correlation coefficient, SEM=standard error
of measurement.

Figure 1.
Rating scale provided for observer judgments. Observers were provided
with the following description of the rating scale anchors: On the
abnormal scale, a rating of 0 represents a person with no push-off. A
rating of 10 on this scale represents someone who has only marginal or
"just abnormal" ankle power generation (ie, the upper limit of
abnormal). The normal scale represents the range of normal variability
reflected within an age-matched population (ie, what you would consider
to be within the normal range of push. off for a person at this age).
A rating of 0 on this scale represents someone who can "just" be
described as having normal push-off (ie, the lower limit of what you
would consider to be normal). A rating of 10 on this scale represents
the upper limits of normal push-off during walking in a person without
known impairments or pathology.

                        ABNORMAL

   0       1   2   3   4   5   6   7   8   9      10
  No                                             Just
Push-off                                       Abnormal

                         NORMAL

   0     1   2   3   4   5   6   7   8   9      10
 Just                                          Upper
Normal                                       Limit of
                                              Normal

* Redlake Corp, 1711 Dell Ave, Campbell, CA 95008.

([dagger]) Advanced Medical Technology Inc, 141 California St, Newton, MA 02158.

([double dagger]) GTCO Carp, 1055 First St, Rockville, MD 20850.

References

(1) Carr J, Shepherd R. Neurological Rehabilitation: Optimising Motor Performance. Oxford, England: Butterworth Heinemann; 1998.

(2) Malouin F. Observational gait analysis. In: Craik RL, Oatis CA, eds. Gait Analysis: Theory and Applications. St Louis, Mo: Mosby; 1995: 112-124.

(3) Miyazaki S, Kubota T. Quantification of gait abnormalities on the basis of a continuous foot-force measurement: correlation between quantitative indices and visual rating. Medical & Biological Engineering & Computing. 1984;22:70-76.

(4) Saleh M, Murdoch G. In defense of gait analysis. J Bone Joint Surg Br. 1985;67:237-241.

(5) Patla A, Clouse S. Visual assessment of human gait: reliability and validity. Rehabilitation Research. October 1988:87-96.

(6) Spencer K, Goldie PA, Matyas TA. Criterion-related validity of visual assessment of the temporal symmetry of hemiplegic gait. In: Proceedings of the Australian Physiotherapy Association National Congress, 1992, Adelaide, Australia. Melbourne, Victoria, Australia: Australian Physiotherapy Association; 1992:68.

(7) Haupstein T, Goldie PA. Visual judgements of steadiness in one-legged stance: reliability and validity. Physiotherapy Research International. 2000;5:137-152.

(8) Bernhardt J, Bate PJ, Matyas TA. Accuracy of observational kinematic assessment of upper-limb movements. Phys Ther. 1998;78:259-270.

(9) Bernhardt J, Bate PJ, Matyas TA. Training novice clinicians improves observation accuracy of the upper extremity after stroke. Arch Phys Med Rehabil. 2001;82:1611-1618.

(10) Goodkin R, Diller L. Reliability among physical therapists in diagnosis and treatment of gait deviations in hemiplegics. Perceptual and Motor Skills. 1973;37:727-734.

(11) De Bruin H, Russell D, Latter J, Sadler J. Angle-angle diagrams in monitoring gait and quantification of gait patterns for children with cerebral palsy. Am J Phys Med. 1982;61:176-192.

(12) Krebs DE, Edelstein JE, Fishman S. Reliability of observational kinematic gait analysis. Phys Ther. 1985;65:1027-1033.

(13) Potter K, Rheault W, Pucci D. Interrater reliability of videotaped observational gait analysis in hemiplegia [abstract]. Phys Ther. 1993;73: S46.

(14) Eastlack M, Arvidson J, Snyder-Mackler L, et al. Interrater reliability of videotaped observational gait analysis assessments. Phys Ther. 1991; 71:465-472.

(15) Hughes K, Bell F. Visual assessment of hemiplegic gait following stroke: a pilot study. Arch Phys Med Rehabil. 1994;75:1100-1107.

(16) Keenan AM, Bach TM. Video assessment of rearfoot movements during walking: a reliability study. Arch Phys Med Rehabil. 1996;77: 651-655.

(17) Riley M, Goodman M, Fritz V. A comparison between observational analysis and temporal distance measurements. South African Journal of Physiotherapy. 1996;52:27-30.

(18) Lord S, Halligan P, Wade D. Visual gait analysis: the development of a clinical assessment and scale. Clin Rehabil. 1998;12:107-119.

(19) Brunnstrom S. Recording gait patterns of adult hemiplegic patients. Phys Ther. 1964;44:11-18.

(20) Ranchos Los Amigos Medical Center. Observational Gait Analysis Handbook. Downey, Calif: The Professional Staff Association; 1989.

(21) Winter D. Biomechanics and Motor Control of Human Gait. Waterloo, Ontario, Canada: University of Waterloo Press; 1991.

(22) Judge J, Davis R, Ouupuu S. Step length reductions in advanced age: the role of ankle and hip kinetics. J Gerontol A Biol Sci Med Sci. 1996;51:M303-M312.

(23) Kerrigan D, Todd M, Della Croce U, et al Biomechanical gait alterations independent of speed in the healthy elderly: evidence for specific limiting impairments. Arch Phys Med Rehabil. 1998;79:317-322.

(24) Olney S, Griffin M, Monga T, McBride I. Work and power in stroke gait. Arch Phys Med Rehabil. 1991;72:309-314.

(25) Kerrigan D, Karvosky M, Riley P. Spastic paretic stiff-legged gait joint kinetics. Am J Phys Med Rehabil. 2001;80:244-249.

(26) Herbert R, Moore S, Moseley A, et al. Making inferences about muscle forces from clinical observations. Australian Journal of Physiotherapy. 1993;39:195-202.

(27) Richards C, Malouin F, Dumas F, Tardif D. Gait velocity as an outcome measure of locomotor recovery after stroke. In: Craik RL, Oatis CA, eds. Gait Analysis: Theory and Application. St Louis, Mo: Mosby; 1995:355-364.

(28) Knudson D, Morrison C. Qualitative Analysis of Human Movement. Champaign, Ill: Human Kinetics Inc; 1997.

(29) Perry J. Gait Analysis: Normal and Pathological Function. Thorofare, NJ: Slack Inc; 1992.

(30) Allingame S, Williams T, Jenkins S, Tucker B. Accuracy and reliability of physiotherapists in the interpretation of tape-recorded lung sounds. Australian Journal of Physiotherapy. 1995;41:179-184.

(31) Brooks D, Thomas J. Interrater reliability of auscultation of breath sounds among physical therapists. Phys Ther. 1995;75:1082-1088.

(32) Cipriany-Dacko L, Innerst D, Johannsen J, Rude V. Interrater reliability of the Tinetti balance scores in novice and experienced physical therapy clinicians. Arch Phys Med Rehabil. 1997;78:1160-1164.

(33) Mann M, Glasheen-Wray M, Nyberg R. Therapist agreement for palpation and observation of iliac crest heights. Phys Ther. 1984;64: 334-338.

(34) Craik RL, Dutterer L. Spatial and temporal characteristics of foot fall patterns. In: Craik RL, Oatis CA, eds. Gait Analysis: Theory and Application. St Louis, Mo: Mosby; 1995:143-158.

(35) Dempster W. Space Requirements of the Seated Operator: Geometrical, Kinematic and Mechanical Aspects of Body With Special Reference to Limbs. Ohio: Wright-Patterson Air Force Base; 1955.

(36) Poulton E. Bias in Quantifying Judgements. London, England: Lawrence Erlbaum Associates Ltd; 1989.

(37) Guilford J, Fruchtor B. Fundamental Statistics in Psychology and Education. New York, NY: McGraw-Hill Inc; 1978.

(38) Shrout P, Fleiss J. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-428.

(39) Matyas TA, Bach TM. The reliability of selected techniques in clinical anthropometrics. Australian Journal of Physiotherapy. 1985;31: 175-199.

(40) Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;1:37-46.

(41) Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.

(42) Patla A, Proctor J, Morson B. Observation of aspects of visual gait assessment: a questionnaire study. Physiotherapy Canada. 1987;39: 311-316.

(43) Goldie PA, Matyas T, Evans O. Deficit and change in gait velocity during rehabilitation after stroke. Arch Phys Med Rehabil. 1996;10: 1074-1082.

(44) Whittle M. Gait Analysis: An Introduction. Boston, Mass: Butterworth-Heinemann Inc; 1991.

JL McGinley, PT, BAppSc (PT), is a doctoral student in the School of Physiotherapy, La Trobe University, Victoria, Australia, and Senior Research Physical Therapist, Geriatric Research Unit, Kingston Centre Rehabilitation and Aged Services Program, Southern Health, Melbourne, Australia. Address all correspondence to Ms McGinley at Geriatric Research Unit, Kingston Centre, Warrigal Rd, Cheltenham, Victoria, Australia 3192 ([e-mail]).

PA Goldie, PT, PhD, is Associate Professor, School of Physiotherapy, La Trobe University.

KM Greenwood, PhD, is Associate Professor, School of Psychological Science, La Trobe University.

SJ Olney, PhD, is Professor and Director, School of Rehabilitation Therapy, and Associate Dean (Health Sciences), Queen's University, Kingston, Ontario, Canada.

Ms McGinley, Dr Goldie, and Dr Greenwood provided concept/research design, writing, data analysis, and project management. Ms McGinley and Dr Olney provided data collection and subjects. Dr Goldie and Dr Olney provided institutional liaisons. Dr Goldie, Dr Greenwood, and Dr Olney provided facilities/equipment and consultation (including review of manuscript before submission).

This research was approved by the La Trobe University Faculty of Health Sciences Human Ethics Committee.

This research was presented at the Sixth International Physiotherapy Congress of the Australian Physiotherapy Association; May 24-27, 2000; Canberra, Australian Capital Territory, Australia.

This article was submitted February 22, 2002, and was accepted September 2, 2002.