Reliability of the Gross Motor Performance Measure.

Key Words: Gross Motor Function, Outcome measure, Pediatrics, Reliability.

Reporting numeric estimates of reliability and the method of their determination is a standard expected of test developers. The assessment of reliability amounts to determining that the results obtained from a measure's administration are reproducible and consistent under a variety of conditions.[1](pp79,106) Although this assessment is a necessary step in establishing the usefulness of a measure, it is not sufficient in and of itself for this purpose and must be accompanied determining the degree to which a meaningful interpretation can the inferred from a measurement (ie, its validity).[1](p106),[2]

In the assessment of a measure's reliability, there are various sources of variation that may interfere with the reproducibility or consistency of scores, and together these sources explain the total variance. Total variance can be partitioned into main components due to subjects, raters (observers), and error.[1](p84) The variation included in the error term generally arises from the environment, the individual being assessed, and the instrument itself.[3] When developing administration guidelines, all potential sources of variation (ie, those arising from the subjects, the raters, the environment, the instrument, and the interactions among these sources) should be identified and controlled as much as possible.

Usually, to determine the reliability of a test where ratings are on a quantitative scale, use of an analysis of variance (ANOVA) and estimates of intraclass correlation coefficients (ICC) are possible and desirable.[4] The ICC type 2.1 (reliability coefficient) is a ratio of the variance components of interest to the sum of the variance components of interest plus error.[5] There are several steps in the process of determining an ICC. First, the variance due to subjects, raters, and random error is analyzed using an ANOVA. Following this analysis, the reliability coefficient is calculated, where 0 indicates no reliability and 1 indicates perfect reproducibility.[1](pp82,83) Intrarater reliability can be estimated from a rater's assessment of individual subjects on one occasion accompanied by videotaping, and a rescoring of the videotapes by the same rater after a period of time. Interrater reliability is estimated on scores obtained by two raters simultaneously assessing the same subject's performance. Finally, test-retest reliability can be estimated when scores are obtained on two or more occasions for the same subject, and by the same rater over a time period when the subject has not changed.

Streiner and Norman state, "Reliability should not be seen as a property which a particular instrument does or does not possess; rather any measure will have a certain degree of reliability when applied to certain populations under certain conditions. The issue which must be address is how much reliability is good enough."[1](p89) Several opinions of minimal levels of reliability are suggested in the literature,[6-8] varying from .65[6] to .94.[8] A recommended approach to reporting reliability is to present not only the reliability coefficient but also the coefficients for the upper and lower limits of the 95% confidence interval. The lower bound of the 95% confidence interval offers the decision maker the opportunity to consider the "practical importance" of the estimate in addition to the "statistical significance" of the reliability coefficient.[9](p16)

The Gross Motor Performance Measure (GMPM) was developed to evaluate quality of movement of children with cerebral palsy (CP). This measure was developed to be used in conjunction with the Gross Motor Function Measure (GMFM).[10,11] The GMFM consists of 88 item organized into five dimensions: lying and rolling; sitting; crawling and kneeling; standing; and; walking, running, and jumping. Although the items were chosen by therapist as important and amenable to change, they are not assessed within different environmental contexts. The distinction between these two measures is that the GMFM measures how much a child can do, whereas the GMPM measures quality of performance, or how well a child performs a subset of the same gross motor tasks. Both measures were developed specifically for their ability to detect change. In the GMFM, the important change measured is intended to be gross motor function, and the GMPM is intended to measure the quality of movement during the gross motor behaviors. Only with measures known to be able to capture change will it become possible to ascertain what magnitude of change is judged to be "clinically important" to clients, clinicians, funding agencies, or others who have a stake in this issue.[12] Clinically important change, in contrast to statiscally significant change, refers to the magnitude of change in a score over tune that matters (is important) to individual clients.

Details of the development of the GMPM, including the method of selecting the GMFM items and the GMPM attributes, are published elsewhere, as is a description of the measure.[13,14] During the development of the GMPM, five attributes of quality (alignment, coordination, dissociated movements, stability, and weight shift) were selected and operationally defined using a series of modified nominal group process Meetings.[14] Also, using the same method, 20 items from the GMFM were selected for use in the GMPM. For each item, three attributes of quality are assessed. The following is an example of one GMFM item and the three attributes measured:

GMFM item 50. High kneeling: attains half kneeling on left knee, arms free, maintains 10 seconds.

Performance attributes to be measured:

Alignment of trunk and pelvis

Dissociated movements of lower extremities

Stability in trunk and left lower extremity

A five-point scale (1=severely abnormal, 2=moderately abnormal, 3=mildly abnormal, 4=inconsistently normal, and 5=consistently normal) is used to score each attribute. In total, a maximum of 60 GMPM items (three attributes for each of 20 GMFM items) can be scored. If a child does not initiate a GMFM item (ie, scores 0 on the item), the quality of the performance is not assessed. The total score is not calculated merely by summing the individual item scores; instead, the mean score obtained for each attribute (regardless of the number of times the attribute is scored) is calculated and converted to a percentage. This is referred to as the attribute percent score. The mean of the five attribute percent scores is then computed to give a total percent score.

Two assumptions were made while deciding on the method of computing this total percent score. First, the GMPM total percent score should not depend on how much a child can do, but rather should be based on how well the items that can be completed are performed. With this approach, a nondisabled infant would have a low score on the GMFM (because of little functional ability), but a high score on the GMPM (because of "normal" quality of movement). Second, each of the five attributes should contribute equally to the total percent score. Guidelines to standardize the administration and scoring of individual items and attributes were developed for the therapists involved in the validation trial but are not published.

The objective of this study was to assess the interrater, intrarater, and test-retest reliability of the GMPM. The methods and results of a study to validate the GMPM's ability to detect changes in quality of movement in children with CP are described in a companion article in this issue (see article by Boyce et al).

Method

Study Sample

Centers from which the children were drawn for inclusion in the study's convenience sample were the Children's Developmental Rehabilitation Program, Chedoke-McMaster Hospitals, Hamilton, Ontario, Canada; the Hugh MacMillan Rehabilitation Centre, Toronto, Ontario, Canada; and the Niagara Peninsula Children's Centre, St Catharines, Ontario, Canada. These are regional pediatric rehabilitation referral centers for areas encompassing a total population of more than 3 million persons.

Twenty-eight children were included in the reliability study: 25 with CP, 2 with no disability, and 1 with head injury (HI). These children were a subset of those involved in the validation study. Because children without disability and children with HI were included in the validation study, it was important to include them in the reliability to determine the reproducibility of their scores. The sample was evenly divided between males and females, with 14 of each gender. The mean age was 4.6 years (SD=2.23), varying from age 1 to 10 years. The 25 children with CP comprised 18 with spasticity, 4 with athetosis, and 1 each with ataxia, dystonia, and hypotonia. Children with either CP or HI were categorized into mild, moderate, and severe groups based on the therapist' clinical impression of the severity of their physical disability. Six children were classified as mildly disabled, 13 as moderately disabled, and 7 as severely disabled. Table 1 summarizes the attribute percent scores and total percent scores that were obtained during GMPM administration at the time of the first assessment. Coordination was not assessed in 1 child, and dissociated movement was not assessed in 3 children. These children were capable of little functional activity, scoring above 0 in only a few of the GMFM items in which coordination and dissociated movements are not assessed. In accordance with institutional guidelines aimed at protecting the rights of human subjects, informed written consent to participate was obtained for all children from a parent or legal guardian.

Table 1. Gross Motor Performance Measure Attribute Percent Scores and Total
Percent Scores at the Time of the First Test Administration

Attribute              n     X     Minimum-Maximum

Alignment              28   60.9        20-100
Coordination           27   54.3        20-100
Dissociated movement   25   60.0        20-100
Stability              28   56.0        20-100
Weight shift           28   54.4        20-100
Total                  28   56.1        20-100

The calculation of the study sample size was based on our experience with the GMFM and the power contours presented by Donner and Eliasziw.[15] Because we expected ICCs for the total scores to vary between .90 and .95, we estimated that approximately 25 subjects would be needed for the reliability study. Twenty-eight children were assessed initially to allow for a small number of dropouts.

Therapists Evaluators

Nineteen therapists from the three children's treatment centers carried out the evaluations. The pediatric experience of these therapists ranged from 2 to 20 years (X=11.4 years). These therapists were also trained and experienced in the use of the GMFM. They were then trained in administration and scoring of the GMPM at a 1-day workshop provided by the project team. They were trained by standard videotapes of children with CP, illustrating various levels of involvement of the five attributes of quality. The therapists were then tested on the use of the GMPM to achieve at least an average weighted Kappa estimate of 0.6 (substantial agreement[15]) with a criterion videotaped assessment before commencing to enroll children in the study. The weighted Kappa statistic was selected to allow us to examine the mean agreement with multiple single items, each scored on the five-point ordinal scale. This statistic is suitable for use with ordinal data and provides for disagreements of varying gravity to be weighted accordingly. For example, we considered a disagreement between a therapist and the criterion score of 2 points (eg, a score of 2 versus a score of 4) as a graver disagreement than 1 point (eg, a score of 2 versus a score of 3).

Procedure

To meet the study objective, the children were studied at three different times. At time 1, each child's treating therapist assessed the child, and the assessment was videotaped. The two nondisabled children were the siblings of disabled children and were also assessed by the study therapists. Approximately 2 weeks later, at time 2, a second assessment was carried out by the same therapist while a second therapist observed and independently scored the performance. At least 6 weeks later, at time 3, the treating therapist viewed and scored the child's performance from the videotape made at time 1.

To estimate intrarater reliability, data from assessments at times 1 and 3 were used; for interrater reliability, data from the two assessments obtained at tune 2 were used; and for test-retest reliability, data from assessments by the treating therapist at times 1 and 2 were used.

To minimize the sources of variability and to maximize the number of true responses, several precautions were taken:

1. To minimize within-subject variation, consideration was given to the comfort level of the child during the administration of the GMPM. To achieve this, the child's treating therapist administered the test in a treatment room familiar to the child. If the child was tired or not feeling well, assessment was postponed. Parents were encouraged to attend the session.

2. To minimize variation between two raters and with the same rater over time, the therapists received training on the administration and scoring of the GMPM, and they were tested to a criterion level of reliability (K>0.6) prior to enrolling children in the study.

3. To minimize variation over time, the testing environment was kept as consistent as possible, including the room and the time of day of the assessment.

4. To minimize variability due to the instrument, objective definitions of performance attributes and standardized scoring and administration guidelines were developed. The assessment involved direct observation only, thus avoiding a "hands-on" assessment that could have biased assessor observations and that could not easily be standardized.

Data Analysis

All data entry and statistical analyses were done using SPSS/PC+ Version 3.1 software(*) on an IBM PS2 55SX.([dagger]) Separate ANOVA tables were generated, and type 2.1 ICCs[5] and their respective 95% confidence limits[9] were calculated for each measure of interrater, intrarater, and test-retest reliability. The ICC type 2.1 allows for the examination of systematic differences due to rater or time. To calculate confidence intervals, a balanced two-factor crossed random model without interaction was used.[9](pp126-131)

Results

Table 2 summarizes the intrarater, interrater, and test-retest reliability coefficients (ICCs) and their 95% confidence limits for the five attribute percent scores and the total percent scores. The videotape did not capture the performance of coordination and dissociated movement attributes of 2 additional children, reducing the sample size to 25 and 23, respectively, for these two attributes. The ICCs for the total percent scores varied from .92 to .96. We established a priori that these ICCs should be at least .90. The ICCs for the five attribute percent scores varied from .90 to .97 for intrarater reliability, from .84 to .94 for interrater reliability, and from .89 to .96 for test-retest reliability. The lower 950% confidence levels of the ICCs for the total percent scores varied from .85 to .93.

[TABULAR DATA OMITTED]

Discussion

This study confirms that the results obtained from the administration of the GMPM are acceptably reproducible and consistent within a single rater, between two different raters, and on two occasions over a short period of time (several weeks). The reliability coefficients are gratifyingly high, considering how difficult quality of movement is to measure. Campbell emphasizes this difficulty when stating that quality of movement "... is difficult to capture and describe because it does not consist of a single factor, but rather is inclusive of coordination, postural control, and balance."[16](p3)

Although no specific reliability is necessarily "good enough,"[1](p89) the results support the conclusion that the total score of the GMPM, when administered by therapists who are familiar with the GMFM and have an additional 1-day training workshop, has acceptable interrater, intrarater, and test-retest reliability for the total percent score. Both the reliability coefficients and the lower bound of the confidence intervals fell above a recommended minimum of .85.[1](p89) The same degree of confidence with the reliability of individual attributes cannot be expressed. The ICCs for the attributes were within acceptable limits, with the exception of the interrater reliability for weight shift, which was .84. The lower 95% confidence intervals fell below the .85 level on four occasions. For the attribute dissociated movement, the lower confidence interval for intrarater reliability fell to .76 and the lower confidence interval for test-retest reliability fell to .77. For both coordination and weight shift, the lower limits of the interrater reliability fell to .72 and .69, respectively. These findings suggest less confidence with the consistency of results obtained from the measurement of a single attribute. Streiner and Norman[1](p89) point out that a test used for making judgments about individuals (this would pertain to clinical measures) should be more reliable than one used for group decisions or research purposes. Further, they note that the final determination of an acceptable level of reliability for a research study is dependent on the sample size required for the project.

In spite of our efforts to minimize the sources of variability in the administration of the GMPM, variability did exist. Both the reliability coefficients and our experience with the measure provide clues about the sources of this variability. Observing task performance from a videotape appears to be a major source of variability. Surprisingly, the highest reliability coefficients were found with repeated assessments that were directly observed over the 2-week time frame. This finding suggests that the variability within a child over this period of time was less than the inconsistencies that arise when a single evaluator attempts to capture an assessment on a videotape, or when two evaluators score a single performance. Taping frequently did not capture the full performance, or the part of the body to be observed was not viewed fully or from the appropriate angle. Evaluators preferred to score the measure while observing task performance directly. Another source of variability lay in the children's cooperation in carrying out tasks, particularly with three repetitions. Evaluators noted considerable noncompliance and variability in performance. The evaluators themselves offered a third probable source. They found it difficult to observe three attributes simultaneously, yet indicated that this was the preferred manner of test administration. Also, as expected, the reliability between two different evaluators scoring a videotape of an assessment was slightly lower than the reliability obtained with a single rater rescoring the videotaped assessment.

The reliability coefficients reported in this article were achieved after therapists were trained in a 1-day workshop and reached acceptable levels of agreement with the criterion videotape. Approximately 10% of the therapists who participated in the original training workshop did not reach the criterion levels of aggreement and therefore did not participate in the study. Many questions relating to training therapists in the use of this measure remain unanswered. For example, how much training does the typical therapist need? What is the most efficient method of training? How should training be provided? In our experience with training therapists to use the GMFM,[17] the answers to these questions will have implications for future use of this measure. Feedback from the therapists who participated in this study indicated that the GMPM is not an easy measure to administer. Suggestions for improving ease of administration will need to be incorporated before recommending the GMPM for clinical use.

Conclusion

This study has three major implications for therapists. First, quality of movement, especially disordered movement as it has traditionally been observed by clinicians, can be described in a formal, standardized manner. Second, pediatric therapists can be trained to assess complex sets of body movements with precision. Third, observational measures can have a high degree of reliability if used by experienced clinicians who are appropriately trained in their administration and scoring. Although this study contributes to physical therapists' body of knowledge about the measurement of motor activities, the real value of such measures awaits demonstration of their validity and responsiveness to meaningful change.

Acknowledgments

We gratefully acknowledge the participation of the physical therapists from the Children's Developmental Rehabilitation Program, Chedoke-McMaster Hospitals; the Hugh MacMillan Rehabilitation Centre; and the Niagara Peninsula Children's Centre. Particularly among these therapists we would like to acknowledge Nancy Plews, Mary Lane, Deborah Harding, and Shelley Potter-Zdrobov.

C Gowland, MHSc, PT is Associate Professor, School of Occupational Therapy and Physiotherapy, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada L8N 3Z5. Address all correspondence to Ms Gowland at the Neurodevelopmental Clinical Research Unit, OT/PT Building T16, Room 103/D, 1280 Main St W, Hamilton, Ontario, Canada L8S 4K1 ([e-mail]).

WF Boyce, MSc, PT, is Assistant Professor, School of Rehabilitation Therapy and Departments of Paediatrics and Community Health and Epidemiology, Queen's University, Kingston, Ontario, Canada K7L 5G2.

V Wright, MSc, PT, is Research Physiotherapist, Hugh MacMillan Rehabilitation Centre, Toronto, Ontario, Canada.

DJ Russel, MSc, is Research Coordinator (Neurodevelopmental Clinical Research Unit), Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University.

CH Goldsmith, PhD, was Professor, Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University, Center for Evaluation of Medicines-Biostatistics, Martha Wing, 50 Charlton Ave E, Hamilton, Ontario, Canada L8N 4A6.

PL Rosenbaum, MD, FRCP(C), is Professor, Department of Pediatrics, Faculty of Health Sciences, McMaster University.

This research was approved by the Research Ethics in Human Experimentation Committee, Chedoke-McMaster Hospitals.

This research was supported by Grant 6606-3740-R, National Health Research and Development Program, Ottawa, Ontario, Canada.

This article was submitted December 16, 1993, and was accepted February 24, 1995.

(*) SPPS, 444 N Michigan Ave, Chicago, IL 60611. ([dagger]) IBM Corp, 951 NW 51st St, Boca Raton, FL 33431.

References

[1] Streiner DL, Norman GR. Health Measurement Scales A Practical Guide to Their Development and Use. New York, NY: Oxford University Press Inc; 1989. [2] Task Force on Standards for Measurement in Physical Therapy. Standards for tests and measurements in physical therapy practice. Phys Ther. 1991;71:589-622. [3] Goldsmith CH. Measurement: Variation and Disagreement. Hamilton, Ontario, Canada: McMaster University; 1987. Unpublished teaching guide. [4] Bartko JJ, Carpenter WT. On the methods and theory of reliability. J Nerv Ment Dis. 1976;163:307-317. [5] Shrout PE, Fleiss JL. Intraclass correlation coefficients: uses' in assessing rater reliability. Psychol Bull. 1979;86:420-428. [6] Law M. Measurement in occupational therapy: scientific criteria for evaluation. Can J Occup Ther. 1987;54:133-138. [7] Weiner EA, Stewart BJ. Assessing Individuals. Boston, Mass: Little, Brown and Company Inc; 1984. [8] Kelley TL. Interpretation of Educational Measurements. Yonkers, NY: World Books; 1927. [9] Burdick RK, Graybill FA. Confidence Intervals on Variance Components. New York, NY: Marcel Dekker Inc; 1992. [10] Russell DJ, Rosenbaum PL, Cadman D, et al. The Gross Motor Function Measure: a means to evaluate the effects of physical therapy. Dev Med Child Neurol. 1989;31:341-352. [11] Russell DJ, Rosenbaum PL, Gowland C, et al. Gross Motor Function Measure (GMFM). A Measure of Gross Motor Function in Cerebral Palsy. Hamilton, Ontario, Canada: McMaster University; 1990. [12] Kirshner B, Guyatt G. A methodological framework for assessing health indices. J. Chronic Dis. 1905;38:27-36. [13] Boyce WF, Gowland C, Hardy S, et al. Development of a quality-of-movement measure for children with cerebral palsy. Phys Ther. 1991;71:820-832. [14] Boyce WF, Gowland C, Russell DJ, et al. Consensus methodology in the development and content validation of a gross motor performance measure. Physiotherapy Canada. 1993;45(2):94-100. [15] Donner A, Eliasziw M. Sample size requirements for reliability studies. Stat Med. 1987;6.441-448. [16] Campbell SK. Measurement in developmental therapy: past, present, future. In: Miller LJ, ed. Developing Norm-referenced Standardized Tests. New York, NY: The Haworth Press; 1989:1-13. [17] Russell DJ, Rosenbaum PL, Lane M, et al. Training users in the Gross Motor Function Measure: methodological and practical issues. Phys Ther. 1994;74:630-636.