Item reliability of the Milani-Comparetti Motor Development Screening Test.

Item Reliability of the Milani-Comparetti Motor Development Screening Test The Milani-Comparetti Motor Development Screening Test was first published in 1967 by Milani-Comparetti and Gidoni as a mechanism for identifying infants with abnormal movement patterns. [1] A testing manual was published by the C Louis Meyer Children's Rehabilitation Institute in 1977 [2] to provide information on test administration and revised in 1987 [3] to include new data on test standardization. The Milani-Comparetti test is a screening tool developed to systematically examine the integration of primitive reflexes and the emergence of volitional movement against gravity. The test assesses spontaneous motor behavior and evoked responses including righting, protective and equilibrium reactions, and select primitive reflexes. [1,4] Specifically, the Milani-Comparetti test is used to 1) identify motor dysfunction in infants up to 2 years of age, 2) establish the basis for an early intervention program, 3) aid in research on motor development, and 4) teach the skillful observation of motor development in handicapped and nonhandicapped children. [3]

Although the Milani-Comparetti test has not been examined with the rigor applied to commonly used tests of infant development, its application as an outcome measure has been documented for premature children with intracranial hemorrhage [5] and children with post-cerebral palsy implantation of cerebellar pacemakers. [6] A major criticism in using the Milani-Comparetti test for research and clinical applications has been the lack of data establishing the test's reliability. [7,8]

VanderLinden [9] reported interester reliability for two experienced physical therapists at 85% for data collected as part of Campbell and Wilhelm's Baby Research and Training Section study. [10] Specific information describing the methodology to establish reliability was not published.

Estimates of interobserver reliability provide information on the accuracy of agreement among examiners on the scoring of items presented during an observation period. Estimates of test-retest reliability, also commonly referred to as intraobserver reliability, provide information on the stability of the scoring of items over repeated tests by the same examiner. The purpose of this study was to determine the level of interobserver and test-retest reliability for items on the Milani-Comparetti test. We hypothesized that an acceptable level of reliability (percentage of agreement greater than 80%) would be obtained using the standardized protocol for administration and scoring.

Method

Subjects

A total of 60 infants, aged 1 through 16 months, participated in the study. The sample represented a distribution of 15 children in each of four age groupings: 1) 1- to 4-month age group (X = 2.1 months, s = 1.1), 2) 5- to 8-month age group (X = 6.2 months, s = 1.1), 3) 9- to 12-month age group (X = 10.6 months, s = 1.1), and 4) 13- to 16-month age group (X = 14.5 months, s = 1.2). Each child was tested within three days of a one-month anniversary. The children did not have a history of prematurity ([is greater than] 2 weeks), significant birth history, neuromuscular impairment, medical complications, or illness at the time of testing. Table 1 lists the demographic information concerning the sample population and the families. The children were recruited for participation in the study through "well baby" follow-up clinic appointments at a university hospital, from an Omaha (Neb) area day-care center, and from siblings of C Louis Meyer Children's Rehabilitation Institute staff members.

Informed consent following university guidelines was obtained prior to videotaping the children. Test results were provided to the parent(s).

Setting

The infants were tested in a quiet examination room at the C Louis Meyer Children's Rehabilitation Institute or at an Omaha area day-care center. The session was videotaped with the infants' parent(s) present. Two examiners were present during the videotape session for approximately one third of the sample. One examiner videotaped the session while the other tested the child. The remaining sessions were videotaped by a technician with only the examining therapist present. Only one examiner was present for the retest session, which was not videotaped. The therapist who tested the child initially retested the child.

Examiners

Four pediatric physical therapists participated in the study. One therapist (WAS) who had over seven years' experience using and teaching the Milani-Comparetti test served as a standard observer for the study. The other therapist had at least three years of pediatric experience. Training in administration and scoring of the Milani-Comparetti test for the therapists participating in the study was provided by the standard observer using the criteria published in the testing manual. [3] All therapists administered, viewed, and scored 10 demonstration videotapes to clarify test scoring criteria. Feedback was provided to each therapist during the training period to ensure consistency in administration of the test items.

Instrumentation

The Milani-Comparetti test consists of 27 items that examine spontaneous motor behaviors and evoked responses. [1] Spontaneous motor behaviors include active movement items such as locomotion and postural control items such as sitting or standing. Evoked responses include 1) tilting (equilibrium) reactions, 2) parachute (protective extension) reactions, 3) righting reactions, and 4) primitive reflexes. Approximately two thirds of the items require direct handling, dependent on the child's age at testing. No specialized equipment is required, except a cushion or tiltboard to assess equilibrium reactions. The authors have written specific behavioral criteria for each item (Figure). [3]

The score form used for the study was a modification of the original form published by Milani-Comparetti. [1] The revised score form was published in the 1977 edition of the Milani-Comparetti test manual [2] and represents a reordering of the test items from the original score form. Twenty-one of the test items are scored as present, absent, or incomplete with 6 of the items including subtests that are each scored as present or absent. For example, the item locomotion includes automatic stepping, rolling, crawling, and walking, which the examiner scores using a dichotomous scale of present or absent. A complicated scoring system for the Milani-Comparetti test has been developed, but was not used in this study because we believe the system has limited application in routine clinical use. [7]

Procedure

Sixty children were videotaped during administration of the Milani-Comparetti test. The Denver Developmental Screening Test (DDST) was also administered during the session to determine whether the child was developing within normal limits and to establish reliability data on the DDST for the testers as part of a further study. The item order on the score form was followed as closely as possible. An attempt was made to administer and score all appropriate items for the child's chronological age. Items greater than two to three months above the child's age were not administered unless advanced development was observed. In general, 19 items were administered to the 1- to 4-month-olds, 24 items to the 5- to 8-month-olds, and all 27 items to the 9- to 16-month-olds.

Forty-three of the 60 children were retested within three to five days by the same tester. Only 43 of the children could be rescheduled to participate in retesting. Test-retest data were collected by two therapists (WAS, PRD). The second session was not videotaped. The Milani-Comparetti test score form results of the first session were not available to the examiner prior to the second session. Test-retest reliability was determined by comparing the therapist's score form results from the first and second session. The score form completed by the therapist during the first session was not used in the determination of interobserver reliability. The DDST was not administered during the second session.

The videotapes were viewed and independently scored by each therapist and the standard observer. The results of the three observers were compared with the results of the standard observer to determine interobserver reliability. The criteria for scoring each item are outlined in the test manual. [3]

Data Analysis

Data were analyzed individually for each of the 27 items of the Milani-Comparetti test to examine test interobserver and test-retest reliability. Table 2 lists the month ranges used to calculate reliability for the individual items. The decision to use the month ranges rather than entire age range was made to eliminate inflation of the reliability level that would be introduced by scoring items that are obviously present or absent. For example, a backward protective extension response cannot be expected prior to age 6 months, and inclusion of the 1- to 6-month-old children to calculate agreement would incorrectly inflate the results. In general, a range of [+ or -] 3 months from the expected date of integration of a primitive reflex or acquisition of a motor milestone was used to set the ranges. The entire range for items such as locomotion was used because the item includes numerous subtest tasks such as rolling, crawling, and walking.

Percentage of agreement and the G statistic were used to obtain an estimate of interobserver reliability. Percentage of agreement (PA) was calculated by comparing the score of the standard observer with that of the other observers using the formula

PA = number of agreements/total paired observations X 100

Because percentage of agreement may provide an overestimate of agreement if change agreement is high, the G statistic was calculated to provide an indication of true reverses chance agreement. For example, if two observers score a behavior with a three-point scoring system, they will agree by chance alone one third of the time. The G statistic accounts for the chance agreement in observations between observers and tests the hypothesis that agreement is greater than chance alone (ie, true agreement).

The G statistic is an appropriate tool when comparing joint agreement of two or more observers with a standard observer. [11] A computer program was written to generate the appropriate standard error and the G statistic. The G statistic, which is a ratio of the difference between observed and expected agreements to the standard error, was then compared with a standard normal distribution to test the hypothesis of random agreement. The G statistic values equal to or greater than 2.56 are significant beyond the .01 level for a two-tailed test.

The use of a standard observer and the G statistic for data analysis were chosen because the three observers were not familiar with the Milani-Comparetti test prior to initiation of the study. Percentages of agreement and Kappa values across all testers could have been calculated; however, it was not the intent of the authors to demonstrate association between the testers but rather agreement between the testers and an accepted standard. We assumed, although it was not within the scope of the study, that the standard used in the study would be representative of any individual who had experience in using the Milani-Comparetti test.

Percentages of agreement and Kappa values were used to obtain an estimate of test-retest reliability. Percentages of agreement were calculated as previously described. Kappa values were calculated using the formula published by Fleiss. [12]

Kappa is an estimate of agreement beyond chance alone and ranges from +1 to -1. The hypothesis of agreement significantly different from chance was tested by calculation of the variance of Kappa and standard error to obtain a z score. A probability level of .01 was used for determination of significance, which corresponds to a z score of 2.56 or greater for a two-tailed test. In general, Kappa values greater than .75 represent excellent agreement beyond chance, values between .40 and .75 represent fair to good agreement, and values below .40 represent poor agreement beyond chance. [13]

Adequate variability on all items of the Milani-Comparetti test was available to allow calculation of both the G and Kappa statistics. Neither the G nor Kappa statistics are valid predictors of agreement when scores are grouped around one response for an item. [14] For an item to have good agreement beyond chance, both a high percentage of agreement and a Kappa (test-retest) or G (interobserver) statistic should be obtained.

Although the Milani-Comparetti test does not provide the examiner with a total or summary score, a mean percentage of agreement for interobserver reliability was calculated to examine between-tester differences for scores for the four age ranges studied. Mean percentage of agreement was calculated by modifying the percentage-of-agreement formula to include all individual items on the test for the children representing the particular age range.

Results

Interobserver reliability results are listed in Table 3. High percentages of agreement and significant G statistics were obtained for all items. The most consistently high level of agreement was noted for active movement and postural control items. Overall, equilibrium reactions demonstrated lower levels of agreement with standing equilibrium being the lowest scored item at 79%.

The mean percentage-of-agreement results of each tester by age range are reported in Table 4. The lowest reliability was obtained for the 5- to 8-month age range by tester 3. In general, the 1- to 4-month and 5- to 8-month age ranges demonstrated lower levels of agreement than the 9- to 12-month and 13- to 16-month age ranges.

Test-retest reliability results are listed in Table 5. High percentages of agreement and significant Kappa values were obtained for all items. Eighty-two percent (20 out of 27) of the items demonstrated excellent test-retest reliability using the Kappa statistic guidelines. Although significant, only fair to good Kappa values were found for the items head righting, landau forward protective, supine equilibrium, and body lying supine. The lower Kappa values for the items were consistent with percentages of agreement in the 80% to 85% range as compared with higher percentages of agreement and Kappa levels for other test items.

Discussion

The results of this study compare favorably with the results of similar studies examining the reliability of commonly used infant screening tools. Mean percentage of agreement between four examiners during the standardization of the DDST was 90% with a range of 85% to 95%. [15] Test-retest percentage of agreement for the DDST ranged from 90% to 100% over a one-week interval between testing sessions. Haley et al reported percentages of agreement and Kappa statistics on the items of the Movement Assessment of Infants (MAI). [13] Using Kappa statistic guidelines as reported in this article, only 2% of the items demonstrated excellent interobserver reliability with 58% in the good range. Ten percent of the items demonstrated excellent test-retest results with 42% in the fair to good range. Eighty-five percent (23 out of 27) of the items for test-retest reliability in this study were in the excellent range (K [is greater than] .75), and the remaining items were in the good range. Interobserver agreement levels were significantly high and comparable to either the MAI [13] or the DDST [15] results.

An acceptable level of accuracy (interobserver agreement) was established for all items on the Milani-Comparetti test. Consistently high levels of agreement were found for items in the subsections of righting or protective reactions, primitive reflexes, and active movement (Tab. 3). In general, lower levels of agreement were found for the equilibrium reactions and for items that involved a more complex coding system because many tasks were grouped into one item.

An example of greater complexity in scoring is the item standing, which includes five criteria to be scored: supporting reactions, astasia, weight bearing, standing posture, and indepent standing. Other items with multiple subtests that demonstrated lower levels of agreement were all fours, body pulled to sitting, and sitting. An inverse relationship between the complexity of scoring and level of agreement has been documented by Mash and McElwee, who studied the ability of trained observers in scoring audiotapes using four- and eight-category scoring systems. [16] Only a three-category system of scoring criteria is used for 20 of the 27 items on the Milani-Comparetti test (ie, the response is present, in transition, or absent). The items with the less complex scoring system demonstrated consistently higher levels of agreement with the exception of the equilibrium reactions.

The lowest interobserver percentage of agreement was found for the item standing equilibrium at 79%. Werner and Bayley reported the lowest levels of test-observer and test-retest agreement for motor scale items on the Bayley Revised Scale of Mental and Motor Development requiring assistance from the examiner. [17] Although items that required assistance did not uniformly demonstrate low levels of agreement, the equilibrium reactions were the most difficult to score accurately. A subtle relationship between the stimulus intensity and the equilibrium response was noted in scoring the videotapes, which made scoring difficult. Additional study is needed to delineate the gradations of a developing response and the appropriate stimulus intensity to elicit the response.

The lower level of interobserver agreement for the children in the 1- to 4-month and 5- to 8-month ranges as compared with the 9- to 12-month and 13- to 16-month ranges reflects the greater number of items in transition during those ranges (Tab. 4). Transition of all primitive reflex items, with the exception of foot grasp, from presence to absence takes place within the first eight months. A majority of items examining postural control and acquisition of motor mile-stones are also scored during the first eight months in comparison with the 8- to 12-month and 9- to 16-month ranges.

The stability estimates (test-retest agreement) were acceptable and corresponded to the levels for the accuracy estimates. Although error is introduced with bias of knowledge of the first session results, the error was minimized by allowing a three- to five-day period between tests. We felt that delaying the retest session for a longer period would introduce inconsistency in results because of maturational changes of the child. The high test-retest scores have important clinical significance because the method used to collect the data was similar to clinical measurement versus recording off of videotapes.

A limitation of the study is the pediatric population that was used. No children with developmental delays were included to assess the observers' ability to score the items on children with abnormal development. Further studies to establish the reliability levels using populations of developmentally delayed children are needed.

Clinical Implications

The Milani-Comparetti test's ability to discriminate changes in motor development is decreased after 12 months of age and is extremely limited past 16 to 18 months of age. Only one item--locomotion--includes a subtest that is scored past 16 months of age. The majority of items are scored within the first 12 months of age, which would empirically be the age range of the Milani-Comparetti test's greatest clinical value.

Items scored by observation of the child's movements rather than through handling the child were found to be the most reliable in this study. minimizing handling should help to obtain more accurate test results.

Although acceptable levels of accuracy and stability were obtained, the generalizability of the results of this study to the clinical setting are in question. Further study is needed to establish the constancy of the motor behaviors and accuracy with which they can be observed in children at risk for developmental delays.

Conclusion

The hypothesis that acceptable levels of interobserver and test-retest reliability for the Milani-Comparetti test would be obtained was supported by the results of this study. A standardized protocol for test administration has been developed and indications for training provided. [3]

The results suggest that, with proper training, the Milani-Comparetti test can be used clinically or in research applications as a reliable tool to assess motor development in the first year and one-half for children not suspected of having motor delays. The level of reliability in screening developmentally delayed infants, specific training requirements for varying degrees of examiner experience, and test validity are variables yet to be determined.

Acknowledgment 

We thank Kashinath Patil, PhD, for his statistical assistance and development of the computer program for the G statistic.

References

[1] Milani-Comparetti A. Gidoni AE: Routine developmental examination in normal and retarded children. Dev Med Child Neurol 9:631-638, 1967

[2] Trembath JT, Kliewer D, Bruce W: The Milani-Comparetti Motor Development Screening Test. Media Resource Center, C Louis Meyer Children's Rehabilitation Institute, University of Nebraska Medical Center, Omaha, NE, 1977

[3] Stuberg WA, Dehne PR, Miedaner JA, et al: Milani-Comparetti Motor Development Screening Test: Test Manual, 1987 ed. Media Resource Center, C Louis Meyer Children's Rehabilitation Institute, University of Nebraska Medical Center, Omaha, NE, 1987

[4] Milani-Comparetti A, Gidoni AE: Pattern analysis of motor development and its disorders. Dev Med Child Neurol 9:625-630, 1967

[5] Bozynski ME, Nelson MN, Rosati-Skertich C, et al: Two-year longitudinal follow-up of premature infants weighing [is less than or equal to] 1,200 grams at birth: Sequelae of intracranial hemorrhage. J Dev Behav Pediatr 5:346-352, 1984

[6] Penn RD: The neurosurgical treatment of cerebral palsy. Pediatr Ann 8(10):72-78, 1979

[7] Ellison PH, Browning CA, Larson B, et al: Development of a scoring system for the Milani-Comparetti and Gidoni method of assessing neurologic abnormality in infancy. Phys Ther 63:1414-1423, 1983

[8] Price MM: Critique of the Milani-Comparetti Motor Development Screening Test. Physical and Occupational Therapy in Pediatrics 1(1):59-68, 1980

[9] VanderLinden D: Ability of the Milani-Comparetti developmental examination to predict motor outcome. Physical and Occupational Therapy in Pediatrics 5(1):27-38, 1985

[10] Campbell SK, Wilhelm IJ: Development from birth to 3 years of age of 15 children at high risk for central nervous system dysfunction. Phys Ther 65:463-469, 1985

[11] Light RJ: Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychol Bull 76:365-377, 1971

[12] Fleiss JL: Statistical Methods for Rates and Proportions, ed 2. New York, NY, John Wiley & Sons Inc, 1981

[13] Haley SM, Harris SR, Tada WL, et al: Item reliability of the Movement Assessment of Infants. Physical and Occupational Therapy in Pediatrics 6(1):21-39, 1986

[14] Plewis I, Bax M: The uses and abuses of reliability measures in developmental medicine. Dev Med Child Neurol 24:388-390, 1980

[15] Frankenburg WK, Dodds JB, Fandal AW: Denver Developmental Screening Test. Denver, CO, LADOCA Project and Publishing Foundation Inc, 1973

[16] Mash EJ, McElwee JD: Situational effects on observer accuracy: Behavioral predictability, prior experience, and complexity of coding categories. Child Dev 45:367-377, 1974

[17] Werner EE, Bayley N: The reliability of Bayley's Revised Scale of Mental and Motor Development during the first year of life. Child Dev 37:39-50, 1966

W Stuberg, MS, PT, is Assistant Professor and Director of Physical Therapy, C Louis Meyer Children's Rehabilitation Institute, University of Nebraska Medical Center, 42nd & Dewey Ave, Omaha, NE 68105 (USA).

P White, MS, PT, is Staff Physical Therapist, C Louis Meyer Children's Rehabilitation Institute.

J Miedaner, MS, PT, is Assistant Professor, Division of Physical Therapy Education, and Chief Therapist, Vince Mosley Clinic, Medical University of South Carolina, Charleston, SC 29425.

P Dehne, MS, PT, is Assistant Instructor and Staff Physical Therapist, C Louis Meyer Children's Rehabilitation Institute.

This article was adapted from a presentation at the Combined Sections Meeting of the American Physical Therapy Association, Atlanta, GA, February 12-15, 1987.

This research was supported in part by Project 405, Division of Maternal and Child Health Services, US Department of Health and Human Services, awarded to the C Louis Meyer Children's Rehabilitation Institute, University of Nebraska Medical Center.

This article was submitted December 14, 1987; was with the authors for revision for 27 weeks; and was accepted December 5, 1988.