School effectiveness of a streamed-school system: a multilevel modelling of the Hong Kong secondary schools.

This study has two distinct features that are different from current studies using multilevel analysis on school effectiveness. First, it uses a very large dataset, and second, the analysis is on a streamed-school system. The initial findings of this study record three differences from studies in the West: (a) the relatively large size of school effect in the Hong Kong schools; (b) the negative correlation between intercepts and slopes for prior attainment; and (c) the differential effectiveness for low and high ability students. The study highlights the issue of whether or not the results of school effectiveness research based on mixed ability schools can be generalised to some countries in East Asia where students are streamed to schools according to their academic ability. It suggests the need to explore whether the contextual factors that influence the effectiveness of schools in a mixed system are duplicated in a streamed-school system.

Introduction

Studies of school effectiveness (Boskers & Witzers, 1995, 1996; Levine, 1992; Purkey & Smith, 1983; Scheerens, 1992; Scheerens & Boskers, 1997) have been well reported for schools that take in students of mixed ability. However, in China (Hong Kong) and, to a great extent, in East Asian countries, schools take in students of similar rather than mixed ability. In these countries, students compete to enter the good primary schools. Primary school students compete to enter the good secondary schools, and then compete to enter universities. One consequence of this highly competitive system of education is that only the best students can enter the good schools, the next best students to the next best schools, and the weak students to the less preferred schools. In other words, students are streamed to schools based on how they do on public examinations.

One purpose of this paper is to examine the effectiveness of schools in Hong Kong. Will school effects be larger as a result of this streaming of students to schools rather than to classes as is the practice in countries such as the United Kingdom, United States of America, and elsewhere?

The streamed school system is also practised in a number of European countries, such as the Netherlands, Belgium and Austria. Daly (1991) did examine school effectiveness in Northern Ireland, where some streaming existed in the public educational system. Interestingly he did not observe any greater school effects. However, in doing this type of study, Gray (1989) raised the need to use a large number of schools to ensure the accuracy in determining school effects.

Schooling in Hong Kong

Public schools in Hong Kong are different from other countries in that the students were streamed to schools according to their academic achievement. Specifically all graduates of primary schools had to sit for the Academic Aptitude Test (AAT) in Chinese and in numerical reasoning. The score distribution of the AAT was used to divide the student population into five bands: Band 1, Band 2, ..., Band 5. A school would be assigned quotas of Band 1, Band 2, ..., Band 5 based on how the students in the school performed on the AAT. The more successful the students of a school were in the AAT, the more Band 1 or Band 2 quota they would gain for their particular school. The less successful they were, the more Band 4 and Band 5 quota the schools would get.

Once the primary schools have been assigned the different quotas, how were students assigned to secondary schools? First, the Primary Six students in a school were ordered on how they did in the school internal examinations on the following six subjects: English, Chinese, mathematics, health education, social studies, civic education and other subjects. Next, the best students in a school were assigned to Band 1 until the quota for the school in Band 1 was filled, the next best students were assigned until Band 2 was filled, and this process continued, until all students were allocated to the five different Bands. This process was known as Secondary Schools Places Allocation (SSPA).

The Hong Kong Government recently introduced a number of measures to reduce the undesirable back-wash effects associated with SSPA. One such measure was to abolish the AAT and to reduce the banding of students from five to three. As from 2000, the Academic Attitude Test was a thing in the past. Instead of asking the students to sit for the public test, the last three-year average of a school in the SSPA is used for allocation. Another measure is to encourage primary schools to be paired up with secondary schools, to allow students to transfer from primary to secondary schools without the need for SSPA. This is known as the `through train'.

Parents are given a choice in this allocation process. Parents choose the secondary schools they wish to send their children to. Obviously this choice is constrained by the actual bands their children gained in SSPA. Many parents choose schools on the basis of how the schools do in the Hong Kong Certificate of Education Examination (HKCEE), the public examination considered by the universities when they admit students. For the same reason, in order to attract Bands 1 and 2 to choose their schools, the schools have an incentive for raising or maintaining their standards. Within the great majority of secondary schools, the students are streamed into different classes according to their ability.

This type of competitive schooling--students competing to get into the better schools and schools competing to raise their status by ensuring their students do well on external examinations to attract the best students--is not unique to Hong Kong, but is quite common in East Asia, such as Singapore, Taiwan, South Korea and Japan. The practice of streaming students across schools and within schools has remained the same after Hong Kong became the Special Administrative Region of China in July 1997.

Within this context, the purposes of this paper are first, to examine the relative effectiveness of streamed schools in Hong Kong, and second, to make comparisons of the relative effectiveness of the streamed schools in Hong Kong with those reported for non-streamed schools in other countries. Mortimore and Sammons (1994) have suggested that differential effect within the school is an important issue in examining school effectiveness. In a mixed ability school system, differential effectiveness implies that a single regression equation may not be appropriate for the whole ability range of students within a school. Instead two or more regression equations may be necessary. When schools are streamed, with some schools covering a particular range of ability and other schools a different range of ability, a question of interest is how differential effectiveness should be represented. A final purpose is to examine the differential effectiveness of schools in Hong Kong.

Related studies on school effectiveness

The earlier studies of the effects of schools on student achievement by Rutter, Maughan, Mortimore, & Ouston (1979) did not employ multilevel analysis. However later studies by Willms (1986), Willms and Raudenbush (1989), Smith and Tomlinson (1989) employed multilevel models in determining the relative contribution of the effects of schools on student achievement. Through a series of meta-analyses, Boskers and Witizers (1996) and Scheerens and Boskers (1997) concluded that the proportion of total variation attributed to schools was in the region of 9 per cent but when prior attainment and other variables were accounted for, the net effect of schools was near to 4 per cent. A key question in estimating the net effect of schooling is what variables should be controlled for.

In a review of earlier studies, Daly (1991) observed that the effects of schools after having adjusted for prior attainment could still drop considerably depending on the school contextual variables that would be added to the equation. In his study of secondary schools in Northern Ireland, Daly observed that, after having adjusted for prior attainment, the effect due to schooling was 10 per cent, but when the school type variable--grammar school or comprehensive school--was introduced, the value dropped to 5 per cent. Similar observations were made in an earlier study by Willms (1987) in his study of Scottish secondary schools. There he noted that the adjusted effect due to schooling, controlling for prior attainment, was around 10 per cent, but this dropped considerably to close to 5 per cent when the mean socioeconomic status (SES) of the students in a school was included as a school variable. Hill and Rowe (1996) also found a large drop in the adjusted school effect after introducing the contextual variable on class/teacher membership, from 7.2 per cent to 3.4 per cent for English and from 8.6 per cent to 0 per cent in mathematics.

These studies have suggested the need to take into account the contextual variables when judging the effects of schools. However other studies (Sammons, Nuttal, Attance, & Thomas, 1993; Thomas & Mortimore, 1996) found that there was minimal change in the effects of schools when contextual variables were included. Thomas and Mortimore (1996) analysed a number of data sets and concluded that, when prior attainment was poorly measured or not finely graded, then introducing additional variables into the multilevel model would matter, irrespective of whether they were school contextual variables, students' variables, or indicators of SES. But when prior attainment was well measured and finely graded, the addition of contextual variables did not significantly change the results of the multilevel analysis. These results would indicate that, even in studies such as the present study, where school contextual variables or SES indicators were not observed, the effects of a school would still be accurately assessed as long as the prior attainments of students were well discriminated and well measured.

In a recent review of school effectiveness research, Coe and Fitz-Gibbon (1998) have identified another important issue. They argued that Coleman et al. (1966) judged effects of schools more on curriculum-free outcomes that were easily available rather than on the curriculum-oriented goals that schools considered important in achieving. When curriculum-free goals were used, the effectiveness of a school might be contingent on the correspondence of what was assessed to what the school happened to teach on the sideline. In this study, the outcome variables were the results of an open, public, school-leaving examination, the HKCEE, in which all schools in Hong Kong are pressured to do well.

Problem of IEA data

The International Association for the Evaluation of Educational Achievement (IEA) conducts regular international comparisons of educational achievement. These IEA data not only compare achievement of students in different countries but also compare the effectiveness of schools of different countries. Woessman (2001) found that the structure of schooling and educational policies do have an effect on student achievement, at least as judged from the Third International Mathematics and Science Study (TIMMS). However, he did not examine the size of the effects of schooling. Nor did he examine whether streaming has an effect on the effectiveness of schools.

Studies by Scheerens, Vermuelen, and Pelgrum (1989) on other IEA data sets related to secondary school students' achievement in mathematics and science found that the unadjusted effects of schools for a streamed school system such as Hong Kong and Thailand were high; the intra-class correlations were .51 and .39 respectively. These unadjusted correlations might over-estimate the effects due to schooling by not adjusting for differences among schools in relation to the ability of students at entry to the schools. Unfortunately no prior attainments were available in these IEA studies. If parental occupations were related to the ability of students at entry to schools, then these occupations could act as proxies in adjusting for the initial differences of students at entry to schools. Scheerens and his team found that occupational levels of parents did not change significantly the size of the intra-class correlation. Left unanswered in this study was whether occupational levels were the appropriate proxies or whether the schools were already similar in the initial abilities of students.

Another difficulty in judging school effectiveness from IEA data is the lack of correspondence between the achievements assessed in the IEA studies and the curriculum goals of the individual countries participating in the study. To allow for cross-national comparisons, a common set of test items has to be given to all countries. Because different countries may have different curriculum targets, the agreed common set of items will not measure exactly the curriculum targets of any one single country.

Sample size

Another shortcoming with previous studies is the observation by Gray (1989) that the size of the school effects depended on the number and the variety of schools sampled in a study. If the sample did not cover well the different types of schools, the school effects size may be reduced. Typically studies of school effectiveness have employed 50 to 150 schools. In some of the studies using large data sets, such as the one by Yang, Goldstein, Rath, and Hill (1998), the data were based on a sample of 161 schools consisting of 6400 Primary-Two students.

The sample used in this study was large. Almost all secondary schools in Hong Kong were included in the study. The only schools and students excluded were those who did not follow the same curriculum and would probably contaminate the findings of this study. These excluded schools and students are described in the next section. The effects of schools on students' achievement in the standardised school leaving examination, the HKCEE, were examined.

The sample

The HKCEE is similar in nature to the O level examination in the United Kingdom. The cohort who took the SSPA in 1992 and HKCEE five years later was included with the following exceptions:

* students from the pre-vocational schools since they followed a practical curriculum (this group is about 6 per cent of the population);

* students who took the English A syllabus (about 10 per cent of the population took Syllabus A English; they are from the Chinese medium secondary schools and the exclusion is to avoid the influence of a different (easier) medium of instruction);

* students who, from 1992 to 1997, repeated a year or more in their schools or who had changed schools (this group is excluded in order to measure the `pure' effect of schools); and

* students from those schools with less than 10 students attending the HKCEE (17 schools with a total of 66 students).

The size of the full-time student cohort was thus reduced from over 80 000 to 42 248 students from 332 schools. A listwise deletion method was used to handle missing data. This method of deletion further reduced the sample to 41 709 with 18 836 boys and 22 873 gifts coming from 332 schools, of which 251 were co-educational, 39 were boys-only and 42 were girls-only schools. The students who were excluded from the study were weaker academically and, as a result, the remaining cohort was likely to represent the characteristics of an academically stronger group of students.

Student achievement

The study traced the development of students at the point of entry into secondary school (age 12) to the time when they left school at the Secondary 5 (age 17) level. The baseline measure of students' academic achievement at the time when they entered secondary school is the score they achieved on the Secondary School Places Allocation (SSPA) exercise. The SSPA scores are standardised scores that summarised how well students achieved on English, Chinese, mathematics, health studies, social studies and civic education of the primary school curriculum.

The outcome measures are scores students obtained on the HKCEE. The HKCEE consists of tests on more than 30 academic subjects, with students permitted to take no more than 10 subjects. Although students are free to decide on the number of subjects, in practice most choose around seven subjects, with English, Chinese and mathematics being considered the core subjects. However, when they move to the matriculation classes (Secondary 6 and 7), their five best subjects in the HKCEE will be counted plus at least a pass in the English language. For this reason, the general ability of students is defined as the students' five best performances on the HKCEE plus English. This score is computed by averaging the standardised English score and the best five standardised subject scores. In addition, standardised scores of English, Chinese and mathematics are employed as the outcome measures. All standardised scores were calculated based on all students (about 110 000, including both part-time students and private candidates) who took the HKCEE in 1997.

Methods of analysis

The value-added models were developed using a multilevel approach. Data were analysed using the MLn computer package (Woodhouse, 1995). The following explanatory variables were introduced into the model:

Explanatory  Level of
variable     variable       Description

SSPA         Student level  A generalised measure of students'
                            academic performance at reception to
                            secondary schools, with scores
                            ranging from 145 to 870

GENDER       Student level  Male (1) or female (0)

INTAKE       School level   Average SSPA of all students entering the
                            school

CO-EDU       School level   Co-educational school (1) or single-sexed
                            school (0)

The scores on the SSPA and the HKCEE were transformed and expressed in standard scores before conducting the multilevel analyses. Multilevel models were set up for each outcome measure to calculate the value-added scores for each school and intra-school correlations. The explanatory variables covered student prior attainment at entry to school (SSPA), student gender (GENDER.), type of school--co-educational or single-sexed (CO-EDU), and the contextual effect of the ability of students streamed to a particular school (INTAKE). In theory, the type of schools ought to be coded as `boys only', `gifts only' and coeducational schools rather than co-educational or single-sex schools. But because of possible problems of collinearity with the variable gender, it was judged to be more appropriate to code `boys only' and `girls only' as single-sexed schools.

Two multilevel models were set-up to estimate the intra-school correlation. In one model, the ability of individual students (SSPA) as well as the average of the ability of students in a school (INTAKE) were introduced as explanatory variables into the multilevel analyses. In another model, only the ability of individual students was used. The reason for introducing the average ability of students in a school (INTAKE) as an explanatory variable was to adjust for differences among schools in the SSPA scores due to allocating students to schools according to their SSPA scores. The school intake would remove any inflation in the variation among schools in the HKCEE, interpreted as the school effects.

Two multilevel models, with or without streaming adjusted, were constructed. Model lb with INTAKE as an explanatory variable removed any artificial inflation of school effects resulting from streaming. Model 1a would not correct for such inflation.

Model 1a [Y.sub.ij] = [[beta].sub.0] + [[beta].sub.1]SSP[A.sub.ij] + [[beta].sub.2][GENDER.sub.ij] + [[beta].sub.3]CO- ED[U.sub.j] + [u.sub.0j] + [e.sub.ij]

Model 1b [Y.sub.ij] = [[beta].sub.0] + [[beta].sub.1]SSP[A.sub.ij] + [[beta].sub.2][GENDER.sub.ij] + [[beta].sub.3]CO- ED[U.sub.j] + [[beta].sub.4][INTAKE.sub.j] + [u.sub.0j] + [e.sub.ij]

[Y.sub.ij] represented the standardised HKCEE subject score of the ith student studying in the jth school. Both SSPA and INTAKE were centred around their grand means.

Following Goldstein (1995), an intra-school correlation is defined as the ratio of the residual variance at school level to the sum of the residual variances at both school and student level. This intra-school correlation can be interpreted as the size of the effects schools have on students' achievement. It ranges between 0 and 1, with 0 indicating minimal or no effect and 1 very large effect. In this study, the intra-school correlation was used to compare the effectiveness of different systems of schooling.

The school-level residual [u.sub.j] is often taken as an indication of the effectiveness of a single school. It is called the relative effectiveness of a school after all other explanatory variables have been adjusted for. When the residual is a positive number, the school is more effective than a typical (average) school. When it is negative, the school is less effective than a typical school. In the United Kingdom and in Hong Kong, there are major studies (Fitz-Gibbon, 1997; Hill, Mok, Jane, Au, & Richardson, 2001) producing statistics for government in supervising the quality of a school.

Results and discussion

Table 1 shows that, on the average, boys did slightly better than girls in the SSPA; a mean of 0.12 for boys as compared with -0.10 for gifts. The very much higher SSPA scores for single-sexed schools probably reflects the fact that they were more selective than co-educational schools. The high mean and the low standard deviation of the SSPA supported the observation that gifts only schools are the most selective schools in Hong Kong. This could be part of the reasons why the gifts schools out-perform all other schools in the HKCEE.

Several observations can be made in relation to how these students performed in the HKCEE. First, except for mathematics, gifts out performed boys in English, Chinese and `the best five plus English'. These results contradicted the argument made by the Education Department of the Hong Kong Government in their recent unsuccessful defence of sex-biased practices in SSPA since the early 1980s. Their defence was that the better performance of gifts in primary schools should be discounted since gifts mature faster than boys in the primary school years. The better performance of gifts in the HKCEE argued against any `catching-up' even by the time boys and gifts completed secondary school. (See Hartmann (2001) on the judgement of the case.)

Secondly, the differences among schools on the HKCEE corresponded to the differences in SSPA among schools. The gifts schools did best, followed by the boys schools and the co-educational schools did the worst.

Thirdly, the students examined in this study were slightly more able and more homogeneous than the population of students who took the HKCEE; the mean standardised HKCEE scores obtained for our sample were greater than zero and their standard deviations were greater than one on a number of HKCEE examinations. The bias was particularly apparent in the gifts schools.

Similar bias has been observed by Thomas and Mortimore (1996) who argued that low ability students were more likely to have missing records in prior attainment and in examination results. Another reason for the bias is our decision to assess `pure' school effects, uncontaminated by students who were repeating the HKCEE or who sat as `private candidates' to the HKCEE.

Models with and without streaming effects

The results of the multilevel analyses of the two multilevel models (models 1a and 1b) were displayed in Table 2. The school-level variances ([[sigma].sub.0.sup.2]) were statistically significant in the models for with- and without- streaming effects (p<0.01), demonstrating strong evidence of school effects on students' achievement. The intra-school correlations ranged from a high value of 0.37 in English to a low but still comparatively high value of 0.13 for mathematics that indicated large school effects.

The adjusted school effects were large relative to what has been reported in the literature of school effectiveness for mixed ability schools; their adjusted intra-school correlations were only around .10. The large school effects in the Hong Kong data may well be the result of a number of factors: the emphasis on teaching to the public examination in the Asian school systems; the wide disparity in the qualities of schools as a result of streaming; and the over-dependence of students on teachers as the main source of learning.

There was a large drop in the intra-school correlation when the contextual effect of the INTAKE variable (mean SSPA of students in a school) was introduced into the model. English dropped from 0.37 to 0.13, `best 5 + English' from 0.28 to 0.12, Chinese from 0.19 to 0.12 and mathematics from 0.11 to 0.08.

There are two possible reasons for the drop. First, the streaming of students according to their academic achievement at entry to secondary school accentuated the differences of academic ability of students among schools. Although this difference was statistically controlled for by including the mean level of SSPA, the larger differences in students' achievement among schools were reduced. The second reason may well be that the contextual effects of streaming were large. Examples of such contextual effects could be the ease of teaching students of similar ability, students' collaboration and morale.

The results in both schools with and without streaming effects indicate that girls generally did better than boys in all the HKCEE other than mathematics. This phenomenon is consistent with findings elsewhere, except that the recent trend in GCSE in the United Kingdom is that girls overtook boys even in science subjects.

The coefficients for CO-EDU in all subjects with the exception of English changed from negative to positive after the adjustment for streaming effects. This indicates that, on the surface, all boys or all girls schools appeared to be more effective than co-educational schools. But one reason for their greater effectiveness is that they have better students to work with.

Models with random intercepts and random slopes

One usual approach in judging effectiveness or value-addedness of schools is to compare the regression lines of schools, where the outcome measure is regressed on prior attainments and other relevant explanatory factors. When the regression lines for schools are parallel, namely the slopes of the regressions are all equal, differences in effectiveness can be based only on the intercepts of the regression lines.

Differential effectiveness occurs when some schools are more effective at teaching students of a particular ability, such as the more able students, and other schools at teaching less able students. The regression lines for schools will not be parallel in such a situation. When this happens, differences in the effectiveness of a school will depend on the point of comparison on the prior attainment. A simple comparison of the intercepts will not do. Differences between intercepts only represent a comparison at the zero value on the prior attainment.

In the previous models, Model la and Model 1b, we have assumed the similarity of the slopes of the regression lines relating prior attainment (SSPA) to eventual outcome (HKCEE). There is increasing evidence that the slope coefficient and the intercept should be allowed to vary in order to reflect the fact that some schools can be more effective than others; these schools are indicated by a larger positive intercept and slope. This differential effectiveness of schools can be modelled in multilevel analysis by assuming a different SSPA slope coefficient and a different intercept coefficient for each school. Instead we assume a random coefficient model, random for the slope coefficient associated with SSPA and random for the intercept coefficient. To test this feature, a linear regression model (Model 2a) is fitted.

[Y.sub.ij] = [[beta].sub.0] + [[beta].sub.1]SSP[A.sub.ij] + [[beta].sub.2][GENDER.sub.ij] + [[beta].sub.3]CO- ED[U.sub.j] + [[beta].sub.4][INTAKE.sub.j] + [u.sub.0j] + [u.sub.1j] SSP[A.sub.ij]+ [e.sub.ij]

Table 3 presents the multilevel analyses associated with the linear model. The statistical significance of the variances associated with the slope coefficient ([u.sub.1j]) of SSPA and the intercept coefficient ([u.sub.0j]) provided support for differential effectiveness in schools.

Furthermore, when the variance among schools was plotted against prior attainment, there was a clear U shape pattern in all the four HKCEE subjects. The U-shaped pattern could be considered as a support for differential effectiveness. A U-shaped pattern implied that the differences among schools in the predicted achievement of students in the HKCEE were the largest at the two extreme ends of the predictor, SSPA. If the prediction or regression lines for schools were parallel, the variance would be constant across the SSPA scores. However, if the regression lines cross-over one another at a common point, then one would expect the cross-over point to have a minimum variance and a large variance at the two extreme ends.

The statistical significance of differential effectiveness was not surprising. As stated earlier, differential effectiveness implies that some schools are better at teaching at a particular range of ability. In mixed ability schooling, differential effectiveness occurs more by chance than by design since, if it were not the case, it would imply that a school has been ignoring the needs of all students to cater exclusively for a subgroup of students.

Here the students were first classified into different ability groupings on the basis of examinations at Primary Six. The SSPA allocation process meant that schools would be admitting students of similar ability banding. A possible purpose of streaming was to allow schools to design teaching curriculum and strategies that would cater best to the ability range of the students they had admitted. For this reason, differential effectiveness in Hong Kong schools occurred more by design than by chance.

Differential effectiveness creates a problem in judging the effectiveness of a school. If some schools are more effective at teaching the more able students and other schools the less able students, then there is a difficulty in deciding at what level of ability of students should the school be compared with other schools.

There is an additional problem in the case of streamed schooling, namely that even if a point of comparison could be agreed on, such comparisons would be invalid because no students could be found at the point of comparison for some schools.

The results also showed that there was a negative correlation between the slope coefficient of the SSPA and the intercept coefficient. However the negative correlation could well have been the result of a ceiling effect of the HKCEE scores. A sizeable number of schools have intercepts near the maximum possible score on the HKCEE. For this reason, the regression line of these schools can only take a small slope in order to cover the range of ability of students above the average score. In order to control the possible ceiling effects, a quadratic function of SSPA was fitted in the fixed part of the model (Model 2b).

[Y.sub.ij] = [[beta].sub.0] + [[beta].sub.1]SSP[A.sub.ij] + [[beta].sub.12]SSP[A.sup.2.sub.ij] + [[beta].sub.2][GENDER.sub.ij] + [[beta].sub.3]CO-ED[U.sub.j] + [[beta].sub.4][INTAKE.sub.j] + [u.sub.0j] + [u.sub.1j] SSP[A.sub.ij]+ [e.sub.ij]

The term SSP[A.sup.2] was found to be significant in all models (p<0.01), and its coefficients were all negative whereas the coefficients for SSPA remained positive. Moreover the correlations were still all negative, and were even more negatively correlated than those in the models without SSP[A.sup.2] in the fixed part. This suggests that the negative correlation cannot be simply due to the ceiling effect.

The negative correlation could be caused by the presence of extreme observations and very little variations in the slope coefficients. An examination of the scatterplot of the slope coefficients and the intercept coefficients (Figure 1) indicated there was very little validity in this possibility. A more reasonable explanation is that the negative correlation represented a feature of the streamed schooling education system.

[FIGURE 1 OMITTED]

Whereas the intercept is often interpreted in school effectiveness research as the value-addedness of a school, the slope coefficient is less often interpreted. Thomas and Mortimore (1996) interpreted the slope coefficient associated with prior attainment to imply the degree of widening by a school of the achievement gap between the able and less able students. One interpretation of the negative correlation found in this study is that it is in the less effective schools that students were pulled further apart in terms of their achievements on the public examinations. But in the more effective schools, the differences in achievements were less accentuated.

One possible explanation is due to the school policies implemented in the Hong Kong schools. Because of the examination-oriented culture in Hong Kong, there is the practice of streaming in most Hong Kong secondary schools. In the less effective schools with many low ability students, these schools focus their resources mainly on those students that are likely to succeed in the HKCEE. Students are streamed according to their ability into classes. The schools then concentrate their resources on helping the class of more able students. As a result, there is a gap between the effect of schools on the performances of the able and the less able students respectively. In the more effective schools, there are very able students who have both the ability and the motivation to do well in the public examinations. Streaming is therefore less effective and the gap in the effects of schools on the performances of the able students is less apparent.

These negative correlations contradicted previous findings in school effectiveness research involving mixed ability schools. Thomas and Sammons (1997) reported fairly strong positive correlations between LRT slopes and intercepts for total score (r = 0.80) and mathematics (r = 0.53). Smith and Tomlinson (1989) found that a more able student gained a greater advantage from a good school than a less able student.

Conclusion

The present study is based on all the secondary schools in Hong Kong with over 40000 students coming from more than 330 schools. When judging value-addedness, it is necessary to control for all factors that may have an influence on students' achievement. In this study, both student and school variables are included in the multilevel model. The contextual variables--school type, mean ability level--were also included. What have not been included are variables related to classes or departments, and economic status. The prior attainment is based on the SSPA which was coded in a standardised examination score. Since this study excluded those students who had repeated in the schools or changed schools, the results provide reliable estimates for pure school effects.

Three major discrepancies between our study and Western studies are noted. These include: (a) the relatively large size of school effects, (b) the negative correlation between intercepts and slopes for prior attainment, and (c) the differential effectiveness for low and high ability students. It highlights the issue of whether or not the results of school effectiveness research based on mixed ability schools can be generalised to countries, such as Singapore, China or Japan, where students are streamed to schools according to their academic ability.

The present study highlights some areas of research into the reasons why the value-addedness of Hong Kong schools differs so greatly among one another. A question of concern is whether streaming of students into schools makes the disparity in quality and effectiveness of schools larger. Another unanswered question is the aptitude treatment interaction. Would students streamed to a school with more able students learn better than if streamed to a school of similar ability? Although it is possible to conduct more sophisticated multilevel analysis on the data set to explore this issue, it has not been attempted in this paper.

In a mixed ability school system, a large proportion of the variation in students' achievement is found at the class level. In this study no data have been collected to investigate the variation in students' achievement at the class level. Nor was information collected regarding teaching strategies or effectiveness of teachers in teaching students of a particular ability. There is a need for research on the large variation among the schools of Hong Kong to see whether this was caused by the factors at the class level.

The presence of differential effectiveness across schools in the streamed school system raised questions of determining value-addedness based on a single regression line. The problem is made difficult by the fact that it may be difficult to define a common point of comparison. A point of comparison appropriate for the high ability schools could be out of range for the low ability schools. New methods will have to be invented to investigate this situation. These may involve the use of piecewise regressions (Ramsay & Silverman, 1997) and other more sophisticated procedures.

Keywords

ability grouping

academic achievement

data analysis

educational attainment 

effect size

school effectiveness

Table 1 Mean standardised subject scores and standard deviation
(in parentheses)

                                          Mean standardised scores of
                                              English, Chinese,
                                           mathematics and `English
                                           + best 5 subjects' in
                Number      Mean SSPA           HKCEE, 1997
                  of        standardised
                students    scores, 1992      English

Students

Boys              18 836     0.12  (1.01)    -0.16  (1.01)
Girls             22 873    -0.10  (0.98)     0.13  (0.98)
Schools
Co-educational    31 233    -0.17  (0.98)    -0.17  (0.98)
Boys               4 964     0.48  (1.00)     0.16  (0.98)
Girls              5 512     0.51  (0.81)     0.80  (0.64)

                       Mean standardised scores of
                            English, Chinese,
                         mathematics and `English
                          + best 5 subjects' in
                               HKCEE, 1997

                                                      English +
                  Chinese          Mathematics         best 5
Students
Boys            -0.21  (0.99)      0.12  (1.01)      -0.07  (1.02)
Girls            0.17  (0.98)     -0.10  (0.98)       0.05  (0.98)
Schools
Co-educational  -0.10  (1.00)     -0.11  (1.00)      -0.13  (0.99)
Boys            -0.07  (0.94)      0.27  (1.00)       0.13  (1.00)
Girls            0.64  (0.76)      0.37  (0.83)       0.64  (0.75)

Table 2 Results of the multilevel analysis of null models and
models with and without streaming effects

                                   Null model

                                          Standard
                              Estimate     error

English
Fixed part
 Intercept                    0.005444    0.473
 SSPA                         --          --
 GENDER                       --          --
  (boys vs. girls)
 CO-EDU                       --          --
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.739       0.05766
 Student-level
  [[sigma].sub.e.sup.2]       0.3613      0.002512
 Intra-school                 0.6716
  correlation
-2 (log-likelihood)           77719.8
Chinese
Fixed part
 Intercept                    0.2033      0.03906
 SSPA                         --          --
 GENDER                       --          --
  (boys vs. girls)
 CO-EDU                       --          --
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.5018      0.03931
 Student-level
  [[sigma].sub.e.sup.2]       0.4538      0.003155
 Intra-school                 0.5251
  correlation
-2 (log-likelihood)           87025.8
Mathematics
Fixed part
 Intercept                    0.169       0.03888
 SSPA                         --          --
 GENDER                       --          --
  (boys vs. girls)
 CO-EDU                       --          --
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.4958      0.03894
 Student-level
  [[sigma].sub.e.sup.2]       0.5733      0.003986
 Intra-school                 0.4638
  correlation
-2 (log-likelihood)           96689.5
English + best 5
Fixed part
 Intercept                    0. 1756     0.04056
 SSPA                         --          --
 GENDER                       --          --
  (boys vs. girls)
 CO-EDU                       --          --
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.5427      0.04236
 Student-level
  [[sigma].sub.e.sup.2]       0.3416      0.002375
 Intra-school                 0.6137
  correlation
-2 (log-likelihood)           75293.3

                              With-streaming-effects
                                    (Model 1a)

                                          Standard
                              Estimate     error

English
Fixed part
 Intercept                    0.5255      0.04598
 SSPA                         0.005763    0.00005715
 GENDER                       -0.3519     0.006192
  (boys vs. girls)
 CO-EDU                       -0.3434     0.05285
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.1675      0.01323
 Student-level
  [[sigma].sub.e.sup.2]       0.2852      0.001983
 Intra-school                 0.3700
  correlation
-2 (log-likelihood)           67446.9
Chinese
Fixed part
 Intercept                    0.558       0.03317
 SSPA                         0.006001    0.00006328
 GENDER                       -0.4184     0.006998
  (boys vs. girls)
 CO-EDU                       -0.07778    0.03809
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.08442     0.006842
 Student-level
  [[sigma].sub.e.sup.2]       0.3673      0.002553
 Intra-school                 0.1869
  correlation
-2 (log-likelihood)           77686.7
Mathematics
Fixed part
 Intercept                    0.2731      0.02919
 SSPA                         0.006678    0.00007043
 GENDER                       0.1402      0.007964
  (boys vs. girls)
 CO-EDU                       -0.06056    0.03347
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.06298     0.00525
 Student-level
  [[sigma].sub.e.sup.2]       0.48        0.003337
 Intra-school                 0.1160
  correlation
-2 (log-likelihood)           88673.1
English + best 5
Fixed part
 Intercept                    0.4858      0.03758
 SSPA                         0.005476    0.00005656
 GENDER                       -0.1584     0.00617
  (boys vs. girls)
 CO-EDU                       -0.1865     0.04319
  (co-edu. vs. single)
 INTAKE                       --          --
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.1108      0.00882
 Student-level
  [[sigma].sub.e.sup.2]       0.284       0.001975
 Intra-school                 0.2806
  correlation
-2 (log-likelihood)           67134.8

                               Without-streaming-
                               effects (Model 1b)

                                          Standard
                              Estimate     error

English
Fixed part
 Intercept                    0.4162      0.02426
 SSPA                         0.005599    0.00005813
 GENDER                       -0.3536     0.006151
  (boys vs. girls)
 CO-EDU                       -0.08785    0.02866
  (co-edu. vs. single)
 INTAKE                       0.004921    0.0001703
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.04286     0.003542
 Student-level
  [[sigma].sub.e.sup.2]       0.2852      0.001983
 Intra-school                 0.1306
  correlation
-2 (log-likelihood)           67002.3
Chinese
Fixed part
 Intercept                    0.4954      0.02596
 SSPA                         0.005778    0.00006596
 GENDER                       -0.4177     0.006972
  (boys vs. girls)
 CO-EDU                       0.06206     0.03063
  (co-edu. vs. single)
 INTAKE                       0.002776    0.0001835
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.04854     0.004042
 Student-level
  [[sigma].sub.e.sup.2]       0.3672      0.002553
 Intra-school                 0.1168
  correlation
-2 (log-likelihood)           77501.8
Mathematics
Fixed part
 Intercept                    0.2196      0.02432
 SSPA                         0.006372    0.00007537
 GENDER                       0.1416      0.007928
  (boys vs. girls)
 CO-EDU                       0.05274     0.02857
  (co-edu. vs. single)
 INTAKE                       0.002389    0.0001772
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.04065     0.003506
 Student-level
  [[sigma].sub.e.sup.2]       0.4797      0.003335
 Intra-school                 0.0781
  correlation
-2 (log-likelihood)           88517.6
English + best 5
Fixed part
 Intercept                    0.3996      0.02289
 SSPA                         0.005288    0.000058
 GENDER                       -0.1599     0.006131
  (boys vs. girls)
 CO-EDU                       0.01139     0.02701
  (co-edu. vs. single)
 INTAKE                       0.003846    0.0001618
Random part
 School-level
  [[sigma].sub.0.sup.2]       0.03777     0.003146
 Student-level
  [[sigma].sub.e.sup.2]       0.2839      0.001974
 Intra-school                 0.1174
  correlation
-2 (log-likelihood)           66782.5

Table 3 Results of the multilevel analysis with random intercept and
slope

                                  Linear function of SSPA
                                         (Model 2a)

                                          Standard    Correlation
                            Estimate       error          (a)

English
Fixed part
 Intercept                0.5234        0.02546
 SSPA                     0.006211      0.0001303
 SSPA x SSPA              --            --
 GENDER                   -0.3619       0.006132
  (boys vs. girls)
 CO-EDU                   -0.09498      0.02989
  (co-edu. vs. single)
 INTAKE                   0.003902      0.0001874
Random part (b)
 School-level
  [[sigma].sub.0.sup.2]   0.04482       0.004055
  [[sigma].sub.01.sup.2]  -0.0001203    0.00002906    -0.277
  [[sigma].sub.1.sup.2]   0.000004219   0.000004137
 Student-level
  [[sigma].sub.e.sup.2]   0.275         0.001919
-2 (log-likelihood)       66044.5
Chinese
Fixed part
 Intercept                0.5536        0.02341
 SSPA                     0.006124      0.0001101
 SSPA x SSPA              --            --
 GENDER                   -0.4223       0.006982
 (boys vs. girls)
 CO-EDU                   0.05708       0.02729
 (co-edu. vs. single)
 INTAKE                   0.001542      0.0001792
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.03825       0.003552
  [[sigma].sub.01.sup.2]  -0.0001147    0.00002214    -0.388
  [[sigma].sub.1.sup.2]   0.000002287   0.0000002746
 Student-level
  [[sigma].sub.e.sup.2]   0.3626        0.002528
-2 (log-likelihood)       77238.8
Mathematics
Fixed part
 Intercept                0.2398        0.02517
 SSPA                     0.006491      0.0000994
 SSPA x SSPA              --            --
 GENDER                   0.1403        0.007965
 (boys vs. girls)
 CO-EDU                   0.05545       0.02953
 (co-edu. vs. single)
 INTAKE                   0.002404      0.0001899
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.04169       0.003883
  [[sigma].sub.01.sup.2]  -0.00004441   0.00001935
  [[sigma].sub.1.sup.2]   0.00000115    0.0000002078
 Student-level
  [[sigma].sub.e.sup.2]   0.4765        0.003322
-2 (log-likelihood)       88436
English + best 5
Fixed part
 Intercept                0.4234        0.02175
 SSPA                     0.005426      0.00008566
 SSPA x SSPA              --            --
 GENDER                   -0.1625       0.006136
 (boys vs. girls)
 CO-EDU                   0.009724      0.02506
 (co-edu. vs. single)
 INTAKE                   0.00327       0.0001669
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.03897       0.003409
  [[sigma].sub.01.sup.2]  -0.0001249    0.00001729
  [[sigma].sub.1.sup.2]   0.000001142   0.0000001596
 Student-level
  [[sigma].sub.e.sup.2]   0.2809        0.0019590000
-2 (log-likelihood)       66568.2

                                 Quadratic function of SSPA
                                         (Model 2b)

                                          Standard
                            Estimate       error      Correlation

English
Fixed part
 Intercept                0.5086        0.0231
 SSPA                     0.006201      0.0000891
 SSPA x SSPA              -0.000008015  0.0000004319
 GENDER                   -0.363        0.006101
 (boys vs. girls)
 CO-EDU                   -0.09342      0.02693
 (co-edu. vs. single)
 INTAKE                   0.003903      0.0001735
Random part (b)
 School-level
  [[sigma].sub.0.sup.2]   0.0394        0.0349
  [[sigma].sub.01.sup.2]  -0.00009512   0.0000177     -0.418
  [[sigma].sub.1.sup.2]   0.000001317   0.0000001768
 Student-level
  [[sigma].sub.e.sup.2]   0.2757        0.001923
-2 (log-likelihood)       65843.2
Chinese
Fixed part
 Intercept                0.5737        0.02186
 SSPA                     0.006239      0.00007723
 SSPA x SSPA              -0.000007996  0.0000003988
 GENDER                   -0.4255       0.006921
 (boys vs. girls)
 CO-EDU                   0.04319       0.02521
 (co-edu. vs. single)
 INTAKE                   0.001822      0.0001677
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.03562       0.00313
  [[sigma].sub.01.sup.2]  -0.000006624  0.00001303    -0.586
  [[sigma].sub.1.sup.2]   0.0000003586  0.0000001085
 Student-level
  [[sigma].sub.e.sup.2]   0.3626        0.002527
-2 (log-likelihood)       76968.9
Mathematics
Fixed part
 Intercept                0.2601        0.02437
 SSPA                     0.006597      0.00009233
 SSPA x SSPA              -0.000003651  0.0000004762
 GENDER                   0.1384        0.007945
 (boys vs. girls)
 CO-EDU                   0.04521       0.02834
 (co-edu. vs. single)
 INTAKE                   0.002186      0.0001848
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.03978       0.003645
  [[sigma].sub.01.sup.2]  -0.00004925   0.00001661    -0.302
  [[sigma].sub.1.sup.2]   0.0000006668  0.0000001631
 Student-level
  [[sigma].sub.e.sup.2]   0.4768        0.003324
-2 (log-likelihood)       88385.7
English + best 5
Fixed part
 Intercept                0.4433        0.02106
 SSPA                     0.005507      0.00007672
 SSPA x SSPA              -0.00003701   0.0000003882
 GENDER                   -0.164        0.006121
 (boys vs. girls)
 CO-EDU                   -0.0003079    0.0241
 (co-edu. vs. single)
 INTAKE                   0.003427      0.0001635
Random part
 School-level
  [[sigma].sub.0.sup.2]   0.03638       0.003152
  [[sigma].sub.01.sup.2]  -0.00009956   0.00001427    -0.64
  [[sigma].sub.1.sup.2]   0.0000006655  0.0000001184
 Student-level
  [[sigma].sub.e.sup.2]   0.2811        0.001959
-2 (log-likelihood)       66492.1

(a) Correlation between SSPA slope and intercept

(b) [[sigma].sub.0.sup.2] = variance of CONS at school-level,
[[sigma].sub.01.sup.2] = covariance of CONS and SSPA at school-level,
[[sigma].sub.01.sup.2] = variance of SSPA at school-level,
[[sigma].sub.e.sup.2] = variance of CONS at student-level

Acknowledgements

The study is supported by the Research Grant Council of Hong Kong.

References

Boskers, R.J. & Witzers, B. (1995). School effects, problems, solutions and meta-analysis. Paper presented at the International Congress for School Effectiveness and School Improvement, Leeuwarden, Netherlands.

Boskers, R.J. & Witzers, B. (1996). The magnitude of school effects. Or: Does it really matter which school a student attends? Paper presented at AERA annual conference, New York.

Coleman, J.S., Campbell, E.O., Hobson, D.F., McPartland, M.M., Mood, A.M., Weinfeld, F.D., & Yourk, R.L. (1966). Equality of educational opportunity. Washington, DC: Department of Health, Education, and Welfare.

Coe, R. & Fitz-Gibbon, C.T. (1998). School effectiveness research: Criticisms and recommendations. Oxford Review of Education, 24(4), 421-438.

Daly, P. (1991). How large are secondary school effects in Northern Ireland? School Effectiveness and School Improvement, 2(4), 305-323.

Fitz-Gibbon, C.T. (1997). The value-added national project: The final report. Durham: SCAA.

Goldstein, H. (1995). Multilevel statistical models. London: Edward Arnold.

Gray, J. (1989). Multilevel models: Issues and problems emerging from their recent application in British studies of school effectiveness. In R.D. Bock (Ed.), Multilevel analysis of educational data. New York: Academic Press.

Hartmann, J. (2001). Judgment on constitutional and administrative law list no. 1555 of 2000: Between equal opportunities commission (applicant) and director of education (respondent). Hong Kong: Justice Department.

Hill, P.W. & Rowe, K.J. (1996). Multilevel modelling in school effectiveness research. School Effectiveness and School Improvement, 7(1), 1-34.

Hill, P.W., Mok, M., Jane, G., Au, W.K., & Richardson, C. (2001). The Quality Education Fund Project on development of performance indicators for measuring value-added improvement of primary and secondary students' academic performance. Melbourne: University of Melbourne, Centre for Applied Educational Research.

Levine, D. (1992). An interpretive review of US research and practice dealing with unusally effective schools. In D Reynolds & P. Cuttance (Eds.), School effectiveness: Research, policy and practice. London: Cassell.

Mortimore, P. & Sammons, P. (1994). School effectiveness and value added measure. Assessment in Education: Principle, Policy & Practice, 1(3), 315-332.

Purkey, S.C. & Smith, M.S. (1983) Effective schools: A review. Elementary School Journal, 83(4), 427-452.

Ramsay, J.O. & Silverman, B.W. (1997). Functional data analysis. New York: Springer-Verlag.

Reynolds, D. (1992) School effectiveness and school improvement: An updated reivew of the British literature. In D. Reynolds and P. Cuttance (Eds.), School effectiveness: Research, policy and practice. London: Cassell.

Rutter, M., Maughan, B., Mortimore, P., & Ouston, J. (1979). Fifteen thousand hours: Secondary schools and their effects on children. London: Open University.

Sammons, P., Nuttal, D., & Cuttance, P. (1993). Differential school effectiveness: Results from a reanalysis of the Inner London Education Authority's Junior School Project data. British Educational Research Journal, 19(4), 381-405.

Sammons, P., Nuttal, D., Cuttance, P., & Thomas, S. (1993). Continuity of school effects: A longitudinal analysis of primary and secondary school effects on GCSE performance. An unpublished draft paper.

Scheerens, J. (1992). Effective school research: Theory and practice. London: Cassell.

Scheerens, J. & Boskers, R.J. (1997). The foundations of educational effectiveness. Oxford: Pergamon.

Scheerens, J., Vermeulen, C.J.A.J., & Pelgrum, W.J. (1989). Generalizability of instructional and school effectiveness indicators across nations. International Journal of Educational Research, 13, 789-800.

Smith, D.J. & Tomlinson, S. (1989). The school effect: A study of multi-racial comprehensives. London: Policy Studies Institute.

Thomas, S. & Mortimore, P. (1996). Comparison of value-added models for secondary-school effectiveness. Research Papers in Education, 11(1), 5-33.

Thomas, S. & Sammons, P. (1997). Differential secondary school effectiveness: Comparing the performance of different pupil groups. British Educational Research Journal, 23(4), 451-469.

Willms, J.D. (1986). Social class segregation and its relationship to pupils' examination results in Scotland. American Sociological Review, 51,224-241.

Willms, J.D. (1987). Differences between Scottish Education Authorities in their examination attainment. Oxford Review of Education, 13(2), 211-232.

Willms, J.D. & Raudenbush, S.W. (1989). A longitudinal hierarchical linear model for estimating school effects and their stability. Journal of Educational Measurement, 26, 209-232.

Woessman, L. (2001). Why students in some countries do better. Education Matters, 1(2), 67-74.

Woodhouse, G. (Ed.). (1995). A guide to MLn for new users. London: University of London, Institute of Education.

Yang, M., Goldstein, H., Rath, T., & Hill, N. (1998). The use of assessment data for school improvement purposes. An unpublished draft paper.

Raymond Lam is a Lecturer, Dr Kam-cheung Wong is a Senior Lecturer, and Dr L. M. Ho is a Senior Research Assistant in the Department of Education, University of Hong Kong, Pokfulam Road, Hong Kong.

E-mail: [e-mail] [e-mail] [e-mail]