State differences in achievement among secondary school students in Australia.

A number of recent national studies of student achievement in secondary school have reported differences between the Australian states and territories. State differences are often viewed as insubstantial or as simply reflecting sociodemographic factors, or differences between the states in the grades or ages of the students sampled. In this article, we show that state differences are larger than generally assumed and cannot be attributed to socioeconomic and demographic factors. Generally, student achievement in reading, mathematics and science are higher in New South Wales than the other states, once demographic and grade differences are taken into account. Of concern, is the increased likelihood that students from Queensland, Victoria and Tasmania have in only reaching the lowest OECD proficiency level in reading. We conclude that state differences are meaningful and do have policy implications.

Introduction

The division of Australia into states and territories (hereafter, referred to as states) is one of the most important aspects of the Australian school system. Constitutionally, the states have jurisdiction over education and a large proportion of their budgets is allocated to schools. Each state has its own curriculum, standards, and certificates. With few exceptions, each school in a state or territory is subject to the standards and organisation of that state, usually, regardless of school sector. This difference in the organisation of education between the states may have implications for student outcomes.

Differences in student outcomes between states have long been a contentious issue. State governments of all political persuasions have been reluctant to allow comparisons between states since state differences in student outcomes may be attributed to the quality of teaching, the level of funding or the organisation of the education system. State governments and state education systems raise a number of plausible reasons why comparisons between states can be misleading. States differ in: when a child is allowed to start school, the presence of a preparatory year before first grade, the number of years of secondary school, the senior secondary school curriculum, and the proportion of students who reach Year 12 and compete for university entrance. Also the states differ in the proportion of rural and Indigenous students and the students' socioeconomic backgrounds (1). These considerations complicate comparisons between the states and undermine conclusions on whether particular states deliver better or worse educational outcomes for their students. It is unfortunate that comparing states is so difficult because it is one area where educational policies can be evaluated; states could share information that may lead to better outcomes for all students.

Previous research

Although there are problems in making comparisons, state differences in achievement among secondary school students appear quite large. In 1964, the mean mathematics score of Queensland students in the First International Mathematics Study was about 0.4 of a standard deviation higher than that for students in Victoria and New South Wales, and the mean performance of Western Australian and Tasmanian students was lower still.

In the 1978 Second International Mathematics Study, the mean mathematics scores of students from the Australian Capital Territory and Queensland were higher than that for the other states. The relative performance of the states had changed since 1964 (Afrassa & Keeves, 1999; Masters, 2003).

In 1983, seven percentage points separated the average scores of students in word knowledge across the states and territories from 68 in the Australia Capital Territory (ACT) to 61 in Victoria. In mathematics, the ACT and Queensland showed the highest scores (73 and 70), and Victoria and the Northern Territory the lowest scores (64 and 62).These differences compare to the one or two percentage point difference between male and female students (Rosier & Banks, 1990, p. 83). In science, the mean score percentage for the ACT (65) was significantly higher than the scores for New South Wales (58), Victoria (56), South Australia (59), Tasmania (59) and the Northern Territory (56). For comparison, male students scored on average four percentage points higher than female students (Rosier & Banks, 1990, pp. 83, 126-127).

In the 1994 Third International Mathematics and Science Study (TIMSS), the average mathematics score of students from the highest performing state, Western Australia, was 60 score points higher than average score for the Northern Territory. Western Australian students scored, on average, 54 score points higher than Victorian students. Considering that students' scores were scaled to a standard deviation of 100, this difference is large. Similar patterns were found for science with the students from Western Australia scoring, on average, 63 points higher than Victorian students. In contrast in both mathematics and science, gender differences were trivial and not statistically significant (Lokan, Ford, & Greenwood, 1996, pp. 32, 35). These substantial state differences in achievement may be due to differences between states in the grades sampled. In Western Australia, South Australia, Queensland and the Northern Territory, grades 8 and 9 were sampled, whereas for the other states grades 7 and 8 were sampled. Students in higher grades show higher average scores than students in lower grades. Although these differences do not account for the strong performance of students from the ACT, it does account for the higher average scores of students from Western Australia, South Australia and Queensland.

The OECD/Programme for International Student Assessment (PISA) study, carried out in 2000, sampled 15-year-olds rather than students in specific grades and found that the ACT, New South Wales and Western Australia showed the highest mean scores in reading and mathematics, and South Australia replaced New South Wales to be among the top three performing states in science (Lokan, Greenwood, & Cresswell, 2001, pp. 121-123).

In the international context, the mean scores of the Australian states are not clustered around the Australian mean. The mean reading score of ACT students is higher than that for any country participating in PISA. Similarly, students in New South Wales, South Australia and Western Australia show average reading scores higher than all countries except Finland. However, the mean scores of students from Queensland, Victoria, Tasmania and the Northern Territory are much lower (see Figure 1). Although there is considerable variation in the mean scores of the Australian states, no state mean was substantially below the OECD mean.

[FIGURE 1 OMITTED]

PISA also showed strong state differences in the proportions of students not reaching basic proficiency (Lokan et al., 2001, pp. 65-72; OECD, 2001a, pp. 34-49, 71-78, 83-87). In the ACT only 8 per cent of students were at or below the lowest proficiency level in reading, compared to 11 per cent in New South Wales, 15 per cent in Victoria, 19 per cent in Tasmania and 23 per cent in the Northern Territory (Lokan et al., 2001, pp. 125, 127, 128). However, these comparisons are also vulnerable to similar problems to the TIMSS study; Western Australia and Queensland, and to a lesser extent South Australia, have a higher proportion of 15-year-old students in Year 11 (see Table 1).

State differences may not disappear when taking into account other factors. Analysing data from the 1975 Australian Studies in School Performance, Bourke and Keeves (1977, pp. 87-88) concluded that state differences in achievement cannot be entirely attributed to differences in year level. Their analyses suggested that state was an important discriminator although the authors note it was difficult to disentangle its effects from age and grade. Williams, Clancy, Batten, & Girling-Butcher (1980, pp. 57-60) analysed the effects of state on achievement, net of sociodemographic factors, but not grade, and concluded that state differences were minor. Compared to students in New South Wales, Queensland students show significantly higher scores in literacy and numeracy and students from the Northern Territory significantly lower scores. The other states tended to show higher scores than New South Wales. A later analysis of the same data that controlled for grade as well as sociodemographic factors, found that all states except Tasmania and the ACT had significantly lower overall levels of achievement than New South Wales (Williams & Carpenter, 1990).

This review indicates that the ACT shows the highest scores and the Northern Territory the lowest scores. Although the strong performance of Queensland students in numeracy in 1975 was consistent with earlier studies (see Williams et al., 1980, p. 59), this is not apparent in more recent studies of student achievement (see Masters, 2003). Western Australia has replaced Queensland as the best performing major state. Victoria and Tasmania tend to show lower average scores than New South Wales. However, there are a number of issues to resolve before these differences between states can be linked to the organisation and policy mix of state education systems. The higher average score of ACT students is likely to be due to their higher socioeconomic backgrounds. The higher proportion of Indigenous students in the Northern Territory may explain why its scores tend to be substantially below the national average. Similarly, the higher proportion of students living in nonmetropolitan rural Tasmania may account for that state's lower average scores. The stronger performance of Queensland in the earlier studies and Western Australia more recently may be due to differences in the distribution of students across grades since in these states there are higher proportions of students in higher grades.

Therefore, the purpose of this article is to examine state differences in achievement among Australian secondary school students and the extent to which these differences can be accounted for by socioeconomic background, the proportion of Indigenous students, the regional distribution of students and students' grade. We examine the state effects on achievement in reading, mathematics and science, and their effects on not reaching basic proficiency in reading. The investigation of the net effects of state has not been performed on recent national achievement data.

Data and measures

Data has been drawn from the PISA 2000 study, which investigated achievement of 15-year-old students in reading, mathematics and science in 32 countries. The Australian data comprises 5477 students from all states and territories. The smaller states were oversampled so that conclusions on state differences would not be undermined by small sample sizes. Schools were sampled with probability according to size and then samples of 15-year-old students were drawn within selected schools. Further details on the data, tests, and sampling procedures can be found in the international PISA and Australian reports (Lokan et al., 2001; OECD, 2001b). Item Response Theory (IRT) modelling was used to create scores which were standardised for each of the three domains--reading, mathematical and scientific literacy--at an international mean of 500 and standard deviation of 100. Details on this procedure are available from the PISA technical report (OECD, 2002).The scale of measures in reading was divided into proficiency levels, to give a clearer view of the knowledge and skills that students possess. In PISA there were five proficiency levels described, Level 1 the most basic and Level 5 the most advanced. There was a small number of students who scored below Level 1, which meant that this assessment was unable to clearly describe their reading skills--it does not mean that these students possess no reading skills. Generally, reasonable proficiency in reading was defined as Level 2 or above. The cut-off point for this was a score of 407.47.

Students' state and territory is indicated by the school's address rather than the students' home address. Socioeconomic background is a composite comprising father's and mother's occupation and education. It was constructed by the sheaf variable method (see Whitt, 1986) which maximised the effects of the four variables on combined achievement score. It is a continuous variable standardised with a mean of zero and a standard deviation of one. Indigenous status was measured by asking students if they considered themselves as Aboriginal or a Torres Strait Islander. Region is measured by the Jones typology distinguishing between students living in mainland state capital, other major urban centre, a provincial city, other regional area, or a remote area (Jones, 2000, 2002). Grade or year level was ascertained by the sample lists of students and centred at Grade 10. The percentages for students in different grades in the various states for the PISA study are shown in Table 1. Age was not included in the analysis since all students were 15 years old and age in months was found to have only a small negative effect on achievement when taking into account grade level.

The effects of state and territory on achievement are presented as Ordinary Least Squares (OLS) regression coefficients that reflect the average difference in achievement score for students from different states relative to students from New South Wales. The standard errors associated with the regression coefficients have been adjusted to take into account the cluster design of the sample and sample stratification. Each regression coefficient and standard error was calculated by averaging the results obtained from separate analyses of the five plausible values obtained from IRT modelling. Logistic regression was used in the analyses of proficiency level. The estimates are log-odds, which are interpreted as odds ratios in the text. The levels of statistical significance for both the OLS and logistic regression coefficients are indicated in the tables (2). All analyses were weighted to reflect the population of 15-year-old students in the country.

State differences in achievement

Table 2 shows the effects of state on reading achievement. The estimates in the first column show the raw differences between the states and are almost exactly the same as the differences presented in the Australian report (Lokan et al., 2001, p. 121). The minor discrepancies are due to minor differences in method. The intercept is the average reading score for students from New South Wales. The only statistically significant state differences in this column are for Victoria with an average score 24 score points lower and the Northern Territory with an average score nearly 40 score points below that of New South Wales.

The second column shows state differences net of students' socioeconomic backgrounds. Controlling for socioeconomic background the difference in average scores between students from the ACT and New South Wales declines from 15 to 0 score points. Similarly, compared to New South Wales, the effects for Queensland and Tasmania declined with little change for Victoria, South Australia, Western Australia and the Northern Territory.

The addition of Indigenous status (column 3) reduced the effect for the Northern Territory, although the difference with New South Wales remained significant. There was little or no change for the estimates for the other states.

The additional control for region (column 4) produced little change except for the Northern Territory where differences with New South Wales decreased.

The addition of grade to the analysis (column 5) caused large changes to the effects, especially for Queensland and Western Australia--which is to be expected, because these two states have the largest proportion of Year 11 students. Students attending school in New South Wales showed significantly higher scores in reading than students in Victoria, Queensland, Western Australia, Tasmania and the Northern Territory. The effects are reasonably large, with effects of over 30 score points for the Northern Territory and Queensland and between 20 and 26 score points for Victoria, Western Australia and Tasmania. The addition of grade to the model, therefore, leads to the greatest change in the results and to the rank order of means of the states.

Similar, but not identical patterns were found for mathematics (Table 3). Corresponding with the results presented in the Australian PISA report, only the Northern Territory was significantly different from New South Wales (Lokan et al., 2001, p. 122). Controlling for socioeconoimc background, Indigenous status and region changed the magnitude of most estimates but no further significant differences were revealed. In contrast, when controlling for grade, mathematics achievement in New South Wales was significantly higher than in Queensland, South Australia, Western Australia, Tasmania and the Northern Territory. Relative to New South Wales, Victoria and to a lesser extent Western Australia performs better in mathematics than reading, while the reverse is true for South Australia. As was the case for reading, students in the New South Wales perform better in mathematics than students in several other states.

In the initial analysis of science achievement without controlling for other factors, only the Northern Territory shows significantly lower scores than New South Wales. Controlling for socioeconomic background moves the stronger performance of Western Australian students into statistical significance. The addition of Indigenous status only marginally reduced the effect for the Northern Territory and adding region did not change the overall pattern. When controlling for grade, Queensland and Tasmanian students performed at a significantly lower level than students from New South Wales and the effect for the Northern Territory increased to over 40 score points.

Table 5 shows the effects of state on level one proficiency. Tasmanian students were 1.8 times more likely to be in this group (rather than not in this group) relative to students in New South Wales. The respective odds ratios for Victorian and Queensland students were 1.6 and 1.3. There were no significant differences for the other states. Controlling for socioeconomic background decreased the effects for Queensland and Tasmania. Indigenous status produced little change in the pattern of state differences and the addition of region increased the effect for Tasmania. The addition of grade to the analysis increases the effect for Queensland, and to a lesser extent Tasmania. With addition of grade to the analysis, the effect for Western Australia becomes larger and significant, indicating that Western Australian students are now 1.5 times more likely to be only at proficiency level 1 or below than students from New South Wales. The effect for Queensland was larger with an odds ratio of 2.1.

Discussion and conclusions

In the context of a move towards greater uniformity in the delivery of education services across Australia, state differences in achievement are non-trivial especially when sociodemographic and educational differences between the states are taken into account. A difference of between 20 and 40 score points is relatively large in the context of the PISA study in which the scores have been standardised to a standard deviation of 100 score points.

It appears that the higher attainments of ACT students can be attributed to their generally higher socioeconomic backgrounds. The weaker performance of students from the Northern Territory cannot be attributed to the higher proportion of Indigenous students or differences in socioeconomic background. When taking into account grade level, New South Wales students show significantly higher average scores than five other states and territories in reading and mathematics and three other states and territories in science. This is consistent with the finding that students from New South Wales also showed higher achievement scores when taking into account grade in data collected 25 years earlier in 1975.

In the analysis of proficiency levels, students from Victoria, Queensland, Tasmania and Western Australia are significantly more likely to be at or below the lowest OECD proficiency level than students from New South Wales. This finding should be of concern to educational administrators in these states. Also of concern is the high proportion of Tasmanian students with achievement scores below 350 (see Figure 1). Improving the performance of weaker students in these states would increase the overall mean achievement levels of these states. It would also have the additional benefit of improving their chances of completing school and their prospects in the labour market.

The next stage is to investigate the reasons for the state differences reported here. However, an investigation of the administrative and policy configurations responsible for these state differences is beyond the scope of this paper. What is it about the provision of teaching and learning in New South Wales that is responsible for its students achieving higher average achievement scores than students in some of the other states? Are there differences in the provision of services to low achieving students that may explain the lower propensity of students from New South Wales to perform at or below the lowest OCED proficiency level? Addressing these questions may lead to the development of policies that improve the achievements of Australian students, especially low achievers who are most at risk of unemployment and other undesirable labour market outcomes.

Keywords

secondary education

reading

academic achievement

mathematics

state norms

science education

References

Afrassa, T. M. & Keeves, J. P. (1999). Changes in students' mathematics achievement in Australian lower secondary schools over time. International Education Journal, 1(1), 1-21.

Bourke, S. F. & Keeves, J. P. (1977). The Mastery of Literacy and Numeracy: Final Report (Vol. 3). Canberra: AGPS.

Jones, R. G. (2000). Development of a Common Definition of and Approach to Data Collection on, the Geographic Location of Students to be Used for Nationally Comparable Reporting of Outcomes of Schooling within the Context of "National Goals for Schooling in the 21st Century". Melbourne: MCEETYA.

Jones, R. G. (2002). Educational Participation and Outcomes by Geographic Location (LSAY Research Report No. 26). Melbourne: Australian Council for Educational Research.

Littell, R. C., Milliken, G. A., Stroup, W. W., & Wolfinger, R. D. (1996). SAS System for Mixed Models. Cary, NC: SAS Institute.

Lokan, J., Ford, P., & Greenwood, L. (1996). Maths and Science on the Line: Australian Junior Secondary School Students' Performance in the Third International Mathematics and Science Study (TIMSS Australia Monograph No. 1). Camberwell, Vic.: Australian Council for Educational Research.

Lokan, J., Greenwood, L., & Cresswell, J. (2001). 15-up and Counting, Reading, Writing, Reasoning: How Literate are Australia's Students? The Pisa 2000 Survey of Students' Reading, Mathematical and Scientific Literacy Skills. Melbourne: Australian Council for Educational Research.

Masters, G. N. (2003). Challenges in national comparisons. Professional Educator, 2(2), 14-16.

OECD. (2001a). Knowledge and Skills for Life: First Results from PISA 2000. Paris: Author.

OECD. (2001b). Knowledge and Skills for Life: First Results from the OECD Programme for International Student Assessment. Paris: Author.

OECD. (2002). PISA 2000 Technical Report. Paris: Author.

Rosier, M. J. & Banks, D. K. (1990). The Scientific Literacy of Australian Students: Science Achievement of Students in Australian Primary and Lower Secondary Schools (Vol. 39). Hawthorn, Vic.: Australian Council for Educational Research.

Whitt, H. P. (1986). The sheaf coefficient: A simplified and expanded approach. Social Science Research, 15, 174-189.

Williams,T. & Carpenter, P. G. (1990). Private schooling and public achievement. Australian Journal of Education, 34, 3-24.

Williams, T., Clancy, J., Batten, M., & Girling-Butcher, S. (1980). School, Work and Career: 17 year olds in Australia (ACER Research Monograph No. 6). Hawthorn,Vic.: Australian Council for Educational Research.

Notes

(1) Masters (2003) notes that possible misinterpretation of state differences had led to decisions not to report or collect relevant data.

(2) The mixed models statistical procedure in SAS was used (Littell, Milliken, Stroup, & Wolfinger, 1996).

Gary Marks is Principal Research Fellow at the Australian Council for Educational Research, Private Bag 55, Camberwell, Victoria 3124. He is also research associate (Associate Professor) at the Melbourne Institute, University of Melbourne.

Email: [e-mail]

John Creswell is a former staff member of the Australian Council for Educational Research. He is currently employed by the OECD, Paris.

Email: [e-mail] 

Table 1 Percentages of students in different grades in
Australia (PISA 2000)

                                Year level (%)

State                            9   10   11

New South Wales                  8   86    5
Victoria                        13   81    5
Queensland                       1   57   42
South Australia                  3   79   17
Western Australia                1   49   49
Tasmania                         7   84    8
Northern Territory               6   80   13
Australian Capital Territory     7   92    0

TOTAL                            7   76   17

Table 2 State differences in Reading Achievement

                                                Net of SES,
Variable             No Controls   Net of SES   Indigenous
                                                  Status

Intercept             539.1 ***    540.9 ***     541.2 ***
Victoria              -23.9 **     -23.5 **      -23.4 **
Queensland            -16.5         -9.0          -8.6
South Australia        -2.0          0.9           1.2
Western Australia       0.5          2.6           3.1
Tasmania              -22.6        -14.9         -14.0
Australian Capital
  Territory            14.7          0.1           0.1
Northern Territory    -39.5 ***    -38.0 ***     -34.9 ***

                                    Net of SES,
                     Net of SES,    Indigenous
Variable             Indigenous       Status,
                       Status,        Region,
                       Region          Grade

Intercept             541.3 ***      542.7 ***
Victoria              -24.4  **      -21.6 **
Queensland             -8.7          -33.9 ***
South Australia         2.4           -7.3
Western Australia       4.2          -25.9 **
Tasmania              -17.8          -20.9 *
Australian Capital
  Territory            -2.9           -1.9
Northern Territory    -29.5 **       -35.1 ***

Regression coefficients. * 0.05>P>0.01; ** 0.01>P>0.001;
*** P<0.001 Reference Group, New South Wales, average
family SES, non-Indigenous, Mainland Capital, grade 10.

Table 3 State differences in mathematics achievement

                                                Net of SES,
Variable             No Controls   Net of SES    Indigenous
                                                   Status

Intercept             541.4 ***    542.7 ***      543.0 ***
Victoria              -13.7        -12.0          -11.9
Queensland            -14.7         -7.9           -7.7
South Australia       -15.7        -11.6          -11.6
Western Australia       6.5          9.3            9.4
Tasmania              -23.5        -17.1          -16.2
Australian Capital
  Territory             6.4         -6.8           -6.9
Northern Territory    -26.3 *      -24.3 *        -21.2 *

                                    Net of SES,
                     Net of SES,    Indigenous
Variable             Indigenous       Status,
                       Status,        Region,
                       Region          Grade

Intercept             543.4 ***      544.5 ***
Victoria              -13.1           -9.6
Queensland             -8.5          -30.2 ***
South Australia       -11.6          -20.9 **
Western Australia       8.8          -18.2 *
Tasmania              -16.4          -18.1 *
Australian Capital
  Territory            -2.7           -0.5
Northern Territory    -28.7 *        -34.5 **

Regression coefficients. * 0.05>P>0.01; ** 0.01>P>0.001;
*** P<0.001 Reference Group, New South Wales, average
family SES, non-Indigenous, Mainland Capital, grade 10.

Table 4 State differences in science achievement

                                                Net of SES,
Variable             No Controls   Net of SES   Indigenous
                                                  Status

Intercept             532.6 ***    534.7 ***     535.1 ***
Victoria              -17.3        -16.6         -16.4
Queensland             -5.9          0.2           0.4
South Australia         6.6          8.2           8.4
Western Australia      14.1         15.6 *        16.0 *
Tasmania              -22.1        -15.8         -15.2
Australian Capital
  Territory            19.8          7.5           7.6
Northern Territory    -36.7 ***    -37.1 ***     -33.6 ***

                                   Net of SES,
                     Net of SES,   Indigenous
Variable             Indigenous      Status,
                       Status,       Region,
                       Region         Grade

Intercept             534.3 ***     535.5 ***
Victoria              -17.8         -16.4
Queensland             -0.3         -22.0 **
South Australia         9.3           0.8
Western Australia      16.5          -8.8
Tasmania              -18.4         -21.2 *
Australian Capital
  Territory             8.8           8.6
Northern Territory    -34.9 ***     -41.2 ***

Regression coefficients. * 0.05>P>0.01; ** 0.01 >P>0.001;
*** P<0.001 Reference Group, New South Wales, average family SES,
non-Indigenous, Mainland Capital, grade 10.

Table 5 State differences in effects on achieving proficiency level
1 or below in reading

                                                Net of SES,
Variable             No Controls   Net of SES   Indigenous
                                                  Status

Intercept             -2.22 ***    -2.41 ***     -2.41 ***
Victoria               0.48 ***     0.51 ***      0.51 ***
Queensland             0.39 **      0.29 **       0.28 *
South Australia        0.06         0.03          0.01
Western Australia      0.04         0.05          0.06
Tasmania               0.60 *       0.52 *        0.49 *
Australian Capital
  Territory           -0.47        -0.22         -0.47
Northern Territory     0.78         0.81          0.74

                                   Net of SES,
                     Net of SES,   Indigenous
Variable             Indigenous      Status,
                       Status,       Region,
                       Region         Grade

Intercept             -2.40 ***     -2.50 ***
Victoria               0.54 ***      0.44 ***
Queensland             0.29 *        0.75 ***
South Australia       -0.04          0.13
Western Australia      0.10          0.43 *
Tasmania               0.64 **       0.72 **
Australian Capital
  Territory           -0.09         -0.11
Northern Territory     0.73          0.84

Logistic Regression coefficients. * 0.05>P>0.01; ** 0.01>P>0.001;
*** P<0.001 Reference Group, New South Wales, average family SES,
non-Indigenous, Mainland Capital, grade 10.