Is Australian educational research worthwhile?

This research examines publication and citation data to assess the contribution of Australian educational research to teaching practice and policy making. Australia makes a significant contribution to international educational research, although its share of publications has been increasing over time, whereas its share of citations has remained relatively constant. Substantial turnover in leading research institutions is found over time. It might be possible to explain this turnover by the productivity of a small number of academic staff members. Finally it is found that Australian educational research is extremely relevant to educational practice.

Introduction

Important volumes detailing the nature of educational research in Australia have been produced by Keeves (1987) and Keeves and Marjoribanks (1999). These works show that a substantial amount of excellent educational research is conducted in this country. Nevertheless, in recent years, lingering doubts about the quality and usefulness of educational research have emerged.

Although review panels have affirmed the quality of Australian educational research (see Lingard & Blackmore, 1998; McGaw, Bowd, Poole, & Warry, 1992), the report by McGaw et al. raised a disturbing issue. It suggested that `educational administrators and practitioners perceive much educational research to be irrelevant to their concerns'.(1) This troublesome notion has been articulated more potently by Harman (1998) who describes a debate over the quality of educational research currently raging in England and hints at its relevance to Australia. Specifically he reports on the ideas of Hargraves who `argued that a great deal of educational research is "frankly second rate" and "bad value for money"'. Add these concerns to Hattie, Print, and Krakowski's (1994) findings that 33 per cent of Australian academics in the area of education have never published anything, and a further 26 per cent have published no more than three papers, and the need for a focused examination on the quality and nature of Australian educational research is clear.

This paper supplements previous research by considering two important aspects of research quality. First, the visibility of Australian research as a component of the international educational research arena is examined. This involves an analysis of the quantity and source of Australian educational research published in major journals as well as a consideration of the recognition it receives in terms of citations by other scholars. This analysis of publication and citation rates will provide useful input to the debate over the standing of Australian research.

Even if Australian educational research is highly cited, however, this does not mean that it is useful to educational practitioners. Indeed, as follows from both the McGaw et al. report and the Harman article, it would hardly be surprising if most teachers and policy makers are found never to consult academic educational research, and believe research has no impact on their practices whatsoever. Even if this is the case, it is still possible that academic research is, in fact, very relevant to teaching practice and government policy. It could be that the dissemination of research to educational practitioners and policy makers follows an unrecognised pathway, for example through the media or teacher training. This study sheds light on this issue by providing a detailed review of Australia's most cited educational research.

Data and methods

Data presented here have been compiled from a subset of the publication and citation data from 1981 to 1996 compiled by the Institute for Scientific Information (ISI) (see Bourke & Butler, 1998, 1993).

Several important aspects of this study should be noted. First, the focus here is limited to Australian research appearing in ISI indexed journals. A report by the National Board of Employment, Education, and Training (1994) suggests that only about 25 per cent of Australian publications in the area of education appear in ISI journals. Clearly a significant amount of Australian research appears in books, monographs and other outlets not considered here. Nevertheless, although comprising only a minority of Australian research, publication in major journals is widely recognised as important, and is a fundamental influence on Australia's international research reputation.

Second, this study only considers publications appearing in educational journals indexed by the Social Sciences Citation Index. While this journal set includes many cross-disciplinary ones (such as Sociology of Education or Anthropology and Education), education related research published in journals primarily focusing on other fields is not examined here.

Finally, there has previously been extensive discussion of methodological issues relating to the analysis of ISI data (e.g. Kostoff, 1998; Phelan, 1999). Regardless of any criticisms, however, the ISI data remain the best measure of international research output available. Nevertheless it is important to keep in mind that although the information presented here provides useful insight into educational research in Australia, publishing in major international journals is only one aspect of a scholarly role.

International standing of Australia

In 1995, about 4 per cent of the research published in major international education journals had authors listing Australian affiliations (see Figure 1). Given that, in the same year, Australia accounted for only .3 per cent of the world's population and only 1.2 per cent of the world's total economic output (World Bank, 1998), our educational research output can be considered an admirable achievement. It is also evident, however, that there is significant variation, year by year, in the share of world publications garnered by Australian authors, ranging from less than 2.5 per cent in 1982 to nearly 4.5 per cent in 1994. The variation in output between adjacent years can be substantial with Australia's share of educational publications changing, at times, more than 1 per cent within the course of a single year. One reason for these volatile swings in output is that Australian authors publish fewer than 200 articles in the international education journals under consideration in a typical year.

[GRAPH OMITTED]

Although the chosen field is relatively small, Australian representation is fairly large, not only in relation to its economy, but also in relation to other academic fields. The total share of Australian publications in major journals is indicated by the line dotted with triangles in Figure 1. The total share in the world's major journals of Australian publications in all fields is about 2.5 per cent, a figure that has remained relatively stable over time. Thus, since 1983, the share of international literature in education has exceeded by a decent margin the average share of international literature garnered by Australian academics in other fields.

In Figure 2, Australia's share of world publications in education is presented in relation to its share of world citations. A linear trend line has been superimposed on each measure. Clearly the output of Australia's educational researchers has increased substantially over time and is on an upward trajectory. What is troublesome, however, is that Australia's world share of citations has not kept up with its publication output. Despite substantial volatility, the overall trend of Australia's share of world educational citations is relatively flat. Moreover it is generally considered an indication of research quality for a nation to achieve a greater share of world citations than its share of publications. This achievement, were it accomplished, would suggest that Australia is attaining more citations than would be expected from the number of publications it produces. Although Australia has achieved this status several times between 1981 and 1995, it does not routinely achieve this level of accomplishment in the field of education.

[GRAPH OMITTED]

Output of institutions

Table 1 presents the percentage of Australian educational publications and citations accruing to the five leading universities over time. If publications and citations were distributed evenly over all Australian universities, we would expect the top five institutions to account for about 13 per cent of them. It is clear that the top five universities account for more than their share of publications (over 30%) and substantially more than their share of citations (nearly 50%). Moreover, whereas the concentration of publications has decreased over time, the concentration of citations has increased.

Table 1 Educational publications and citations at leading Australian
  universities (percentage of total)

1981 to 1986                 1987 to 1992

Publications

1. Monash          8.7%      1. Monash           9.1%
2. Sydney          6.9%      2. Deakin           6.0%
3. La Trobe        6.2%      3. Queensland       5.9%
4. Queensland      6.0%      4. New England      5.2%
5. Curtin U.T.     5.1%      5. Sydney           4.8%

Top five          33.1%      Top five           31.0%

Citations

1. Sydney         14.6%      1. Curtin U.T.     13.8%
2. New England    11.4%      2. Sydney          10.3%
3. Monash          7.3%      3. Deakin           9.4%
4. Macquarie       6.8%      4. West. Aust.      7.7%
5. A.N.U.          6.8%      5. Monash           7.0%

Top five          46.9%      Top five           48.2%

There is also indication that universities that produce the most publications are not always the same ones that receive the most citations. In addition, there appears to be a reasonable amount of change over time, both in terms of which universities produce the most publications and which garner the most citations.

The issue of change is an interesting one. Because of the time lag before citations are received, it is difficult to ferret out the relationship between publication and citation rates. It is clear that there is not a small group of elite universities that has dominated publication output and citation receipt over time. Over all 37 institutions examined, the correlation between publications in the earlier period and citations in the later period (r=.67) was stronger than the correlation between citations in the earlier period and citations in the later period (r=.59). This is to say that the number of publications a university has is a better predictor of the quality of its research in the future, than is the quality of its research at present. It will be interesting to see if the top publishing universities in recent years will lead citation counts in the future. Although not shown in Table 1, the four leading publishers from 1993 to 1995 are universities that were not ranked in the top five from 1987 to 1992 (Queensland University of Technology, Griffith University, Curtin University of Technology, and La Trobe). It will be several more years until adequate data are available in order to determine if the recognition these institutions receive for their research is commensurate with the amount of research they produce.

Productivity of individual authors

Details about Australian publications for three time periods are presented in Table 2. Total counts include all publications with an Australian author in ISI indexed journals. About 35 per cent of these papers have more than a single author, and this proportion has been increasing over time (see panel C). On around 95 per cent of papers with Australians as authors, the first author is also Australian.

Table 2 Australian educational publications in ISI journals over time

                                        1981-85           1986-90

All educational publications                660               756
ISI indexed education articles             2.6%              3.4%

Authors
Total number of authors                     560               690
Number of first authors                     407               488
Publications/1st author                    1.62              1.55
% Australian first authors                96.4%             94.2%
Pubs by 1% all authors                     8.8%              9.9%

Collaboration
I Author                                  73.3%             68.5%
2 Authors                                 20.5%             23.0%
3 Authors                                  4.4%              7.0%
More than 3 authors                        1.8%              1.5%

Papers authored (all)
10 or more                                 0.5%              0.3%
5 to 9                                     3.4%              6.8%
3 or 4                                     8.4%              5.8%
2                                         17.9%             15.1%
1                                         69.8%             72.0%

Most authorships (1st)           BJ Fraser (13)     KG Tobin (13)
Most authorships (all)           BJ Fraser (15)     KG Tobin (20)

                                       1991-95             Total

All educational publications                837              2253
ISI indexed education articles             3.8%              3.3%

Authors
Total number of authors                     894              1717
Number of first authors                     567              1144
Publications/1st author                    1.48              1.97
% Australian first authors                93.3%             94.5%
Pubs by 1% all authors                     9.8%             13.2%

Collaboration
I Author                                  59.4%             66.5%
2 Authors                                 27.4%             23.9%
3 Authors                                  8.1%              6.7%
More than 3 authors                        5.1%              2.9%

Papers authored (all)
10 or more                                 0.3%              1.7%
5 to 9                                     3.2%              6.1%
3 or 4                                     7.4%             10.0%
2                                         17.1%             16.2%
1                                         71.9%             66.0%

Most authorships (1st)            HW Marsh (10)     BJ Fraser (27)
Most authorships (all)           BJ Fraser (12)     BJ Fraser (46)

The average publication rate per author is less than two papers over the entire 15-year period and 66 per cent of those who published produced only a single paper. However, these data are only for publications in ISI indexed educational journals, and a number of authors may be publishing in other fields too. Nevertheless these findings support previous research (Hattie, Print, &: Krakowski, 1994) which suggests that strong research productivity in Australia tends to be concentrated on a relatively small group of researchers. Over the 15-year period, 1 per cent of Australian authors have produced more than 13 per cent of the published research and, in particular, the names Fraser, Tobin, and Marsh appear again and again.

The extraordinary productivity of a few authors is better shown in Table 3, which presents data on the ten leading educational researchers in Australia. Of course, a number of technical issues confront any ranking of authors. Evaluation of productivity using just publication counts permits authors who produce many articles that achieve little recognition to be thought of as productive even though the quality of that production may be limited. Evaluation of only citation counts, on the other hand, ranks as productive authors who might have produced only one or two articles during their careers, but struck it rich in terms of citations. And evaluation of all authorships instead of just first authorships allows minor authors to be ranked as major educational researchers. For example, a research assistant who was third author on three highly acclaimed papers is one of Australia's most cited authors by this measure. A better measure of productivity takes into account both publication output and the ability to attract citations. To tap into publication activity characteristics that are likely to be most valued by universities (many publications attaining many citations), an Author Productivity Index (API) was developed as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where (p.sub.i) equals the total number of first authored publications and (c.sub.1) equals the total number of citations from first authored publications. The API was multiplied by 100 for ease of reference. Thus this measure ranges from zero for authors who have no citations to a theoretical limit of 100 in the unlikely case of a single author in a field accounting for all of its publications and citations.

Table 3 Leading Australian educational researchers, 1981-1996

                            First authorships

Author                      API      Pubs     Cits

1 Marsh, Herbert W.         55.8      28       345
2 Fraser, Barry J.          37.7      27       155
3 Tobin, Kenneth G.         31.0      22       136
4 Watkins, David            18.9       8       138
5 Smyth, W. John            15.5      15        50
6 White, Richard T.         11.8      11        39
7 Walker, Jim C.            11.6      10        42
8 Freebody, Peter           11.3       5        79
9 Over, Ray                 10.6       8        44
10 Hatton, Elizabeth J.     10.2      10        32

                           All authorship   All fields

Author                      Pubs    Cits      Pubs   Cits

1 Marsh, Herbert W.           32     350       147   2825
2 Fraser, Barry J.            47     213        57    272
3 Tobin, Kenneth G.           31     185        32    209
4 Watkins, David               8     138        31    264
5 Smyth, W. John              16      51        16     51
6 White, Richard T.           16      76        16     76
7 Walker, Jim C.              10      42        12     51
8 Freebody, Peter              9      97        21    127
9 Over, Ray                   11      46        85    373
10 Hatton, Elizabeth J.       12      32        12     32

Irrespective of which measure of author productivity, or what time interval is examined, Marsh, Fraser, and Tobin consistently rate as the most productive authors of Australian educational research. In fact, the three of them together account for nearly 20 per cent of the citations that Australian educational research receives.

In addition to examining author productivity in terms of publications and citations in education journals, for leading educational scholars, publications and citations received by them for publications in any field were also tabulated. The findings relating to Marsh are especially noteworthy. Despite publishing more in major educational journals than anyone else in Australia, nearly 80 per cent of his work is published in journals attributed to other fields (mostly psychology). During the 15-year period examined, Marsh collected an astonishing 2825 citations for all of his work. Over, Freebody, and Watkins have also published a majority of their work in non-educational journals.

Are `academic superstars' essentially responsible for the performance of their entire university's output of educational research? To take Marsh as an example, he began to list his affiliation on educational articles as Western Sydney rather than Sydney on papers published from 1991. How did his transfer from one university to another affect their respective outputs of educational research? During the period between 1985 and 1990, Marsh was an author on 18 per cent of the articles in educational journals attributable to the University of Sydney. From 1991 to 1996, he was an author on 47 per cent of the educational literature produced at Western Sydney. Nevertheless, in the six years after he left Sydney, the total output of educational research at Sydney actually increased by around 10 per cent, so his departure was not fatal to them by this measure. His arrival at Western Sydney, on the other hand, was a substantial boost to them. Western Sydney's educational publication count almost tripled in the six years after he arrived (from the number attained jointly in the prior six-year period by the three institutions that unified during 1989).

It is only when we examine citations, however, that the full extent of Marsh's contribution to his university is completely recognised. In the period between 1985 and 1990, Marsh accounted for fully 76 per cent of the citations to educational papers published at the University of Sydney during that period. In the six-year period after Marsh departed, citations to University of Sydney educational papers dropped by 40 per cent despite the increase in publications. By way of contrast, the colleges that formed the University of Western Sydney had no citations at all to educational papers in the six-year period prior to Marsh's arrival. After Marsh's arrival, however, Western Sydney attained 25 per cent more citations than Sydney did, and it attained these citations with 43 per cent fewer articles. More importantly, almost 90 per cent of the citations that Western Sydney received were to articles on which Marsh was an author. This example has shown that it is clearly possible for even a single scholar to have an extraordinarily strong influence on the output ranking of a university. Statistics on departmental performance should be viewed with this possibility in mind.

Characteristics of highly cited research

A list of the 50 most cited Australian educational articles in each of two time intervals, 1981-1987 and 1988-1995 was compiled. Because of ties in the number of citations, the final list included 104 highly cited articles. Details concerning these publications are presented in Table 4.

Table 4 Details about highly cited Australian educational research

                            1981-1987   1988-1993

Gender
Male first authors             80.8%        79.4%
Male any author                78.3%        77.8%

Quantitative                   74.1%        55.1%

Journal origin
Australian                     13.0%         4.3%
British                        11.1%        10.6%
American                       53.7%        48.9%
International                  22.2%        36.2%

Number of authors
One                            44.4%        39.6%
Two                            42.6%        47.9%
Three                           7.4%         8.3%
More than three                 5.6%         4.2%

Target
Tertiary                       31.5%        37.5%
Secondary                      31.5%        27.1%
Primary                        11.1%        10.4%
Primary & secondary            16.7%        16.7%
General                         9.2%         8.3%

Almost 80 per cent of these most cited publications were authored by men and this percentage has barely changed over time. Additionally, women were only slightly more likely to be second or later author than they were to be first author. The lack of increase in female authorship over time is especially striking. This could be explained by the fact that highly cited authors tend to emerge from the more senior ranks in which women are still underrepresented. A related explanation is that the gatekeepers of acceptable journal content are often men who might favour (to the extent there is any difference) the sort of content that men are more likely to produce. Although these data are not adequate to explore this issue, it deserves additional attention in future research.

Whereas most highly cited research is quantitative, the proportion of research that is non-quantitative increased considerably over time to reach 45 per cent in the later period. In addition, although the largest share of our most cited research is published in American journals, there is a trend towards publishing in international ones (typically European). The 104 most-cited papers that were reviewed were distributed across 31 different journals. The American Educational Research Journal (with 12 papers), the Australian Journal of Education and the Journal of Research in Science Teaching (each with 9 papers) were the journals that published the greatest number of highly cited articles. Finally, highly cited research is likely to have multiple authors. This is not the case with less cited research.

The last issue considered was whether highly cited Australian educational research mostly consisted of research within tertiary institutions. If this were true, it would explain why teachers might not find educational research useful. More than 60 per cent of the research, however, was more directly relevant to schools than to tertiary institutions. Therefore, if teachers think that educational research is irrelevant, the reason must lie in its content rather than the level of institution it targets.

Content of highly-cited research

Each of the 104 highly cited articles in our sample was classified according to major foci. Some articles had multiple central foci and were classified in more than one category. Only the major themes of each paper, however, were classified. For example, studies of gender and computing (Clarke & Chambers, 1989; Hattie & Fitzgerald, 1987) were classified under both areas because the central theme intertwined the two topics. Although each paper also presented data on achievement in computing classes, they were not classified as articles on achievement because achievement was not a central theme. This method of classification resulted in 24 categories to classify the content of the 104 articles. A substantial number of these categories, such as special education, computing, memory, health, and perception, had only a few entries.

Figure 3 combines the ten most commonly researched topics in each of the two time intervals studied. This procedure resulted in a total of thirteen categories as there were seven categories that were frequently researched in both time periods and an additional six categories that were frequently researched in one time period but not in the other.

[GRAPH OMITTED]

It is evident from Figure 3 that large proportions of the most highly cited papers in both intervals deal with the issues of teaching, learning, and measurement. The data also indicate an increase in the percentage of papers focusing on teaching, from less than 40 per cent in the early period to more than 50 per cent in the later period. The amount of research focusing on gender, self concept/self esteem, attainment, classroom environment, reading, and administration has dropped off in recent years, whereas the amount of research pertaining to the psychology and attitudes of students, assessment, educational theory and teacher education has increased.

Research on teaching

A good example of scholarship that makes a substantial potential contribution to teaching practice is the work of Tobin. His articles on `wait time' (Tobin, 1986, 1987) show that student learning can be significantly enhanced if teachers would simply pause for a few seconds after each sentence to allow the students to absorb the information. Other articles on teaching focus on ideas such as the use of analogies to improve teaching (Treagust, 1988; Treagust et al., 1992), the use of `peer leaders' (Perry, Klepp, Hawkins, Murray, & Halper, 1986), the effectiveness of workbooks in comparison with direct teaching (Patching et al., 1983), and the use of small groups to enhance teaching (Jackson & Prosser, 1989). Overall most of the articles on teaching practice focus on primary and secondary education.

A number of highly cited papers focus on reflective education or on action research (Baird, 1986; Gore, 1987; Gore & Zeichner, 1991; Smyth, 1989) and, more specifically, several papers with an action or reflective approach are identified with `classroom environment' research which relates very closely to teaching practice (Fisher & Fraser, 1983; Fraser, 1981, 1989; Fraser & O'Brien, 1985; Trigwell & Prosser, 1991). A common theme in much of this research is that effective teaching does not involve the regurgitation of a large body of information that students essentially learn by rote. Rather teachers should provide their students with learning strategies to assist them to learn themselves (Rams&n, 1991; Tobin & Fraser, 1989; White, 1992). Teacher education is a third field in which research is closely related to that focusing on teaching practice and classroom environment. Calderhead and Robson's (1991.) study, which followed student primary teachers through the first year of their degree program and draws conclusions about the obstacles they face, is a good example of research in this category with strong relevance to educational practice.

Learning and study practice

Research on learning or study practices is the second most common focus of highly cited articles for both time intervals. Like research on teaching, classroom environment and teacher education, this research tends to have direct application to classroom practices, although some of it tends to be methodological. Most of the work in this area has focused on student study practices. It is generally found that many students use disorganised or superficial study methods that tend to lead to less achievement or lower grades (Baird & White, 1982; Thomas & Bain, 1982; Watkins, 1982, 1983a, 1983b; Watkins & Hattie, 1985).

Measurement issues

A strong focus on measurement issues was often evident in research concerned with the testing and development of the questionnaires used to assess approaches to study (mentioned above) as well as in the work of Marsh and his colleagues. Marsh's research has often concerned the development of testing instruments to assess self-concept (Marsh, 1986; Marsh, Barnes, & Parker, 1985; Marsh, Barnes, & Smith, 1983; Marsh & Holmes,1990) or student ratings (Marsh, 1981; Marsh & Hocevar, 1984, 1991).

Education and gender

Gender-related studies are a substantial component of Australian educational research, although gender as a primary focus in the most highly cited research has diminished somewhat over time. Highly cited papers in the area of gender discuss issues such as the failure of single-sex schools to promote better learning by females (Carpenter & Hayden, 1987; Marsh, Smith, Owens, & Marsh, 1988; Willis & Kenway, 1986), the under-representation of women in Australian universities (Over, 1981), the low degree of female student interest in computing (Clarke & Chambers, 1989; Hattie & Fitzgerald, 1987), the perpetuation of sexual stereotypes by teachers (Davies, 1989; Kessler, Dowsett, Connell, & Ashenden, 1995; Tobin & Garnett, 1987), the differences in sociopolitical attitudes between females and males (Marjoribanks, 1981) and gender differences in mathematics and verbal ability (Marsh, 1989). In a twist from the usual focus, Connell (1989) discusses the development and implications of masculinity in men.

Relevance of Australian educational research

The assumed implication of policy makers and teachers finding educational research irrelevant is that there is something wrong with the research. It must either focus on arcane matters of no practical import or is so abstract and incomprehensible that it is useless to practitioners. This view has been expressed well by the Chief Inspector of Schools in England who claims never to read educational literature and writes, `There is too much to do in the real world with real teachers and real schools to worry about methodological quarrels or waste time decoding unintelligible, jargon-ridden prose to reach (if one is lucky) a conclusion that is often so transparently partisan as to be worthless' (quoted by Harman, 1998).

To research this question, all articles reviewed were classified into one of seven categories depending on how directly relevant the work was to researchers, educational administrators, policy makers and teachers. This categorisation, which was performed on research for each of the two time intervals, is presented in Figure 4. In considering these categories, it should remain clear that all articles were research documents. The question considered here is whether these research articles are also relevant and accessible to applied audiences and to what degree. Specifically the research was divided into that with direct relevance and easy accessibility, and that which still has strong relevance to practitioners, but the elements of interest emerge from a research process that is likely to be of little interest to them. Finally, some of the research, although it still may provide findings potentially useful to practitioners, or lead to later work useful to practitioners, in and of itself is likely to be difficult for practitioners to access or be of little interest. This latter type of research is the sort that comes to mind when claims are made that educational research is `irrelevant'.

[GRAPH OMITTED]

Relevance to teaching

Figure 4 suggests that Australian educational research has substantial potential for application to teaching practices. Although a reasonable proportion of papers are directly relevant to teaching issues (around 18% in the early period and 8% in the later period), an increasing proportion are related to teaching in a more peripheral manner. Papers directly relating to teaching focus on issues of immediate interest to teachers, and do not dwell on technical issues. Tobin and Fraser's (1990) paper providing case studies which demonstrate exemplary teaching and Gallagher and Tobin's (1987) study on the role of teachers and their interaction with students both provide excellent examples of accessible research with clear and direct application to teaching practice.

Most papers (18% in the early period and nearly 35% in the later period), although still offering useful information relating to teaching practices, are somewhat less direct or accessible in their approach than the two papers mentioned above. An article by Schibeci and Riley (1986) provides a good example of research that teachers might dismiss as irrelevant at first glance, but in fact provides potentially useful information. Schibeci and Riley evaluate the influence of student background and perceptions on achievement and attitudes towards science. Using a structural equation model, the authors find that attitudes tend to influence achievement, rather than the other way around. They then argue that teachers should pay more attention to encouraging positive student attitudes, and trying to make science more exciting. This strategy, they argue, will pay off in terms of improved student performance.

Many research articles focusing on teaching practices follow the same general model as does the work of Schibeci and Riley. First they deal with sampling and methodological issues that are unlikely to be of much interest to most teachers, but eventually draw conclusions from their research providing practical information that teachers would value if it were made available to them. Although, for example, it might not be immediately evident to a teacher that the direction of causality between performance and attitudes has important implications for teaching practice, research on issues such as this make an important potential contribution, if only teachers could learn of these findings. At any rate, articles of this type have been classified as being of peripheral interest to teachers and they make up a substantial portion of highly cited educational research.

Relevance to policy

About 15 per cent of educational research during the early period and over 20 percent of research in the later period was policy related. Examples of papers with direct policy relevance are Waugh and Punch's (1987) summary of research findings regarding teacher receptivity to system educational change, Crossley's (1984) work on international transfer of educational innovations, and White's (1992) paper examining the relationship between research and practice.

As with research relating to teaching, policy relevant research is increasingly becoming less directly accessible. A good example of a study with peripheral policy implications is Over's (1982) paper on the productivity of scientists as they get older. It is clear that the issue has implications for retirement policy although this is not the central focus of the study. Other examples of research with peripheral policy implications include studies of the process by which school leavers develop career plans (Saha, 1982), research relevant to encouraging condom use by homosexuals (Ross & McLaws, 1992), and Tobin and Fraser's (1989) study of a non-traditional high school.

Relevance to administration

Research relevant to school and university administrators is similar to research categorised as policy relevant. The difference is that policy relevant work has national implications whereas work relevant to administrators has an institutional focus. Overall, much work with potential interest to administrators has to do with the teaching profession, or with curriculum development. Moses's (1986) paper on the dissatisfaction of university staff with promotion practices, Laughlin's (1984) study of teacher stress, and Tinning and Fitzclarence's (1992) paper about the crises in physical education in Australian secondary schools are examples of papers which potentially are of direct interest to administrators.

Numerous other papers relate to educational administration, but not as centrally as the research just mentioned. Examples of research with peripheral relevance to administration include Hart and Robottom's (1990) paper about a pending paradigm shift in science education, Gronn's (1982) critique of studies of school principals, and papers suggesting the importance of educational institutions offering instruction in study methods (Baird & White, 1982; Watkins, 1983a).

Relevance of abstract theory and methodological research

The most interesting set of research might well be that classified as not having easily discernible relevance to educational practice. Around 18 per cent of highly cited Australian educational research was classified as being only of interest to researchers in the 1981 to 1987 period, falling to around 14 per cent in the 1988 to 1995 period.

Much of the research in this category is methodological. Although publications in this category might not be immediately relevant to educational practice, they often involve the development of measurement instruments essential for later research with a strong practical component. The remainder of research in this category, only a few papers, was strongly theoretical and quite challenging. It is this latter sort of paper, the rarest of all types of educational research, that one gets the feeling some policy makers believe constitutes virtually the entire production of Australian educational researchers.

Turning first to methodological research, as mentioned previously, a large amount of this has been conducted by Marsh and his colleagues in their effort to develop methods to assess self-concepts, or to improve student rating instruments. Even though this research is methodological, much of it still makes noteworthy practical contributions. Good examples are Marsh and his colleagues' work on the impact of coeducation on male and female self-concepts and achievement (Marsh et al., 1988), Marsh's (1991a) study of students working after school, and his counter-intuitive work finding that high-ability schools do not offer advantages for most students (Marsh, 1991b).

The final type of research examined was deeply theoretical and would probably be challenging for many educational practitioners. Teachers might have difficulty, for example, perceiving the relevance of Levi-Strauss's concept of `bricolage' to their work (Hatton, 1988, 1989) and most of them probably do not care that Willis's theory of cultural production suffers from the errors of essentialism and dualism (Walker, 1986). As these three papers were the only ones in the sample that were so strongly theoretical, and they were classified as being of interest to researchers only, it is worth asking if they actually do contain or inspire ideas directly relevant to educational practice.

Citations to Hatton and Walker were followed up to determine if their theoretical work contributed to later research that teachers might more easily recognise as relevant. Robinson (1994) picking up on the same issue in a paper on the practical promise of critical research (which cites Walker's paper) provides suggestions as to how this theory can more effectively influence educational practices. Walker (1987) himself used his earlier paper as the theoretical foundation for a more applied and easy to comprehend study of multiculturalism in schools. Hatton's theoretical papers were cited by a number of other researchers examining the teaching profession. Warton, Goodnow, and Bowes (1992), for example, extended Hatton's earlier work in their article on the orientation of teachers to industrial action resulting partly from a feeling of a lack of respect towards them by state governments. It can be concluded that strongly theoretical work good enough to be extensively cited will ultimately have practical influence.

Moreover sophisticated theoretical discussions can turn up in unexpected places. For example, in his discussion of pedagogy, Tinning (1991) provides an easily understood discussion of critical theory and post-modernism and proceeds to apply skilfully these ideas to teacher training in the area of physical education.

Conclusion

This paper has examined both the standing and content of Australian educational research. It was found that Australia is an important producer of educational research in the journals indexed by the Institute for Scientific Information. Moreover Australia's share of world educational publications has been increasing over time. At the same time as publication output has been increasing, however, Australia's share of world citations has remained relatively stable. Two developments may have influenced this occurrence. First, the conversion of colleges of advanced education to universities in the eighties may have encouraged greater emphasis on publication without a concomitant increase in research quality. Second, although the Research Quantum composite index for the allocation of funding promotes increased publication activity, it does not provide any additional incentive to invest the effort that can be required to produce research acceptable to top journals (for a related discussion, see Bourke, 1997).

Another finding is that several recently established universities have joined older institutions in being relatively productive and that there is substantial turnover in lead institutions. In general, this can be considered a sign of a healthy and competitive university system. At the same time, however, it is evident that a relatively small number of individuals produce a disproportionately large amount of Australia's most acclaimed educational research. Unfortunately the productivity of many Australian academics, at least in terms of research published in major international education journals, is relatively low.

In addition, it has been shown that a single productive scholar can be responsible, almost single-handedly, for the majority of ISI indexed research produced by an entire department. This finding suggests that care may need to be taken in the interpretation of university rankings based on aggregate measures of research productivity. The turnover in leading universities could simply reflect the migration of a small number of leading researchers from one institution to another over the course of their careers. This issue deserves attention in future research.

Finally it was found that Australian educational research tends to be highly relevant to educational practice and is potentially accessible to applied audiences. More research needs to be conducted to determine how this research is disseminated and how much of it eventually reaches non-academic audiences. This would probably involve surveys of teachers and/or policymakers as well as an analysis of research dissemination mechanisms such as teacher training or the media.

As for policy makers questioning the quality of Australian educational research, several scholars have attempted to explain this. Husen (1984) points out that the differing nature of the core endeavour of educational researchers and policy makers may explain why educational research appears to have so little practical value. Also Caplan (1991) has noted that policy makers tend to reject research information because it can contradict what they believe to be true, or suggest strategies that are politically unfeasible. Another possibility is that they simply do not read it.

Keywords

academic staff publishing educational research publications citations evaluation research tools

References

Academy of the Social Sciences in Australia. (1998). Challenges for the social sciences and Australia. Canberra: Australian Research Council.

Baird, J.R. (1986). Improving learning through enhanced metacognition: A classroom study. European Journal of Science Education, 8, 263-282.

Baird, J.R. & White, R.T. (1982). Promoting self-control of learning. Instructional Science,11,227-247.

Bourke, P. (1997). Evaluating university' research: The British research assessment exercise and Australian practice (Commissioned Report No. 56). Canberra: National Board of Employment, Education and Training.

Bourke, P. & Butler, L. (1993). A crisis for Australian science (Performance Indicators Project, Monograph Series No. 1). Canberra: Australian National University.

Bourke, P. & Butler, L. (1998). Institutions and the map of science: Matching university departments and fields of research. Research Policy, 26, 711-718.

Calderhead, J. & Robson, M. (1991). Images of teaching: Student-teachers' early conceptions of classroom practice. Teaching and Teacher Education, 7, 1-8.

Caplan, N. (1991). The use of social research knowledge at the national level. In D.S. Anderson & B.J. Biddle (Eds.), Knowledge for policy. London: Falmer Press.

Carpenter, P. & Hayden, M. (1987). Girls' academic achievements: Single-sex versus coeducational schools in Australia. Sociology of Education, 60, 156-167.

Clarke, V.A. & Chambers, S.M. (1989). Gender-based factors in computing enrollments and achievement: Evidence from a study of tertiary students. Journal of Educational Computing Research, 5, 409-429.

Connell, R.W. (1989). Cool guys, swots and wimps: The interplay of masculinity and education. Oxford Review of Education, 15, 291-304.

Crossley, M. (1984). Strategies for curriculum change and the question of international transfer. Journal of Curriculum Studies, 16, 75-88.

Davies, B. (1989). The discursive production of the male-female dualism in school settings. Oxford Review of Education, 15, 229-242.

Fisher, D.L. and Fraser, B.J. (1983). A comparison of actual and preferred classroom environments as perceived by science teachers and students. Journal of Research in Science Teaching, 20, 55-61.

Fraser, B.J. (1981). Using environmental assessments to make better classrooms. Journal of Curriculum Studies, 13, 131-144.

Fraser, B.J. (1989). 20 years of classroom climate work: Progress and prospect. Journal of Curriculum Studies, 21,307-327.

Fraser, B.J. & O'Brien, P.O. (1985). Student and teacher perceptions of the environment of elementary-school classrooms. Elementary School Journal, 85, 567-580.

Gallagher, J.J. & Tobin, K. (1987). Teacher management and student engagement in high-school science. Science Education, 71, 535-555.

Gore, J.M. (1987). Reflecting on reflective teaching. Journal of Teacher Education, 38, 33-39.

Gore, J.M. & Zeichner, K.M. (1991). Action research and reflective teaching in preservice teacher education: A case study from the United States. Teaching and Teacher Education, 7, 119-136.

Gronn, P.C. (1982). Neo-Taylorism in educational administration. Educational Administration Quarterly, 18, 17-3.5.

Harman, G. (1998, July 15-21). Education research: UK attempts a value judgement. Campus Review.

Hart, E.P. & Robottom, I.M. (1990). The science-technology society movement in science education: A critique of the reform process. Journal of Research in Science Teaching, 27, 575-588.

Hattie, J. & Fitzgerald, D. (1987). Sex differences in attitudes, achievement and use of computers. Australian Journal of Education, 31, 3-26.

Hattie J., Print, M., & Krakowski, K. (1994). The productivity of Australian academics in education. Australian Journal of Education, 38, 201-218.

Hatton, E.J. (1988). Teachers work as bricolage: Implications for teacher education. British Journal of Sociology of Education, 9, 337-357.

Hatton, E.J. (1989). Levi-Strauss, bricolage and theorizing teachers' work. Anthropology and Education Quarterly, 20, 74-96.

Husen, T. (1984). Issues and their background. In T. Husen & M. Kogan (Eds.), Educational research and policy: How do they relate? Oxford:Pergamon.

Jackson, M.W. & Prosser, M.T. (1989). Less lecturing, more learning. Studies in Higher Education, 14, 55-68.

Keeves, J.P. (Ed.). (1987). Australian education: Review of recent research. Sydney: Allen & Unwin.

Keeves, J.P. & Marjoribanks, K. (Eds.). (1999). Australian education: Review of research 1965-1995. Melbourne: ACER Press.

Kessler, S., Dowsett, G.W., Connell, R.W., & Ashenden, D.J. (1985). Gender relations in secondary schooling. Sociology of Education, 58, 34-48.

Kostoff, R.N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics, 43, 27-43.

Laughlin, A. (1984). Teacher stress in an Australian setting: The role of biographical mediators. Educational Studies, 10, 7-22.

Lingard, B. & Blackmore, J. (1998). Education. In Academy of the Social Sciences in Australia, Challenges for the social sciences and Australia. Canberra: Australian Research Council.

Marjoribanks, K. (1981). Sex-related differences in sociopolitical attitudes: A replication. Educational Studies, 7, 1-6.

Marsh, H.W. (1981). Students' evaluations of tertiary instruction: Testing the applicability of American surveys in an Australian setting. Australian Journal of Education, 25, 177-193.

Marsh, H.W. (1986). Verbal and math self-concepts: An internal external frame of reference model. American Educational Research Journal, 23, 129-149.

Marsh, H.W. (1989). Sex differences in the development of verbal and mathematics constructs: The high school and beyond study. American Educational Research Journal, 26, 191-226.

Marsh, H.W. (1991a). Employment during high school: Character building or a subversion of academic goals. Sociology of Education, 64, 172-189.

Marsh, H.W. (1991b). Failure of high ability high schools to deliver academic benefits commensurate with their students' ability levels. American Educational Research Journal, 28, 445-480.

Marsh, H.W., Barnes, J., & Parker, J. (1985). Multidimensional adolescent self-concepts: Their relationship to age, sex, and academic measures. American Educational Research Journal, 22, 422-444.

Marsh, H.W., Barnes, J., & Smith, I.D. (1983). Multitrait-multimethod analyses of the Self-Description Questionnaire: Student-teacher agreement on multidimensional ratings of student self-concept. American Educational Research Journal, 20, 333-357.

Marsh, H.W. & Hocevar, D. (1984). The factorial invariance of student evaluations of college teaching. American Educational Research Journal, 21,341-366.

Marsh, H.W. & Hovevar, D. (1991). The multidimensionality of students' evaluations of teaching effectiveness: The generality off actor structures across academic discipline, instructor level, and course level. Teaching and Teacher Education, 7, 9-18.

Marsh, H.W. & Holmes, I.W.M. (1990). Multidimensional self-concepts: Construct-validation of responses by children. American Educational Research Journal, 27, 89-118.

Marsh, H.W., Smith, I.D., Owens, L., & Marsh, M. (1988). The transition from single-sex to coeducational high-schools: Effects on multiple dimensions of self-concept and on academic achievement. American Educational Research Journal, 25, 237-270.

McGaw, B., Boud, D., Poole, M., & Warry, R. (1992). Educational research in Australia. Canberra: AGPS.

Moses, I. (1986). Promotion of academic stale Reward and incentive. Higher Education, 15, 135-149.

National Board of Employment, Education, and Training. (1994). Quantitative indicators of Australian academic research. Canberra: AGPS.

Over, R. (1981). Women academics in Australian universities. Australian Journal of Education, 25, 166-176.

Over, R. (1982). Does research productivity decline with age? Higher Education, 11, 511-520.

Patching, W., Gersten, R., Colvin, G., Kameenui, E., & Carnine, D. (1983). Direct instruction in critical-reading skills. Reading Research Quarterly, 18, 406-418.

Perry, C.L., Klepp, K.I., Hawkins, K.G., Murray, D.M., & Halper, A. (1986). A process evaluation study of peer leaders in health education. Journal of School Health, 56, 62-67.

Phelan, T.J. (1999). A compendium of issues for citation analysis. Scientometrics, 45, 117-136.

Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The course experience questionnaire. Studies in Higher Education, 16, 129-150.

Robinson, V. (1994). The practical promise of critical research in educational administration. Educational Administration Quarterly, 30, 56-76.

Ross, M.W. and McLaws, M.L. (1992). Subjective norms about condoms are better predictors of use and intention to use than attitudes. Health Education Research, 7, 335-340.

Saha, L.J. (1982). Gender, school attainment and occupational plans: Determinants of aspirations and expectations among Australian urban school leavers. Australian Journal of Education, 26, 247-265.

Schibeci, R.A. & Riley, J.P. (1986). Influence of students' background and perceptions on science attitudes and achievement. Journal of Research in Science Teaching, 23, 177-187.

Smyth, J. (1989). Developing and sustaining critical reflection in teacher education. Journal of Teacher Education, 40, 2-9.

Thomas, P.R. & Bain, J.D. (1982). Consistency in learning strategies. Higher Education, 11, 249-259.

Tinning, R. (1991). Teacher education pedagogy: Dominant discourses and the process of problem setting. Journal of Teaching in Physical Education. 11, 1-20.

Tinning, R. & Fitzclarence, L. (1992). Postmodern youth culture and the crisis in Australian secondary school physical education. Quest, 44, 287-303.

Tobin, K. (1986). Effects of teacher wait time on discourse characteristics in mathematics and language arts classes. American Educational Research Journal, 23, 191-200.

Tobin, K. (1987). The role of wait time in higher cognitive level learning. Review of Educational Research, 57, 69-95.

Tobin, K. & Fraser, B.J. (1989). Barriers to higher level cognitive learning in high-school science. Science Education, 73, 659-682.

Tobin, K. & Fraser B.J. (1990). What does it mean to be an exemplary science teacher? Journal of Research in Science Teaching, 27, 3-25.

Tobin, K. & Garnett, P. (1987). Gender related differences in science activities. Science Education, 71, 91-103.

Treagust, D.F. (1988). Development and use of diagnostic tests to evaluate students' misconceptions in science. International Journal of Science Education, 10, 159-169.

Treagust, D.F., Duit, K., Joslin, P., & Lindauer, I. (1992). Science teachers use of analogies: Observations from classroom practice. International Journal of Science Education, 14, 413-422.

Trigwell, K. & Prosser, M. (1991). Improving the quality of student learning: The influence of learning context and student approaches to learning on learning outcomes. Higher Education, 22, 251-266.

Walker, J.C. (1986). Romanticizing resistance, romanticizing culture: Problems in Willis's theory of cultural production. British Journal of Sociology of Education, 7, 59-80.

Walker, J.C. (1987). School sport, ethnicity and nationality: Dimensions of male youth culture in an inner city school. Australian Journal of Education, 31, 303-316.

Warton, P.M., Goodnow, J.J., & Bowes, J.M. (1992). Teaching as a form of work: Effects of teachers' role and role definitions on working to rule. Australian Journal of Education, 36, 170--180.

Watkins, D. (1982). Identifying the study process dimensions of Australian university students. Australian Journal of Education, 26, 76-85.

Watkins, D. (1983a). Assessing tertiary study processes. Human Learning, 2, 29-37.

Watkins, D. (1983b). Depth of processing and the quality of learning outcomes. Instructional Science, 12, 49-58.

Watkins, D. & Hattie, J. (1985). A longitudinal study of the approaches to learning of Australian tertiary students. Human Learning, 4, 127-141.

Waugh, R.F. & Punch, K.F. (1987). Teacher receptivity to systemwide change in the implementation stage. Review of Educational Research, 57, 237-254.

White, R.T. (1992). Implications of recent research on learning for curriculum and assessment. Journal of Curriculum Studies, 24, 153-164.

Willis, S. & Kenway, J. (1986). On overcoming sexism in schooling: To marginalize or mainstream. Australian Journal of Education, 30, 132-149.

World Bank. (1998). Size of the economy. In World development indicators book. Internet Site: http://www.worldbank.org/wdi/

Dr Thomas Phelan works on the Science and Technology Project, Social Sciences Computing Annex, 2041 Public Policy, University of California, Los Angeles, CA 90025. email: [e-mail]