Australian research output in economics and business: high volume, low impact?

Abstract:

This paper investigates publication patterns of Australian academics in Economics & Business. I show that this discipline follows the general Australian trend of declining impact, measured as citations per paper, from the mid-1990s. However, the gap in Australia's ranking of publication quantity (number of papers) and publication quality (impact) is much wider in Economics & Business than in other disciplines. The discipline combines the highest ranking in quantity with the lowest ranking in quality. Seven possible explanations for this pattern are discussed.

Keywords:

RESEARCH OUTPUT; ACADEMIC RESEARCH; AUSTRALIAN ACADEMICS; PUBLICATION PATTERNS; PUBLICATION QUALITY.

1. Introduction

In the past decades there has been a growing demand for research assessment. The main reasons for this are accountability and value-for-money considerations on the part of government as the provider of public funding for research (Murphy 1996). Until 1993 the Australian Federal Government mainly used research input measures (i.e. competitive research grants) to evaluate the research performance of academic institutions. However, in 1994 the formula for distributing infrastructure funding was supplemented with research output measures (i.e. the number of publications). The quality of publications was not taken into account. Each publication was given the same weight, although books were weighted more heavily than journal articles, conference proceedings and book chapters. As early as 1996 there were calls to develop measures for discriminating between journals, but it was not until May 2004 that the government initiated a formal consultation process about this. This process should result in a recommendation to the Minister by December 2005. A Research Quality Framework is to be established as part of the Backing Australia's Ability--Building our Future through Science and Innovation Initiative (DEST 2004a http://www.dest.gov.au/resqual/default.htm).

Bibliometric analyses conducted by ANU researchers under the Research Evaluation and Policy Project show that Australia's share of ISI publications (1) has increased significantly in the 1990s--from 2.2% to nearly 2.8%--but that Australia's share of citations is falling further behind most other comparable OECD countries. As a result Australia's ranking on the relative citation index (share of world citations divided by the share of world publications) has dropped from seventh in 1988 to eleventh in 1993 (Butler 2001). The BIE (1996) report shows a decline in relative citation rates from the 9th place in the 1981-1985 and 1985-1989 period to 16th in the 1990-1994 period. King (2004) claims that Australia ranks 14th for the 1993-2002 period on citations per paper. Although these sources provide slightly different figures due to slight differences in calculation methods, the downward trend is very clear.

Butler (2001, 2002a, 2003a) explores a number of possible reasons for this decline and argues that increased performance evaluation with a focus on publication output rather than publication quality is the most likely reason for the adverse impact on Australia's relative citation index. Since the introduction of publications as a quantitative indicator of research output, university output has jumped considerably in spite of stable staff numbers and tight funding. Even more concerning is the fact that this rise in output was much stronger for lower-level journals than for top journals. Between 1993 (2) and 1999 Australian universities' share of SCI publications rose by around 20% in the top two quartiles and by 50% in the third quartile, whereas publication in the bottom quartile has doubled (Butler 2002b). This pattern was consistent across all fields of research and occurred only in the university sector, not in the other research sectors that were not subject to the same funding formula (Butler 2003a).

The Social Sciences and Arts & Humanities are noticeably absent from the multitude of bibliometric analyses that have been conducted recently. This is understandable for Art & Humanities where books are typically more important as research output than are journal articles. As a result, the Arts & Humanities Citation Index only includes a small number of journals and data are therefore volatile. However, there are no fewer than 1714 journals listed in the Social Science Citation Index, while 5907 journals are listed in the Science Citation Index. As the Social Sciences cover only three of the 22 ISI disciplines (Business & Economics, Psychology & Psychiatry, Social Sciences General), while the Sciences cover the remaining 19 disciplines, an analysis covering the Social Sciences would seem long overdue. This article will focus on one of the Social Sciences: Business & Economics.

2. Methodology

Two aspects of research output are measured: quantity and quality. Quantity is measured by the number of papers published by country in the different disciplines. Quality is measured by the average number of citations per paper. Although citation impact scores are by no means a perfect measure of quality (3), there is a strong relationship between journal impact scores and perceived journal quality (Butler 2002a) (4) and there is probably no better method to compare scientific performance across countries (BIE 1996). Both measures were sourced from the ISI Web of Knowledge's Essential Science Indicators. The ESI indicators cover a ten-year-plus-six-month period, January 1, 1994 to June 30, 2004.

I must caution that the ISI coverage of research output varies by discipline. DEETYA 1995 data show that the proportion of journal articles varies from 21.4% in Information Technology, Computing & Communication to 83.9% in Chemistry. Bourke and Butler (1996) show that the use of journal citation rates in the Social Sciences and Humanities is problematic as the ISI's coverage of journal output in this type of research is far less complete. (5) However, this should not necessary bias our comparison within disciplines between countries or any analysis of relative performance over time. Also, printed refereed journals were seen as essential by 97% of academics in the Social Sciences. Only in Arts and Humanities were books seen as more important than journals. (Education for Change Ltd 2002) Finally, even though ISI journals might not cover the majority of output in the Social Sciences, most academics aspire to publish in ISI journals as these are generally seen as higher quality journals. It is true that countries differ in terms of the importance attached to journal publications. However, while this might impact on the total number of papers (quantity), it shouldn't necessarily influence the impact of the papers that are published (quality).

The Web of Knowledge covers 22 disciplines. The multidisciplinary category was excluded from this study, as there were only eleven countries with more than 500 papers in this category. I included the top-20 countries in terms of either the number of papers published or the number of citations per paper for each discipline, which resulted in a total of 35 countries. (6) Countries with fewer than 500 papers in a particular discipline were excluded for that discipline for the citations per paper ranking.

3. Results

Table 1 shows the comparative ranking of Australian research output in terms of quantity and quality for 21 disciplines. It is clear that Australia's quality ranking generally lags behind its quantity ranking. In fact, virtually all developed Western countries have higher impact scores than Australia. If we perform an overall analysis averaging impact scores for all disciplines, the only countries in this group that Australia leaves behind are New Zealand, Greece, Spain and Ireland.

In addition, however, table 1 also shows that there are substantial differences between disciplines. Space Sciences, Chemistry and Mathematics show a more than negligible higher quality ranking than quantity ranking. The drop in ranking from quantity to quality is particularly pronounced in Economics & Business. The next most significant drop is for Psychiatry/Psychology and this is only half of the drop for Economics & Business. In Economics & Business, Australia shows a rather unique combination of having the highest discipline ranking in terms of the number of papers (ex equo with Social Sciences/General) and the lowest but one discipline ranking for citations per paper (except for Physics). The pattern of a high quantity ranking and a low quality ranking seems fairly typical for the Social Sciences. Butler (2003b) showed that although a substantial increase in Australia's share of papers was present in both the Social Science citation Index and the Science Citation Index since 1993, the increase was much more dramatic in the former case.

The fact that papers in the Social Sciences generally have lower impact scores than papers in the Medical, Natural and Physical Sciences and Engineering is not unexpected. Much of the research in the Social Sciences is more context-dependent than research in the other disciplines. Hence research conducted in a particular context might not be relevant (and hence not cited) by researchers working in another context. In addition, the Social Sciences are probably more ideologically based, resulting in a stronger fragmentation into sub-disciplines that work in relative isolation, and hence would not cite each other. Finally, research in the other sciences would seem to resemble a perfect market more closely in that new research findings are distributed more quickly and on a broader scale. In some fast-moving research fields, researchers need to consult on-line working papers on a weekly basis. Contrast that with research in Economics & Business that will be presented at conferences, but might not be available to the general academic public until several years (and sometimes even 5-10 years) after the research was conducted. (7)

However, while these differences might explain differential impact scores for different disciplines, table 2 shows that Australian academics in Economics & Business fare much worse in terms of impact than academics in the same discipline in other countries. The only countries we leave behind are Japan, Germany, China and Taiwan, and only Germany and Japan show a similar drop in position when comparing quantity (number of papers) with quality (impact). Table 2 also clearly shows the dominance of the Anglophone countries in the Social Sciences. The USA, UK and Canada are responsible for nearly 70% of the papers published in Economics & Business. The context-dependency of these disciplines is likely to lead many academics in this field to publish in national journals, most of which are not included in the SSCI.

Because conferences are an important means of knowledge dissemination and network building (and subsequent citations) in the Social Sciences, a lack of conference participation might explain why for some countries quality rankings are lower than quantity rankings. Table 2 therefore also ranks countries in terms of their relative participation at the top conference in Management (one of the areas in Economics & Business) for the years 2002-2004. Although we cannot make any general claims from looking at one subdiscipline only, in the Management area the lack of conference participation seems to offer some explanation for the drop of position of Germany and Japan when quantity is compared with quality. They are among the four countries with the lowest per capita conference participation. A relatively high level of conference participation for academics from Singapore and Israel might explain why their quality ranking is higher than their quantity ranking. However, this does not explain Australia's position, since in terms of conference participation Australian academics rank nearly as highly as in terms of the number of published papers. Both measures far exceed Australia's position in terms of quality.

The drop in ranking when comparing research quantity and research quality is also apparent at the level of individual Australian universities. Table 3 compares the ranking of the top three Australian universities in Economics & Business in terms of the number of papers, number of citations and the number of citations per paper. While these universities score rather highly in terms of the number of papers--ranked 48 to 67 worldwide--their ranking in terms of citations per paper is much lower. To put these impact figures into perspective, I compared the impact factor for the University of Melbourne with a range of other universities. The average number of citations per article in Business & Economics for the University of Melbourne is 2.45. This is about one fifth of the average number of citations per article for top US universities such as Harvard, MIT, Princeton, Stanford and the University of Chicago, and one third of the University of Virginia--Melbourne's benchmark university--and top European universities such as INSEAD, LBS, the Free University of Brussels, Stockholm University and Oxford University. It also ranks below all Dutch universities with a Faculty of Economics & Business.

So what might be the reason behind this combination of high quantity, low quality in Business & Economics for Australian universities? One explanation could be that even though Australian academics do publish regularly in journals that are listed in the Social Science Citation Index, their publications are concentrated in journals with lower citation rankings and hence their articles draw a lower level of citations. Another explanation could be that even if Australian academics publish in high-ranking journals, their work is cited less than the work of academics from other countries.

In order to investigate both explanations, I looked at the number of articles that Australian academics have published in top journals and the citations resulting from these articles. For each of the four sub-disciplines in Economics & Business, I selected the top 5 journals in terms of their citation ranking, that is, the average number of citations drawn each year by an article in that journal. (8) This information was drawn from the Journal Citation Reports in the Web of Knowledge. The journals included are listed in table 4. Information on the number of citations was drawn from the Web of Science database by aggregating the number of citations by country for all articles published in top-20 journals. I compared Australia with other Anglophone countries (US, England, Canada) and with the Netherlands, a country similar in terms of the size of the population. Only regular articles were included in this analysis, that is, no book reviews or literature reviews. The results of this analysis can be found in table 5.

Table 5 shows that US and to a lesser extent Canadian academics clearly dominate in the top-20 journals. However, Australia lags behind England and the Netherlands as well, both in terms of the number of top-level publications per capita and in terms of the proportion of top-level publications over total publications. Even more importantly, whereas the Netherlands and to a lesser extent England have experienced a very substantial growth in the number of top publications over a 20 year period, Australia has only just matched the US growth in this respect (56% vs. 53%). Especially in the second half of the 1990s, Australia seems to lag behind the England and the Netherlands in terms of high-quality publications. The start of this period coincides with the introduction of quantity-based research evaluation in Australia. Therefore, Butler's (2001, 2002a, 2003a) conclusion of changing publication patterns in response to changing funding formulas clearly seems to apply to the Social Sciences as well. In contrast, the Netherlands and the UK have introduced research assessments that focused more on quality, typically asking for the submission of an individual academic's best papers. (9)

In terms of citations, the picture is slightly different, but not much more positive for Australia. For each country, the top 10% most cited articles make up around 50% of all citations. However, citations per article are higher for the US, Canada and England regardless of whether top 10% articles are included or not. Australia fares better than the Netherlands in terms of citations. However, Dutch academics only started publishing in top journals recently and hence their articles have not yet had the same amount of time to gather citations. So Australian academics tend to publish less in highly-ranked journals and even when they do their articles get cited less frequently.

Having North American or British co-authors positively influences the number of citations per article. Of the ten most highly cited articles (10) with Australian authors in top-20 journals in Business & Economics (listed in appendix 1), seven had a North American or British co-author and in four of these seven cases they were also the first author. Two articles had an author who worked at a British university, but had a secondary affiliation with an Australian university. Only one article was authored by Australian academics only, but even in this case the first author is someone who was born and trained outside Australia and worked at several international universities. This is true for a number of the other 'Australian' authors as well. The pattern for the Netherlands in this respect was slightly more positive. Four of the five most highly cited articles for Dutch academics had Dutch authors only. However, of the remaining six articles in the top-10, five were co-authored with US academics (one was co-authored with a Belgian academic). In all cases non-Dutch author was first author.

4. Discussion

There are several possible explanations for the Australian publication pattern of high volume, low impact and the low level of publications in top journals. First, it is of course not fair to expect Australian academics to be able to compete with academics in top North American institutions, which generally allocate much more resources to research, including from the business community. Government funding for universities has steadily declined in Australia. It is now on average well below 50% and falling. This is below the levels in the UK and Canada and is approaching the average level for US public universities. However, the difference between US public universities and Australian universities is that half of the income of the former comes from sources other than government funding and tuition fees (including for instance investment income, donations, and bequests), whereas in Australia this source of income only accounts for about a fifth of the total budget. Australian (public) universities match US private universities in terms of the proportion of income drawn from tuition fees. (11) The large proportion of income from tuition fees might have negative implications for research quality on three counts. First, it is generally not money that is invested in research. Second, it means universities have to spend a lot of their time and resources on attracting fee-paying (international) students. Third, it means that student/staff ratios are very high and include a large proportion of international students, both of which increase work pressure for academics, leaving them less time to do research. These negative implications are likely to be particularly pronounced in the Business & Economics discipline because faculties in this discipline draw a far larger proportion of their income from fees than faculties in other disciplines. (12)

Second, and related to the first argument: Australian universities are at a disadvantage to US universities in terms of the salaries they can offer. US universities can generally offer substantially higher salaries in order to attract top researchers from all over the world. Even though salaries according to the official pay scales might be comparable in purchasing power (Ong & Mitchell 2000) (13), US universities generally have a far greater flexibility in their pay scales to reward academic high-flyers (Stevens 2004). There is also greater variation in pay across levels and disciplines in the US (Stevens 2004). (14) Graduates in Economics & Business can typically find high-paying jobs in industry with much greater ease than graduates in other disciplines. Hence attracting top academics in these fields becomes even more difficult. These differences, however, would not explain why academics in Canada, the UK, and the Netherlands--countries in which academic wages are also lower and more compressed than in the US--still produce more high-quality research output than academics in Australia.

Third, Australian research might be tied to the Australian context and hence be less interesting to international researchers. This might hinder both publication of Australian research in high-level journals as well as limit the number of citations for articles that are published. Without data on the number of Australian manuscripts submitted, we cannot assess the extent of the former problem. Moreover, Australian researchers might self-select and simply do not submit articles drawing on the Australian context to international journals. Inspection of the 230 articles that were published in top journals showed that only a handful of them focused specifically on Australia in terms of context or sample. So whereas the lack of articles with an Australian focus might provide some evidence of the problem of getting Australian research published in top journals, it cannot explain why articles that are published are cited less. In order to alleviate the first problem, Australian academics might need to do a better job in explaining what it is about the Australian context that makes the article interesting to a larger audience (see also Kulik 2005). Of course, North American researchers in Business & Economics--who would account for the bulk of citations--might display a perceptual bias against Australian researchers. This might be a simple case of a not-invented-here syndrome. Partial support for this is found in the fact that the most highly cited articles by Australian academics are those co-authored with North American or British academics and the fact that many of the Australian most highly published academics originated from other countries. However, this would not explain why Dutch (and British) academics fare better in this respect.

Fourth, business education in Australia simply has a shorter history than in North America and the UK. However, while business schools might generally have been established much later in Australia than in the other countries, faculties of Economics & Commerce have a far longer history. Melbourne's Faculty of Economics & Commerce was established 80 years ago. And again this does not explain why Dutch academics fare better and it certainly does not explain why Australian academics have particularly lagged behind in the last two decades.

Fifth, local Australian research might simply reflect local managerial practice. It is now generally recognised in Australia that Australian managers have never been at the forefront of innovations in management. A much publicised report in the mid 1990s (Karpin 1995) was highly critical about management skills in Australia and claimed Australian managers lacked the very skills that would make successful managers in the 21st century. Most important among those were leadership skills which include teamwork and empowerment, people management skills including managing a diverse workforce, strategic skills, and an international orientation. If Australian management lags behind its international counterparts, it might be more difficult for Australian academics to do innovative research to produce high quality articles.

Sixth, as discussed above, and in contrast to for instance the UK, the Netherlands and New Zealand (15), the current Australian academic climate seems to reward quantity over quality. Part of research funding by the government is based on the number of publications and one 'DEST point' is given regardless of the quality of the publication. Publications in local conference proceedings with very limited peer review and very high acceptance rates carry the same reward as a journal publication in highest ranked journal in the field with extensive peer review and very low acceptance rates. (16) Butler (2003a) claims that every DEST point is now worth $3,000. It is unclear why this policy would impact on Economics & Business more so than other disciplines, although the discipline's focus on instrumental rationality, efficiency & money might lead academics in this discipline to respond more strongly to financial incentives than e.g. Egyptologists. In addition, the pressured existence of academics in these disciplines might lead them to go for the alternative that appears to maximise return on time invested in terms of DEST points and promotion: publication in journals that are ISI listed, but have relatively low quality standards and high acceptance rates. Another reason might lie in the restructuring of higher education in the late 1980s when the separation between universities and colleges of advanced education and institutes of technology was removed. This was accompanied many mergers and the creation of new universities. As staff in the former colleges and institutes were typically not research-active, the average productivity per researcher declined markedly (Butler 2003b). And because the colleges and institutes had a heavy focus on the social and applied sciences, many academics in business faculties in Australia were drawn from non-research active population. These new academics were typically not well-trained to do research, but were subjected to the same performance evaluation system. I consider it highly likely that their response to this was publication in low-level journals. Butler (2003b) provides an interesting analysis of how different university policies can shape publication results by comparing the University of Western Australia (UWA) with the University of Queensland (UQ). (17) In the late 1980s UWA introduced a new formula to distribute research funds, a major component of which was a publication count. The formula was certainly more sophisticated than the later DEST formula. However, the emphasis was on quantity, not quality. UQ followed a very different strategy, recruiting bright young researchers, including a substantial number from abroad, and providing them with good resources. Both universities succeeded in lifting their publication output per member of staff, but in the case of UWA publications increasingly appeared in lower impact journals, with a subsequent decline in citation impact score that was stronger than the overall decline for Australian universities. UQ academics on the other hand succeeded in publishing in higher impact journals and improving their citation impact score, against the general trend of decline for Australian universities. This shows that university policies can mitigate (undesirable) government policies and hence senior university management can play a big role in shaping an institution's profile.

Finally, I would normally expect government-funded research to provide higher quality output than research that has to be conducted with internal university funding or without funding. A comparison of ARC Discovery Grant (18) funding over the last four years (2002-2005) shows that Economics (RFCD codes 34000) accounted for 2.6% of the grants awarded, while Commerce, Management, Tourism & Services (RFCD codes 350000) accounted for 2.1% of the grants awarded (ARC 2001-2004). For Economics this percentage is broadly similar to the proportion of academic staff (teaching-only and research & teaching) working in this discipline, which for 2004 was 1.6% (DEST 2004b). (19) In comparison, 12.6% of academic staff in Australian universities work in Management and Commerce and hence funding levels lag substantially behind in this discipline. A comparison with one of the Science disciplines brings this picture into even starker relief. Engineering & Technology (RFCD codes 290000) received 14.5% of the grants awarded for 6.9% of staff. It is important to note that these figures look at the number of grants, not the dollar value of grants. Given that the average amount awarded for successful applicants for 2005 was about 66% higher for Engineering & Technology (approx. $111,000) than for Economics, Commerce & Management (approx. $67,000) the difference would be even more pronounced if we look at the dollar value of grants instead. Arguably the best conditions for high-quality research would be presented by fellowships, that allow the Fellow to devote his/her time to research only for 3-5 years. Between 2002-2005 1.3% of the 682 fellowships (APD, ARF, QEII and APF) were awarded in Economics--a proportion broadly equal to the proportion of staff in this discipline. However, a mere 0.4% (2 APDs and 1 APF) of fellowships was awarded in Commerce & Management, while Engineering & Technology received 16% of the fellowships. So forty times as many fellowships were awarded in Engineering & Technology, even though there are only half as many academics in this area as in Management & Commerce. It is clear that, in comparison to other disciplines, academics in Management & Commerce are not well endowed to do high-quality research. Finally, when national research priorities were introduced in 2002 and the ARC was required to commit 33% of funding in these priority areas, their focus was squarely in Science and Engineering. Although the recent introduction of the priority goals 'promoting an innovation culture and economy', 'understanding our region and the world' and 'strengthening Australia's social and economic fabric' give some opening to the Social Sciences and Humanities, the other 18 priority goals remain focused on Science and Engineering.

5. Conclusions and Recommendations

In this paper I investigated publication patterns of Australian academics in Economics & Business. I showed that this discipline followed the general Australian trend of declining impact from the mid-1990s. However, the gap in Australia's ranking of publication quantity and publication quality was much wider in Economics & Business than in other disciplines. The discipline combines the highest ranking in quantity with the lowest but one ranking in quality. In Economics & Business, even the top-3 Australian universities lag very far behind top US and European universities. A comparison with the Netherlands showed that all universities with a Faculty of Economics & Commerce ranked above all Australian universities. A comparison of publication patterns in the top-20 journals in Business & Economics found Australia lagging behind not only the US and Canada, but also England and the Netherlands. Australian academics have a lower tendency to publish in top journals, and even if they do, their articles get cited less than articles from academics in other Anglophone countries. Seven possible explanations for this high volume/low impact pattern shown by Australian research in Economics & Business were discussed. Many of these explanations refer to circumstances that are difficult to change.

However, a change in research evaluation to a system that rewards quality as well as quantity might resolve some of these problems. In addition, the examples of UQ and UWA showed that senior management in universities could play an important role in counteracting the impact of undesirable government policies. Further, the increasing demand for methodological rigour in top journals in Economics & Business puts Australian academics--usually trained in research-only degrees--at a disadvantage in comparison to US academics who have typically undergone rigorous coursework programmes in their PhD studies. Inclusion of formal coursework into PhD programmes in Australia--as is increasingly the case in the UK and the Netherlands--might therefore better prepare the next generation of Australian academics for publication in top journals. If this would be accompanied by a higher level of government funding of research in Economics & Business--especially in terms of fellowships--this might allow Australian academics in this field to devote themselves to top-quality research and increase their international impact.

Appendix 1

Top-10 Australian Publications in Top-20 Journals by Number of Citations

Breusch, T.S. (UK) & Pagan, A.R. (Australian National University), 1979, 'Simple test for heteroscedasticity and random coefficient variation', Econometrica, vol. 47, no. 5, pp. 1287-94, CITES: 493.

Ball, R. (USA) & Brown, P. (University of Western Australia), 1968, 'Empirical evaluation of accounting income numbers', Journal of Accounting Research 6 (2): 159-178, CITES: 310.

Knetsch, J.L. (Canada) & Sinden, J.A. (University of New England), 1984, 'Willingness to pay and compensation demanded: Experimental-evidence of an unexpected disparity in measures of value', Quarterly Journal of Economics, vol. 99, no. 3, pp. 507-21, CITES: 234.

Wood, R. (Australian Graduate School of Management) & Bandura, A. (USA), 1989, 'Social cognitive theory of organizational management', Academy of Management Review, vol. 14, no. 3, pp. 361-84, CITES: 178.

Mizon, G.E. (UK/Australian National University) & Richard, J.F. (Belgium), 1986, 'The encompassing principle and its application to testing nonnested hypotheses', Econometrica, vol. 54, no. 3, pp. 657-78, CITES: 138.

Watson, R.T. (Western Australia College of Advanced Education), Desanctis, G. (USA) & Poole, M.S. (USA), 1988, 'Using A GDSS to facilitate group consensus: Some intended and unintended consequences', MIS Quarterly, vol. 12, no. 3, pp. 463-78 CITES: 111.

Midgley, D.F. (Australian Graduate School of Management) & Dowling, G.R. (University of Newcastle), 1978, 'Innovativeness: Concept and its measurement', Journal of Consumer Research, vol. 4, no. 4, pp. 229-42, CITES: 101.

Sanchez, R. (University of Western Australia) & Mahoney, J.T. (USA), 1996 'Modularity, flexibility, and knowledge management in product and organization design', Strategic Management Journal, vol. 17, special issue, pp. 63-76, CITES: 92.

Porac, J.F. (USA), Tomas, H. (Scotland) & Wilson, F. (Curtin University of Technology), et al. 1995, 'Rivalry and the industry model o Scottish knitwear producers', Administrative Science Quarterly, vol. 40, no. 2, pp. 203-27, CITES: 92.

Hammond, P.J. (UK/Australian National University), 1976, 'Equity, arrows conditions, and rawls difference principle', Econometrica, vol. 44, no. 4, pp. 793-804, CITES: 91.

The author would like to thank the following people for their thoughtful suggestions: Glyn Davis, Carol Kulik, Christina Cregan and Tatiana Zalan (all University of Melbourne), Arjen van Witteloostuijn (University of Groningen), Phyllis Tharenou (University of South Australia) and Marcel Wissenburg (Radboud University Nijmegen). Thanks are also due to the Economics Area Editor of AJM, Chongwoo Choe, for assigning a very knowledgeable reviewer to this paper. This anonymous reviewer's suggestions have helped me to clarify several aspects of the paper.

(Date of receipt of final transcript: April 18, 2005. Accepted by Chongwoo Choe, Area Editor.)

References

Amin, M. & Mabe, M. 2000, 'Impact factors: Use and abuse', Perspectives in Publishing, no. 1, October.

ANU 2004 Review of The Australian National University, http://info.anu.edu.au/Discover_ANU/Review/index.asp, accessed January 2005.

ARC 2001-2004, 'Data on funded Discovery Grants and Fellowships', http://www.arc.gov.au/funded_grants/selection_discovery_projects.htm and http://www.arc.gov.au/funded_grants/selection_discovery_fellow.htm

Azar, O.H. 2004, 'Rejections and the importance of first response times', International Journal of Social Economics, vol. 31, no. 3, pp. 259-74.

Bourke, P. &, Butler, L. 1996, 'Publication types, citation rates and evaluation', Scientometrics, vol. 37, no. 3, pp. 473-94.

Bureau of Industry Economics (BIE), 1996, Australian Science. Performance from published papers', Report 96/3, Australian Government Publishing Service, Canberra.

Butler, L. 2001, Monitoring Australia's Scientific Research, Australian Academy of Science, Canberra.

Butler, L. 2002a, 'Identifying 'highly-rated' journals: An Australian case study', Scientometrics, vol. 53, no. 2., pp. 207-27.

Butler, L. 2002b, 'A list of published papers is no measure of value', Nature, vol. 419, 31 October, pp. 877.

Butler, L. 2003a, 'Academic reactions: Modifying publication practices in response to funding formulas', Research Evaluation, vol. 12, no. 1, April, pp. 39-46.

Butler, L. 2003b. 'Explaining Australia's increased share of ISI publications: The effects of a funding formula based on publication counts', Research Policy, vol. 32, pp. 143-55.

DEST, 2004a, Information on Research Quality Framework, http://www.dest.gov.au/resqual/default.htm, accessed January 2005.

DEST, 2004b, Higher Education Statistics 2004. FTE for Full-Time and Fractional Full-Time Staff by State, Insitution and Function in an Academic Organisational Group, http://www.dest.gov.au/highered/statpubs.htm#staffpubs, accessed December 2004.

Education for Change Ltd. 2002, Researchers' Use of Libraries and other Information Sources: current patterns and future trends', SIRU, University of Brighton, The Research Partnership, http://www.rslg.ac.uk/research/libuse/, accessed January 2004.

Karpin, D.S. 1995 'Enterprising nation: Renewing Australia's managers to meet the challenges of the Asia-Pacific century', Executive summary report of the Industry Task Force on Leadership and Management Skills, Canberra : Australian Govt. Pub. Service.

King, D.A. 2004, 'The scientific impact of nations. What different countries get for their research spending', Nature, vol. 430, 15 July, pp. 311-16.

Kulik, C.T. 2005, 'On editing in an international context', Journal of Management, vol. 31, no. 2, pp. 162-5.

Murphy, P. 1996, 'Determining measures of the quality and impact of journals', Commissioned Report No. 49, Australian Government Publishing Service, Canberra.

Ong, L.L. & Mitchell, J.D. 2000, 'Professors and hamburgers: An international comparison of real academic salaries', Applied Economics, vol. 32, pp. 869-76.

Stevens, 2004, 'Academic salaries in the UK and US', National Institute Economic Review, no. 190, October, pp. 104-13.

(1.) ISI publications are publications in journals listed in the Citation Indices (Science Citation Index, Social Science Citation Index, Arts & Humanities Citation Index) published by ISI (Institute for Scientific Information). The ANU and other analyses mentioned in this paragraph only covered the Science Citation Index.

(2.) Although publication output was not included in the funding formula until 1995, data on the number of publications has been collected since 1993.

(3.) A potential problem with bibliometric analyses is that individual papers could skew the overall results. Papers could be highly cited because authors cite their own work or because the paper in question has been discredited. However, given the large number of papers included in bibliometric analyses, these potential distortions are likely to be small (King 2004). For a discussion of the limitations of the use of impact factors see Amin & Mabe (2000).

(4.) I compared four journal rankings (collected by the Universities of Aston, Cranfield, Queensland and Vienna) based on surveys of academics in my collated Journal Quality List (see www.harzing.com) with the associated Journal Citation scores from ISI. As lower ranked journals typically didn't have ISI scores, I only included the top three categories from each of the four journals rankings. Each of the four rankings showed very significant differences in ISI scores between the three ranks (F values ranging from 13.223 to 63.072) in the expected order (i.e. higher ranked journals having higher ISI scores).

(5.) Although this is probably true, ISI coverage in Economics & Business might have increased over the years. Well over half of the journals in my Journal Quality List (a compilation of a dozen journal rankings) are ISI-ranked. As indicated in the introduction, there are 1714 journals listed in the Social Science Citation Index overall.

(6.) Hong Kong was excluded from this analysis. During the 1997-2004 period its papers were progressively included under the Peoples Republic of China. This has happened gradually as it relied on authors to add PRC to their country affiliation. If PRC was not included ISI still assigned papers to Hong Kong. However, no papers are listed for Hong Kong after 2001. As a result, Hong Kong's share of papers is underestimated. Its impact score, however, is very high as pre-2001 papers continue to gather citations and the citations per paper metric is not diluted by recent papers with few citations.

(7.) This observation is mainly based on conversations with colleagues in areas such as Physics and Medicine as well as colleagues in Economics & Business and hence is to a large extent anecdotal. To the best of my knowledge there are no studies comparing publication delays in different disciplines. However, long publication delays in major journals in the field in Economics & Business are easily confirmed by comparing the date of first submission, which many journals list for each article, with the final publication date. In addition, publication delays in Economics journals have been documented by Azar (2004). He indicates an average delay of submission to publication of 26 months, increasing to 31 months for the top-5 Economics journals. It is important to note that this publication delay does not include the delays caused by rejection of the paper by other journals before acceptance. According to Azar (2004) the average paper is submitted between three and six times prior to publication and the average 'first response time'--that is, the time it takes to decide whether to reject the paper or advise a revise & resubmit or acceptance as is--is five months. Assuming the average paper is submitted four times before acceptance, this would add at least another 15 months to the publication delay. Actual delays are likely to be much longer as the authors will usually revise the paper before sending it to another journal. This would bring the average publication delay to 41-46 months plus the time authors take to revise the paper. A publication delay of 4-5 years from first submission would hence not seem unusual in Economics. Please note this delay does not yet include the time it takes to complete the study and write the first version of the paper.

(8.) While I do realise that some academics in Business & Economics publish in Psychology journals, I have chosen not to expand the analysis to these journals as it would mean including the many academics working in Psychology, which would obscure the focus of this paper.

(9.) Research assessment was first introduced in the Netherlands in 1993, while the first Research Assessment Exercise (RAE) in the UK took place in 1986 (then named Research Selectivity Exercise).

(10.) The 4th most cited article on this list was actually written by two US academics. Sydney, Australia had accidentally been added to the affiliation of one of the authors. I therefore included the 11th most cited article instead. In this analysis ISI only lists the number of correct citations to the articles in questions. Nearly all articles in the ISI database have additional citations to them with e.g. incorrect page numbers, journal titles, incomplete author initials etc. However, I assume this would not drastically change this analysis as this would be a problem for all articles.

(11.) This information is drawn from data presented in The Inaugural Melbourne Politics Lecture by Professor Glyn Davis, the incoming Vice Chancellor of the University of Melbourne.

(12.) As an example: At the University of Melbourne, the Faculty of Economics and Commerce is targeted to raise 77.5% of its income from fees in 2005 (up from the 75.4% target in 2004), while the average 2005 target for the proportion of fee income for the other faculties lies around 36% (identical to the 2004 targets). With a ratio of nearly 25 in 2004, its student/staff ratio is the highest of the university, though it must be said that Architecture, Building & Planning, Law, and Education also have student/staff ratios above 20, while the ratio for Arts is 16.5. Lower ratios are found for the Sciences and Engineering (ranging from 5.8 for Medicine, Dentistry & Health Science to 11.7 for Engineering). Another way to look at this is to compare the percentage of total academic staff working in the Faculty of Economics & Business and Melbourne Business School (7.5%) to the percentage of the total number of students (EFTSU) served by these faculties (15.7%).

(13.) Even this comparison has been criticised though as purchasing power comparisons were based on the Big Mac index only.

(14.) Although this article involved a comparison between the UK and the US, this would apply even more so to Australian academic salaries that are even more compressed than British salaries.

(15.) New Zealand only introduced the Performance Based Research Funding Scheme in 2003 and the proportion of research funding linked to it gradually increase to 100% in 2007. It is therefore too early to assess whether the introduction of this scheme has had an impact on publication patterns in New Zealand.

(16.) It is interesting that one Australian university--the Australian National University has taken the step to independently assess the quality of its research and education in a very comprehensive exercise that could function as a model for a more meaningful assessment of research in Australia. The final report was published in September 2004 (http://info.anu.edu.au/Discover_ANU/Review/index.asp).

(17.) This analysis was for the two universities as a whole and did not focus specifically on Business & Economics.

(18.) Discovery Grants were chosen as these grants and associated fellowships are open to all disciplines and arguably more prestigious than most other grants. It is important to note that in addition to ARC funding many of the Sciences have alternative sources of funding (NHMRC) that are not open to academics in the Social Sciences.

(19.) Data on the DEST website only lists information by generic AOU. As Economics & Econometrics is a subcategory of the Society and Culture Academic Organisational Unit Group, data on the proportion of academic staff in this discipline were sourced directly from DEST.

Anne-Wil Harzing, The University of Melbourne, Parkville Campus, VIC, 3010. Email: [e-mail]

Table 1
Ranking of Australia in Terms of Quantity (no of papers) and
Quality (citations per paper) in Comparison to Other Countries

Discipline                     Rank by      Rank by      Change of
                                No of    Citations Per    Position
                               Papers        Paper       (Quality--
                                                          Quantity
                                                          Ranking)

Economics & Business              4/5         19          -14/-15
Agricultural Sciences             9           14           -5
Biology & Biochemistry           10           14           -4
Chemistry                        16           12           +4
Clinical Medicine                 9           13           -4
Computer Science                 11           16           -5
Engineering                      13           16           -3
Environment/Ecology               5           10           -5
Geosciences                       8            6           +2
Immunology                       10            9           +1
Materials Science                15           16           -1
Mathematics                      11            8           +3
Microbiology                     10           13           -3
Molecular Biology & Genetics     12           15           -3
Neuroscience & Behaviour         11           17           -6
Pharmacology & Toxicology        11           11           --
Physics                          19           20           -1
Plant & Animal Science            7           13           -6
Psychiatry/Psychology             5           13           -8
Social Sciences/General           4           11           -7
Space Sciences                   11            5           +6

Table 2
Business & Economics Country Ranking for Research Quantity and
Quality and Conference Participation *

  Country     Rank by    Rank by    Change of Position
               No of    Citations   (Quality--Quanity)
              Papers    Per Paper

USA              1          1               --
UK               2          5              -3
Canada           3          4              -1
Germany          4/5       21             -16/17
Australia        4/5       19             -14/15
France           6         10              -4
Netherlands      7          8              -1
Italy            8         12              -4
Spain            9         17              -8
Japan           10         20             -10
Sweden          11          3              +8
Israel          12          2             +10
Belgium         13          6              +7
China           14         22              -8
Switzerland     15          7              +8
Denmark         16         14              +2
South Korea     17          9              +8
Norway          18         11              +7
Taiwan          19         23              -4
New Zealand     20         18              +2
Austria         22         13              +9
Singapore       23         16              +8
Ireland         26         15             +11

  Country       % of     % of Impact   AoM Participation
              Research   (USA= 100%)    2002-04 Per 1m
               Output                       People

USA            50.2%       100%           10.08 (2)
UK             13.5%        66.2%          4.16 (11)
Canada          5.7%        71.5%          7.38 (3)
Germany         3.4%        40.7%          0.60 (20)
Australia       3.4%        43.9%          6.90 (5)
France          3.2%        56.9%          1.50 (15)
Netherlands     3.1%        64.3%          5.99 (9)
Italy           1.8%        50.1%          0.63 (18)
Spain           1.8%        46.1%          1.09 (17)
Japan           1.5%        41.5%          0.15 (22)
Sweden          1.5%        72.6%          4.08 (12)
Israel          1.4%        84.6%          6.18 (7)
Belgium         1.3%        67.1%          1.64 (14)
China           1.2%        35.7%          0.02 (23)
Switzerland     1.0%        65.2%          5.99 (8)
Denmark         0.9%        48.4%          5.05 (10)
South Korea     0.8%        57.5%          0.37 (21)
Norway          0.8%        53.3%          6.19 (6)
Taiwan          0.8%        35.5%          1.03 (19)
New Zealand     0.7%        45.8%          7.09 (4)
Austria         0.6%        49.6%          1.18 (16)
Singapore       0.6%        46.3%         14.24 (1)
Ireland         0.4%        48.2%          3.53 (13)

Note: * In the ranking of citations per paper I excluded the
countries that had less than 500 published papers. These were
usually countries with a very small number of publications, for
example, Macao and Ecuador had the highest citation/paper score
but published only 5 and 9 papers respectively.

Table 3
Ranking of Three Australian Universities in Economics & Business

          University              Rank      Rank         Rank
                                 No of      No of     Citations/
                                 Papers   Citations     Paper

Australian National University     48        116         148
University of Melbourne            59        135         149
University of New South Wales      67        103         141

Table 4
Top-5 Journals in Each Business & Economics Sub-Discipline
According to Impact Score

                                               JCR
                                              2003

           Finance & Accounting

Journal of Accounting Economics               3.844
Journal of Finance                            3.267
Journal of Financial Economics                2.723
Review of Finance Studies                     2.200
Journal of Accounting Research                1.524

                Management

Academy of Management Review                  4.415
Academy of Management Journal                 3.343
MIS Quarterly                                 2.811
Strategic Management Journal                  2.723
Administrative Science Quarterly              2.721

                 Marketing

Journal of Marketing                          2.611
Journal of Consumer Research                  2.585
Journal of Marketing Research                 2.143
Marketing Science                             1.898
Journal of the Academy of Marketing Science   1.321

                 Economics

Journal of Economic Literature                5.243
Quarterly Journal of Economics                4.756
Journal of Economic Perspectives              2.677
Economic Policy *                             2.250
Econometrica *                                2.250

Note: * These journals were added as two higher ranked journals
were already included under Finance & Accounting.

Table 5
Publications in Top-20 Journals in Economics & Business and
Resulting Citations for Five Countries *

                                  Australia              USA

Sub-Disciplines
  Economics                       74 (32%)           3,841 (22%)
  Finance & Accounting            54 (23%)           5,472 (31%)
  Management                      58 (25%)           4,197 (24%)
  Marketing                       44 (19%)           4,334 (24%)
  Total                          230                17,844
  Top Publications Per 1          11.55                 60.90
    Million Inhabitants
Year of Publications
  1956-1980                       55                 4,315
  1980-1984                       34                 2,180
  1985-1989                       24 (-29%)          2,374 (+9%)
  1990-1994                       27 (+13%)          2,735 (+15%)
  1995-1999                       37 (+37%)          2,913 (+7%)
  2000-2004                       53 (+43%)          3,327 (+7%)
  Growth in Top Publications       56%                 53%
    2000-2004/1980 1984

Top Publications/Total         97/4,131 = 2.4%   6,783/61,128 = 11.1%
  Publications (1994 2004)

Citations
  Average Number of                22.88               32.26
    Citations/ Article
  Idem, Excluding Top 10%          11.24               15.66
    Most Cited Articles
  Average Citations/Article         2.56                5.83
    for All SSCI Listed
  Articles 1994-2004

                                    Canada             England

Sub-Disciplines
  Economics                       247 (20%)           430 (49%)
  Finance & Accounting            342 (28%)           173 (20%)
  Management                      388 (32%)           195 (22%)
  Marketing                       249 (20%)            82 (9%)
  Total                         1,226                 880
  Top Publications Per 1           37.71               18.43
    Million Inhabitants
Year of Publications
  1956-1980                       301                 183
  1980-1984                       172                 112
  1985-1989                       193 (+12%)           83 (26%)
  1990-1994                       170 (12%)            84 (+1%)
  1995-1999                       180 (+6%)           173 (+106%)
  2000-2004                       210 (+17%)          245 (+42%)
  Growth in Top Publications       22%                119%
    2000-2004/1980 1984

Top Publications/Total         424/6,935 = 6.1%   439/13,868 = 3.2%
  Publications (1994 2004)

Citations
  Average Number of                35.67               32.17
    Citations/ Article
  Idem, Excluding Top 10%          16.61               14.05
    Most Cited Articles
  Average Citations/Article         4.17               4.02
    for All SSCI Listed
  Articles 1994-2004

                                 Netherlands

Sub-Disciplines
  Economics                        61 (28%)
  Finance & Accounting             24 (11%)
  Management                       57 (26%)
  Marketing                        78 (35%)
  Total                           220
  Top Publications Per 1           13.48
    Million Inhabitants
Year of Publications
  1956-1980                       15
  1980-1984                       10
  1985-1989                       10 (-%)
  1990-1994                       22 (+120%)
  1995-1999                       65 (+195%)
  2000-2004                       98 (+51%)
  Growth in Top Publications     880%
    2000-2004/1980 1984

Top Publications/Total         169/3,787 = 4.5%
  Publications (1994 2004)

Citations
  Average Number of               19.58
    Citations/ Article
  Idem, Excluding Top 10%          9.83
    Most Cited Articles
  Average Citations/Article        3.75
    for All SSCI Listed
  Articles 1994-2004

* This analysis was conducted in January 2005. Additional articles
published in 2004 might have been listed in the SSCI since.