Towards research performance in the humanities.

ABSTRACT

THIS PAPER DESCRIBES A general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application-oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on our study of scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The study illustrates that bibliometrics is much more than conducting citation analyses based on the indexes produced by the Institute for Scientific Information (ISI), since citation data do not play a role in the study. Interaction with scholars in the fields under consideration and openness in the presentation of the quantitative outcomes are the basic features of the methodology. Bibliometrics should be used as an instrument to create a mirror. While not a direct reflection, this study provides a thorough analysis of how scholars in the humanities and social sciences structure their activities and their research output. This structure can be examined empirically from the point of view of its consistency and the degree of consensus among scholars. Relevant issues can be raised that are worth considering in more detail in followup studies, and conclusions from our empirical materials may illuminate such issues. We argue that the principal aim of the development and application of bibliometric indicators is to stimulate a debate among scholars in the field under investigation on the nature of scholarly quality, its principal dimensions, and operationalizations. This aim provides a criterion of "productivity" of the development process. We further contend that librarians are not infrequently requested to provide assistance in collecting data related to research performance assessments, and that the methodology described in the paper aims at offering a general framework for such activities, and can be used by librarians as a line of action whenever they become involved.

1. INTRODUCTION

The study presented in this paper focused on the fundamental questions: How does one recognize a "good" scholar? How does one recognize an "important" scholarly contribution? The approach adopted in this study can be defined as bibliometric. It aims at identifying characteristics of scholarly publications that can validly be assumed to reflect the "quality" or "importance" of a scholar or a scholarly work. Therefore, a first answer to the question "How does one recognize a `good' scholar?" is: One should examine his or her scholarly publications. In other words, in a bibliometric approach, it is assumed that important contributions to scholarly progress are sooner or later communicated in scholarly publications. This is considered to be a universal characteristic of scholarly development in natural sciences, life sciences, social sciences, and humanities.

A bibliometric approach is a quantitative approach. It attempts to calculate statistics of quantitative aspects derived from scholarly publications. Bibliometric indicators result from the statistical analysis of bibliographic information retrieved from the scholarly literature. This determines both their strength and their limitations. The strength of the bibliometric method is that, once established, it can be applied in a uniform or objective manner, eliminating the influence of subjective or personal factors. On the other hand, being a statistical method, it cannot take into account all particularities or special features of the objects to be assessed. As a consequence, bibliometric data should always be applied in combination with qualitative knowledge about the scholars involved and the subdisciplines in which they are active.

Bibliometric indicators have been successfully applied in many subdisciplines in the natural and life sciences. Data from the Science Citation Index (SCI), produced by the Institute for Scientific Information (ISI), play an important role in analyses of research performance in these subdisciplines (e.g., van Raan, 1996; Van Den Berghe et al., 1998). Thus far, the social sciences and humanities have not often been subjected to such analyses. At the same time, the academic authorities of many universities have expressed the need to obtain an insight into the research performance of all faculties and in all fields of scholarship.

Fundamental differences exist between the natural and life sciences, on the one hand, and the humanities and social sciences, on the other hand, with respect to the research object, the methodologies applied, and the structure of scholarly communication. As a result, those who are involved in the development of performance indicators for the humanities and social sciences are confronted with the following situation.

Firstly, they need to develop methodological tools to assist evaluation agencies or policymakers in carrying out their tasks, in the same way that the current SCI-based methodologies provide supplementary research assessment tools in the natural and life sciences. Secondly, this methodology should take into account the characteristics of the field of scholarship, the nature of the scholarly research object, and particularly the communication practices among scholars and the structure of the communication system in their fields. A study dealing with these challenges can indeed be considered an endeavour.

This paper attempts to describe a general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on a study on scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The background, setup, and methodological framework are presented in Section 2. It is followed by a concise review in Section 3 of earlier studies on research performance in this field of scholarship.

Section 4 presents a number of characteristic outcomes of the study. Its principal aim is to illustrate the methodology outlined in Section 2. A detailed overview of the study is presented in a research report by Luwel et al. (1999). The study illustrates that bibliometrics is much more than conducting citation analyses based on the ISI citation indexes, as citation data do not play a role in this study.

Finally, Section 5 gives a critical discussion of the methodology, in the light of the experiences collected in the study. This discussion includes a short overview of the comments of scholars and of the followup of our study, and summarizes the main features of our methodology.

2. BACKGROUND, SETUP, AND METHODOLOGICAL FRAMEWORK

2.1 Background and Setup of the Study

The study presented in this paper was a pilot study commissioned by the Flemish Inter-University Council (VLIR) for developing a methodology to assess research performance in the social sciences and humanities. The disciplines selected by the VLIR for this study were Law and Linguistics. This paper discusses only the study on Law. The Catholic University of Leuven, the University of Gent, the Flemish-speaking Free University of Brussels, and the University of Antwerp decided to participate in the study, which was partially funded by a grant approved by the Flemish Minister-President, who is also in charge of science and technology policy.

At the start of the project, in early 1997, a project team was set up, consisting of the research staff of the VLIR, researchers of the Centre for Science and Technology Studies (CWTS) at Leiden University (the Netherlands), and a staff member of the Science and Innovation administration of the Ministry of the Flemish Community.

The activity of the project team was supported by a university expert group, set up for each of the two disciplines, and composed of senior academic staff members of the departments of Law and Linguistics at the four participating universities. The expert groups assisted the project team during the elaboration of the project, and played an active role in their respective universities in a series of activities, such as data collection, the development of classification systems, and commenting on drafts of the final report.

The first stage of the project work plan was a clear boundary setting of the two disciplines. For the Law faculties, this operation was relatively simple. Academic staff data were extracted from the universities' central administration databases, including year of birth, gender, starting and ending date of their appointments, rank, length of appointment (e.g., 40 percent, that is, two days a week), funding source, and year of Ph.D. granting.

In a subsequent phase, a questionnaire was prepared to collect quantitative data. It was sent to all researchers, both junior and senior, active in one of the four Flemish universities at the end of the year 1996. The members of the two expert groups were of the opinion that the study should not be limited to research activities only, but that all academic activities should be taken into account, analyzing also the fraction of work dedicated to research. The most important data, analyzed in this paper, were lists of publications.

In tandem with the analysis of the first questionnaire, a second questionnaire was elaborated, again in collaboration with the expert groups, in order to collect more qualitative information on leading publications,journals, publishers, and Flemish scholars in the two disciplines. The main objective was to obtain insight into scholars' perceptions on scholarly work quality, and to assess to what extent the scholars' opinions corroborated the outcomes of the quantitative indicators. For Law, this questionnaire was sent to professors working at Flemish, Dutch, and Belgian French-speaking universities, as well as to Belgian senior magistrates. Respondents were asked to indicate "outstanding," "good, yet not outstanding," and "less good" journals, and the names of Flemish scholars whose work is currently very important to their subdiscipline.

A draft report was sent to and discussed with the two expert groups, and the results of these discussions were incorporated into the final version of the report.

2. 2 Methodological Framework

Our study was primarily a methodological one. Rather than making comparative evaluation statements on research performance at the four universities, it explores methods to provide a clear insight into scholarly research activities and proposes indicators for measuring relevant aspects of scholarly performance. It examines the validity of such indicators and explores the type of data needed to construct them, taking into account the availability and reliability of such data.

In this study, the participants were confronted with the problem of lack of standardization in the publication practices of Law scholars. This is a problem in many subfields in the social sciences and humanities. By contrast, from interviews with scientists in molecular biology, it appears that this subfield has a strong consensus of how research materials should be published, and which journals are the most prestigious. All important research output is published in English, in international journals. There are some five to ten journals that are generally acknowledged as "top"journals (e.g., Van Den Berghe et al., 1998). Many publication lists of scientists have a standard format, and small contributions, such as meeting abstracts or editorials, are not even listed.

However, in the humanities, and particularly in Law, important contributions are often published in commemorative books with a narrow circulation. There are many types of publications, and publication lists of scholars are often not ordered by type. Publications are often in the mother language, and many activities have an applied nature.

In our study, a thorough analysis of the publication output was conducted. The quality of the bibliographic information was assessed. The scholars providing the publication data added several types of additional information to each publication, using classification systems of types of publications and subdisciplines. This additional information was examined carefully, from the point of view of its accuracy, embedded structural relationships, the degree of consensus among scholars, and the extent to which differences existed among subdisciplines.

Typical examples of more specific research questions were: How important is the role of books in the communication among scholars? How can one reliably measure the number of books published by a scholar during a given time period? How important is the role of journals? Are there differences among subdisciplines? In which ways do publications classified by juridical scholars as "substantial contributions" differ from "small contributions"? Do the two types of publications reveal different bibliometric characteristics? How consistently was this distinction made among scholars? Are there any criteria to discriminate between scholarly journals and journals of a more applied nature? Assuming that, in the field of Law, it is appropriate to make a distinction between a "scholar" and a "practitioner," are there any bibliometric indicators that can be used in helping to discriminate between the two types?

What is the perception of Flemish, Belgian French-speaking, and Dutch scholars on the quality of individual Law journals? Were there significant differences between the perceptions of Flemish scholars and those of their colleagues abroad? What does a tentative ranking of journals look like, based on their quality, as perceived by scholars, and their international visibility? What is the position of Flemish journals in such ranking? To what extent do actual publication strategies of Flemish scholars conform to a "quality standard" as expressed in their own ratings of scholarly journals?

3. A CONCISE REVIEW OF EARLIER STUDIES ON RESEARCH PERFORMANCE IN Law

Several authors have addressed the measurement of research performance in the social sciences and humanities from a general perspective (Cole, Cole, & Dietrich, 1978; Garfield, 1979, 1986; Cole, 1983; Nederhof et al., 1989; Kyvik, 1989; Finkenstaedt, 1990; Nederhof & Zwaan, 1991; Nederhof & Noyons, 1992; Hemlin, 1996; Hemlin & Gustafsson, 1996; Wood, 1998). Law, in particular, has been called "the birthplace of citation study" (Shapiro, 1992, p. 339). For instance, according to Shapiro, in 1894 a table showing the comparative citation frequency of the Federal, English, and State decisions was produced in Boston, and in 1817 a first count of the volume of English Law reports was made. Notwithstanding this very early start, a literature search revealed very few articles referring to the measurement of scholarly performance in Law (cf. Justiss, 1993).

Swygert & Gozansky (1985) studied the productivity of 1,950 U.S. senior Law faculty members (full-time appointed, full professors) by examining their publications in the Legal Resource Index (LRI) and the database of the Online Computer Library Centre, Inc. (OCLC) during three and four years, respectively. The coverage by LRI (mostly articles and book reviews) and OCLC (book titles) was deemed nearly complete. Swygert & Gozansky (1985, p. 378) included a wide variety of publications: "Articles, books, book reviews, casebooks, teacher manuals, practice manuals, textbooks, monographs, treatises, supplements and compilations, as well as edited and co-authored works," provided the entry was five pages or longer in length. Testimonials, obituaries, reports or proceedings, bibliographies, newspaper columns, recordings and any title labelled "bar review notes" were not included.

The results showed that 44 percent of the faculty members had no publications whatsoever, while 65 percent had no more than 1 publication. Only 15 percent had four or more publications. The mean number of publications was 1.5 per faculty member over a period of three to four years, or less than 1 publication in two years. A school or university published on average about 18 items, with the faculty at Chicago (mean = 5.1 publications per member), Cornell, and New York University ranking highest in average productivity, followed by Berkeley, Stanford, Yale, and Harvard (all with 3.0 publications or more per faculty member). Schools with less senior faculty members were less productive on average (as indicated by the Pearson correlation coefficient, r = 0.53). The authors assumed that smaller-sized schools had a higher teaching load per head. It should be noted that the productivity of faculty members of lower age (the mean was fifty-one years), and striving for a tenured position, may well be considerably higher.

Other less extensive studies analyzed the productivity of institutions by recording author affiliation in journals of high prestige (e.g., Ellman, 1983; Sorensen, 1994).

Two publications related to research performance assessment in juridical research are of particular interest to the study presented in this report. The first is the November 1996 final report of the Inter-University Committee of the Flemish Faculties of Law, entitled "The Assessment of Performance in Juridical Research." This report presents a classification of scholarly publications in the field of juridical research. In our study, a classification scheme was applied that is principally based upon that of this committee. Therefore, our study can be viewed as a first large-scale experiment with this classification system.

According to the Inter-University Committee of the Flemish Law Faculties, the published book ranks first in the scholarly juridical publication output. A book is viewed as the result of an often individual and personal synthesis of legislation,jurisdiction, and juridical theory in a subdiscipline. A book often reflects continuous, intensive scholarly research, conducted for many years. In the Committee's view, the same is true for doctoral theses. Therefore, as a rule, a doctoral thesis deserves publication as a book.

The Committee also made a distinction between substantial scholarly contributions and scholarly contributions of a limited size, published in accepted scholarly journals, anniversary volumes, seminar reports, and collective works. Typical examples of the first type are: A leading article, a review on jurisdiction, or a thorough annotation. A short annotation, a thorough book review, or an intervention as panel member or participant in a conference are examples of scholarly contributions of a limited size.

The Committee did not succeed in developing a classification of scholarly journals in terms of their quality or reputation. The main impediment to such a ranking was that most Law journals show large variations in the quality of the papers published. In addition, some subdisciplines are covered by a limited number of national journals only, for which no definitive ranking could be made.

A second report that bears a high relevance to our study is the "Quality Assessment of Research--Rechtsgeleerdheid," published in April 1996, by the Review Committee on Juridical Research, set up by the Association of Universities in the Netherlands (VSNU). The report presents an assessment of research activities in Law at universities in the Netherlands. In the publication output assessment, the Dutch VSNU Committee applied several criteria to identify the most valuable scholarly works in the mass of publications listed. Firstly, the Committee made a distinction between first editions and later editions of single- or multiauthored books. Interestingly, this aspect is not mentioned in the report of the Committee of the Flemish Law Faculties.

Secondly, in order to discriminate between substantial and small scholarly contributions, the VSNU Committee took into account the publications length as reflected in the number of pages. Publications with a length of more than five pages were regarded as "substantial" contributions. For each research programme to be assessed, the Committee regarded the number of single- or multiauthored books (first editions only), doctoral theses, and articles of which the number of pages exceeded five, as the most significant productivity measure. In addition, the total number of publications (of all types) was determined.

In its final report, the VSNU Committee expressed the need for clear guidelines and criteria for selecting and structuring the information on publication output. Such criteria should first of all specify the type of publications to be included in a performance or quality assessment. In addition, the Committee stressed that attempts should be made to distinguish between "genuine" scholarly contributions, on the one hand, and informative publications primarily aimed at providing social services, on the other. Genuine scholarly publications conform to criteria of methodological soundness, thoroughness, and significance. In the Committee's view, it is the first category of publications that distinguishes between a juridical scholar and a practitioner or a professional legal expert. Academic scholars should be primarily evaluated according to their contribution to scholarly progress, rather than to their practical activities.

The relationship between juridical research and practice is also addressed in the report by the Inter-University Committee of the Flemish Law Faculties. This committee stated that juridical research primarily serves the practice, a basic characteristic that creates difficulties in distinguishing between fundamental and applied juridical research.

4. RESULTS

The core of our analyses can be denoted as bibliometric, and related to publications. In this section, the main findings are summarized related to the Flemish juridical scholars' publication output. A detailed account is given in the research report by Luwel et al. (1999).

4.1 Units of Analysis

We agree with McGrath's (1996) statement that it is crucial in any bibliometric study to define carefully its units of analysis. This study deals with several units of analysis. The first is the individual publication. The main aspect is the classification of individual publications. In view of the main interest of our study--research performance assessment--an attempt was made to rank the various types of publications according to their importance or size of contribution to scholarly progress. A second unit of analysis is that of the journal. Journals were classified on the basis of characteristics of the publications included, and also on the basis of judgments by peers obtained from a questionnaire. The distinction between scholarly journals and journals of an applied nature or directed towards a broad audience is a crucial element. A third unit of analysis is the individual scholar. Here, the distinction between "scholar" and "practitioner" is important, based on an analysis of the type of publications made, the type of journals used, and also on quality judgments obtained from a questionnaire.

4.2 Publication Output: Classifications

In the first questionnaire, respondents listed their complete publication output during the 1992-1996 time period. The total number of publications listed amounted to 3,753. All publications were arranged into types. The classification of publications in Law applied in this study is largely based on the 1996 final report of the Inter-University Committee of the Flemish Law Faculties (see Section 3). The classification system is presented in Table 1.

In this system, a book is the most important publication, as it is often the reflection of continuous, intensive scholarly research, conducted for many years. In the view of the Committee, the same is true for doctoral theses. One of the key elements in the system is the distinction between substantial contributions (about 33 percent of all publications listed), small contributions (17 percent), publications for a wide audience (11 percent), and other publications (13 percent).

The classification system of publication types was not always applied correctly by the respondents. Many relevant examples of erroneous classifications were collected. At times, multiauthored books were classified as single-authored. In addition, respondents often listed both the unpublished and the published versions of their Ph.D. thesis. Further, it is questionable whether reports of advisory committees or committees preparing legislation can be qualified as books. The same is true for teaching course notes. In view of the great importance of books as publications in juridical research, the Inter-University Committee of Flemish Law Faculties--or any committee dealing with this issue--was suggested to specify more precisely the criteria to be applied in determining whether or not a publication should be classified as a book.

A main problem related to book publications (about 9 percent of the total number of publications listed) is that books can have different editions. In the data provided by the respondents, in 78 percent of the cases, no information was given on the edition number. Obviously, publishing the first edition of a book is a much greater achievement than publishing a slightly revised version of an existing book. It was suggested allowing only first editions or completely revised editions of existing books to be classified as genuine book publications. In our analysis of book publications, a Flemish online inter-university library catalogue was used. This has proven to be a most useful tool in verifying publication lists, particularly book publications or book chapters. As a rule, information on the edition number of a book is available in that catalogue.

Bibliometric characteristics of articles classified as substantial contributions were examined in more detail. Table 2 illustrates that 84 percent of substantial contributions had a page length greater than five. For the three other types this percentage is near 80 percent. Among the 16 percent of substantial contributions containing five pages or less, there were several with a page length of one or two. It is questionable whether such publications can be marked as substantial contributions. It is worthwhile considering whether or not it is appropriate to set a minimum page length for a publication in order to be classified as a substantial contribution. One could even take into account differences in the number of printed characters (or words) per page in the various sources.

Another important phenomenon observed in the publication lists was that the same publication might be published twice by the same author, in different sources. The two versions may be entirely identical or show only small differences. A bibliometric tool was developed to identify candidate identical publications, that is, publications published by the same author that are probably identical. The method compares the titles of pairs of publications, and determines the number of words two publication titles have in common. Publications with similar titles are then selected. However, prior to drawing definite conclusions, it is appropriate to collect printed copies and compare these manually. In the class of books, doctoral theses, and substantial contributions, almost 8 percent of the publications written in Dutch, and listed by the same authors, were found to have very similar titles. A detailed analysis of printed copies revealed that most of the pairs were actually (almost) identical publications.

4.3 Publication Output in Journals

The role of journals was found to be less prominent in communicating research results in juridical research than it is in many fields in the natural and life sciences. The percentage of journal articles among all publications listed by the respondents to the first questionnaire, and published during 1992-1996, amounted to 59 percent. In the category substantial contributions it was 60 percent. Nevertheless, this percentage was considered sufficiently high to justify a separate analysis of journals.

In our analysis of journals, two statements of the above mentioned Inter-University Committee of Flemish Law Faculties are of particular interest. The first deals with the relationship between juridical research and practice. The second statement will be discussed in Section 4.4. The Flemish Committee stated that juridical research primarily serves the practice, which makes it difficult, if not impossible, to distinguish between fundamental and applied juridical research.

The data collected in our study made it possible to distinguish between scholarly journals and journals of a more applied nature, or journals directed to a wide audience. The distinction is based on an analysis of the classification of publications into types given by the respondents themselves. By arranging the classified publications by journal, one obtains an indirect insight into the scholars' perceptions of the nature of the journals. A basic assumption underlying this approach is that scholarly journals should contain a certain minimum number of publications classified by the respondents as substantial contributions.

Table 3 presents typical results from this analysis. The second, third, and fourth columns give, for each journal, the total number of publications, the number of substantial contributions, and the percentage of substantial contributions, respectively. The table shows that the journals Fiskoloog: Nieuwsbrief over Fiscaliteit en Belastingen (Newsletter on Fiscal Matters and Taxes), Balans: Nieuwsbrief voor Accountancy en Financieel Management (Newsletter on Accountancy and Financial Management), and en Milieurecht Info (Information on Environmental Law) include very few or no articles denoted by the scholars themselves as substantial contributions. These journals are typical examples of applied journals or journals directed to a wide audience.

In addition, findings from the first questionnaire on input, output, and recognition, and from the second questionnaire on quality perceptions, were combined. The number of journal publications made by respondents in the first questionnaire was compared to the number of times the respective journal was nominated in the second questionnaire. This analysis included only nominations made by Flemish scholars. A more complete picture of the quality perceptions of journals, including the views of Belgian French-speaking and Dutch scholars, will be presented in Section 4.4.

The last four columns in Table 3 give the total number of Flemish nominations and the total number of times the journal was qualified by Flemish respondents as "outstanding" (qualification A), "good but not outstanding" (qualification B) or "not often containing high quality contributions" (qualification C). It should be noted that there is a substantial overlap between the respondents in the first and the second questionnaire. Our findings enabled us to examine the consistency of their responses.

Table 3 presents the ten journals in which the respondents to the first questionnaire have published the largest number of publications during the 1992-1996 time period. The journals were ranked by descending total number of publications. It can be seen that journals in which the Flemish scholars published relatively few substantial contributions, or no such contributions at all, were hardly nominated by the Flemish respondents in the second questionnaire.

It should be noted that general journals tend to be nominated more frequently than more specialized ones. For instance, Rechtskundig Weekblad (Juridical Weekly) comprises many--if not all--juridical subdisciplines. Therefore, it is perhaps not surprising that so many Flemish respondents mentioned it. The data presented in Table 3 can be used as a first step in a process of evaluating and qualifying journals used by Flemish juridical scholars. In principle, it could be used to give weights to juridical journals, enabling one to calculate weighted indicators of publication output, taking into account the orientation and quality of the journal. A first attempt to assign such weights is presented below.

4.4 Ranking of Journals and Weighting of Publication Output

A second statement of the Inter-University Committee of Flemish Law Faculties concerning journals relates to journal quality. The Committee did not succeed in developing a classification of scholarly journals in terms of their quality or reputation due to the fact that most Law journals show large variations in the quality of papers published and some subdisciplines are covered by a limited number of national journals only.

An analysis of the data from the second questionnaire enabled us to make at least a first step in the process of evaluating and qualifying journals used by Flemish juridical scholars. The overall response rate was about 33 percent.

The distribution of the quality ratings per journal was examined, as well as the journals' national or international visibility. Both aspects were quantified. The product of these two weights provides a third weight factor, which is assumed to indicate a journal's quality, as perceived by the respondents' nominations, and its visibility, as reflected in the geographical spread of the respondents.

The A, B, and C scores given by the 144 respondents were combined in a simple quality weight for a particular journal: Quality weight = [ (3. A) + (2 * B) + (1 * C)] - (A + B + C). The quality weight varies between 1 (only C nominations) and 3 (only A nominations). For example, a journal with 6 A nominations, 2 B nominations, and 1 C nomination was weighted as follows: [(3.6) + (2 * 2) +(1 * 1)] + (6 + 2 +1) = 23 + 9 = 2.56. Overall, this journal was scored slightly closer to A than to B. A detailed overview of the method, including a discussion on possible biases in view of the modest response rate, is presented in Nederhof, Luwel, & Moed (2001).

The second weight involved the international visibility of journals. Here, the nationalities of the respondents nominating a journal were decisive. For instance, journals nominated more than fourteen times by at least two Belgian and two Dutch nominators received a weight of 2.0. Thus, four nominators can be sufficient for a journal to be weighted--because of its international visibility, and, it is assumed, its correspondingly larger potential public. A lower weight is assigned to journals nominated by at least one Belgian and one Dutch respondent.

To illustrate the application of journal weights, these have been linked to the substantive output of Flemish scholars. Table 4 lists ten journals in which the Flemish scholars have published the largest number of substantial contributions. The left end of Table 4 contains the number of nominations (total and from the three samples), and the number of A, B, and C nominations obtained in the questionnaire on quality perceptions. These data were used to compute a quality weight (Quality), an (inter)national visibility weight (Int. Vis.), and an index combining these two weights (Weight).

Table 4 shows that the two journals in which Flemish scholars published most of their substantial contributions, Rechtskundig Weekblad (Juridical Weekly) and Tijdschrift voor Privaatrecht (Journal of Private Law), were also the journals that obtained the highest weight. For example, a substantial contribution in one of these journals is weighted approximately 3 times as high as a single substantial contribution in Tijdschrift voor Rechtspersoon en Vennootschap (Journal of Corporate Body and Partnership).

These examples show that the application of journal weights--based on quality perceptions of international samples of scholars--to the output of Flemish scholars might yield quite differentiating results. A few publications in journals with high weights can weight more heavily than many publications in journals with lower quality and visibility ratings. Also, similar output levels can be weighted quite differently, depending upon the weights of the journals. In our view, our rankings provide a sound basis for a thorough discussion among Flemish juridical scholars on the quality of journals in Law.

4.5 International Orientation

The main publication language of Flemish publications in Law is Dutch. 81 percent of all publications were written in Dutch, and 10 percent in English. Interestingly, publications published in English were rather unevenly distributed among the various subdisciplines. In Information Technology Law and Informatics, Public International Law, Economics of Law, Private International Law, and European Community Law the share of publications in English was greater than 25 percent. Subdisciplines with more than 200 publications during 1992-1996, and with less than 5 percent of publications in English, were: Tax Law, Judicial Law, Contract Law, and Administrative Law.

A high percentage of publications in Dutch is often assumed to reflect the national (or regional) character and relevance of juridical research. It is indeed plausible to assume that the international orientation of a subdiscipline is related to the object of research in that subdiscipline. From this point of view, it is perhaps not surprising that subdisciplines such as International Law, International Private Law, and European Community Law show a relatively high percentage of publications written in English. In addition, Information Technology Law, Legal Informatics, and Law and Economics focus on issues with a growing international interest within the framework of globalization.

In the final report, we maintained that genuine scholarly research, regardless of the subdiscipline and the object of research, leads to results the relevance and implications of which go beyond a purely national viewpoint or interest. This may be less so for contributions of a more applied or practical nature. Therefore, outcomes of genuine scholarly research, even those primarily related to national aspects, deserve to be communicated--in an appropriate form--to scholars in other countries as well. This does not imply that all publications should be directed towards an international scholarly public, but rather that at least some publications should go beyond a purely national or local viewpoint and should be exposed to criticisms from a wide international scholarly audience.

If one is willing to agree with the line of reasoning outlined above, it follows that the international orientation or, more specifically, the extent to which research findings are communicated across national or cultural boundaries, is a relevant criterion of scholarly performance in all subdisciplines.

We concluded that the percentage of publications in English can be used as an indicator of international orientation, but that two comments should be made here. First, possibly other indicators are equally valid or even more valid for measuring this aspect of research performance. Perhaps the percentage of publications in non-Flemish media is a more appropriate indicator of international orientation. A further discussion on this topic with Law scholars could provide more insight. Secondly, we did not wish to imply that publications written in English are generally of better quality than publications in other languages merely because English was used as the publication language, nor did we maintain that all Flemish publications of good quality were published in English in the past, or should be published in English in the future.

4. 6 Indicators of Research Performance

A detailed comparison was made of the results from the second questionnaire on nominations of Flemish scholars with several bibliometric indicators based on publications, calculated for those Flemish scholars who replied to the first questionnaire. For instance, the number of publications made by Flemish scholars receiving three or more nominations was compared with the number of publications by scholars nominated once or twice, or with that of scholars not nominated at all. Results are presented in Table 5.

Table 5 illustrates that scholars receiving three or more nominations have published a significantly higher number of books, Ph.D. theses, and substantial contributions than scholars who were not nominated at all in the questionnaire on quality perceptions. Analyzing the total number of pages produced, a significant difference was observed between the class of scholars without any nominations, on the one hand, and the classes of scholars with one to two or three to ten, on the other. With respect to the total number of publications or the total number of pages, no significant differences were found among the three classes of nominations.

If one considers the number of nominations received as a measure of scholarly quality, as perceived by colleagues or peers, our statistical analysis suggests that the number of books, Ph.D. theses, and substantial contributions is a more appropriate indicator of research performance than the total number of publications. This outcome provides an empirical confirmation of the indicative rank order of types of publications given by the Inter-University Committee of Flemish Law Faculties. It also gives grounds for the definition of books, Ph.D. theses, and substantial contributions as juridical core publications.

These findings also have implications for statements made by the Review Committee on Juridical Research, set up by VSNU, in their 1996 report "Quality Assessment of Research--Rechtsgeleerdheid" (see Section 3). This committee stressed the need to distinguish between "genuine" scholarly contributions, on the one hand, and informative publications primarily aimed at providing social services, on the other. Genuine scholarly publications conform to criteria of methodological soundness, thoroughness, and significance. In the view of the Committee, it is the first category of publications that distinguishes between juridical scholars--who should primarily be evaluated according to their contribution to scholarly progress--and practitioners or professional legal experts.

From this perspective, it was suggested to use the number of core juridical publications as defined above--that is, the number of books, Ph.D. theses, and substantial contributions--as an indicator for discriminating between scholars and practitioners. This can be achieved either by counting the number of core publications per scholar, or by following a more qualitative approach, aimed at visualizing publication profiles of individual scholars in terms of types of publications.

5. DISCUSSION AND CONCLUSIONS

5.1 Practical Conclusions

If the publication lists obtained in our study constitute a sufficiently representative sample, we conclude that in the field of Law the publication lists should be rather carefully screened, analyzed, and verified prior to any counting or calculation of performance indicators based on publication counts. It should be noted that the members of the expert committee were very surprised by the observed large percentage of (semi-)duplicates, and they considered this unacceptable.

Our findings enabled us to propose the following procedure for calculating bibliometric indicators, at least at the level of aggregates of scholars, such as departments or faculties:

1. Collect raw data per scholar on publication output in electronic form.

2. De-duplicate entries using a semi-automatic approach.

3. Identify book publications; look up all books in university library catalogues, checking authors and editions, and selecting only first or totally revised editions.

4. Identify Ph.D. theses, but avoid double-counting as a book.

5. Identify all publications of which the number of pages exceeds five.

6. Add up the number of items found under points 3, 4, and 5, determining the "raw" number of "core" publications.

7. Calculate a weighted number of core-publications, by weighting journal articles with the journal weights described in Section 4.3, and book publications by a factor obtained by dividing the number of pages by sixteen, the median page length of a substantial contribution.

In view of the findings obtained in our study, this approach was qualified as a good one, both in terms of validity and practical feasibility, which could be applied if no better alternatives were available. In many places in the final report, the Inter-University Committee of Flemish Law Faculties was invited to comment on our findings.

a. Official comments by the Inter-University Committee of Flemish Law Faculties. In May 2000, about ten months after publication of the final report, the Committee gave its comments, in an official letter to the VLIR. It should be noted that the members of this committee are the deans of the Law faculties involved. Since a deanship lasts normally two or four years, the members who issued the comments were not the same as those who prepared the Committee's report on the assessment of performance in juridical research in 1996.

Although the Committee was unhappy with the fact that the research report was written in English, it expressed its approval of the work, and stated that many results from it are useful for the development of performance evaluation criteria. The Committee's letter addressed three main points.

The first related to the statements in our report on international orientation (see Section 4.4). Although the Committee agrees that the international orientation of Flemish juridical research needs to be stimulated, it issues a warning that this aspect should not be assessed merely on the basis of publication language. It should be noted that this warning was already included in the report's section on comments of members of the expert group participating in our study.

The Committee stated that contributions in English are often not of a fundamental nature, but are rather popularizing works--for example, aimed at providing an introduction to the Belgian or Flemish Law system for a larger, foreign audience. Rewarding publication in English would stimulate "legal journalism." Publications in English should only be given a higher weight when they are published in journals of which the quality guarantees that they go beyond legal journalism.

A second point concerned rankings of journals. Although the Committee stated in its 1996 report that it is impossible to rank journals, it is now willing to reconsider this. In its view, rankings should be primarily determined by expert opinion, and based on criteria such as international orientation, severity of review procedure, a journal's circulation, and its citation impact.

The third major point related to the operationalization of the concept of substantial contribution. The Committee argued that the page length gives a certain indication, but that other criteria should be developed as well: Descriptive-systematic, analytical, comparative, evaluative, innovative, critical, or interdisciplinary.

Finally, the Committee stated that it will continue to work on the development of criteria for measuring research performance in Law, and that it would be regrettable if findings from our report would be applied "in a premature way" in university research policy.

b. General methodological discussion. A fundamental assumption is that the concepts of research performance and research quality do have a meaning in all fields of scholarship, particularly also in the social sciences and humanities. As a result, differences in research quality among individual scholars or groups of scholars do exist. To the best of our knowledge, none of the scholars involved in our study has questioned this assumption. This is, in itself, a significant outcome of our study.

The principal aim of the development and application of bibliometric indicators is to stimulate a debate among scholars in the field under investigation on the nature of scholarly quality, its principal dimensions, and operationalizations. This aim provides a criterion of productivity for the process. A development process in which such discussions do not take place is to be considered as unproductive and unsuccessful. This would particularly be the case when developers calculate quantitative indices, which are used by evaluators or policy makers for evaluating research and making policy decisions above the scholars' heads.

A productive process enables scholars to express their views on scholarly quality more explicitly and clearly than in the beginning. In other words, a productive process establishes conditions for a more profound reflection upon what is most valuable and less valuable in scholarly research. Applying this criterion, we are inclined to conclude that our study, and particularly the methodology applied, has been successful.

The relationship between the views and perceptions of scholars and the development of bibliometric indicators is rather complex. On the one hand, scholars in the field under study should participate in all stages of the developmental process. In fact, their views and impressions are indispensable for developing valid and useful performance indicators. On the other hand, validity and utility cannot be assessed merely on the basis of scholars' views. An indicator is not valid merely because scholars say it is. The developer of bibliometric indicators should have independent tools to examine and test scholars' perceptions. In addition, a view of perception is not a static entity, but may change during the process, particularly in view of outcomes of bibliometric analyses. Utility should be evaluated from the point of view of specific policy issues and objectives, which are expressed not only by scholars but also by policy makers.

The essential elements of our methodology can be summarized in the following points. First, one should collect documents containing statements of scholars in the field under study on how assessment of research performance should be conducted, and, of course, on how it should not be conducted. Reports on research assessments conducted in the past constitute the most fruitful basis for such an inventory. The analyst should identify the main aspects of research quality involved, issues that were raised, problems that remained unsolved, operationalizations that were applied or rejected.

Secondly, scholars from the field should be involved in all stages of the study. They should be stimulated to propose or develop--even preliminary--classification systems, and to structure their own research output accordingly. Such an effort, though time consuming, is essential for making progress towards standardization of research output.

Next, bibliometrics should be used as an instrument to create a mirror. While not a direct reflection, this study is a thorough analysis of how scholars in the humanities and social sciences structure their activities and their research output. This structure can be examined empirically from the point of view of its consistency and the degree of consensus among scholars. Relevant issues can be raised that are worth considering in more detail in followup studies and conclusions from our empirical materials can be derived that may illuminate such issues. It is essential to recognize the need to develop adequate classification systems for scholarly activities and research output prior to any comparative measurement of scholarly performance.

Finally, the analyst presents to the scholarly community what he or she believes is the "best" approach for structuring and measuring research output, in the light of the outcomes of the study. Given the constraints imposed on any study in terms of time and manpower, he/she should acknowledge that not all issues raised during the study can be solved during that study. It is essential that he/she exercises a sufficient degree of openness in his/her presentation, both towards the scholars and to policy makers.

It is up to the scholarly community and its committees to discuss and evaluate the outcomes of the study. The process summarized above may then start again. Thus, an interactive, open process is created for developing performance indicators in the social sciences and humanities.

5.2 Relevance for Bibliometric/Scientometric Theory and for Librarians

Our paper showed that the development of bibliometric research performance indicators in general, and in the humanities in particular, is a systematic, scientific, and even scholarly activity. It can be denoted as scientific as it embraces empirical-analytical approaches to the analysis of publication practices and quality perceptions of scholars in the field of study. The scholarly dimension is the opportunity for scholars to reflect upon their publication strategies and to sharpen and make more explicit their qualitative views.

The classification system of publications in Law explored in this study is of interest to librarians, even though it has a preliminary status and needs further clarification and operationalization. It should be noted that publication cultures in the humanities show specific national characteristics, and differ considerably among subfields. Therefore, the extent to which it can be validly extended to research outputs in other subfields of the humanities, or to publication activities in the field of Law in other countries is open to further research.

In our view, the principal relevance of our study to librarians should primarily be found in the following. Librarians are not infrequently requested to provide assistance in collecting data related to research performance assessments, or even to participate as contributor in such assessments. Our methodology aims at providing a general framework for such activities, and can be used by librarians as a line of action whenever they become involved. It should also be noted that librarians are, in principle, well equipped to conduct the empirical-analytical tasks outlined in this paper, including development and testing of classification systems, analyzing their structural properties, and collecting data through questionnaires.

In view of this, librarians could make important contributions to the process of structuring research activities, particularly research publications in the humanities, by developing classification systems and by assigning weights to the various entities reflecting perceived quality or importance in the subfield under investigation. As argued above, such a process can only be fruitful if the scholars themselves are willing to participate.

From a general policy point of view, it is in the interest of the humanities that their scholars do actually participate in such processes. It ensures the development of adequate methods for indicating research performance, taking into account the proper characteristics of these fields of scholarship. This would generally increase research performance in the humanities.

Table 1. Classification Scheme of Scholarly Publications in Law,
Applied in the Study.

No.   Description

 1    Book published as single author
 2    Published doctoral (Ph.D.) thesis
 3    Book published as coauthor
 4    Unpublished doctoral (Ph.D.) thesis
 5    Substantial scholarly contribution, published in accepted
        scholarly journals, anniversary volumes, seminar reports, and
        collective works, such as a leading article; review on
        jurisdiction; thorough annotation
 6    Edited book or collective work
 7    Published integral contribution to international conferences
 8    Published abstract of lecture at international conferences
 9    Published integral contribution to national conferences
10    Published abstract of lecture at national conferences
11    Scholarly contribution of a limited size, published in accepted
        scholarly journals, anniversary volumes, seminar reports and
        collective works, such as a short annotation; a thorough book
        review; an intervention as panel member or disputant on a
        conference
12    Teaching course notes
13    Scholarly edition of codes of Law, jurisdiction volumes,
        bibliographies
14    Research report circulated in the scholarly community
15    Internal research report or report on commissioned work
16    Published inaugural or valedictory lecture
17    Other publication, such as an introduction, editorial
        contribution, letter to the editor, commemorative article,
        correction, descriptive or introductory book review
18    Juridical publications for a wide audience
Table 2. Statistics on the Page Length of Four Publication Types.

                                % Pub. with Num. Pages
Publication type                 < = 5             > 5

Substantial Contributions        16%               84%
Small Contributions              77%               23%
Other Publications               82%               18%
Publications for Wide Audience   83%               17%
Table 3. Journals Used and Nominated Most Frequently by Flemish
Scholars.

                                   Total   Subst.   % Subst.
Journal                            Pub.    Contr.    Contr.

Rechtskundig Weekblad              456     135      29.6
Fiskoloog: Nieuwsbrief voor
  Fiscaliteit & Belast.            275       0       0.0
Tijdschrift voor Privaatrecht      122      39      32.0
Algemeen Juridisch Tijdschrift      91      18      19.8
Revue Historique de Droit
  Francais et Etranger              55       0       0.0
Recente Arresten van het
  Hof van Cassatie                  51      34      66.7
Balans--Nieuwsbrief
  Accountancy en Fin. Manag.        50       0       0.0
Tijdschrifl voor Rechtsdocum.
  & -Informatie                     50       0       0.0
Milieurecht Info.                   47       5      10.6
Tijdschrift voor Rechtspersoon e
  n Vennootschap                    39      29      74.4

                                   Flemish Nominations

Journal                            Total   A    B    C

Rechtskundig Weekblad              42      26   12   4
Fiskoloog: Nieuwsbrief voor
  Fiscaliteit & Belast.             0       0    0   0
Tijdschrift voor Privaatrecht      16      12    2   2
Algemeen Juridisch Tijdschrift     10       0    7   3
Revue Historique de Droit
  Francais et Etranger              0       0    0   0
Recente Arresten van het
  Hof van Cassatie                  8       4    2   2
Balans--Nieuwsbrief
  Accountancy en Fin. Manag.        0       0    0   0
Tijdschrifl voor Rechtsdocum.
  & -Informatie                     1       0    0   1
Milieurecht Info.                   0       0    0   0
Tijdschrift voor Rechtspersoon e
  n Vennootschap                     6       1    3   2

Total Pub.: Total number of publications. Subst. Contr.: Number of
substantial contributions. % Subst. Contr.: Percentage of substantial
contributions (relative to total number of publications).

Flemish nominations: A: "outstanding"; B: "good but not outstanding";
C: "not often containing high quality contributions."
Table 4. Weights of Journals in Which Flemish Scholars Publish Most
Frequently.

Total   Tot. Fl   Tot. BFs   Tot. NL   Tot. A   Tot. B   Tot. C

 57       42         12         3        35       17        5
 21       16          2         3        14        4        3
 10        8          2         0         4        4        2
 13        7          2         4         2        9        2
  6        6          0         0         1        3        2
 26       19          7         0        14        6        6
  8        5          1         2         5        2        1
 11       10          1         0         0        8        3
  7        6          0         1         1        5        1
  2        2          0         0         2        0        0

Total   Journal

 57     Rechtskundig Weekblad
 21     Tijdschrift Voor Privaatrecht
 10     Recente Arresten Van Het Hof Van Cassatie
 13     Panopticon
  6     Tijdschrift Voor Rechtspersoon En Vennootschap
 26     T. Voor Bestuurswetenschappen En Publikrecht
  8     Tijdschrift Voor Milieurecht
 11     Algemeen Juridisch Tijdschrift
  7     T. Voor Onderwijsrecht En Onderwijsbeleid
  2     Notarieel En Fiscaal Maanblad

Total   N Pubs.   Quality   Int. Vis.   Weight

 57     135       2,6        2,00       5,2
 21      39       2,6        2,00       5,3
 10      34       2,3        1,00       2,3
 13      30       2,0        1,75       3,5
  6      29       1,8        1,00       1,8
 26      22       2,4        1,00       2,4
  8      21       2,7        1,75       4,7
 11      18       1,9        1,00       1,9
  7      16       2,0        1,25       2,5
  2      14       2,5        1,00       2,5

Legend: Journals weights applied to journals with at least three
substantial contributions.

N Pubs.: Number of substantial contributions published by the
respondents in the first questionnaire during the time period
1992-1996. Total: Total number of nominations. Tot. Fl.: Total number
of nominations by Flemish respondents. Tot. BFs: Total number of
nominations by Belgian French speaking respondents. Tot. NL: Total
number of nominations by respondents from the Netherlands. Tot. A:
Total number of A-nominations. Tot. B: Total number of B-nominations.
Tot. C: Total number of C-nominations. Quality: Perceived quality
measure, based on the distribution of nominations among A-, B-, and
C-nominations (see main text). Int. Vis.: International visibility (see
main text). Weight: Product of Quality and Int. Vis. measures.

Journals titles printed in bold are nominated at least twice by
Flemish/Belgian French speaking and by Dutch scholars.
Table 5. Statistical Relationship Between Number of Publications and
Number of Nominations

    Nr.          Nr.     Total   Core    Total   Core
Nominations   Scholars    Pub.   Pub.    Pages   Pages

   3-10          20       9.2    4.4      504    270
   1-2           21       8.7    3.4      400    261
   0             20       5.6    2.3 *    321    143 **

* : Significantly different from mean score in class with 3-10
nominations, according to Duncan's multiple range test with
alpha = 0.05. ** : Significantly different from mean score in class
with 3-10 and class with 1-2 nominations.

Total Pub.: The total number of publications per year during 1992-1996.
Core Pub.: The number of single- and multiauthored books, Ph.D. theses,
and substantial contributions per year.

Total Pages, Core Pages: The average number of pages per year in all
publications and in core publications, respectively.

ACKNOWLEDGEMENTS

The project presented in this paper has been partially funded by a grant approved by the Flemish Minister-President, who is also in charge of science and technology policy. The authors are grateful to V. De Samblanx, IL Verbruggen, L. J. van der Wurff, W. Geerts, J. Van Der Perre, F. Colson, and the members of the Flemish Law Expert Committee for their important contributions to the study.

REFERENCES

Cole, S. (1983). The hierarchy of the sciences? American Journal of Sociology, 89, 111-139.

Cole, S.; Cole, J. R.; & Dietrich, L. (1978). Measuring the cognitive state of scientific disciplines. In Y. Elkana, J. Lederberg, & IC Merton (Eds.), Toward a metric of science: The advent of scientific indicators (pp. 209-251). New York: Wiley.

Ellman, I. A. (1983). Comparison of law faculty production in leading law reviews. Journal of Legal Education, 33(4), 681.

Finkenstaedt, T. (1990). Measuring research performance in the humanities. Scientometrics, 19(5-6), 409-417.

Garfield, E. (1979). Most-cited authors in the arts and humanities, 1977-1978. Current Contents, 32. In E. Garfield, Essays of an information scientist, volume 4 (pp. 238-244). Philadelphia: Institute for Scientific Information.

Garfield, E. (1986). The 250 most-cited authors in the Arts and Humanities Citation Index, 1976-1983. Current Contents, 48, 3-10.

Hemlin, S. (1996). Social studies of the humanities: A case study of research conditions and performance in ancient history and classical archaeology, and English. Research Evaluation, 6(1), 53-61.

Hemlin, S., & Gustafsson, M. (1996). Research production in the arts and humanities. A questionnaire study of factors influencing research performance. Scientometrics, 37(3), 417-432.

Inter-University Committee of the Flemish Faculties of Law (1996). The assessment of performance in juridical research. Final Report (in Dutch).

Justiss, L. K. (1993). A bibliometric study of Texas law reviews. Law Library Journal, 85, 407-408.

Kyvik, S. (1989). Productivity differences, fields of learning, and Lotka's Law. Scientometrics, 15(3-4), 205-214.

Luwel, M.; Moed, H. F.; Nederhof, A. J.; De Samblanx V.; Verbrugghen, K.; & van der Wurff, L. J. (1999). Towards indicators of research performance in the social sciences and humanities: An exploratory study in the fields of law and linguistics at Flemish Universities. Brussels: Vlaamse Universitaire Raad, Depot nr. d/1999/2939/9.

McGrath, W. E (1996). The unit of analysis (objects of study) in bibliometrics and scientometrics. Scientometrics, 35(2), 257-264.

Nederhof, A. J.; Luwel M.; & Moed, H. F. (2001) Assessing the quality of scholarly journals in linguistics: An alternative to citation-based journal impact factors. Scientometrics, 51(1), 241-265.

Nederhof, A. J., & Noyons, E. C. M. (1992). International comparison of departments' research performance in the humanities. Journal of the American Society for Information Science, 43(3), 249-256.

Nederhof, A. J., & Zwaan, R. A. (1991). Quality judgements of journals as indicators of research performance in the humanities and the social and behavioral sciences. Journal of the American Society for Information Science, 42(5), 332-340.

Nederhof, A. J.; Zwaan, R. A.; de Bruin, R. E; & Dekker, P. J. (1989). Assessing the usefulness of bibliometric indicators for the humanities and tile social behavioral sciences. Scientometrics, 15(5-6), 423-435.

Review Committee on Juridical Research, Association of Universities in the Netherlands (1996). Quality assessment of research--Rechtsgeleerdheid (in Dutch). Utrecht: Vereniging van Samenwerkende Nederlandse Universiteiten (VSNU).

Shapiro, F. R. (1992). Origins of bibliometrics, citation indexing, and citation analysis: The neglected legal literature. Journal of the American Society for Information Science, 43(5), 337-339.

Sorensen, J. R. (1994). Scholarly productivity in criminal justice: Institutional affiliation of authors in the top ten criminal justice journals. Journal of Criminal Justice, 22(6), 535-547.

Swygert, M. I., & Gozansky, N. E. (1985). Senior law faculty publication study: Comparisons of law school productivity. Journal of Legal Education, 35(3), 373-394.

Van Den Berghe, H.; de Bruin, R. E.; Houben, J. A.; Kint, A.; Luwel, M.; Spruyt, E.; & Moed, H. F. (1998). Bibliometric indicators of research performance in Flanders. Journal of the American Society for Information Science, 49(1), 59-67.

van Raan, A. F. J. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36(3), 397-420.

Wood, J. B. (1988). The growth of scholarship: An online bibliometric comparison of dissertations in the sciences and the humanities. Scientometrics, 13(1-2), 53-62.

Henk F. Moed, Centre for Science and Technology Studies (CWTS), Leiden University, P.O. Box 9555, 2300 KB Leiden, the Netherlands

Mark Luwel, Science and Innovation Administration, Ministry of the Flemish Community, Boudewijnlaan 30, Brussels, Belgium

A. J. Nederhof, Centre for Science and Technology Studies (CWTS), Leiden University, P.O. Box 9555, 2300 RB Leiden, the Netherlands

MARC LUWEL is currently Senior Policy Advisor of the Flemish Minister of Education in Brussels, Belgium. He is author of a number of publications focusing on the assessment of Flemish universities and other research institutions.

HENK F. MOED is a senior staff member at the Centre for Science and Technology Studies, Leiden University, the Netherlands. He is the author of numerous articles dealing with quantitative assessment of research performance, development, and application of science and technology indicators and creation of scholarly literature databases for bibliometric analysis.

A.J. NEDERHOF is a senior staff member at the Centre for Science and Technology Studies, Leiden University, the Netherlands. He is the author of numerous articles dealing with quantitative assessment of research performance, development, and application of science and technology indicators and creation of scholarly literature databases for bibliometric analysis.