Printer Friendly

The publication incidence of replications and critical commentary in economics.

"That which isn't worth replicating isn't worth knowing." (Mittlestaedt and Zorn, 1984, p. 14.)

In a recently published article, Dewald, Thursby, and Anderson (1986) drew attention to the critical role of replication in the development of the economics discipline.(1) In doing so, they have made an important contribution to the literature. Replication of previous findings safeguards the integrity of a discipline's substantive empirical base. Indeed, the principle of replicability has been called the touchstone of the scientific method (Kane, 1984), the Supreme Court of the scientific system (Collins, 1985), and the most important criterion of genuine scientific knowledge (Rosenthal and Rosnow, 1984).

Dewald, Thursby, and Anderson (1986, p. 590) stated that:

"Overall, however, professional journals in economics have not adopted editorial policies to facilitate replication, and there have been few attempted replications." (Emphasis ours.)

If this statement is true, then the veracity of the empirical literature in economics simply cannot be taken for granted. Of course, it is difficult to know how many replications are attempted. We can, however, estimate how many are published in leading journals.

The present paper complements Dewald, Thursby, and Anderson's work by seeking to answer the following five questions: (1) To what extent are replications and replications with extensions (modifications of the original research design) published in the economics literature? (2) How does this publication incidence compare with other social and business sciences? (3) Has the publication of replications increased or decreased over time? (4) To what degree do replication studies support, partially support, or conflict with the results of previous work? (5) How much discussion concerning the authenticity of published studies occurs in the field of economics?

1. The Publication Incidence of

Replications in Economics

"Replication" and "replication with extension" are defined following Brown and Coney (1976) and Reid, Soley, and Wimmer (1981). Thus, a replication is a substantial duplication of a previously published empirical research project that is chiefly concerned with increasing the internal validity of the research design. An example would be repeating the experiment with another sample drawn from the same population. A replication with extension is a duplication of a previously published empirical research project that is primarily concerned with increasing the external validity (generalizability) of the research design. The intent is not to alter the conceptual relationships investigated in the earlier study but to test them differently by modifying the original research design. Examples would be changing either the manipulated (independent) or measured (dependent) variables, but not both, including an additional variable or variables for analysis, or repeating the study with a sample drawn from a different population. These definitions correspond respectively to what Kane (1984) calls the econometric audit and improvisational replication. They are also generally consistent with Mittelstaedt and Zorn's (1984) fourfold classification of replication types.

We examined the incidence of replications and replications with extensions in the economics literature by content-analyzing a 25% random sample of issues of the American Economic Review (AER), the Review of Economics and Statistics (RES), and the Journal of Political Economy (JPE), published during 1965-1989. More specifically, the sampling frame consisted of a listing of all 100 issues of AER (proceedings were omitted) and RES for the period 1965-1989, as well as all 150 issues of JPE for the same time period. We used simple random sampling procedures to select 25 issues each of AER and RES, and 38 issues of JPE.(2)

AER, JPE, and RES were chosen for inclusion in the present study because they are prestigious, and therefore likely to exert the greatest influence on scholarly thought and practice. For example, studies ranking the quality or impact of economics journals, using unweighted and weighted citation data from the Social Sciences Citation Index (SSCI), almost invariably result in either AER or JPE occupying the first two places, with RES usually in the top ten (see Laband and Sophocleus, 1985; Liebowitz and Palmer, 1984). Being prestigious, they typically enjoy the widest readership; AER, for example, had a 1989 subscription rate of over 27,000 (AER, 1990). Replications and replications with extensions published in these journals are more likely to attract attention than those published in the lower tiers of the journal hierarchy.

The content analysis involved the inspection of 1,698 research reports (articles and notes), 942 of which were empirical. The latter were read and classified independently by each of the authors, and resulted in a high level of agreement as measured by Cohen's (1960) kappa (k = .85, [zeta] = 11.9, p< .001).

We encountered no replications in our sample (Table 1). Similar conclusions have been drawn in other social and business sciences.(3) Evidence regarding the publication frequency of replications with extensions is more encouraging. An average of some 5.4% (92) of all reports published in the three journal included in this study belonged to this category (Table 1).(4) This result compares favorably with Brown and Coney's (1976) finding that only 2% of marketing publications are replications with extensions, and it is almost identical to the 5.2% of such studies published in the advertising literature (Reid, Soley, and Wimmer, 1981).


A breakdown of published replications with extensions by journal shows that AER (8.0%) led the way, followed by JPE (4.3%) and RES (3.0%).(5) When investigating the occurrence of replications based on empirical studies only, AER's (21.1%) dominance over JPE (9.1%) and RES (3.5%) increases appreciably. On the average, almost 10% of the empirical literature in these three journals deals with replications (Table 1).

Informal consensus exists among some researches that because replications compete with original studies for a limited amount of journal-space, editors are reluctant to publish them. We therefore determined what percentage of journal pages devoted to research reports were allocated to replications. Overall, this figure was only 4% (Table 1). Again, AER (7.2%) ranked first, followed by JPE (2.4%) and RES (1.8%).

II. Trends in the Publication Incidence

of Replications

Troubled by the prospect that the economics literature was contaminated with an inordinate number of Type I errors, a by-product of journal editorial policies favoring the publication of the statistically significant result, Feige (1975) argued that more attention should be directed toward improving the relevance and rigor of research designs. The editors of JPE responded in 1975 by agreeing to publish replications and other reexaminations of previously published works in a new section titled "Confirmations and Contradictions."(6)

Did this change in editorial policy lead to the publication of more replications in JPE after 1975? Furthermore, taking the lead from JPE, did other economics journals also publish a larger proportion of replications after 1975? Table 2 portrays the frequency of published replications by journal for the time periods 1965-1975 and 1976-1989. From this table it can be seen that the percentage of all JPE papers devoted to replications increased from 2.7% to 5.7%, while the percentages based only on empirical papers increased from 6.4% to 11.2%. Neither of these increases, however, were statistically significant at the .05 level ([zeta] = 1.58 and 1.20, respectively). The proportion of all papers in RES that were replications increased marginally (2.1% to 3.5%), as did the proportion of empirical paper (2.7% to 3.9%). Whether considered as a percentage of all papers (9.5% to 7.0%), or only those empirical ones (24.8% to 18.5%), AER's rates declined. None of these differences are significant.


However, it is unreasonable to expect an increase in AER's publication incidence of replications. This journal already publishes more replications than JPE and RES combined. Moreover, AER is competitive with the Quarterly Journal of Business and Economics (QJBE) when it comes to publishing replications. What is illuminating about this observation is that in 1984 the QJBE editors announced bold changes in publication policies. Chief among these was the explicit commitment to publish replications. We content-analyzed a fifty-percent random sample of QJBE issues between 1984-1989. Of the 52 empirical papers included in our sample, 17.3% were replications. During that same time period, 1984-1989, some 13.4% of all empirical articles appearing in AER qualified as replications.

Between 1965-1975 and 1976-1989, the percentage of AER journal-space allotted to replications fell from 8.7% to 6.1% (Table 2). JPE (2.1% to 2.6%) and RES (1.6% to 1.9%) showed slight increases. On average across all three journals, replications consumed less journal-space during 1976-1989 (3.7%) than they did during 1965-1975 (4.5%).

III. Do Replicated Studies Replicate?

Given that highly replicable results promote confidence in a discipline's cumulative empirical foundation, to what extent do replications published in the economics literature provide confirmatory findings? Table 3 presents some potentially disquieting evidence. Of the 92 replications appearing in AER, JPE, and RES between 1965-1989, only 18 (19.6%) fully supported the original work. In contrast, 14 studies (15.2%) conferred partial support, while 60 (65.2%) contradicted earlier results. These results are based upon the conclusions drawn by the authors conducting the replications. The high frequency of conflicting outcomes underscores Mayer's (1980) contention that empirical findings in economics may be nonreplicable.
Outcomes of Replications with Extensions in Economics: 1965-1989
 Results of Replications with Extensions
Journals Conflict Partial Support Support Total
AER 38 6 12 56
JPE 12 4 4 20
RES 10 4 2 16
Totals(a) 60 14 18 92
 (65.2) (15.2) (19.6)
(a) Values in parentheses are percentages.

Of course, the large proportion of conflicting findings does not automatically imply that the earlier articles are flawed. Discrepancies in results could arise because of the modifications in the original research design introduced by the replication study. Again, it is possible that journal editors reveal a preference for publishing replications that yield contradictory outcomes.

Notwithstanding, it is sobering to consider Dewald, Thursby, and Anderson's (1986) conclusions following their attempted econometric audits of several raw data sets association with the Journal of Money, Credit and Banking Data Storage and Evaluation Project. They (1986, pp's. 587-88) cautioned that ". . . inadvertent errors in published empirical articles are a commonplace rather than a rare occurrence."

IV. The Role of the Research Comment

and Rejoinder

The routine practice of replication offers the most powerful method for assessing the accuracy of any discipline's empirical output. However, the research comment and rejoinder also provided a forum for publicly debating the authenticity of research findings.

Table 4 presents information on the number of comments and rejoinders published in AER, JPE, and RES for the time period 1965-1989. AER (32.5%) published more of such works than did JPE (13.2%) and RES (10.7%). Interestingly, an analysis of the publication frequency of comments and rejoinders over time points to a downward trend. Combining the results of all three journals, one observes an approximately one-third decrease in their occurrence between 1965-1975 (25.2%) and 1976-1989 (17.0%). All three journals registered declines over the two time periods, some quite dramatic (e.g., RES from 19.6% to 5.8%; JPE from 18.1% to 8.9%) and others less so (e.g., AER from 34.5% to 31.2%).


The average amount of journal-space accorded to research commentary also diminished substantially between 1965-1975 (10.4%) and 1976-1989 (5.8%). Table 4 shows how the three journals compared in this regard.

V. Conclusions

Replication serves to prevent specious empirical results from being uncritically incorporated into the literature. Prior to the present study, estimates concerning the number of replications published in economics have been impressionistic. Our analysis revealed that only 5% of all (10% of empirical) research reports appearing in AER, JPE, and RES between 1965 and 1989 were replications. Moreover, these works accounted for a mere 4% of journal-space.

An argument could be made that our sample estimates of the publication incidence of replications in economics are biased downwards because they are based on some of the most respected journals in the discipline. A greater frequency of replications might be observed in the middle and lower levels of the journal hierarchy. Perhaps, but two points are worth offering. First, AER, arguably the most prestigious of economics journals, actually publishes substantial proportions of replications, whether these are calculated over the total number of research reports (8.0%) or only those empirical one (21.1%). Second, this is an interesting proposition, and fortunately one that is amenable to empirical testing.

Dewald, Thursby, and Anderson (1986) raised concerns about the amount of replication that occurs in economics. Our paper has attempted to address some of these concerns. Replications must be encouraged, and the practice afforded a respectability commensurate with its pivotal role in the development of economics as a science.


(1.) See also the subsequent exchange between Merrick, Jr. (1988) and Dewald, Thursby, and Anderson (1988). (2.) The specific journal issues included in our sample are as follows. American Economic Review: Sept. 1965; March 1966; June 1967; June 1968; June, Dec. 1969; June 1970; Sept. 1972; March, June 1973; Dec. 1975; Dec. 1976; March 1977; June 1979; Sept. 1979; Sept. 1980; June 1983; March, Dec. 1984; Sept. 1985; June, Sept. 1986; March 1987; June 1988; March 1989. Journal of Political Economy: Oct., Dec. 1965; Feb. 1966; Oct. 1967; Dec. 1968; May, Sept. 1969; May, Nov. 1970; March, Sept. 1971; Jan., Nov. 1972; March, Sept. 1974; April, Dec. 1975; Feb., Dec. 1976; Feb., Aug., Sept. 1977; June 1978; Oct. 1979; April 1980; Feb. 1981; Feb., Dec. 1982; April 1983; Dec. 1984; June 1985; June, Oct., Dec. 1986; June 1987; Feb., April 1988; Aug. 1989. Review of Economics and Statistics: Nov. 1965; Feb. 1966; Feb. 1967; Aug. 1968; May, Aug. 1969; Aug. 1970; Nov. 1972; Feb., Aug., 1973; May 1975; May 1976; Feb. 1977; Aug., Nov. 1979; Nov. 1980; Aug. 1983; Feb., May 1984; Nov. 1985; Aug., Nov. 1986; Feb. 1987; Aug. 1988; May 1989. (3.) See, for example, the studies by Sterling (1959) and Bozarth and Roberts (1972) in psychology; Brown and Coney (1976) in marketing; and Reid, Soley, and Wimmer (1981) in advertising. (4.) The percentage of the literature devoted to the publication of replications was calculated using both the total number of research reports (because replications must compete with all other studies for a finite amount of journal space), as well as only the empirical ones. (5.) Hereafter, it will be understood that for space-economizing reasons the term "replications" also refers to replications with extensions. (6.) It is important to note that not all of the papers appearing in this section qualify as replications as defined in this paper.


AER, "Report of the Secretary for 1989," American Economic Review, May 1990, 80, 461. Bozarth, J.D., and R. R. Roberts, Jr., "Signifying Significant Significance," American Psychologist, August 1972, 27, 774-775. Brown, S. W., and K. A. Coney, "Building a Replication Tradition in Marketing," in K. L. Bernhardt, ed., Marketing 1776-1976 and Beyond, Chicago, IL: American Marketing Association, 1976, 622-625. Cohen, J., "A Coefficient of Agreement for Nominal Scales," Educational and Psychological Measurement, Spring 1960, 20, 37-46. Collins, H. M., Changing Order: Replication and Induction in Scientific Practice, Beverly Hills: Sage Publications, 1985. Dewald, W. G., J. G. Thursby, and R. G. Anderson, "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, September 1986, 76, 587-603. _____, _____, _____, "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project: Reply," American Economic Review, December 1988, 78, 1162-1163. Editors, "Editorial Comment," Journal of Political Economy, December 1975, 83, 1295-1296. Feige, E. L., "The Consequences of Journal Editorial of Policies and a Suggestion for Revision," Journal of political Economy, December 1975, 83, 1291-1295. Kane, E. J., "Why Journal Editors Should Encourage the Replication of Applied Econometric Research," Quarterly Journal of Business and Economics, Winter 1984, 23, 3-8. Laband, D. N., and J. P. Sophocleus, "Revealed Preference for Economics Journals: Citations as Dollar Votes," Public Choice, 1985, 46, 317-324. Liebowitz, S. J., and J. P. Pal,er, "Assessing the Relative Impacts of Economics Journals," Journal of Economic Literature, March 1984, 32, 77-88. Mayer, T., "Economics as a Hard Science: Realistic Goal or Wishful Thinking? Economic Inquiry, April 1980, 18, 165-178. Merrick, Jr., J. L., "Replication in Empirical Economics Project: Comment," American Economic Review, December 1988, 78, 1160-1161. Mittelstaedt, R. A., and T. S. Zorn, "Econometric Replication: Lessons from the Experimental Sciences," Quarterly Journal of Business and Economics, Winter 1984, 23, 9-15. Reid, L. N., L. C. Soley, and R. D. Wimmer, "Replication in Advertising Research: 1977, 1978, 1979," Journal of Advertising, 1981, 10, 3-13. Rosenthal, R., and R. L. Rosnow, Essentials of Behavioral Research: Methods and Data Analysis, New York: McGraw-Hill, 1984. Sterling, T. D. "Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance - or Vice Versa," Journal of the American Statistical Association, March 1959, 54, 30-34.
COPYRIGHT 1992 Omicron Delta Epsilon
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1992 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Hubbard, Raymond; Vetter, Daniel E.
Publication:American Economist
Date:Mar 22, 1992
Previous Article:An estimation of target effective exchange rates: the case of the U.S.
Next Article:Intersegment and racial differences in nonemployment mobility.

Related Articles
Replication Vs. Duplication.
Author order and research quality.
Statistical Abstract of the United States: 2004-2005, 124th ed.
Jews and American conservatism.
"International Journal of Technology Policy and Law" from Inderscience.
The economics of crime; 3v.
Elsevier Science (Philadelphia, PA) will begin the publication of the "Journal of Clinical Lipidology" in March.
BioMed Central (London), the publisher of open access STM journals, will begin the publication of "Biotechnology for Biofuels" with an issue this...
WIDER publications.

Terms of use | Copyright © 2017 Farlex, Inc. | Feedback | For webmasters