In Defence of "Replication Research".
The notion of replication research has been addressed in the literature from the perspective of publication in academic journals. In effect there appears to be a stigma attached to replication research (Easley, Madden & Dunn, 2000) with the majority of highly ranked (top-tier) journals (Neuliep & Crandall, 1990). Hubbard, Vetter and Little (1998) found that over the first, second and third tier of journals in the management disciple had significantly less replication research published compared to the total research papers published over the period 1976 to 1995. Similar disparities have been reported in the advertising (Arndt, 1976), marketing (Hubbard & Armstrong, 1994), and consumer behaviour (Mahoney, 1987) literature.
Research is predominantly about testing theories to determine their validity. Now the most succinct explanation is that theories should explain and predict some specific observable phenomena, and should be capable replication (Rudner, 1966). Hence, replication is fundamental part of the testing of a theory and the various acts of replication serve to protect against errors in empirical results (Hubbard & Vetter, 1996), and replication with extension provides evidence of generalisability as well as building on knowledge (Galtung, 1967; Hubbard & Vetter, 1996; Lindsay & Ehrenberg, 1993). The ability to replicate research findings has been referred to as the hallmark of science (Blaug, 1992) and the key criterion to genuine scientific knowledge (Rosenthal & Rosnow, 1984). See for example the controversy surrounding the creation of cold fusion as a result of replication having failed to reach the same result (Jones & Hansen, 1995). Other examples concerned with the identification of errors are: the research on the minimum wage by Welch (1974) for which the error was identified by Siskind (1977); the research on trends in manufacturing by Adelman (1951) for which the error was identified by Means (1962). An interesting point is the amount of time before the errors were identified and made public, effectively, during the intervening years the original research findings would have been considered valid and may well have been cited to support research that in retrospect would be misleading.
The importance of replication is intrinsically linked to the scientific paradigm and its grounding in empirical research. To paraphrase the Falsification Theory of Popper (1959) (1) truth is transient and the best one can hope to achieve is to empirically prove that something is not false at a particular point in time, and may we add given the limited knowledge and technology available at that point of time. Which of course is the reason for the use of the Null Hypothesis the test should be to falsify not confirm, hence the result of testing leads to the null hypothesis being either rejected or accepted--if the null hypothesis is accepted then the theoretical basis is not correct (it is false), alternatively if it is rejected it does not prove the that the theoretical basis true it is just not false. (2) With regards to the transient nature of truth the simple example is that in Europe it was considered to be true that all swans were white at least until the discovery of black swans by Dirk Hartog. He happened upon the black swans in a river (the swan river) in what he referred to as New Holland (now Western Australia). With regards to examples of theories that were held to be true but were eventually proven to be wrong--effluvial theory of static electricity, vibratory theory of heat, humoral theory of medicine, they appeared to explain and predict the observable phenomena but were eventually found to be incorrect.
The literature in support of replication research is consistent in arguing that replication in all its forms is essential for the confirmation of research findings. Hubbard and Armstrong (1994) argued that research findings that are not replicated are nothing more than speculative regardless of how statistically significant the results may be. Lindsay and Ehrenberg (1993, 219) went further in their criticism of unreplicated research stating that it was 'virtually meaningless and useless'. Replication does occur in different forms firstly there is pure-replication and then there is the more diverse replication extension. Pure-replication has been defined as 'a duplication of a previously published empirical study that is concerned with assessing whether similar findings can be obtained upon repeating the study' (Hubbard & Armstrong, 1994, 236). Note, we use the term "pure-replication" since it is distinctly concerned with replicating the original research. Effectively, it involves a substantial duplication that is chiefly concerned with increasing internal validity--duplicating all facets of the original research to determine whether the results are reproducible (Hubbard, Vetter & Little, 1998). Conversely, the definition of replication with extension is 'a duplication of a previously published empirical research project that serves to investigate the generalizability of earlier research findings' (Hubbard & Armstrong, 1994, 236). In this situation the primary concern is with increasing external validity or generalisability--this takes the form of modifying aspects of the independent (manipulated) or dependent (measured) variables (Hubbard, Vetter & Little, 1998).
Taking this notion of justification to the next level, it is in the interest of all researchers to be acquainted with the relevant contributions that the different types of replication can make. When confronted with the need to provide an explanation or statement about proposed research which is based upon a form of replication it would seem advantageous to provide clarification. To provide some guidance for the clarification a researcher would likely have to be specific on the issues and Table 2 is intended to provide a basic check list to assist in this endeavour.
There are two distinct types of science for which replication research is applicable. Natural science which is concerned with naturally occurring phenomena, where the focus is on non-random events (Harre & Secord, 1973). By contrast, social science is concerned with phenomena that is not naturally occurring but rather socially constructed and the research seeks to explain actions mediated by meanings (Harre & Secord, 1973). Gergen (1978) put it more succinctly, in that social science pertains to human behaviour and natural science does not. The difference between natural science and social science does not make the need for replication any less valuable or relevant.
One other point that needs to be addressed is that replication tends to be discussed from the perspective of quantitative or empirical research with the assumption that statistical testing is used to arrive at the findings. Interestingly, a means for evaluating replication results using statistical analysis to establish the effect size of replication research is meta-analysis which provides a quantitative assessment (Glass, 1976; Brinberg & Jaccard, 1986; Easley, Madden & Dunn, 2000). Meta-analysis is not in its self a replication but rather a means for evaluating replication. In the domain of quantitative empirical research meta-analysis is a far more obvious choice for the evaluation as compared to the alternative of a qualitative literature review.
This notion of replication being more consistent with quantitative research than qualitative research is rather misleading. Although replication may be more difficult in some forms of qualitative research it no less important to have the findings checked and validated as it plays such a vital role in justifying the theoretical basis and supports the development of the body of knowledge in the particular field of study.
Examples exist in the literature of various disciplines calling for greater use and publication of replication research. For example, Advertising (Reid, Soley & Winner, 1981; Madden, Easley & Dunn, 1995).; Business and Economics (Kane, 1984; Fuess Jr, 1996; Evanschitzky, Baumgarth, Hubbard & Armstrong, 2007).; Information Systems (Kettinger & Lee, 1999; Berthon, Pitt, Ewing & Carr, 2002).; Management (Singh, Ang & Leong, 2003; Tsang & Kwan, 1999; Mezias & Regnier, 2007); Nursing (Connelly, 1986; Beck, 1994).; Psychology (Smith, 1970; Campbell & Jackson, 1979; Amir & Sharon, 1990; Lamal, 1990; Thompson, 1994; Bonett, 2012); Political Science (Herrnson, 1995); Other (Muma, 1993). Subsequently, any argument for replication research should also include reference to published papers in the specific discipline and this would further benefit by showing its consistency with other disciplines thus providing a wide-ranging justification for the replication research.
Adelman, M. (1951). The Measurement of Industrial Concentration, Review of Economics and Statistics, 33, 269-296.
Amir, Y., & Sharon, I. (1990). Replication research: A" must" for the scientific advancement of psychology. Journal of Social Behavior and Personality, 5(4), 51.
Arndt, J. (1976). What's Wrong with Advertising Research?, Journal of Advertising Research, 16 (June), 9-18.
Beck, C. T. (1994). Replication strategies for nursing research. Image: the Journal of Nursing Scholarship, 26(3), 191-194.
Berthon, P., Pitt, L., Ewing, M., & Carr, C. L. (2002). Potential research space in MIS: A framework for envisioning and evaluating research replication, extension, and generation. Information Systems Research, 13(4), 416-427.
Blaug, M. (1992). The Methodology of Economics: Or How Economists Explain, Cambridge University Press: New York.
Bonett, D. G. (2012). Replication-extension studies. Current Directions in Psychological Science, 21(6), 409-412.
Brinberg, D. & Jaccard, J. (1986). Meta-AnalysisTechniques for the Quantitative Integration of Research Findings (pp. 155-180), in Perspectives on Methodology in Consumer Research, David Brinberg and Richard J. Lutz, eds., Springer-Verlag: New York.
Bronstein, R. F. (1990). Publication politics, experimenter bias and the replication process in social science research. Journal of Social Behavior and Personality, 5(4), 71.
Campbell, K. E., & Jackson, T. T. (1979). The role of and need for replication research in social psychology. Replications in Social Psychology, 1(1), 3-14.
Connelly, C. E. (1986). Replication research in nursing. International Journal of Nursing Studies, 23(1), 71-77.
Easley, R. W., Madden, C. S., & Gray, V. (2013). A tale of two cultures: Revisiting journal editors' views of replication research. Journal of Business Research, 66(9), 1457-1459.
Easley, R., Madden, C. & Dunn, M. (2000). Conducting Marketing Science: The Role of Replication in the Research Process, Journal of Business Research, 48, 83-92.
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007). Replication research's disturbing trend. Journal of Business Research, 60(4), 411-415.
Fuess Jr, S. M. (1996). On replication in business and economics research: The QJBE case. Quarterly Journal of Business and Economics, 35, 3-13.
Galtung, J. (1967). Theory and Methods of Social Research, Columbia University Press: New York.
Gergen, K. (1978). Toward Generative Theory, Journal of Personality and Social Psychology, 36, 1344-1360.
Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research, Educational Researcher, 5, 3-8.
Harre, R. & Secord, P. (1973). The Explanation of Social Behavior, Littlefield, Adams & Co.: Totowa, NJ.
Herrnson, P. S. (1995). Replication, verification, secondary analysis, and data collection in political science. PS: Political Science & Politics, 28(03), 452-455.
Hubbard, J. & Armstrong, J. (1994). Replications and Extensions in Marketing: Rarely Published but Quite Contrary, International Journal of Research in Marketing, 11, 233-248.
Hubbard, R., & Vetter, D. (1996). An Empirical Comparison of Published Replication Research in Accounting, Economics, Finance, Management, and Marketing, Journal of Business Research, 35, 153-164.
Hubbard, R., Vetter, D. & Little, E. (1998). Replication in Strategic Management: Scientific Testing for Validity, Generalizability, and Usefulness, Strategic Management Journal, 19, 243-254.
Jones, S. & Hansen, L. (1995). Examination of claims of Miles et al. in Pons-Fleischmann-type cold fusion experiments, Journal of Physical Chemistry, 99(18), 6966-6972.
Kane, E. J. (1984). Why journal editors should encourage the replication of applied econometric research. Quarterly Journal of Business and Economics, 23, 3-8.
Kaynak, H., & Hartley, J. L. (2006). Using replication research for just-in-time purchasing construct development. Journal of Operations Management, 24(6), 868-892.
Kettinger, W. J., & Lee, C. C. (1999). Replication of measures in information systems research: the case of IS SERVQUAL. Decision Sciences, 30(3), 893-899.
King, G. (1995). Replication, replication. PS: Political Science & Politics, 28(03), 444-452.
Lamal, P. A. (1990). On the importance of replication. Journal of Social Behavior and Personality, 5(4), 31.
Lindsay, R. & Ehrenberg A. (1993). The Design of Replicated Studies, American Statistician, 47, 217-228.
Madden, C. S., Easley, R. W., & Dunn, M. G. (1995). How journal editors view replication research. Journal of Advertising, 24(4), 77-87.
Mahoney, J. (1987). Scientific Publication and Knowledge Politics, Journal of Social Behavior and Personality, 2, 165-176.
Means, G. (1962). Thoughts on Concentration, Proceedings of the Business and Economic Statistics Section of the American Statistical Association, 718-726.
Mezias, S. J., & Regnier, M. O. (2007). Walking the walk as well as talking the talk: replication and the normal science paradigm in strategic management research. Strategic Organization, 5(3), 283-296.
Muma, J. R. (1993). The need for replication. Journal of Speech, Language, and Hearing Research, 36(5), 927-930.
Neuliep, J. & Crandall, R. (1990). "Editorial Bias Against Replication Research", Handbook of Replication Research, Sage Publications, Inc: Newbury Park, CA.
Popper, K. (1959). The Logic of Scientific Discovery. London: Hutchinson. This is an English translation of Logik der Forschung, Vienna: Springer (1935).
Popper, K. (1963). Conjectures and Refutations, Routledge and Keagan Paul: London.
Reid, L. N., Soley, L. C., & Winner, R. D. (1981). Replication in advertising research: 1977, 1978, 1979. Journal of Advertising, 10(1), 3-13.
Rosenthal, R. & Rosnow, R. (1984). Essentials of Behavioral Research: Methods and Data Analysis, McGraw-Hill: New York.
Rosenthal, R. (1990). Replication in behavioral research. Journal of Social Behavior and Personality, 5(4), 1.
Rudner, R. (1966). The Philosophy of Social Science, Prentice Hall: Englewood Cliffs, N.J.
Singh, K., Ang, S. H., & Leong, S. M. (2003). Increasing replication for knowledge accumulation in strategy research. Journal of Management, 29(4), 533-549.
Siskind F. (1977). Mimimum Wage Legislation in the United States: Comment, Economic Inquiry, 15, 135-138.
Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970.
Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62(2), 157-176.
Tsang, E. W., & Kwan, K. M. (1999). Replication and theory development in organizational science: A critical realist perspective. Academy of Management review, 24(4), 759-780.
Welch, F. (1974). Mimimum Wage Legislation in the United States, Economic Inquiry, 12, 285-318.
Gregory Kenneth Laing
Syme Business School, Australia
Ronald William Perrin
Faculty of Business
University of Wollongong, Australia
Sharon Margaret Laing
Statistics Research & Psychological Consultancy Services, Australia
(1) Karl Popper was an eminent philosopher and mathematician who argued for the application of deductive reasoning. This is based upon Popper's falsification theory which can be traced to his original book "Logik der Forschung", published (1935) and the English translation in 1959 "The Logic of Scientific Discovery".
(2) Empirical research should therefore always be based on testing the null hypothesis to do otherwise in effect indicates a poor understanding of Popper's falsification theory and the relationship it has with deductive reasoning and the principle intent in the Scientific paradigm.
JEL Classifications: C90
PsycINFO Classifications: 2260
FoR Codes: 2202
ERA Journal ID #: 40840
Table 1: Replication Levels Level Research Design Research Focus Research Justification 1 Pure-Replication Duplication of a * To assess whether previously published similar findings can be research study. obtained upon repeating the research study; * To assess internal validity; * To test for possible errors or omissions. 2 Replication Modified duplication * To investigate the extensions of a previously generalizability of published research earlier research study: findings'; * Different subjects, * To assess external population, sample; validity; * Different * To build on the statistical test theoretical model applied; /framework. * Different time periods; * Different instruments; * Different geographical areas; * Etc. Table 2: Replication Check List Type of Replication Contribution Justification Internal Audit for External Theory Validity Errors Validity Modification Pure-Replication [check] [check] Manipulations: Add Moderating Variables [check] [check] Add Intervening Variables [check] [check] Add New Dependent [check] [check] Variable(s) Add New Independent [check] [check] Variable(s) Different Statistical Tests [check] [check] Different Time Frame [check] [check] Different Instrument(s) [check] [check] Different Demographics: Subjects Age [check] [check] Subjects Sex Gender [check] [check] Subjects Education [check] [check] Subjects Intelligence [check] [check] Subjects Knowledge [check] [check] Familiarity with topic [check] [check] Geographical area [check] [check] (country or region) Sample Size [check] [check] Type of Replication Knowledge Advancement (Development) Pure-Replication Manipulations: [check] Add Moderating Variables [check] Add Intervening Variables [check] Add New Dependent Variable(s) [check] Add New Independent Variable(s) [check] Different Statistical Tests [check] Different Time Frame [check] Different Instrument(s) Different Demographics: [check] Subjects Age [check] Subjects Sex Gender [check] Subjects Education [check] Subjects Intelligence [check] Subjects Knowledge [check] Familiarity with topic [check] Geographical area (country or region) [check] Sample Size
|Printer friendly Cite/link Email Feedback|
|Author:||Laing, Gregory Kenneth; Perrin, Ronald William; Laing, Sharon Margaret|
|Publication:||Journal of New Business Ideas and Trends|
|Date:||Sep 1, 2018|
|Previous Article:||Will Islamic Marketing Survive in Today's World?|
|Next Article:||Biometric Identification and Authentication in Computers: Keystroke Dynamics.|