Why all researchers should report effect sizes and their confidence intervals: paving the way for meta-analysis and evidence-based management practices.
The past three decades have seen an exponential increase in the number of empirical studies investigating entrepreneurial phenomena. Thus, scientific progress increasingly hinges on researchers' ability to make sense of findings across studies. Since the late 1970s, the advent of meta-analysis (MA) has introduced quantitative approaches to estimate effect sizes from the reported effects in multiple quantitative empirical studies (Glaser, 1976; Schmidt & Hunter, 1977). Supported by the success of MA in other fields, management scholars have started to embrace the opportunities of MA--especially scholars promoting evidence-based management practices. Consequently, the number of published MA studies has been increasing steadily both in management and entrepreneurship journals (Figure 1).
Meta Analyses and the "Perfect Study" Fallacy
The goal of an empirical study is to provide an approximation of the "true effect," which is the effect researchers would observe executing a perfect research design with an infinitely large sample and measures unaffected by statistical artifacts (e.g., measurement error). Unfortunately, the studies researchers conduct are never perfect. All empirical studies suffer from limitations, and all research findings are inherently probabilistic. Different studies, however, have different limitations. Hence, combining the results from several studies creates opportunities to address these limitations and to correct for some of the distortions caused by measurement errors, sampling errors, research design, and research context (Schmidt & Hunter, 2014). Before the advancement of MA, scholars solely depended on qualitative approaches to compare and aggregate the findings from empirical studies. These qualitative approaches can work quite effectively when dealing with a small number of prior studies, but they start to face severe challenges when the number of prior studies increases. The introduction of MA extended researchers' methodological "toolbox" to include quantitative approaches to estimate "true effects" based on the reported findings across a large number of prior studies (Schmidt & Hunter)--a situation that researchers now encounter with increasing frequency in the entrepreneurship field. Under these conditions, MA promises important input for the development of theory, predictive models, and evidence-based management practices. The positive experience with MA methodologies in other fields of social science research, such as biomedical research and practice (Hunt, 1997; Moher & Olkin, 1995), provides additional encouragement to embrace and explore related opportunities.
How to Advance and Support MA
Since their inception in the late 1970 (Glaser, 1976; Schmidt & Hunter, 1977), meta-analytic approaches have been continuously advanced. One primary focus has been the refinement and improvement of MA methodologies to increase their accuracy and usefulness. A second focus has been the introduction of statistical software packages to help execute MAs (e.g., Hunter-Schmidt Meta-Analysis Programs; Comprehensive Meta-Analysis Software; and Metafor Meta-Analysis Package for R).
To obtain the full benefits of MA approaches, however, the academic research community needs to adjust its publication practices to support MAs. One straightforward subject is to have researchers report empirical results in ways that are most useful for future MA efforts. A second more complex issue is how the research community can encourage scholars to conduct and publish the types of empirical studies that create excellent data for future MAs.
How to Report Research Results
For the quantitative estimation of effects, MA depends on the information provided in prior quantitative empirical studies. Prior studies hamper any MA application if they do not systematically and accurately report necessary information. At the core, MA uses three bits of information from each prior study: (1) effect size, (2) confidence interval of the effect size, and (3) sample size. If prior studies, however, differ in research design and execution beyond sample size, any estimation of potential moderating effects of these differences requires that original research reports communicate these differences. Carefully reporting such differences across studies promises not only to increase the accuracy of MA effect-size estimates, but creates opportunities to identify the relevance of moderating factors and boundary conditions using MAs.
Currently most empirical studies do not report effect size measures and their confidence intervals. Instead, studies hypothesize the direction of effects and report corresponding statistical significance using p-values. When studies do not report needed effect-size information, MA researchers have to contact authors or try to construct such effect-size estimates from the published information. To address related issues, some MA software includes algorithms to estimate appropriate effect-size measures and their confidence intervals from commonly reported information. For example, only reporting p-values "smaller than" instead of exact p-values is unnecessarily vague--and should always be avoided. Even reporting exact p-values, however, forces MA researchers to perform assumption-based transformations to estimate confidence intervals with higher levels of error. Hence, to support future MAs, all empirical studies should explicitly report appropriate effect-size measures and their 95% confidence intervals and for all hypotheses tested--significant or not. Not providing such information may bias MAs and make them less accurate. Pressure on researchers to focus more on effect sizes and confidence intervals has also been steadily increasing based on various other reasons (American Psychological Association, 2010; Cohen, 1994; Gigerenzer, 2004; Schmidt & Hunter, 2002; Schwab, Abrahamson, Fidler, & Starbuck, 2011; Tukey, 1991). The crucial usefulness of such information for MA only adds another good argument to this already long list of good arguments in favor of reporting effect sizes and confidence intervals.
What Effect-Size Measures to Report?
Methodology scholars have developed and proposed a broad range of effect-size measures. Although each of the various effect-size measures may have value in certain applications, only a few of these measures provide useful inputs for MA. For example, nonstandardized effect-size measures capture the level or change in the outcome variable in original units (e.g., change in number of successful start-up firms, fraction of retained employees). These measures focus on means, differences between means, and nonstandardized regression weights (B). The measurement in original units offers advantages for the intuitive evaluation if a change is substantively relevant. For the quantitative aggregation of effect-size estimates across different studies, however, differences between measures used in different studies create substantial aggregation challenges. Hence, current MA approaches focus on the following standardized and unit-free effect-size measures: (1) Pearson's product-moment correlation coefficient "r," (2) Cohen's mean difference "d," (3) odds ratio, and (4) risk ratio. Odds or risk ratios, however, have been rarely used in management studies.
Several solid textbooks are now available that offer "hands-on" instructions on how to estimate effect sizes with a focus on management and related social sciences (Cumming, 2011; Ellis, 2010). Free web resources can help with related calculations (http://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-ESTypes .php). In addition, techniques and software are available to estimate standardized unit-free effect-size information, including their confidence intervals, from information typically provided in empirical studies (e.g.. Borenstein, Hedges, Higgins, & Rothstein, 2011, for more details).
If such estimation and conversion techniques are available, why should researchers bother to report MA relevant effect-size estimates? Three reasons! First, explicitly providing such information reduces the efforts needed to complete future MAs. If MAs support more reliable conclusions, such as better empirically grounded management practices, any empirical researcher should want to facilitate such future MAs by providing the necessary information.
The second reason is that providing comprehensive and detailed effect-size information in all studies increases the accuracy of MA estimations. If studies do not provide the information MAs require, MA researchers have to estimate this effect-size information indirectly, which decreases the accuracy and confidence in their meta-analytic conclusions.
Finally, reporting effect-size information for future MAs represents an important shift toward more meta-analytic thinking; i.e., a step away from the tempting, but deceptive notion, that single empirical studies can provide conclusive answers to research questions. Meta-analytic thinking guides us back to more incremental and accumulative empirical research philosophies (Platt, 1964; Tukey, 1991) that may prove essential for the development of stronger evidence-based management knowledge (Pfeffer & Sutton, 2006; Rousseau, 2012).
What Should Journals Do?
Journals and the publication process set and reinforce standards for the reporting of empirical research results (Orlitzky, 2012). Hence, journals should require authors to provide standardized effect-size information, such as Pearson's r and Cohen's d, and related confidence intervals for each initially hypothesized effect--including hypothesis tests that produced not statistically significant results. In addition, journals should ask authors to describe other potentially MA-relevant key features and characteristics of any study's research design, research execution, and empirical context. Current publication norms favor extremely concise descriptions of research design and execution. Researchers rarely report more complex information, such as reliability estimates for key measures or comprehensive discussions of potential boundary conditions. Such information, however, enables MAs to estimate and statistically control for related moderating effects.
In the past, limited journal space has been a factor preventing a more comprehensive and detailed reporting of empirical findings. Today, online archives create opportunities to efficiently collect and disseminate additional information related to any specific study. Developing and implementing the corresponding submission guidelines and procedures will require adjustments by everybody involved. The time, however, seems ripe for journals to initiate and explore such opportunities to better support MAs.
Anything Else? Where Shall We Go From Here?
Sometimes looking over the fence can lead to important new insights. Hence, the field of entrepreneurship research should consider learning from other fields of research--fields with substantial experience on how to support MAs and how to develop better evidence-based practices.
The field of medical research, for example, benefited tremendously from the creation of a nonprofit nongovernmental initiative, called the Cochrane Collaboration (http:// www.cochrane.org), which creates and publishes up-to-date systematic reviews based on all the conducted medical trials for a specific treatment or drug. For this purpose, 31,000 volunteers in more than 120 countries collaborate to collect, archive, analyze, and disseminate the information from all conducted medical studies meeting minimum quality standards (e.g., randomized controlled designs). The Cochrane Collaboration has promoted and used MA as the primary methodology to aggregate findings across studies and to formulate evidence-based recommendations and best medical practices. In social policy research, the Campbell Collaboration (http://www.campbellcollaboration.org) represents a similar nonprofit initiative to develop evidence-based recommendations for social policy makers based on aggregating empirical findings across studies. Both initiatives illustrate the potential value of institutionalized collaborative efforts among researchers to engage in meta-analytic investigations and to create infrastructure to collect, share, and analyze the accumulated evidence. These initiatives represent an alternative way to produce scientific knowledge that goes beyond the publication of empirical results in top-level academic journals. Academic journals, however, can play a key supportive role by requiring authors to submit relevant information to such initiatives before publication. Such field-wide initiatives would also relieve journals from each developing and managing their own empirical data repositories.
Entrepreneurship as a field of research has become quite successful in obtaining the necessary resources to conduct an increasing number of empirical studies to investigate key questions related to entrepreneurial opportunity creation, recognition, and exploitation. In spite of the increasing body of empirical data, researchers have been far less effective in integrating the findings from these studies into a comprehensive body of evidence-based entrepreneurship knowledge and practices. The advent of MA methodologies is creating new opportunities for the quantitative integration of findings across prior empirical studies. Such quantitative integration promises to be especially powerful in discovering and confirming effect patterns across large numbers of empirical studies. Obviously, MA faces its own limitations and challenges. For one, advancements of MA design practices and statistical analyses continue. Another, equally important, area of advancement relates to empirical studies providing the necessary information to enable and facilitate later meta-analytic investigations. The necessary changes in reporting empirical findings are relatively simple and. In recognition of these opportunities, all empirical researchers should report MA-relevant effect-size measures and their confidence intervals. In addition, they should provide specific information about potential measurement errors, moderating factors, and boundary conditions--again with future meta-analytic investigations in mind. In addition, journal editorial boards and publication guidelines should demand this. Such institutional pressures promise to support a swift adjustment of research reporting norms to include MA-relevant information. Finally, the success of discipline-wide institutions to collect, archive, analyze, and disseminate meta-analytic information in other fields of science, such as the Cochrane Collaboration in medical research, deserve attention and consideration. Similar collective efforts related to entrepreneurship research are feasible and desirable.
From an epistemological perspective, the proposed adjustments of reporting practices in original empirical studies imply a step toward more meta-analytic mindsets that acknowledges and embraces the often incremental and slow nature of empirical research progress. In the end, the patient and systematic accumulation of empirical evidence across numerous studies represents our most promising road toward better evidence-based entrepreneurship knowledge.
American Psychological Association. (2010). Publication manual of the American Psychological Association. Washington, DC: Author.
Borenstein, M" Hedges, L.V., Higgins, J.P.T., & Rothstein, H.R. (2011). Introduction to meta-analysis. Chichester, UK: Wiley.
Cohen, J. (1994). The earth is round (p < 0.05). American Psychologist, 49(12), 997-1003.
Cumming, G. (2011). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
Ellis, P.D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis and the interpretation of research results. New York: Cambridge University Press.
Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606.
Glaser, G.V. (1976). Primary, secondary and meta-analysis of research. Education Researcher, 10, 3-8.
Hunt, M. (1997). How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.
Moher, D. & Olkin, I. (1995). Meta-analysis of randomized controlled trials: A concern for standards. JAMA: The Journal of the American Medical Association, 274(24), 1962-1964.
Orlitzky, M. (2012). How can significance tests be deinstitutionalized? Organizational Research Methods, 15(2), 199-228.
Pfeffer, J. & Sutton, R.I. (2006). Hard facts, dangerous half-truths and total nonsense: Profiting from evidence-based management. Cambridge, MA: Harvard Business School Press.
Platt, J. (1964). Strong inference. Science, 746(3642), 347-353.
Rousseau, D.M. (2012). The Oxford handbook of evidence-based management. New York: Oxford University Press.
Schmidt, F. & Hunter, J. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
Schmidt, F. & Hunter, J. (2002). Are there benefits from NHST? American Psychologist, 57(1), 65-66.
Schmidt, F. & Hunter, J. (2014). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: SAGE Publications.
Schwab, A., Abrahamson, E., Fidler, E, & Starbuck, W.H. (2011). Researchers should make thoughtful assessments instead of null-hypothesis significance tests. Organization Science, 22(4), 1105-1120.
Tukey, J.W. (1991). The philosophy of multiple comparisons. Statistical Science, 6(1), 100-116.
Andreas Schwab is an associate professor and dean's fellow in the Management Department at Iowa State University, 3315 Gerdin Business Building, Ames, IA 50011, USA.
This editorial has benefited from the comments and suggestions provided by Frank Schmidt, Bill Starbuck. Mike McDaniel, and Howard Van Auken. Any errors and omissions are the responsibility of the author.
Please send correspondence to: Andreas Schwab, tel.: 515-294-8119; e-mail: firstname.lastname@example.org.
|Printer friendly Cite/link Email Feedback|
|Publication:||Entrepreneurship: Theory and Practice|
|Date:||Jul 1, 2015|
|Previous Article:||Struggling with market entry and expansion: parlance Communications in Mexico.|
|Next Article:||Start-up difficulties in early-stage peripheral clusters: the case of IT in an emerging economy.|