Invited Editorial Methodological and Statistical Recommendations: Option or Necessity?
We ended up our chapter with these methodological and statistical recommendations:
* "Make explicit the difference between exploratory or pilot experiments and formal ones; Make explicit the primary and the secondary hypotheses to be tested before collecting any data; Report all experimental conditions, including failed manipulations;
* Make explicit the initial choice of the sample size(s), and provide an explanation if it was not met;
* If possible, explain the rationale for the sample size(s), including a power analysis;
* Whenever possible, report confidence intervals and effect sizes along with or instead of
If Bayesian methods are used, be explicit about all priors, including the prior distribution represented in the alternative hypothesis;
* Exact and conceptual replications are welcomed, but explain which one is being attempted;
* Pre-registration of confirmatory hypotheses is recommended, for example posting them on www.openscienceframework.org and/or wwwkoestler_parapsychology.psyed.ac.uk/irialRegistry
Statistical analyses: When using the frequentist Null Hypothesis Significant Testing approach, adopt the APA 2010 and APS statistical guidelines (Cumming, 2014): "Consideration of whether or not to reject the null hypothesis should be carried out using parameters' confidence intervals, equivalence testing or model comparison procedures (see suggested readings and resources), except for hypotheses that are not about a single parameter, such as chi-square goodness-of-fit tests or tests based on the sum of ranks."
Two years later, Etzel Cardena, one of the editors of the Handbook, was appointed Editor of the Journal of Parapsychology (JP) and, among the changes introduced to the JP, the following statistical guidelines were presented to the authors:
Descriptive statistics (e.g., mean, standard deviation) must be reported in addition to inferential statistics (e.g., t tests), which should also include the specific p value and measures of effect size (authors might consider consulting the Statistical Guidelines for Empirical Studies by Tressoldi and Utts published in the Parapsychology: A handbook for the 21st century edited by Cardena, Palmer, and Marcusson-Clavertz, 2015). Although not mandatory, it is strongly recommended that all research, exploratory and even more so confirmatory, be preregistered, for instance through koestlerunit.wordpress.com/study-registry and that data be made available to other potential researchers through a depository such as data.world. Meta-analyses are encouraged when multiple studies have used the same variables.
In the meantime, what was happening in the scientific world? Most of the scientific fields, from psychology to medicine, ecology, and economics, were and still are in the middle of the so-called credibility revolution" (Vazire, 2018) as a consequence of the "replicability crisis" (Fanelli, 2018; Munafo et al., 2017; Pashler & Wagenmakers, 2012), which exploded literally around 2011 even if the "symptoms" were present many years before (Ioannidis, 2005).
What are many scientific journals doing to "cure" the replicability crisis and favor the credibility revolution? Among other changes, they are requesting new methodological and statistical requirements to submitted papers. In the Appendix, we have added the links of some of the top-tier scientific journals. A rapid comparison with those of the JP confirm that they are very similar (emphasizing descriptive statistics, effect size, confidence intervals, statistical power, maximum transparency in all choices, etc.) and consequently that the JP is contributing to the credibility revolution.
Are the JP methodological and statistical recommendations an option or a necessity? If we want JP papers to reach the quality standards of those published in the most prestigious scientific journals, the answer is quite simple, they must be adopted by the authors and checked by the reviewers.
Cardena, E., Palmer, J., & Marcusson-Clavertz, D. (2015). Parapsychology: A handbook for the 21st century. Jefferson, NC: McFarland.
Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2628-2631. doi. org/10.1073/pnas.1708272114 Ioannidis, J. P A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi. org/10.1371/journal.pmed.0020124
Munafo, M. R., Nosek, B. A., Bishop, D. V M., Button, K. S., Chambers, C. D., Percie du Sert, N., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1 (1), 0021. doi.org/10.1038/ s41562-016-0021
Pashler, H., & Wagenmakers, E. (2012). Editors' introduction to the special section on replicability in psychological science. Perspectives on Psychological Science, 7,528-530. doi.org/10.1177/1745691612465253
Utts, J., & Tressoldi, P (2015). Statistical guidelines for empirical studies. In E. Cardena, J. Palmer, J., & D. Marcusson-Clavertz, D. (2015). Parapsychology: A handbook for the 21st century (pp. 83-93). Jefferson, NC: McFarland.
Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411-417. doi.org/10.1177/1745691617751884
Statistical Guidelines of some of the Top-tier Scientific Journals
Psychological Science, the flagship journal of the Association for Psychological Science: www.psychologicalscience.org/publications/psychological_science/ps-submissions#STAT
Science Statistical Analysis www.sciencemag.org/authors/science-journals-editorial-policies
Psychonomic Society Statistical Guidelines featuredcontent.psychonomic.org/psychonomic-society-statistical-guidelines-updated
University of California, Irvine
Patrizio E. Tressoldi
Universite di Padova
(1) Send correspondence to: Jessica Utts, Ph. D., Department of Statistics. University of California. Donald Bren Hall, Irvine, CA 92697, USA. firstname.lastname@example.org
|Printer friendly Cite/link Email Feedback|
|Author:||Utts, Jessica; Tressoldi, Patrizio E.|
|Publication:||The Journal of Parapsychology|
|Date:||Sep 22, 2019|
|Previous Article:||Editorial: Errors of the Third Kind.|
|Next Article:||In Memoriam: Elizabeth A. Rauscher (1937-2019).|