Printer Friendly

Invited Editorial Methodological and Statistical Recommendations: Option or Necessity?

In 2015 we published the chapter Statistical Guidelines for Empirical Studies (Utts & Tressoldi, 2015, p. 83) with the aim "to convince the reader of the importance of adopting sound methodological and statistical principles as described in this paper."

We ended up our chapter with these methodological and statistical recommendations:

* "Make explicit the difference between exploratory or pilot experiments and formal ones; Make explicit the primary and the secondary hypotheses to be tested before collecting any data; Report all experimental conditions, including failed manipulations;

* Make explicit the initial choice of the sample size(s), and provide an explanation if it was not met;

* If possible, explain the rationale for the sample size(s), including a power analysis;

* Whenever possible, report confidence intervals and effect sizes along with or instead of

If Bayesian methods are used, be explicit about all priors, including the prior distribution represented in the alternative hypothesis;

* Exact and conceptual replications are welcomed, but explain which one is being attempted;

* Pre-registration of confirmatory hypotheses is recommended, for example posting them on www.openscienceframework.org and/or wwwkoestler_parapsychology.psyed.ac.uk/irialRegistry

Statistical analyses: When using the frequentist Null Hypothesis Significant Testing approach, adopt the APA 2010 and APS statistical guidelines (Cumming, 2014): "Consideration of whether or not to reject the null hypothesis should be carried out using parameters' confidence intervals, equivalence testing or model comparison procedures (see suggested readings and resources), except for hypotheses that are not about a single parameter, such as chi-square goodness-of-fit tests or tests based on the sum of ranks."

Two years later, Etzel Cardena, one of the editors of the Handbook, was appointed Editor of the Journal of Parapsychology (JP) and, among the changes introduced to the JP, the following statistical guidelines were presented to the authors:
   Descriptive statistics (e.g., mean, standard deviation) must be
   reported in addition to inferential statistics (e.g., t tests),
   which should also include the specific p value and measures of
   effect size (authors might consider consulting the Statistical
   Guidelines for Empirical Studies by Tressoldi and Utts published in
   the Parapsychology: A handbook for the 21st century edited by
   Cardena, Palmer, and Marcusson-Clavertz, 2015). Although not
   mandatory, it is strongly recommended that all research,
   exploratory and even more so confirmatory, be preregistered, for
   instance through koestlerunit.wordpress.com/study-registry and that
   data be made available to other potential researchers through a
   depository such as data.world. Meta-analyses are encouraged when
   multiple studies have used the same variables.


In the meantime, what was happening in the scientific world? Most of the scientific fields, from psychology to medicine, ecology, and economics, were and still are in the middle of the so-called credibility revolution" (Vazire, 2018) as a consequence of the "replicability crisis" (Fanelli, 2018; Munafo et al., 2017; Pashler & Wagenmakers, 2012), which exploded literally around 2011 even if the "symptoms" were present many years before (Ioannidis, 2005).

What are many scientific journals doing to "cure" the replicability crisis and favor the credibility revolution? Among other changes, they are requesting new methodological and statistical requirements to submitted papers. In the Appendix, we have added the links of some of the top-tier scientific journals. A rapid comparison with those of the JP confirm that they are very similar (emphasizing descriptive statistics, effect size, confidence intervals, statistical power, maximum transparency in all choices, etc.) and consequently that the JP is contributing to the credibility revolution.

Are the JP methodological and statistical recommendations an option or a necessity? If we want JP papers to reach the quality standards of those published in the most prestigious scientific journals, the answer is quite simple, they must be adopted by the authors and checked by the reviewers.

References

Cardena, E., Palmer, J., & Marcusson-Clavertz, D. (2015). Parapsychology: A handbook for the 21st century. Jefferson, NC: McFarland.

Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2628-2631. doi. org/10.1073/pnas.1708272114 Ioannidis, J. P A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi. org/10.1371/journal.pmed.0020124

Munafo, M. R., Nosek, B. A., Bishop, D. V M., Button, K. S., Chambers, C. D., Percie du Sert, N., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1 (1), 0021. doi.org/10.1038/ s41562-016-0021

Pashler, H., & Wagenmakers, E. (2012). Editors' introduction to the special section on replicability in psychological science. Perspectives on Psychological Science, 7,528-530. doi.org/10.1177/1745691612465253

Utts, J., & Tressoldi, P (2015). Statistical guidelines for empirical studies. In E. Cardena, J. Palmer, J., & D. Marcusson-Clavertz, D. (2015). Parapsychology: A handbook for the 21st century (pp. 83-93). Jefferson, NC: McFarland.

Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411-417. doi.org/10.1177/1745691617751884

Appendix

Statistical Guidelines of some of the Top-tier Scientific Journals

Psychological Science, the flagship journal of the Association for Psychological Science: www.psychologicalscience.org/publications/psychological_science/ps-submissions#STAT

Nature: www.nature.com/nature/for-authors/initial-submission

Science Statistical Analysis www.sciencemag.org/authors/science-journals-editorial-policies

Psychonomic Society Statistical Guidelines featuredcontent.psychonomic.org/psychonomic-society-statistical-guidelines-updated

http://doi.org/10.30891/jopar.2019.02.02

Jessica Utts

University of California, Irvine

Patrizio E. Tressoldi

Universite di Padova

(1) Send correspondence to: Jessica Utts, Ph. D., Department of Statistics. University of California. Donald Bren Hall, Irvine, CA 92697, USA. jutts@uci.edu
COPYRIGHT 2019 Parapsychology Press
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2019 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Utts, Jessica; Tressoldi, Patrizio E.
Publication:The Journal of Parapsychology
Article Type:Editorial
Date:Sep 22, 2019
Words:913
Previous Article:Editorial: Errors of the Third Kind.
Next Article:In Memoriam: Elizabeth A. Rauscher (1937-2019).

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters