Back to Search
Start Over
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
- Source :
- PLoS Biology, PLoS Biology, Vol 15, Iss 3, p e2000797 (2017), PLoS Biology, Vol 19, Iss 3, p e3001151 (2021)
- Publication Year :
- 2017
- Publisher :
- Public Library of Science, 2017.
-
Abstract
- We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64–1.46) for nominally statistically significant results and D = 0.24 (0.11–0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.<br />Author summary Biomedical science, psychology, and many other fields may be suffering from a serious replication crisis. In order to gain insight into some factors behind this crisis, we have analyzed statistical information extracted from thousands of cognitive neuroscience and psychology research papers. We established that the statistical power to discover existing relationships has not improved during the past half century. A consequence of low statistical power is that research studies are likely to report many false positive findings. Using our large dataset, we estimated the probability that a statistically significant finding is false (called false report probability). With some reasonable assumptions about how often researchers come up with correct hypotheses, we conclude that more than 50% of published findings deemed to be statistically significant are likely to be false. We also observed that cognitive neuroscience studies had higher false report probability than psychology studies, due to smaller sample sizes in cognitive neuroscience. In addition, the higher the impact factors of the journals in which the studies were published, the lower was the statistical power. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
- Subjects :
- Basic science
Developmental cognitive neuroscience
Social Sciences
Behavioral neuroscience
Empirical Research
Behavioral Neuroscience
0302 clinical medicine
Empirical research
Surveys and Questionnaires
Statistics
Range (statistics)
Psychology
Biology (General)
Statistic
Statistical Data
General Neuroscience
Applied Mathematics
Simulation and Modeling
05 social sciences
Publications
Physical Sciences
Meta-Research Article
Probability distribution
General Agricultural and Biological Sciences
Statistics (Mathematics)
Algorithms
Statistical Distributions
QH301-705.5
Imaging Techniques
Cognitive Neuroscience
Neuroimaging
Cognitive neuroscience
Research and Analysis Methods
050105 experimental psychology
General Biochemistry, Genetics and Molecular Biology
03 medical and health sciences
Functional neuroimaging
Replication (statistics)
Humans
0501 psychology and cognitive sciences
p-value
Probability
General Immunology and Microbiology
Correction
Biology and Life Sciences
Probability Theory
Sample size determination
Cognitive Science
Null hypothesis
030217 neurology & neurosurgery
Mathematics
Neuroscience
Subjects
Details
- Language :
- English
- ISSN :
- 15457885 and 15449173
- Volume :
- 15
- Issue :
- 3
- Database :
- OpenAIRE
- Journal :
- PLoS Biology
- Accession number :
- edsair.doi.dedup.....8a5049bff05c37989107164e7dd186b3