Search

Registered reports are a publication format that involves peer reviewing studies both before and after carrying out research procedures. Although registered reports were originally developed to combat challenges in quantitative and confirmatory study designs, today registered reports are also available for qualitative and exploratory work. This article provides a brief primer that aims to help researchers in choosing, designing, and evaluating registered reports, which are driven by qualitative methods.

Effective science communication is challenging when scientific messages are informed by a continually updating evidence base and must often compete against misinformation. We argue that we need a new program of science communication as collective intelligence—a collaborative approach, supported by technology. This would have four key advantages over the typical model where scientists communicate as individuals: scientific messages would be informed by (a) a wider base of aggregated knowledge, (b) contributions from a diverse scientific community, (c) participatory input from stakeholders, and (d) better responsiveness to ongoing changes in the state of knowledge.

This study reports an independent replication of the findings presented by Smaldino and McElreath (Smaldino, McElreath 2016 R. Soc. Open Sci. 3 , 160384 ( doi:10.1098/rsos.160384 )). The replication was successful with one exception. We find that selection acting on scientist’s propensity for replication frequency caused a brief period of exuberant replication not observed in the original paper due to a coding error. This difference does not, however, change the authors’ original conclusions. We call for more replication studies for simulations as unique contributions to scientific quality assurance.

The push for open science has produced many initiatives that arguably improve the quality of science, such as preregistration of research methods, more general availability of scientific articles in the form of preprints and open access articles, and the availability of (meta)data and code. Open science can have direct benefits for the career prospects of the individual through higher quality of the work and increased chance of publication (Allen & Mehler, 2019). However, being an outspoken advocate of open science might also have indirect positive effects on individual scientific careers, in the form of status in a scientific community, decisions for tenure, and eligibility for certain kinds of funding. In light of these benefits, it may be profitable for individual labs to appear to engage in open science practices, without actually putting in the associated effort or doing only the bare minimum. In this article, we explore two types of academic behavior through a dynamic computational model (cf. Smaldino & McElreath, 2016) of an academic community that rewards open science: 1) actively practicing open science and/or 2) advocating open science. Crossing these two types of behavior leads to four different kinds of labs and we examine which of them thrive in this academic community. Implications of the model results are discussed.

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N = 227; 56.8%), and about one-fifth of studies contain changed hypotheses (N = 82; 19%). We found only a small number of studies with demoted hypotheses (N = 2; 1%) and no studies with promoted hypotheses. In all, 59% of studies include at least one hypothesis in one or more of these categories, indicating a substantial bias in presenting and selecting hypotheses by researchers and/or reviewers/editors. Contrary to our expectations, we found that added hypotheses and changed hypotheses were not more likely to be statistically significant than non-selectively reported hypotheses. For the other types of selective hypothesis reporting, no powerful test of the relationship with statistical significance could be carried out. Finally, we found that replication studies were less likely to include selectively reported hypotheses than original studies. Thus, selective hypothesis reporting is problematically common in psychological research and may partly be explained by the fact that authors do not specifically and clearly enough formulate hypotheses in preregistrations and papers. We urge researchers, reviewers, and editors to ensure that hypotheses outlined in preregistrations are clearly formulated and accurately presented in the corresponding papers.

A large number of systematic replication attempts in the social sciences have failed to support the claims in the original studies. These surprising results have inspired a body of metascientific research aimed at understanding these failures to replicate and ensuring future research is credible. In this article, we relate these new insights from metascience to empirical legal research. Specifically, a recent effort to replicate three influential empirical legal studies published in law journals found results that diverged from the originals. The replicators suggested their results were caused by the changing social context and did not explicitly consider whether the original effects were overstated or false positives. We re-analysed the data from the replications to attempt to confirm their results (i.e., computational reproducibility) with mixed success. When possible, we combined the data from the replications (i.e., meta-analysis) to leverage the greater precision that comes with large sample sizes. In one case, we found an effect where the replicators did not. Overall, however, our re-analysis and review of the broader social scientific context suggests that empirical legal research suffers from the same challenges plaguing other fields – small sample sizes and undisclosed flexibility have produced untrustworthy results.

ARTICLE PUBLISHED (OPEN ACCESS) IN BRITISH JOURNAL OF SOCIAL PSYCHOLOGY. Qualitative data sharing practices in psychology have not developed as rapidly as those in parallel quantitative domains. This is often explained by numerous epistemological, ethical, and pragmatic issues concerning qualitative data types. In this essay, I provide an alternative to the frequently expressed (often reasonable) concerns regarding the sharing of qualitative human data by highlighting three advantages of qualitative data sharing. I argue that sharing qualitative human data is not by default “less ethical,” “riskier,” and “impractical” compared to quantitative data sharing, but in some cases more ethical, less risky, and easier to manage for sharing because 1) informed consent can be discussed, negotiated, and validated, 2) the shared data can be curated by special means, and 3) the privacy risks are mainly local instead of global. I hope the provided alternative perspective further encourages qualitative psychologists to share their data when it is epistemologically, ethically, and pragmatically possible.

Forecasting the reproducibility of research findings is one of the key challenges of Metascience. To date, reliable above-chance predictions have mainly been achieved by pooling the subjective ratings of experts through surveys or via prediction markets. Obtaining such data is laborious, however, and unlikely to increase the theoretical understanding of reproducibility.Here we show that empirical measures of K, a principled metric of knowledge, are correlated with reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), an ongoing replication of 60 studies that use three common methodologies in the life sciences. For each study, we calculated the K value by dividing the effect size, measured in bits of Shannon entropy explained, by the descriptive complexity of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol.K values were statistically associated with subjective predictions about studies’ replication probabilities and relative effect sizes. This relation was robust to controlling for study methodology, and was a stronger predictor than other plausible covariates, including the originally reported effect size, subjective ratings of a study’s methodological difficulty, and raters’ self-reported expertise. The superior fit of K may be due to its particular structure and metrics, which are rooted in information theory and may more accurately reflect the implicit calculations made by raters.This is the first evidence that an objective metric hypothesized to quantify scientific knowledge captures subjective judgments about reproducibility. This finding gives the first independent support of K’s underlying assumption that scientific knowledge is a process of information compression, and it may offer a new tool for metaresearch.

Changes in statistical practices and reporting have been documented by Giofrè et al. (2017), who investigated ten statistical and open practices in two high ranking journals (Psychological Science [PS] and Journal of Experimental Psychology-General [JEPG]): These included null hypothesis significance testing; confidence or credible intervals; meta-analysis of the results of multiple experiments; confidence intervals interpretation; effect size interpretation; sample size determination; sample size stopping rule; data availability; materials availability; and preregistered design and analysis plan.The investigation was based on an analysis of all papers published in these journals between 2013 and 2015.The aim of the present study is to follow up changes in both PS and JEPG in subsequent years, from 2016 to 2020, adding code availability as further open practice.