Search

Registered reports are a publication format that involves peer reviewing studies both before and after carrying out research procedures. Although registered reports were originally developed to combat challenges in quantitative and confirmatory study designs, today registered reports are also available for qualitative and exploratory work. This article provides a brief primer that aims to help researchers in choosing, designing, and evaluating registered reports, which are driven by qualitative methods.

The importance of replication in the social and behavioural sciences has been emphasized for decades. Various frequentist and Bayesian approaches have been proposed to qualify a replication study as successful or unsuccessful. One of them is meta-analysis. The focus of the present study is on the way meta-analysis functions as a replication success metric. To investigate this, original and replication studies that are part of two large-scale replication projects were used. For each original study, the probability of replication success was calculated using meta-analysis under different assumptions of the underlying population effect when replication results were unknown. The accuracy of the predicted overall replication success was evaluated once replication results became available using adjusted Brier scores. Our results showed that meta-analysis performed poorly when used as a replication success metric. In many cases, quantifying replication success using meta-analysis resulted in the conclusion where the replication was deemed a success regardless of the results of the replication study.

A large amount of scientific literature in social and behavioural sciences bases their conclusions on one or more hypothesis tests. As such, it is important to obtain more knowledge about how researchers in social and behavioural sciences interpret quantities that result from such hypothesis tests, such as p-values and Bayes factors. In the present study, we will explore the relationship between obtained statistical evidence and the degree of belief or confidence that there is a positive effect in the population of interest. In particular, we are interested in the existence of a so-called cliff effect: A qualitative drop in the degree of belief that there is a positive effect around certain threshold values of statistical evidence. We will compare this relationship for p-values to the relationship for corresponding degrees of evidence quantified through Bayes factors, and we will examine whether this relationship is affected by two different modes of presentation.

We plan a collaborative research analysis of the original ‘many analysts’ study by Silberzahn et al.1 testing whether European soccer referees were skin-tone biased in giving red cards. Their study concluded that the answer has great model dependency, and in this sense is unclear. Auspurg and Brüderl2 re-analysed the methods and data and argued the answer is clearer when attending to causal inference and outliers. Researchers involved in the original study and follow-up agreed to an adversarial collaboration to better answer this pertinent research question and sources of variability in answers. They recruited other scholars to provide independent methodological and theoretical insights. The result is this study where together we jointly tackle the skin-tone bias hypothesis and the ‘parsing problem’ of identifying meaningful moderators that explain effect size dispersion across a multiverse of alternative specifications. We build on Auspurg and Brüderl’s proposal that analysts in the original study had very different interpretations of the conceptual research question, leading them to different empirical research questions, which in turn produced results that are not directly comparable. This pre-analysis plan describes our motivation, methods, and plan to contextualize our findings in larger discussions of race bias, multiverse analysis parsing, and subjectivity in science.

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or Preregistration Badge to 193 related studies that were not preregistered. In contrast with our theoretical expectations and prior research, we did not find that preregistered studies had a lower proportion of positive results (Hypothesis 1), smaller effect sizes (Hypothesis 2), and fewer statistical errors (Hypothesis 3) than non-preregistered studies. Supporting our Hypotheses 4 and 5, we found that preregistered studies more often contained power analyses and typically had higher sample sizes than non-preregistered studies. Finally, concerns about the publishability and impact of preregistered studies seem unwarranted as preregistered studies did not take longer to publish and scored better on several impact measures. Overall, our data indicate that preregistration has beneficial effects in the realm of statistical power and impact, but we did not find robust evidence that preregistration prevents p-hacking and Hypothesizing After the Results are Known (HARKing).

In recent years, the scientific community has called for improvements in the credibility, robustness, and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (1) students’ scientific literacies (i.e., students’ understanding of open research, consumption of science, and the development of transferable skills); (2) student engagement (i.e., motivation and engagement with learning, collaboration, and engagement in open research), and (3) students’ attitudes towards science (i.e., trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lack methodological details and that undisclosed deviations from preregistered plans are frequent. Combining the strictness and consistency results highlights that biases due to researcher degrees of freedom are prevalent and likely in many preregistered studies. More comprehensive registration templates typically yielded stricter and hence better preregistrations. We did not find that effectiveness of preregistrations differed over time or between original and replication studies. Furthermore, we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrations and published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to write stricter preregistrations, adhere to these preregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.

Generalizations strengthen in traditional sciences, but in psychology (and social and behavioral sciences, more generally) they decay. This is usually viewed as problem requiring solution, but it could be viewed as a law-like phenomenon. Generalization decay cannot be squelched because human behavior is metastable and all behavioral data collected thus far have resulted from a thin sliver of human time.

Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al. 2023) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments. Attempts to debias data-driven inferences by “zooming out” are also challenged by the inseparability of the results sampled from a design space from the assumptions that produce them, and the difficulty of rendering dependencies explicit without sacrificing interpretability.

Urgent attention is needed to address generalizability problems in psychology. However, the current dominant paradigm, which centers dichotomous results and rapid discoveries, may not provide the solution. Instead, we propose a paradigm shift towards a model-centric approach, which can aid in understanding the sources of generalizability and promote systematic exploration. In a model-centric paradigm, scientific activity involves iteratively building and refining theoretical, empirical, and statistical models that communicate with each other. This approach is organic, transparent, and efficient in addressing generalizability issues. We illustrate the nature of scientific activity in a model-centric system and its potential for advancing the field of psychology.

Effective science communication is challenging when scientific messages are informed by a continually updating evidence base and must often compete against misinformation. We argue that we need a new program of science communication as collective intelligence—a collaborative approach, supported by technology. This would have four key advantages over the typical model where scientists communicate as individuals: scientific messages would be informed by (a) a wider base of aggregated knowledge, (b) contributions from a diverse scientific community, (c) participatory input from stakeholders, and (d) better responsiveness to ongoing changes in the state of knowledge.

Integrative experiment design assumes that we can effectively design a space of factors that cause contextual variation. However, this is impossible to do so in a sufficiently objective way, resulting inevitably in observations laden with surrogate models. Consequently, integrative experiment design may even deepen the problem of incommensurability. In comparison, one-at-a-time approaches make much more tentative assumptions about the factors excluded from experiment design, hence still seem better suited to deal with incommensurability.

We expect that consensus meetings, where researchers come together to discuss their theoretical viewpoints, prioritize the factors they agree are important to study, standardize their measures, and determine a smallest effect size of interest, will prove to be a more efficient solution to the lack of coordination and integration of claims in science than integrative experiments.

Almaatouq et al. claim that the integrative experiment design can help “develop a reliable, cohesive, and cumulative theoretical understanding.” I will contest this claim by challenging three underlying assumptions about the nature of scientific theories. I propose that the integrative experiment design should be viewed as an exploratory framework rather than a means to build or evaluate theories.

Publishing study results in scientific journals has been the standard way of disseminating science. However, getting results published may depend on their statistical significance. The consequence of this is that the representation of scientific knowledge might be biased. This type of bias has been called publication bias. In the present study, we make an attempt to get more insight into publication bias by examining it at the author, reviewer, and editor level. Additionally, we make a direct comparison between publication bias induced by authors, by reviewers, and by editors. Our findings suggest that statistically significant findings have a higher likelihood to be published than statistically non-significant findings, because (1) authors are more likely to write up and submit articles with significant results compared to articles with non-significant results; (2) reviewers give more favourable reviews to articles with significant results compared to articles with non-significant results; and (3) editors are more likely to accept for publication articles with significant results compared to articles with non-significant results. Evidence on differences in the relative contributions to publication bias by authors, reviewers, and editors is ambiguous.

This study reports an independent replication of the findings presented by Smaldino and McElreath (Smaldino, McElreath 2016 R. Soc. Open Sci. 3 , 160384 ( doi:10.1098/rsos.160384 )). The replication was successful with one exception. We find that selection acting on scientist’s propensity for replication frequency caused a brief period of exuberant replication not observed in the original paper due to a coding error. This difference does not, however, change the authors’ original conclusions. We call for more replication studies for simulations as unique contributions to scientific quality assurance.

The rules and procedures regulating the admission of potentially unreliable expert evidence have been substantially weakened over the past several years. We respond to this trend by focusing on one aspect of the rules that has not been explicitly curtailed: unfair prejudice. Unfair prejudice is an important component of trial judges’ authority to exclude evidence, which they may do when that unfair prejudice outweighs the evidence’s probative value. We develop the concept of unfair prejudice by first examining how it has been interpreted by judges and then relating that to the relevant social scientific research on the characteristics of expertise that can make it prejudicial. In doing so, we also discuss the research behind a common reason that judges admit expert evidence despite its prejudice, which is that judicial directions help jurors understand and weigh it. As a result, this article provides two main contributions. First, it advances knowledge about unfair prejudice, which is an important part of expert evidence law that has received relatively little attention from legal researchers. Second, it provides guidance to practitioners for challenging expert evidence under one of the few avenues left to do so.

The push for open science has produced many initiatives that arguably improve the quality of science, such as preregistration of research methods, more general availability of scientific articles in the form of preprints and open access articles, and the availability of (meta)data and code. Open science can have direct benefits for the career prospects of the individual through higher quality of the work and increased chance of publication (Allen & Mehler, 2019). However, being an outspoken advocate of open science might also have indirect positive effects on individual scientific careers, in the form of status in a scientific community, decisions for tenure, and eligibility for certain kinds of funding. In light of these benefits, it may be profitable for individual labs to appear to engage in open science practices, without actually putting in the associated effort or doing only the bare minimum. In this article, we explore two types of academic behavior through a dynamic computational model (cf. Smaldino & McElreath, 2016) of an academic community that rewards open science: 1) actively practicing open science and/or 2) advocating open science. Crossing these two types of behavior leads to four different kinds of labs and we examine which of them thrive in this academic community. Implications of the model results are discussed.

The ongoing reproducibility crisis in psychology and cognitive neuroscience has sparked increasing calls to re-evaluate and reshape scientific culture and practices. Heeding those calls, we have recently launched the EEGManyPipelines project as a means to assess the robustness of electroencephalography (EEG) research in naturalistic conditions and experiment with an alternative model of conducting scientific research. 168 analyst teams, encompassing 397 individual researchers from 37 countries, independently analyzed the same unpublished, representative EEG dataset to test the same set of pre- defined hypotheses, and then provided their analysis pipelines and reported outcomes. Here, we lay out how large-scale scientific projects can be set up in a grass-roots, community-driven manner with- out a central organizing laboratory. We explain our recruitment strategy, our guidance for analysts, the eventual outputs of this project and how it might have a lasting impact on the field.

Whether intellectually gifted children have a greater emotional response when tested is still unclear. This may be due to the marked heterogeneity of this particular population, and the fact that most studies lack the power to reduce the noise associated with this heterogeneity. The present study examined the relationship between performance and emotional response in 468,423 Italian fifth-graders taking a national test on mathematics and language. Analyses were performed using statistical models with polynomial terms. Special attention was paid to estimating the mean emotional response of the children who were gifted (1.5-2.5 standard deviations above the mean) or highly gifted (more than 2.5 standard deviations above the mean). The results showed that, although a lower emotional response correlated with a higher achievement, this relationship is nonlinear, and the estimates for gifted and highly gifted children were virtually the same. Girls showed a greater emotional response than boys on all levels of performance. The theoretical and practical implications of these findings are discussed.

The preprint is about how participatory research movement and neurodiversity movement can benefit the open scholarship movement and vice versa, leading to a more generalisable and accurate science of human behaviour and cognition.

International large-scale assessments (ILSAs) in education provide researchers with opportunities to study educational phenomena in a large set of contexts, educational systems, domains, and student populations. This project evaluates this potential and develops and illustrates methodological approaches and guidance to incorporating ILSAs in meta-analyses.

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N = 227; 56.8%), and about one-fifth of studies contain changed hypotheses (N = 82; 19%). We found only a small number of studies with demoted hypotheses (N = 2; 1%) and no studies with promoted hypotheses. In all, 59% of studies include at least one hypothesis in one or more of these categories, indicating a substantial bias in presenting and selecting hypotheses by researchers and/or reviewers/editors. Contrary to our expectations, we found that added hypotheses and changed hypotheses were not more likely to be statistically significant than non-selectively reported hypotheses. For the other types of selective hypothesis reporting, no powerful test of the relationship with statistical significance could be carried out. Finally, we found that replication studies were less likely to include selectively reported hypotheses than original studies. Thus, selective hypothesis reporting is problematically common in psychological research and may partly be explained by the fact that authors do not specifically and clearly enough formulate hypotheses in preregistrations and papers. We urge researchers, reviewers, and editors to ensure that hypotheses outlined in preregistrations are clearly formulated and accurately presented in the corresponding papers.

A large number of systematic replication attempts in the social sciences have failed to support the claims in the original studies. These surprising results have inspired a body of metascientific research aimed at understanding these failures to replicate and ensuring future research is credible. In this article, we relate these new insights from metascience to empirical legal research. Specifically, a recent effort to replicate three influential empirical legal studies published in law journals found results that diverged from the originals. The replicators suggested their results were caused by the changing social context and did not explicitly consider whether the original effects were overstated or false positives. We re-analysed the data from the replications to attempt to confirm their results (i.e., computational reproducibility) with mixed success. When possible, we combined the data from the replications (i.e., meta-analysis) to leverage the greater precision that comes with large sample sizes. In one case, we found an effect where the replicators did not. Overall, however, our re-analysis and review of the broader social scientific context suggests that empirical legal research suffers from the same challenges plaguing other fields – small sample sizes and undisclosed flexibility have produced untrustworthy results.

ARTICLE PUBLISHED (OPEN ACCESS) IN BRITISH JOURNAL OF SOCIAL PSYCHOLOGY. Qualitative data sharing practices in psychology have not developed as rapidly as those in parallel quantitative domains. This is often explained by numerous epistemological, ethical, and pragmatic issues concerning qualitative data types. In this essay, I provide an alternative to the frequently expressed (often reasonable) concerns regarding the sharing of qualitative human data by highlighting three advantages of qualitative data sharing. I argue that sharing qualitative human data is not by default “less ethical,” “riskier,” and “impractical” compared to quantitative data sharing, but in some cases more ethical, less risky, and easier to manage for sharing because 1) informed consent can be discussed, negotiated, and validated, 2) the shared data can be curated by special means, and 3) the privacy risks are mainly local instead of global. I hope the provided alternative perspective further encourages qualitative psychologists to share their data when it is epistemologically, ethically, and pragmatically possible.

A considerable proportion of psychological research has not been replicable, and estimates range from 9% to 77% for nonreplicable results. The extent to which vast proportions of studies in the field are replicable is still unknown, as researchers lack incentives for publishing individual replication studies. When preregistering replication studies via the Open Science Foundation website (OSF, osf.io), researchers can publicly register their results without having to publish them and thus circumvent file-drawer effects. We analyzed data from 139 replication studies for which the results were publicly registered on the OSF and found that out of 62 reports that included the authors’ assessments, 23 were categorized as “informative failures to replicate” by the original authors. 24 studies allowed for comparisons between the original and replication effect sizes, and whereas 75% of the original effects were statistically significant, only 30% of the replication effects were. The replication effects were also significantly smaller than the original effects (approx. 38% the size). Replication closeness did not moderate the difference between the original and the replication effects. Our results provide a glimpse into estimating replicability for studies from a wide range of psychological fields chosen for replication by independent groups of researchers. We invite researchers to browse the Replication Database (ReD) ShinyApp, which we created to check for whether seminal studies from their respective fields have been replicated. Our data and code are available online: https://osf.io/9r62x/

Promoting the economic empowerment of women requires more than just helping them get a foot in the door and hoping for the best. We need women to remain in the workplace, to thrive in their work environments, job descriptions and careers, and to derive the self-efficacy and emotional fulfillment that comes from work. To achieve this, we need to recognize that few workplaces are gender neutral. Instead, they harbor norms and protocols that accommodate male preferences and emotional needs, while neglecting those of women. This bias will push women out or undermine their success. Unfortunately, we feel that the international development agenda has not paid sufficient attention to the importance of female workplace retention or the emotional toil that mismatched norms can inflict. This research agenda hopes to address this failing by outlining ways we can identify, understand, and correct norms and patterns that make it difficult for women to continue working. We feel that this is particularly important in many workplaces of the Global South, where norms prevail over formal procedures, and it will become more relevant as businesses in LICs formalize and establish organizational rules.

In the wake of the replication crisis, there have been calls to increase the clarity and precision of theory in the social sciences. Here, we argue that the effects of these calls may be limited due to systematic and structural factors, and focus our attention on incentives favoring ambiguous theory. Intentionally or not, scientists can exploit theoretical ambiguities to make support for a claim appear stronger than is warranted. Practices include ‘theory stretching’, interpreting an ambiguous claim more expansively to absorb data outside of the scope of the original claim, and ‘post-hoc precision’, interpreting an ambiguous claim more narrowly so it appears more precisely aligned with the data. These practices lead to the overestimation of evidence for the original claim and create the appearance of consistent support and progressive research programs, which may in turn be rewarded by journals, funding agencies, and hiring committees. Selection for ambiguous research can occur outside of scientists’ awareness and even when scientists act in good faith. We supplement our verbal argument with a simple mathematical model. Although ambiguity might be inevitable or even useful in the early stages of theory construction, scientists should aim for increased clarity as knowledge advances. Science benefits from transparently communicating about known ambiguities. To attain transparency about ambiguity, we provide a set of recommendations for authors, reviewers, and journals. We conclude with suggestions for research on how scientists use strategic ambiguity to advance their careers and on the ways in which norms, incentives, and practices favor strategic ambiguity.

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices; (4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.Wagenmakers and colleagues describe seven statistical procedures that increase transparency in data analysis. These procedures highlight common ground among data analysts from different schools and find inspiration in Merton's ethos of science.

Online forums with multiple users and groups are generally known for the lack of coherent patterns in their content. They have hundreds of thousands of members, most of them anonymous. The content that is posted on a regular basis are also fairly random. This is especially true for mega forums like Reddit where each of the sub Reddits (subs) themselves often have millions of members. While the subs individually would function as chat groups or niche forums, at a platform level, that is at the level of Reddit itself, it is almost impossible to foresee any pattern coming out of the interactions of millions of users of the thousands of subs. Yet, on a sub called \textbf{r/place}, we are witnessing a steady decline of randomness. Patterns are emerging and even without the use of mathematical formulae, even with the lay eye it is feasible to understand how the patterns are being formed. In fact, that actual patterns are coming up is itself nothing short of a miracle, or an exception.

Forecasting the reproducibility of research findings is one of the key challenges of Metascience. To date, reliable above-chance predictions have mainly been achieved by pooling the subjective ratings of experts through surveys or via prediction markets. Obtaining such data is laborious, however, and unlikely to increase the theoretical understanding of reproducibility.Here we show that empirical measures of K, a principled metric of knowledge, are correlated with reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), an ongoing replication of 60 studies that use three common methodologies in the life sciences. For each study, we calculated the K value by dividing the effect size, measured in bits of Shannon entropy explained, by the descriptive complexity of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol.K values were statistically associated with subjective predictions about studies’ replication probabilities and relative effect sizes. This relation was robust to controlling for study methodology, and was a stronger predictor than other plausible covariates, including the originally reported effect size, subjective ratings of a study’s methodological difficulty, and raters’ self-reported expertise. The superior fit of K may be due to its particular structure and metrics, which are rooted in information theory and may more accurately reflect the implicit calculations made by raters.This is the first evidence that an objective metric hypothesized to quantify scientific knowledge captures subjective judgments about reproducibility. This finding gives the first independent support of K’s underlying assumption that scientific knowledge is a process of information compression, and it may offer a new tool for metaresearch.

Changes in statistical practices and reporting have been documented by Giofrè et al. PLOS ONE 12(4), e0175583 (2017), who investigated ten statistical and open practices in two high-ranking journals (Psychological Science [PS] and Journal of Experimental Psychology-General [JEPG]): null hypothesis significance testing; confidence or credible intervals; meta-analysis of the results of multiple experiments; confidence interval interpretation; effect size interpretation; sample size determination; data exclusion; data availability; materials availability; and preregistered design and analysis plan. The investigation was based on an analysis of all papers published in these journals between 2013 and 2015. The aim of the present study was to follow up changes in both PS and JEPG in subsequent years, from 2016 to 2020, adding code availability as a further open practice. We found improvement in most practices, with some exceptions (i.e., confidence interval interpretation and meta-analysis). Despite these positive changes, our results indicate a need for further improvements in statistical practices and adoption of open practices.

What drives public beliefs about the credibility of a scientific field? This question is increasingly important, with recent discussion of a “reproducibility crisis” affecting many fields. Such discussions are vital in forensic science, a discipline that has experienced severe scrutiny from both the media and large oversight bodies. In this paper, we make three contributions to this discussion. First, we bring together and compare several studies in which laypeople were asked about the reliability of forensic science practices. This review suggests that forensic practices do not enjoy uniformly high reliability ratings from the public and these ratings are not calibrated with the scientific consensus. We then review three empirically-tested ways that other fields are dealing with their own crises, all centred around transparency and openness. Finally, we make recommendations for how forensic science can leverage transparency and openness to improve and maintain its long-term credibility. As part of these recommendations, we find that empirical research supports the Houston Forensic Science Center’s recent claims that it has improved its credibility through openness and transparency.

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

There is a great need for foster carers in Sweden and the number of existing foster homes is not enough for everyone who needs it. Many young people are therefore placed in institutions instead despite current evidence suggesting that the long-term prospects for institutional care are worse than placement in foster homes. Prospective foster homes undergo an investigation where their suitability is assessed. The most common Swedish method is the New Kälvesten method (Willén & Kronkvist, 2021). The method was scientifically reviewed for the first time by Willén and Kronkvist who found it unethical with a high risk of subjectivity. At the same time, an underuse can be noted of the evidence-based Swedish methods Familjehemsvinjetter and BRA-fam. International research has also shown an underutilized potential regarding the application of psychological measurements in assessments of prospective foster homes. The current project intends to develop and test an investigation model consisting of several individual methods (e.g. BRA-fam and Familjehemsvinjetter), each of which is supported by scientific research. Study 1 examines and tests the potential for application of psychological instruments (e. g. personality tests). Study 2 develops and tests a structured assessment model based on the results from Study 1 and implementing knowledge derived from the scientific field of investigative psychology and investigative practical contexts (e.g. asylum, insurance and criminal investigations). A handbook and project website will be developed as part of the investigative model. Study 2 also prepares for long-term follow-up of approved families. Both studies will be ethically reviewed and preregistered before data collection. Anonymised datasets will be made freely available as well as the project's research and training materials.

Meta-analytic models for dependent effect sizes have grown increasingly sophisticated over the last few decades, which has created challenges for a priori power calculations. We introduce power approximations for tests of average effect sizes based upon several common approaches for handling dependent effect sizes. In a Monte Carlo simulation, we show that the new power formulas can accurately approximate the true power of meta-analytic models for dependent effect sizes. Lastly, we investigate the Type I error rate and power for several common models, finding that tests using robust variance estimation provide better Type I error calibration than tests with model-based variance estimation. We consider implications for practice with respect to selecting a working model and an inferential approach.

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings. ispartof: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA vol:119 issue:44 ispartof: location:United States status: published

The events of recent years have made especially plain the inequities between Global North and South. One such inequality is in the way some social groups are often missing from the data, so that knowledge, policies, services and products do not take proper account of them. This bias exists strongly in the conceptual development of the behavioral sciences, and it makes the research base fragile, with papers making claims of universality that do not stand up to scrutiny. In order to deal with concerns about replication, external validity and the strength of this research base, there are now many initiatives to improve experimentation in the social sciences. However, truly generalizable findings come about when institutions are present in the long term to aggregate across studies. That works even better when those institutions have a deep understanding of the contexts in which they seek to generalise from and to. Busara was founded as an institution to apply context-specific behavioral science literature, and this work has been a constant throughout our history. We now propose to launch a structured three-year Open Science investigation of the gaps in the understanding of canonical patterns of behavior, cognitive processes, preferences, beliefs, and decision-making processes in the Global South, comparing our work in multiple contexts and exploring variance across time, place, and demography. At the conclusion of this project, we will begin integrating our findings into wider theories of global cultural, psychological and microeconomic heterogeneity.

Changes in statistical practices and reporting have been documented by Giofrè et al. (2017), who investigated ten statistical and open practices in two high ranking journals (Psychological Science [PS] and Journal of Experimental Psychology-General [JEPG]): These included null hypothesis significance testing; confidence or credible intervals; meta-analysis of the results of multiple experiments; confidence intervals interpretation; effect size interpretation; sample size determination; sample size stopping rule; data availability; materials availability; and preregistered design and analysis plan.The investigation was based on an analysis of all papers published in these journals between 2013 and 2015.The aim of the present study is to follow up changes in both PS and JEPG in subsequent years, from 2016 to 2020, adding code availability as further open practice.

Academic publishing has changed substantially in the past 30 years due to the advent of the internet. Unlike print publications, digital publications have the ability to provide additional information to visitors about the publication and previous readers via metadata like download counts. This study investigated the effect of this metadata on the development of download inequality and unpredictability of success in an experimental academic literature marketplace. We found that presence of an accurate download count increased inequality in article downloads, meaning fewer papers accumulated a larger share of the total download count. We also found that the presence of download count increased the unpredictability of success, meaning across identical instances, different papers became the most popular. Finally, an exploratory analysis found papers were more highly rated when download counts were present. Together, these findings provide insight into how the download behaviors of previous academic readers may influence literature choice.

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Research synthesis is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can be debated. Yet this probability is not only a function of recurring results, but also of the quality and consistency in the empirical procedures that produced those results and that any meta-analysis necessarily inherits. Standardized protocols in data collection, analysis, and interpretation are important empirical properties, and a healthy sign of a discipline's maturity.This manuscript proposes that meta-analyses as typically applied in psychology benefit from complementing their aggregates of observed effect sizes by systematically examining the standardization of methodology that deterministically produced them. Potential units of analyses are described and two examples are offered to illustrate the benefits of such efforts. Ideally, this synergetic approach emphasizes the role of methods in advancing theory by improving the quality of meta-analytic inferences.

BACKGROUND. Experimental vignette research methods have been used to study a diverse range of theoretical and practical issues. Vignettes are designed to create hypothetical cultural/normative contexts for the study of variation in self-reported attitudes. A key problem in such research, however, is potential social desirability response bias. METHOD. A vignette experimental test of an hypothesis derived from a dual-process theory (the MODE framework initially developed by Fazio) linking explicit vs. implicit self-reported attitude measurement and social desirability response bias is reported here. RESULTS. The data show that measuring the social approval of a central vignette character explicitly results in greater social desirability responding than measuring such approval implicitly, supporting MODE theory. CONCLUSIONS. Vignette research methodologies provide a rich, flexible toolkit for studying many important social psychological topics, including issues of inequality and equity. However, researchers can and should design a measurement strategy that carefully manages inferences drawn in light of conditions likely to produce social desirability response bias.

This paper explores judgements about the replicability of social and behavioural sciences research, and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited using a structured iterative approach for eliciting judgements from groups, called the IDEA protocol (‘Investigate’, ‘Discuss’, ‘Estimate’ and ‘Aggregate’). Five groups of five people separately assessed the replicability of 25 ‘known-outcome’ claims. That is, social and behavioural science claims that have already been subject to at least one replication study. Specifically, participants assessed the probability that each of the 25 research claims will replicate (i.e. a replication study would find a statistically significant result in the same direction as the original study). In addition to their quantitative judgements, participants also outlined the reasoning behind their judgements. To start, we quantitatively analysed some possible correlates of predictive accuracy, such as self-rated understanding and expertise in assessing each claim, and updating of judgements after feedback and discussion. Then we qualitatively analysed the reasoning data (i.e., the comments and justifications people provided for their judgements) to explore the cues and heuristics used, and features of group discussion that accompanied more and less accurate judgements.

Outcome reporting bias (ORB) refers to the biasing effect caused by researchers selectively reporting outcomes based on their statistical significance. ORB leads to inflated average effect size estimates in a meta-analysis if only the outcome with the largest effect size is reported due to ORB. We propose a new method (CORB) to correct for ORB that includes an estimate of the variability of the outcomes' effect size as a moderator in a meta-regression model. An estimate of the variability of the outcomes' effect size can be computed by assuming a correlation among the outcomes. Results of a Monte-Carlo simulation study showed that effect size in meta-analyses may be severely overestimated without any correction for ORB. The CORB method accurately estimates effect size when overestimation caused by ORB is the largest. Applying the new method to a meta-analysis on the effect of playing violent video games on aggressive cognition showed that the average effect size estimate decreased when correcting for ORB. We recommend to routinely apply methods to correct for ORB in any meta-analysis. We provide annotated R code and functions to facilitate researchers to apply the CORB method.

Previous work suggested that humans’ sophisticated speech perception abilities stem from an early capacity to pay attention to speech in the auditory environment. Previous studies have therefore tested if infants prefer speech to other sounds at a variety of ages, but provided contrasted results. In this paper, we make the hypothesis that speech is initially encoded similarly to other natural or vocal sounds, and that infants tune to speech during the first year of life as they acquire their native language. To test this hypothesis, we conducted a meta-analysis of experiments testing speech preference in infants, sorting experiments by whether they used native or foreign speech on the one hand, and vocal or non-vocal, natural or artificial sound on the other hand. Synthesizing data from 775 infants across 38 experiments, we found a medium effect size, confirming at the scale of the literature that infants reliably prefer speech over other sounds. However, this preference was not significantly moderated by the language used, nor vocal quality, or naturalness of the competitor. Strinkingly, we found no effect of age: infants showed the same strength of preference throughout the first year of life. Speech therefore appears to be preferred from birth, even to other natural or vocal sounds. These results suggest that speech is processed in a specific way by an innate dedicated system, dictinct from other sounds processing.

Visual short-term memory (VSTM) links perception with higher cognitive processes by maintaining visual information that is absent from the environment. Yet, it remains unclear if sensory visual cortex is a necessary component of the brain network that underlies short-term maintenance of visual information. Previous reviews remain inconclusive and open to interpretation. Here, we aimed to systematically identify and review studies that have investigated the role of the sensory visual cortex in VSTM using transcranial magnetic stimulation (TMS), a method that allows exploration of causal relationships, and to quantitatively explore the effect of TMS interference on the sensory visual cortex during VSTM using meta-analytic methodology. Thirteen studies were identified and qualitatively reviewed. Out of those, seven studies provided sufficient statistical data for meta-analysis and yielded a total of 30 effect sizes, which were included in the meta-analyses. Two meta-analyses were conducted, one regarding the encoding phase of VSTM (19 effect sizes), and one regarding the maintenance phase of VSTM (11 effect sizes). The results from the systematic review and the two meta-analyses indicate that the sensory visual cortex is likely involved in both the encoding and maintenance phase of VSTM. In some cases, evidence did not show significant effects of TMS, however, this is suggested to be due to low memory load or low perceptual task demands. Overall, these findings support the idea that sensory visual areas are part of the brain network responsible for successfully maintaining information in short-term memory when no physical stimulus is present in the environment.

We explore the promise of statistical reform by starting from the assumption that most researchers would endorse Mertons ethos of science as reflected in the four norms of communalism, universalism, disinterestedness, and organized skepticism. Translated to data analysis, these norms imply a need for transparency, a fair acknowledgement of uncertainty, and openness to alternative interpretations. We discuss seven statistical procedures, both old and new, that we believe can positively impact statistical practice in the social and behavioral sciences.

Replication studies that contradict prior findings may facilitate scientific self-correction by triggering a reappraisal of the original studies; however, the research community’s response to replication results has not been studied systematically. One approach for gauging responses to replication results is to examine how they affect citations to original studies. In this study, we explored postreplication citation patterns in the context of four prominent multilaboratory replication attempts published in the field of psychology that strongly contradicted and outweighed prior findings. Generally, we observed a small postreplication decline in the number of favorable citations and a small increase in unfavorable citations. This indicates only modest corrective effects and implies considerable perpetuation of belief in the original findings. Replication results that strongly contradict an original finding do not necessarily nullify its credibility; however, one might at least expect the replication results to be acknowledged and explicitly debated in subsequent literature. By contrast, we found substantial citation bias: The majority of articles citing the original studies neglected to cite relevant replication results. Of those articles that did cite the replication but continued to cite the original study favorably, approximately half offered an explicit defense of the original study. Our findings suggest that even replication results that strongly contradict original findings do not necessarily prompt a corrective response from the research community.