Search

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Sherman and Rivers (this issue) summarize conceptual and definitional problems with the term “social priming” and identify potential characterizations that distinguish replicable and more questiona...

Additional co-authors: Ivan Ropovik, Balazs Aczel, Lena F. Aeschbach, Luca Andrighetto, Jack D. Arnal, Holly Arrow, Peter Babincak, Bence E. Bakos, Gabriel Banik, Ernest Baskin, Radomir Belopavlovic, Michael H. Bernstein, Michal Bialek, Nicholas G. Bloxsom, Bojana Bodroža, Diane B. V. Bonfiglio, Leanne Boucher, Florian Bruhlmann, Claudia C. Brumbaugh, Erica Casini, Yiling Chen, Carlo Chiorri, William J. Chopik, Oliver Christ, Antonia M. Ciunci, Heather M. Claypool, Sean Coary, Marija V. Cˇolic, W. Matthew Collins, Paul G. Curran, Chris R. Day, Anna Dreber, John E. Edlund, Filipe Falcao, Anna Fedor, Lily Feinberg, Ian R. Ferguson, Maire Ford, Michael C. Frank, Emily Fryberger, Alexander Garinther, Katarzyna Gawryluk, Kayla Ashbaugh, Mauro Giacomantonio, Steffen R. Giessner, Jon E. Grahe, Rosanna E. Guadagno, Ewa Halasa, Rias A. Hilliard, Joachim Huffmeier, Sean Hughes, Katarzyna Idzikowska, Michael Inzlicht, Alan Jern, William Jimenez-Leal, Magnus Johannesson, Jennifer A. Joy-Gaba, Mathias Kauff, Danielle J. Kellier, Grecia Kessinger, Mallory C. Kidwell, Amanda M. Kimbrough, Josiah P. J. King, Vanessa S. Kolb, Sabina Kolodziej, Marton Kovacs, Karolina Krasuska, Sue Kraus, Lacy E. Krueger, Katarzyna Kuchno, Caio Ambrosio Lage, Eleanor V. Langford, Carmel A. Levitan, Tiago Jesse Souza de Lima, Hause Lin, Samuel Lins, Jia E. Loy, Dylan Manfredi, Łukasz Markiewicz, Madhavi Menon, Brett Mercier, Mitchell Metzger, Venus Meyet, Jeremy K. Miller, Andres Montealegre, Don A. Moore, Rafal Muda, Gideon Nave, Austin Lee Nichols, Sarah A. Novak, Christian Nunnally, Ana Orlic, Anna Palinkas, Angelo Panno, Kimberly P. Parks, Ivana Pedovic, Emilian Pekala, Matthew R. Penner, Sebastiaan Pessers, Boban Petrovic, Thomas Pfeiffer, Damian Pienkosz, Emanuele Preti, Danka Puric, Tiago Ramos, Jonathan Ravid, Timothy S. Razza, Katrin Rentzsch, Juliette Richetin, Sean C. Rife, Anna Dalla Rosa, Kaylis Hase Rudy, Janos Salamon, Blair Saunders, Przemyslaw Sawicki, Kathleen Schmidt, Kurt Schuepfer, Thomas Schultze, Stefan Schulz-Hardt, Astrid Schutz, Ani N. Shabazian, Rachel L. Shubella, Adam Siegel, Ruben Silva, Barbara Sioma, Lauren Skorb, Luana Elayne Cunha de Souza, Sara Steegen, L. A. R. Stein, R. Weylin Sternglanz, Darko Stojilovic, Daniel Storage, Gavin Brent Sullivan, Barnabas Szaszi, Peter Szecsi, Orsolya Szoke, Attila Szuts, Manuela Thomae, Natasha D. Tidwell, Carly Tocco, Ann-Kathrin Torka, Francis Tuerlinckx, Wolf Vanpaemel, Leigh Ann Vaughn, Michelangelo Vianello, Domenico Viganola, Maria Vlachou, Ryan J. Walker, Sophia C. Weissgerber, Aaron L. Wichman, Bradford J. Wiggins, Daniel Wolf, Michael J. Wood, David Zealley, Iris Žeželj, Mark Zrubka, and Brian A. Nosek

This meta-analysis evaluated theoretical predictions from balanced identity theory (BIT) and evaluated the validity of zero points of Implicit Association Test (IAT) and self-report measures used to test these predictions. Twenty-one researchers contributed individual subject data from 36 experiments (total N = 12,773) that used both explicit and implicit measures of the social–cognitive constructs. The meta-analysis confirmed predictions of BIT’s balance–congruity principle and simultaneously validated interpretation of the IAT’s zero point as indicating absence of preference between two attitude objects. Statistical power afforded by the sample size enabled the first confirmations of balance–congruity predictions with self-report measures. Beyond these empirical results, the meta-analysis introduced a within-study statistical test of the balance–congruity principle, finding that it had greater efficiency than the previous best method. The meta-analysis’s full data set has been publicly archived to enable further studies of interrelations among attitudes, stereotypes, and identities.

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or U.S. versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

This manuscript contains our responses to several commentaries about the Many Labs Project (Klein et al., 2014).

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary – the replication was either a success or a failure – and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

UNSTRUCTURED REMOVE

Sharing clinical trial data can provide value to research participants and communities by accelerating the development of new knowledge and therapies as investigators merge data sets to conduct new analyses, reproduce published findings to raise standards for original research, and learn from the work of others to generate new research questions. Nonprofit funders, including disease advocacy and patient-focused organizations, play a pivotal role in the promotion and implementation of data sharing policies. Funders are uniquely positioned to promote and support a culture of data sharing by serving as trusted liaisons between potential research participants and investigators who wish to access these participants' networks for clinical trial recruitment. In short, nonprofit funders can drive policies and influence research culture. The purpose of this paper is to detail a set of aspirational goals and forward thinking, collaborative data sharing solutions for nonprofit funders to fold into existing funding policies. The goals of this paper convey the complexity of the opportunities and challenges facing nonprofit funders and the appropriate prioritization of data sharing within their organizations and may serve as a starting point for a data sharing toolkit for nonprofit funders of clinical trials to provide the clarity of mission and mechanisms to enforce the data sharing practices their communities already expect are happening.

To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time. To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time.

Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.

Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In three studies, we show how ideological beliefs impair people’s ability to (1) recognize logical validity in arguments that oppose their political beliefs and (2) recognize the lack of logical validity in arguments that support their political beliefs. We observed belief bias effects among liberals and conservatives who evaluated the logical soundness of classically structured logical syllogisms supporting liberal or conservative beliefs. Both liberals and conservatives frequently evaluated the logical structure of entire arguments based on the believability of arguments’ conclusions, leading to predictable patterns of logical errors. As a result, liberals were better at identifying flawed arguments supporting conservative beliefs and conservatives were better at identifying flawed arguments supporting liberal beliefs. These findings illuminate one key mechanism for how political beliefs distort people’s abilities to reason about political topics soundly.

Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In seven preregistered studies (total N > 7,000), we investigated whether asking participants to avoid one social bias affected that and other social biases. Participants selected honor society applicants based on academic credentials. Applicants also differed on social categories irrelevant for selection: attractiveness and ingroup status. Participants asked to avoid potential bias in one social category showed small but reliable reductions in bias for that category ( r = .095), but showed near-zero bias reduction on the unmentioned social category ( r = .006). Asking participants to avoid many possible social biases or alerting them to bias without specifically identifying a category did not consistently reduce bias. The effectiveness of interventions for reducing social biases may be highly specific, perhaps even contingent on explicitly and narrowly identifying the potential source of bias.

The authors and the co-signatories of this open letter are scientists dedicated to transparency and integrity of research. We support the work needed to investigate potential errors and possible misconduct and believe the scientific community can do more to protect whistleblowers against harassment and threats. Individual researchers can provide vocal support for whistleblowers and against harassment to shift norms. Journals, funders, policymakers, and institutions can make explicit policies that protect whistleblowers and establish fair, judicious, and transparent processes for addressing potential misconduct that protect all participants. With this letter, we show our support for post-publication peer-review, to Elisabeth Bik and her work, and to all the whistleblowers that help maintain quality, honesty, integrity and trustworthiness of scientific advances.

Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if there were rapid, scalable, accurate credibility indicators to guide attention and resource allocation for further assessment. The SCORE program is creating and validating algorithms to provide confidence scores for research claims at scale. To investigate the viability of scalable tools, teams are creating: a database of claims from papers in the social and behavioral sciences; expert and machine generated estimates of credibility; and, evidence of reproducibility, robustness, and replicability to validate the estimates. Beyond the primary research objective, the data and artifacts generated from this program will be openly shared and provide an unprecedented opportunity to examine research credibility and evidence.

We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.

In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs such as reducing novelty. Here, 353 researchers peer reviewed a pair of papers from 29 published RRs from psychology and neuroscience and 57 non-RR comparison papers. RRs numerically outperformed comparison papers on all 19 criteria (mean difference 0.46, scale range -4 to +4) with effects ranging from RRs being statistically indistinguishable from comparison papers in novelty (0.13, 95% credible interval [-0.24, 0.49]) and creativity (0.22, [-0.14, 0.58]) to sizeable improvements in rigour of methodology (0.99, [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]) and overall paper quality (0.66, [0.30, 1.02]). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature.

Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised protocols in multiple laboratories (Median number of laboratories per original study = 6.5; Range 3 to 9; Median total sample = 1279.5; Range 276 to 3512) for high-powered tests of each original finding with both protocols. Overall, Revised protocols produced similar effect sizes as RP:P protocols following the preregistered analysis plan (Δr = .002 or .014, depending on analytic approach). The median effect size for Revised protocols (r = .05) was similar to RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than the original studies (r = .37). The cumulative evidence of original study and three replication attempts suggests that effect sizes for all 10 (median r = .07; range .00 to .15) are 78% smaller on average than original findings (median r = .37; range .19 to .50), with very precisely estimated effects.

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigor-enhancing practices: confirmatory tests, large sample sizes, preregistration, and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (p

UNSTRUCTURED Sharing clinical trial data can provide value to research participants and communities by accelerating the development of new knowledge and therapies as investigators merge data sets to conduct new analyses, reproduce published findings to raise standards for original research, and learn from the work of others to generate new research questions. Nonprofit funders, including disease advocacy and patient-focused organizations, play a pivotal role in the promotion and implementation of data sharing policies. Funders are uniquely positioned to promote and support a culture of data sharing by serving as trusted liaisons between potential research participants and investigators who wish to access these participants’ networks for clinical trial recruitment. In short, nonprofit funders can drive policies and influence research culture. The purpose of this paper is to detail a set of aspirational goals and forward thinking, collaborative data sharing solutions for nonprofit funders to fold into existing funding policies. The goals of this paper convey the complexity of the opportunities and challenges facing nonprofit funders and the appropriate prioritization of data sharing within their organizations and may serve as a starting point for a data sharing toolkit for nonprofit funders of clinical trials to provide the clarity of mission and mechanisms to enforce the data sharing practices their communities already expect are happening.

A hallmark of science is the open exchange of knowledge. At this time of crisis, it is more important than ever for scientists around the world to openly share their knowledge, expertise, tools, and technology. Scientific models are critical tools for anticipating, predicting, and responding to complex biological, social, and environmental crises, including pandemics. They are essential for guiding regional and national governments in designing health, social, and economic policies to manage the spread of disease and lessen its impacts. However, presenting modeling results alone is not enough. Scientists must also openly share their model code so that the results can be replicated and evaluated. Given the necessity for rapid response to the coronavirus pandemic, we need many eyes to review and collectively vet model assumptions, parameterizations, and algorithms to ensure the most accurate modeling possible. Transparency engenders public trust and is the best defense against misunderstanding, misuse, and deliberate misinformation about models and their results. We need to engage as many experts as possible for improving the ability of models to represent epidemiological, social, and economic dynamics so that we can best respond to the crisis and plan effectively to mitigate its wider impacts. We strongly urge all scientists modeling the coronavirus disease 2019 (COVID-19) pandemic and its consequences for health and society to rapidly and openly publish their code (along with specifying the type of data required, model parameterizations, and any available documentation) so that it is accessible to all scientists around the world. We offer sincere thanks to the many teams that are already sharing their models openly. Proprietary black boxes and code withheld for competitive motivations have no place in the global crisis we face today. As soon as possible, please place your code in a trusted digital repository ([ 1 ][1]) so that it is findable, accessible, interoperable, and reusable ([ 2 ][2]). 1. [↵][3]CoMSES Network, Trusted Digital Repositories ([www.comses.net/resources/trusted-digital-repositories/][4]). 2. [↵][5]1. M. D. Wilkinson et al ., Sci. Data 3, 160018 (2016). [OpenUrl][6] Correction (1 May 2020): A second affiliation was added for J.-A.A., and the competing interests declarations were updated. All authors have signed on behalf of the listed organizations only. E.B. advises but does not represent the UK Engineering and Physical Sciences Research Council (EPSRC). B.F. is affiliated with but does not represent the Advanced Systems Analysis Program at the International Institute for Applied Systems Analysis in Austria. P.L.M. is a paid consultant for Robert Wood Johnson Foundation Systems for Action (S4A), chair of the board of directors for Computer Simulation and Advanced Research Technologies, and an advisory board member of Systems Science in Public Health and Economics Research, but she does not represent these organizations. [1]: #ref-1 [2]: #ref-2 [3]: #xref-ref-1-1 "View reference 1 in text" [4]: http://www.comses.net/resources/trusted-digital-repositories/ [5]: #xref-ref-2-1 "View reference 2 in text" [6]: {openurl}?query=rft.jtitle%253DSci.%2BData%26rft.volume%253D3%26rft.spage%253D160018%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.

Summary of post-event evaluation (survey and participant interviews) of Metascience 2019 Symposium (https://www.metascience2019.org/), presented by Fetzer Franklin Fund (https://www.fetzer-franklin-fund.org/) September 5th-8th, 2019 at Stanford University.

Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research. This definition reduces emphasis on operational characteristics of the study and increases emphasis on the interpretation of possible outcomes. The purpose of replication is to advance theory by confronting existing understanding with new evidence. Ironically, the value of replication may be strongest when existing understanding is weakest. Successful replication provides evidence of generalizability across the conditions that inevitably differ from the original study; Unsuccessful replication indicates that the reliability of the finding may be more constrained than recognized previously. Defining replication as a confrontation of current theoretical expectations clarifies its important, exciting, and generative role in scientific progress., What is replication? This Perspective article proposes that the answer shifts the conception of replication from a boring, uncreative, housekeeping activity to an exciting, generative, vital contributor to research progress.

Preprints (early, complete versions of manuscripts made available online before journal-organized peer review) are shifting the scholarly publishing model by accelerating open access and, potentially, open review. Making these tools interoperable with preprint infrastructure will increase the confidence of the scientific community in preprints. In addition, it will influence commercial publishing services to embrace new business models to innovate towards openness.OSF Preprints is open-source software maintained by the Center for Open Science (COS). OSF Preprints hosts 26 community-run services, providing the ideal conditions to integrate with open review platforms such as Peer Community In (PCI), PREreview, and Hypothes.is and to assess whether a fully open model can compete with and disrupt scholarly publishing across disciplines. We integrated hypothes.is and now propose to add the other platforms.With support from the Alfred P. Sloan Foundation, we are investigating how to improve trust in preprints with signals of credibility to improve chances that an open model can innovate scholarly publishing. Making those signals standard, interoperable, and machine-readable with validation will facilitate adoption and impact. We propose to build open-source open science badges for preprints to be interoperable with other scholarly publishing platforms.In 2020, COS advanced its sustainability plan for the preprints infrastructure with a distributed cost model for the shared infrastructure. Six of the services primarily serve under-resourced scholarly communities in the developing world. We request the costs for maintenance of those services for 2020 to extend their opportunity to develop institutional support for sustaining their services.

Significance The ethical practice of research requires researchers to give reasons and justifications for their actions, both to the other members of their research team as well as to external audiences. We developed a project-based training curriculum intended to make ethics discourse a routine practice in university science and engineering laboratories. Here, we report the results of a randomized control trial implemented among science and engineering laboratories in two research-intensive institutions. We demonstrate that, compared with the control laboratories, treatment laboratory members perceived improvements in the quality of discourse on research ethics within their laboratories as well as enhanced awareness of the relevance and reasons for that discourse for their work as measured in surveys administered 4 mo after the intervention., We report a randomized trial of a research ethics training intervention designed to enhance ethics communication in university science and engineering laboratories, focusing specifically on authorship and data management. The intervention is a project-based research ethics curriculum that was designed to enhance the ability of science and engineering research laboratory members to engage in reason giving and interpersonal communication necessary for ethical practice. The randomized trial was fielded in active faculty-led laboratories at two US research-intensive institutions. Here, we show that laboratory members perceived improvements in the quality of discourse on research ethics within their laboratories and enhanced awareness of the relevance and reasons for that discourse for their work as measured by a survey administered over 4 mo after the intervention. This training represents a paradigm shift compared with more typical module-based or classroom ethics instruction that is divorced from the everyday workflow and practices within laboratories and is designed to cultivate a campus culture of ethical science and engineering research in the very work settings where laboratory members interact.

Being able to replicate scientific findings is crucial for scientific progress1,2,3,4,5,6,7,8,9,10,11,12,13,14,15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that researchers in special education adopt emerging open-science reforms, such as preprints, data and materials sharing, preregistration of studies and analysis plans, and Registered Reports. Adoption of these practices will require shifts in cultural norms, guidelines, and incentives. We discuss how adopting open-science practices can advance the quality of research and, consequently, policy and practice in special education.

Many areas of social psychological research investigate how social information may bias judgment. However, most measures of social judgment biases are (1) low in reliability because they use a single response, (2) not indicative of individual differences in bias because they use between-subjects designs, (3) inflexible because they are designed for a particular domain, and (4) ambiguous about magnitude of bias because there is no objectively correct answer. We developed a measure of social judgment bias, the Judgment Bias Task, in which participants judge profiles varying in quality for a certain outcome based on objective criteria. The presence of ostensibly irrelevant social information provides opportunity to assess the extent to which social biases undermine the use of objective criteria in judgment. The JBT facilitates measurement of social judgment biases by (1) using multiple responses, (2) indicating individual differences by using within-subject designs, (3) being adaptable for assessing a variety of judgments, (4) identifying an objective magnitude of bias, and (5) taking 6 min to complete on average. In nine pre-registered studies (N > 9000) we use the JBT to reveal two prominent social judgment biases: favoritism towards more physically attractive people and towards members of one's ingroup. We observe that the JBT can reveal social biases, and that these sometimes occur even when the participant did not intend or believe they showed biased judgment. A flexible, objective, efficient assessment of social judgment biases will accelerate theoretical and empirical progress.

All disciplines should follow the geosciences and demand best practice for publishing and sharing data, argue Shelley Stall and colleagues. All disciplines should follow the geosciences and demand best practice for publishing and sharing data, argue Shelley Stall and colleagues.

Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised protocols in multiple laboratories (Median number of laboratories per original study = 6.5; Range 3 to 9; Median total sample = 1279.5; Range 276 to 3512) for high-powered tests of each original finding with both protocols. Overall, Revised protocols produced similar effect sizes as RP:P protocols following the preregistered analysis plan (Δr = .002 or .014, depending on analytic approach). The median effect size for Revised protocols (r = .05) was similar to RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than the original studies (r = .37). The cumulative evidence of original study and three replication attempts suggests that effect sizes for all 10 (median r = .07; range .00 to .15) are 78% smaller on average than original findings (median r = .37; range .19 to .50), with very precisely estimated effects.

The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research, primarily in association with the SBE Directorate. Reproducibility is the ability to obtain independent evidence supporting scientific findings. Recent investigations suggest that reproducibility of published findings is lower than expected or desired, thus interfering with research progress. COS’s Open Science Framework (OSF) enables rigorous, reproducible science by providing collaboration, registration, and data management support across the entire research lifecycle. With support from NSF 19-537, OSF will become essential mainstream infrastructure supporting researchers’, institutions’, and federal agencies’ priorities to maximize return on investment in research. Adoption of open science behaviors via OSF will enable rigor and reproducibility in harnessing the data revolution, increase accessibility and inclusivity of research for all, stimulate discovery, and accelerate progress in science. Seeded with private philanthropy in 2012, the open-source OSF’s user base and key performance indicators have shown rapid, non-linear growth since inception. OSF is now a popular, robust, and important infrastructure for registering studies and analysis plans; managing and archiving research data, code, and materials; and accelerating open or controlled sharing of research outcomes and all supporting research contents. This mid-scale NSF research infrastructure grant will build on OSF’s success by meeting four major objectives: 1. Expand OSF functionality and workflows to meet the needs of a broad number of research disciplines and communities; 2. Integrate OSF with other established scholarly infrastructure to make search, discovery, transfer, and preservation of scholarly content and metadata more efficient; 3. Manage OSF design for efficient and sustainable long-term service; 4. Provide training to improve effectiveness of registration and results reporting and ensure inclusivity of beneficiaries to these innovations.

In this manuscript, we (a) briefly describe proposed open-science practices to increase transparency of research in special education and related disciplines, and (b) provide recommendations for research funders, professional societies, journal editors and publishers, and individual researchers to support awareness, exploration, and adoption of open science.

Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open science (OS) collaborations comprise of a subset of open practices including open access publication, open data sharing and the absence of restrictive intellectual property rights with which institutions, firms, governments and communities are experimenting in order to overcome these concerns. We gathered two groups of international representatives from a large variety of stakeholders to construct a toolkit to guide and facilitate data collection about OS and non-OS collaborations. Ultimately, the toolkit will be used to assess and study the impact of OS collaborations on research and innovation. The toolkit contains the following four elements: 1) an annual report form of quantitative data to be completed by OS partnership administrators; 2) a series of semi-structured interview guides of stakeholders; 3) a survey form of participants in OS collaborations; and 4) a set of other quantitative measures best collected by other organizations, such as research foundations and governmental or intergovernmental agencies. We opened our toolkit to community comment and input. We present the resulting toolkit for use by government and philanthropic grantors, institutions, researchers and community organizations with the aim of measuring the implementation and impact of OS partnership across these organizations. We invite these and other stakeholders to not only measure, but to share the resulting data so that social scientists and policy makers can analyse the data across projects.

Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires good judgment and humility in making claims.

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied. UCR::Vicerrectoría de Investigación::Unidades de Investigación::Ciencias Sociales::Instituto de Investigaciones Psicológicas (IIP)

Sharing clinical trial data can provide value to research participants and communities by accelerating the development of new knowledge and therapies as investigators merge data sets to conduct new analyses, reproduce published findings to raise standards for original research, and learn from the work of others to generate new research questions. Nonprofit funders, including disease advocacy and patient-focused organizations, play a pivotal role in the promotion and implementation of data sharing policies. Funders are uniquely positioned to promote and support a culture of data sharing by serving as trusted liaisons between potential research participants and investigators who wish to access these participants’ networks for clinical trial recruitment. In short, nonprofit funders can drive policies and influence research culture. The purpose of this paper is to detail a set of aspirational goals and forward thinking, collaborative data sharing solutions for nonprofit funders to fold into existing funding policies. The goals of this paper convey the complexity of the opportunities and challenges facing nonprofit funders and the appropriate prioritization of data sharing within their organizations and may serve as a starting point for a data sharing toolkit for nonprofit funders of clinical trials to provide the clarity of mission and mechanisms to enforce the data sharing practices their communities already expect are happening.

Evaluative conditioning research has investigated the effect of mere stimuli co-occurrence on evaluation and found an assimilative effect—a novel stimulus acquires the valence of the co-occurring stimulus. However, most learning episodes include stimulus co-occurrence with additional relational information. For instance, viewers learn that Batman co-occurs with crime and that he fights crime. Does co-occurrence with crime increase negativity toward Batman parallel to an increase of positivity because Batman fights crime? We examined whether co-occurrence influences evaluation above and beyond the effect of relational qualifiers. We review initial supporting evidence and report five novel experiments (total N = 505) that suggest that co-occurrence with affective stimuli has an assimilative effect on evaluation even when explicit relational information suggests the opposite valence. We suggest further empirical directions for studying the effects of co-occurrence versus other relational information on evalu...

One challenge academic health centers face is to advance female faculty to leadership positions and retain them there in numbers equal to men, especially given the equal representation of women and men among graduates of medicine and biological sciences over the last 10 years. The purpose of this study is to investigate the explicit and implicit biases favoring men as leaders, among both men and women faculty, and to assess whether these attitudes change following an educational intervention.The authors used a standardized, 20-minute educational intervention to educate faculty about implicit biases and strategies for overcoming them. Next, they assessed the effect of this intervention. From March 2012 through April 2013, 281 faculty members participated in the intervention across 13 of 18 clinical departments.The study assessed faculty members' perceptions of bias as well as their explicit and implicit attitudes toward gender and leadership. Results indicated that the intervention significantly changed all faculty members' perceptions of bias (P.05 across all eight measures). Although, as expected, explicit biases did not change following the intervention, the intervention did have a small but significant positive effect on the implicit biases surrounding women and leadership of all participants regardless of age or gender (P = .008).These results suggest that providing education on bias and strategies for reducing it can serve as an important step toward reducing gender bias in academic medicine and, ultimately, promoting institutional change, specifically the promoting of women to higher ranks.