Search

Multi-lab direct replication projects like Many Labs (e.g., Klein et al., 2014) or the Registered ReplicationReports (e.g., Wagenmakers et al., 2016) are a valuable form of scientific collaboration, produceoutstanding data sets and are a great resource for third-party researchers. Their data may be reanalyzedand used in research synthesis. The repositories and code could provide guidance to future projects ofthis kind. But, while large-scale collaborations are similar in their structure and aggregate their data inmeta-analyses, they deploy a variety of different solutions regarding the storage structure in the repositories,the way the (analysis) code is structured and the file-formats they provide. Continuing this trendimplies: Anyone who wants to work with data from multiple of these projects, or combine their datasets, is faced with an ever increasing complexity. Some of that complexity could be avoided. Here, we introduceMetaPipeX, a standardized framework to analyze, document, harmonize and visualize multi-labdata. It features a pipeline conceptualization of the analysis and documentation process, an R-packagethat implements both and a Shiny App that allows users to harmonize and visualize these data sets. Weintroduce the framework by describing its components and applying a practical example to it. Engagingwith this form of collaboration and integrating it more into research practice will certainly be beneficialto psychological science and we hope the framework provides a structure and tools to reduce effort foranyone who creates, harmonizes or learns about multi-lab replication projects.

In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research (Section “An empirically based minimal reporting guideline”).

This manuscript has been published in the Journal of Abnormal Psychology (https://psycnet.apa.org/doi/10.1037/abn0000540).Biased social cognition towards an enhanced processing of negative social information might contribute to instability in interpersonal relationships. Such interpersonal dysfunctions are important for the understanding of several mental disorders, among them borderline personality disorder (BPD). To experimentally test enhanced memory retrieval of negative social information, using a newly developed variant of a looking-at-nothing paradigm, 45 BPD patients and 36 healthy women learned positive and negative personality traits of different target persons. In a translational memory test, participants were asked to use the learned information to evaluate statements about the target person. In addition to behavioral measures of memory performance, we investigated eye gaze patterns to decompose memory retrieval processes. We hypothesized that BPD patients would retrieve negative as compared to positive person information more accurately than healthy controls, and show increased eye gaze towards spatial locations where negative information was provided during the learning phase. Results pointed to a more accurate retrieval of negative person attributes in the patient group as compared with healthy controls (HC), thereby corroborating a negativity bias in social cognition in an exemplary sample of patients with interpersonal problems. Interestingly, the observed negativity bias for person memory was associated with BPD severity, stronger expectancies to be rejected by others, and social detachment. No group differences regarding eye fixation behaviour were found. We propose that enhanced retrieval of negative person information might be associated with dysfunctional cognitive schemas as well as reduced behavioral trust, and be of relevance for mental disorders characterized by interpersonal difficulties.

The study of cognitive processes is built on a close mapping between three components: overt gaze behavior - overt choice - covert processes. To validate this overt-covert mapping in the domain of decision making we collected eye-movement data during decisions between risky gamble problems. Applying a forward inference paradigm, participants were instructed to use specific decision strategies to solve those gamble problems (maximizing expected values or applying different choice heuristics) during which gaze behavior was recorded. We revealed differences between overt behavior, as indicated by eye movements, and covert decision processes, instructed by the experimenter. However, our results indicate that the overt-covert mapping is for some eye-movement measures not as close as expected by current decision theory, and hence question reverse inference as being prone to fallacies due to a violation of its prerequisite, that is, a close overt-covert mapping. We propose a framework to rehabilitate reverse inference.

We develop a reporting guideline for eye-tracking research in the behavioral sciences. To this end, we coded 215 articles on behavioral decision-making published between 2009 and 2017 and extracted a list of reported items. The coded articles were from a broad range of disciplines linked to judgement and decision making, such as cognitive science, marketing, economics, developmental research, vision research, and human–computer interaction. We then had a panel of eye-tracking experts rate the necessity of each item for reproducing a reported study. From these two sources, we generated a guideline containing 31 items that are judged as 'necessary' by the majority of experts for reproducing an eye-tracking study. None of the 215 coded articles report all identified items and approximately 70 percent of the articles report less than 50 percent of the 'necessary' items. We provide the data and list of recommendations as a hands-on shiny app to allow for an easy adoption of the proposed reporting guideline to improve transparency and reproducibility in eye-tracking research.

Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates observed in the social sciences. Both of these problems do not only increase the proportion of false positives in the literature but can also lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect and correct such bias in meta-analytic results. We present an evaluation of the performance of six of these tools in detecting bias. To assess the Type I error rate and the statistical power of these tools we simulated a large variety of literatures that differed with regard to underlying true effect size, heterogeneity, number of available primary studies and variation of sample sizes in these primary studies. Furthermore, simulated primary studies were subjected to different degrees of publication bias. Our results show that the power of the detection methods follows a complex pattern. Across all simulated conditions, no method consistently outperformed all others. Hence, choosing an optimal method would require knowledge about parameters (e.g., true effect size, heterogeneity) that meta-analysts cannot have. Additionally, all methods performed badly when true effect sizes were heterogeneous or primary studies had a small chance of being published irrespective of their results. This suggests, that in many actual meta-analyses in psychology bias will remain undiscovered no matter which detection method is used.

Zusammenfassung. Die Replikationskrise innerhalb der Psychologie hat eine Diskussion über gängige Praktiken im Forschungsprozess und die beteiligten Institutionen ausgelöst. Wir stellen Maßnahmen vor, die dazu beitragen können, Forschung effizienter und Forschungsergebnisse belastbarer zu machen, indem sie eine strategische Ausrichtung und stärkere Verzahnung von Forschungsprozessen in Gang setzen. Die notwendigen Veränderungen setzen dabei auf der Ebene des Individuums sowie der Institutionen an und betreffen die Inhaltsbereiche Theorie, Empirie und Evidenzakkumulation. Über die bekannten eher spezifischen Maßnahmen zur Verbesserung der Reproduzierbarkeit hinaus, zielen unsere Vorschläge darauf ab, die Voraussetzungen für effizientes wissenschaftliches Arbeiten zu verbessern, indem u. a. (i) die exakte Spezifikation und die kritische Testung und Revision von Theorien wieder stärker in den Mittelpunkt der Forschung gerückt werden, (ii) eine Kultur der Transparenz, akzeptierten Fehlbarkeit und Offenheit für Empirie- aber insbesondere auch Theorierevision etabliert wird, (iii) diese Kultur und die damit verbundene Methodik als integrale Bestandteile durchgängig in der Lehre verankert werden und (iv) Evidenz und Theorien in fortlaufend dezentral aktualisierten und miteinander verknüpften Theorie- und Empirie-Datenbanken zusammengeführt werden.

A growing number of people use the Internet to obtain health information, including information about vaccines. Websites that allow and promote interaction among users are an increasingly popular source of health information. Users of such so-called Web 2.0 applications (e.g. social media), while still in the minority, represent a growing proportion of online communicators, including vocal and active anti-vaccination groups as well as public health communicators. In this paper, the authors: define Web 2.0 and examine how it may influence vaccination decisions; discuss how anti-vaccination movements use Web 2.0 as well as the challenges Web 2.0 holds for public health communicators; describe the types of information used in these different settings; introduce the theoretical background that can be used to design effective vaccination communication in a Web 2.0 environment; make recommendations for practice and pose open questions for future research. The authors conclude that, as a result of the Internet and Web 2.0, private and public concerns surrounding vaccinations have the potential to virally spread across the globe in a quick, efficient and vivid manner. Web 2.0 may influence vaccination decisions by delivering information that alters the perceived personal risk of vaccine-preventable diseases or vaccination side-effects. It appears useful for public health officials to put effort into increasing the effectiveness of existing communication by implementing interactive, customized communication. A key step to providing successful public health communication is to identify those who are particularly vulnerable to finding and using unreliable and misleading information. Thus, it appears worthwhile that public health websites strive to be easy to find, easy to use, attractive in its presentation and readily provide the information, support and advice that the searcher is looking for. This holds especially when less knowledgeable individuals are in need of reliable information about vaccination risks and benefits.

This paper examines conditions that do or do not lead to accurate judgments of frequency (JOF) and judgments of duration (JOD). In three experiments, duration and frequency of visually presented stimuli are varied orthogonally in a within-subjects design. Experiment 1 reveals anasymmetricjudgment pattern. JOFs reflected actual presentation frequency quite accurately and were unbiased by exposure duration. Conversely, JODs were almost insensitive to actual exposure duration and were systematically biased by presentation frequency. We show, however, that a tendency towards asymmetricjudgment pattern can be obtained by manipulating encoding conditions. Sustaining attention during encoding (Experiment 2) or enhancing richness of the encoded stimuli (Experiment 3) increases judgment sensitivity in JOD and yields biases in both directions (JOF biased by exposure duration, JOD biased by presentation frequency). The implications of these findings for underlying memory mechanisms are discussed.

IntroductionReproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.RationaleThere is concern about the rate and predictors of reproducibility, but limited evidence. Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results. Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding and is the means of establishing reproducibility of a finding with new data. We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science.ResultsWe conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. There is no single standard for evaluating replication success. Here, we evaluated reproducibility using significance and P values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes. The mean effect size (r) of the replication effects (Mr = 0.197, SD = 0.257) was half the magnitude of the mean effect size of the original effects (Mr = 0.403, SD = 0.188), representing a substantial decline. Ninety-seven percent of original studies had significant results (P < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.ConclusionNo single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes. Moreover, correlational evidence is consistent with the conclusion that variation in the strength of initial evidence (such as original P value) was more predictive of replication success than variation in the characteristics of the teams conducting the research (such as experience and expertise). The latter factors certainly can influence replication success, but they did not appear to do so here. Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.