Search

Multi-lab direct replication projects like Many Labs (e.g., Klein et al., 2014) or the Registered ReplicationReports (e.g., Wagenmakers et al., 2016) are a valuable form of scientific collaboration, produceoutstanding data sets and are a great resource for third-party researchers. Their data may be reanalyzedand used in research synthesis. The repositories and code could provide guidance to future projects ofthis kind. But, while large-scale collaborations are similar in their structure and aggregate their data inmeta-analyses, they deploy a variety of different solutions regarding the storage structure in the repositories,the way the (analysis) code is structured and the file-formats they provide. Continuing this trendimplies: Anyone who wants to work with data from multiple of these projects, or combine their datasets, is faced with an ever increasing complexity. Some of that complexity could be avoided. Here, we introduceMetaPipeX, a standardized framework to analyze, document, harmonize and visualize multi-labdata. It features a pipeline conceptualization of the analysis and documentation process, an R-packagethat implements both and a Shiny App that allows users to harmonize and visualize these data sets. Weintroduce the framework by describing its components and applying a practical example to it. Engagingwith this form of collaboration and integrating it more into research practice will certainly be beneficialto psychological science and we hope the framework provides a structure and tools to reduce effort foranyone who creates, harmonizes or learns about multi-lab replication projects.

In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research (Section “An empirically based minimal reporting guideline”).

This manuscript has been published in the Journal of Abnormal Psychology (https://psycnet.apa.org/doi/10.1037/abn0000540).Biased social cognition towards an enhanced processing of negative social information might contribute to instability in interpersonal relationships. Such interpersonal dysfunctions are important for the understanding of several mental disorders, among them borderline personality disorder (BPD). To experimentally test enhanced memory retrieval of negative social information, using a newly developed variant of a looking-at-nothing paradigm, 45 BPD patients and 36 healthy women learned positive and negative personality traits of different target persons. In a translational memory test, participants were asked to use the learned information to evaluate statements about the target person. In addition to behavioral measures of memory performance, we investigated eye gaze patterns to decompose memory retrieval processes. We hypothesized that BPD patients would retrieve negative as compared to positive person information more accurately than healthy controls, and show increased eye gaze towards spatial locations where negative information was provided during the learning phase. Results pointed to a more accurate retrieval of negative person attributes in the patient group as compared with healthy controls (HC), thereby corroborating a negativity bias in social cognition in an exemplary sample of patients with interpersonal problems. Interestingly, the observed negativity bias for person memory was associated with BPD severity, stronger expectancies to be rejected by others, and social detachment. No group differences regarding eye fixation behaviour were found. We propose that enhanced retrieval of negative person information might be associated with dysfunctional cognitive schemas as well as reduced behavioral trust, and be of relevance for mental disorders characterized by interpersonal difficulties.

The study of cognitive processes is built on a close mapping between three components: overt gaze behavior - overt choice - covert processes. To validate this overt-covert mapping in the domain of decision making we collected eye-movement data during decisions between risky gamble problems. Applying a forward inference paradigm, participants were instructed to use specific decision strategies to solve those gamble problems (maximizing expected values or applying different choice heuristics) during which gaze behavior was recorded. We revealed differences between overt behavior, as indicated by eye movements, and covert decision processes, instructed by the experimenter. However, our results indicate that the overt-covert mapping is for some eye-movement measures not as close as expected by current decision theory, and hence question reverse inference as being prone to fallacies due to a violation of its prerequisite, that is, a close overt-covert mapping. We propose a framework to rehabilitate reverse inference.

We develop a reporting guideline for eye-tracking research in the behavioral sciences. To this end, we coded 215 articles on behavioral decision-making published between 2009 and 2017 and extracted a list of reported items. The coded articles were from a broad range of disciplines linked to judgement and decision making, such as cognitive science, marketing, economics, developmental research, vision research, and human–computer interaction. We then had a panel of eye-tracking experts rate the necessity of each item for reproducing a reported study. From these two sources, we generated a guideline containing 31 items that are judged as 'necessary' by the majority of experts for reproducing an eye-tracking study. None of the 215 coded articles report all identified items and approximately 70 percent of the articles report less than 50 percent of the 'necessary' items. We provide the data and list of recommendations as a hands-on shiny app to allow for an easy adoption of the proposed reporting guideline to improve transparency and reproducibility in eye-tracking research.

This chapter provides a list of reporting standards meant to increase reproducibility of eye-tracking research in judgment and decision making. It addresses a wide set of reporting criteria across the typical sections in research papers: introduction, methods, data preprocessing, and discussion of the results. The list of reporting criteria specifies the minimum information needed for successfully reproducing a study. Thus, by simply reporting the model, producer, and type of eye tracker, other researchers have the chance to consult online documentation of the specific eye-tracking devices and appraise the eye tracker’s quality and it’s fit to the respective research design. Reporting on data quality in eye-tracking studies can refer to several measures most of which are common to all eye-tracking studies. The Minimal Reporting Standards is thought of as a tool not only for researchers, but also for reviewers, editors, and funders.

Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates observed in the social sciences. Both of these problems do not only increase the proportion of false positives in the literature but can also lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect and correct such bias in meta-analytic results. We present an evaluation of the performance of six of these tools in detecting bias. To assess the Type I error rate and the statistical power of these tools we simulated a large variety of literatures that differed with regard to underlying true effect size, heterogeneity, number of available primary studies and variation of sample sizes in these primary studies. Furthermore, simulated primary studies were subjected to different degrees of publication bias. Our results show that the power of the detection methods follows a complex pattern. Across all simulated conditions, no method consistently outperformed all others. Hence, choosing an optimal method would require knowledge about parameters (e.g., true effect size, heterogeneity) that meta-analysts cannot have. Additionally, all methods performed badly when true effect sizes were heterogeneous or primary studies had a small chance of being published irrespective of their results. This suggests, that in many actual meta-analyses in psychology bias will remain undiscovered no matter which detection method is used.

Zusammenfassung. Die Replikationskrise innerhalb der Psychologie hat eine Diskussion über gängige Praktiken im Forschungsprozess und die beteiligten Institutionen ausgelöst. Wir stellen Maßnahmen vor, die dazu beitragen können, Forschung effizienter und Forschungsergebnisse belastbarer zu machen, indem sie eine strategische Ausrichtung und stärkere Verzahnung von Forschungsprozessen in Gang setzen. Die notwendigen Veränderungen setzen dabei auf der Ebene des Individuums sowie der Institutionen an und betreffen die Inhaltsbereiche Theorie, Empirie und Evidenzakkumulation. Über die bekannten eher spezifischen Maßnahmen zur Verbesserung der Reproduzierbarkeit hinaus, zielen unsere Vorschläge darauf ab, die Voraussetzungen für effizientes wissenschaftliches Arbeiten zu verbessern, indem u. a. (i) die exakte Spezifikation und die kritische Testung und Revision von Theorien wieder stärker in den Mittelpunkt der Forschung gerückt werden, (ii) eine Kultur der Transparenz, akzeptierten Fehlbarkeit und Offenheit für Empirie- aber insbesondere auch Theorierevision etabliert wird, (iii) diese Kultur und die damit verbundene Methodik als integrale Bestandteile durchgängig in der Lehre verankert werden und (iv) Evidenz und Theorien in fortlaufend dezentral aktualisierten und miteinander verknüpften Theorie- und Empirie-Datenbanken zusammengeführt werden.

Background. As the number of individuals who search for health information in interactive online environments is increasing, patient networks deserve more scientific attention. Objective. To quantitatively examine if and how reading statistical and/or narrative information as typically displayed in patient networks (e.g., patientslikeme.com ) affects decisions for pharmaceuticals. Previous work suggests that narrative information (e.g., about vaccine adverse events, VAE) affects risk perceptions and intentions. The authors compare the effect of narrative and statistical information about VAE on vaccination decisions and examine if a disclaimer reduces the narrative bias as well as if low numeracy leads to increased use of the narratives. Method and Design. In an online experiment, 458 participants were randomly assigned to a 3 (relative frequency of vaccine adverse events in 1, 7, or 17 of 20 cases) × 3 (type of information: narratives, summary statistics, or both) × 2 (bias awareness v. control disclaimer) between-subjects design. Measurements. Perceived risk, vaccination intention, and subjective numeracy. Results. A higher relative frequency of cases reporting VAE decreased the intention to get vaccinated. This relation was mediated by increased risk perception. The type of information moderated the contents’ impact: Summary statistics had the smallest impact, whereas narrative information was more influential, and the presence of both types of information had the greatest impact on risk perception. Individuals who received the bias-awareness disclaimer were less influenced by the patient network. Highly numerate individuals were generally more sensitive to the provided information independent of its format. Conclusions. Patient networks can influence vaccination decisions by delivering risk-related information. Disclaimers may help to reduce the influence if desired.

A growing number of people use the Internet to obtain health information, including information about vaccines. Websites that allow and promote interaction among users are an increasingly popular source of health information. Users of such so-called Web 2.0 applications (e.g. social media), while still in the minority, represent a growing proportion of online communicators, including vocal and active anti-vaccination groups as well as public health communicators. In this paper, the authors: define Web 2.0 and examine how it may influence vaccination decisions; discuss how anti-vaccination movements use Web 2.0 as well as the challenges Web 2.0 holds for public health communicators; describe the types of information used in these different settings; introduce the theoretical background that can be used to design effective vaccination communication in a Web 2.0 environment; make recommendations for practice and pose open questions for future research. The authors conclude that, as a result of the Internet and Web 2.0, private and public concerns surrounding vaccinations have the potential to virally spread across the globe in a quick, efficient and vivid manner. Web 2.0 may influence vaccination decisions by delivering information that alters the perceived personal risk of vaccine-preventable diseases or vaccination side-effects. It appears useful for public health officials to put effort into increasing the effectiveness of existing communication by implementing interactive, customized communication. A key step to providing successful public health communication is to identify those who are particularly vulnerable to finding and using unreliable and misleading information. Thus, it appears worthwhile that public health websites strive to be easy to find, easy to use, attractive in its presentation and readily provide the information, support and advice that the searcher is looking for. This holds especially when less knowledgeable individuals are in need of reliable information about vaccination risks and benefits.

We validate an eye-tracking method applicable for studying memory processes in complex cognitive tasks. The method is tested with a task on probabilistic inferences from memory. It provides valuable data on the time course of processing, thus clarifying previous results on heuristic probabilistic inference. Participants learned cue values of decision alternatives that were arranged within spatial frames. Later, they were told about the validities of cue dimensions and performed memory-based binary choice tasks: first, according to spontaneously adopted decision strategies and, subsequently, according to instructed decision strategies (a noncompensatory lexicographic strategy and a compensatory equal weighting strategy). During decision making, participants saw only the empty spatial frames without cue values. The spontaneously adopted and instructed decision strategies were reflected in discriminable gaze patterns on the empty spatial frames. When retrieving information no longer visible, participants tended to fixate on locations at which information was visible during the learning phase (the looking-at-nothing phenomenon). Gaze patterns were consistent with cue-wise and alternative-wise patterns of information search predicted for the instructed decision strategies as well as for the spontaneously adopted strategies identified based on decision outcomes. These findings extend previous results on the connection between memory and gaze. Furthermore, the successful application of memory indexing suggests its wider applicability in studying memory-based tasks.

Background. Health-related information found on the Internet is increasing and impacts patient decision making, e.g. regarding vaccination decisions. In addition to statistical information (e.g. incidence rates of vaccine adverse events), narrative information is also widely available such as postings on online bulletin boards. Previous research has shown that narrative information can impact treatment decisions, even when statistical information is presented concurrently. Objectives. As the determinants of this effect are largely unknown, we will vary features of the narratives to identify mechanisms through which narratives impact risk judgments. Methods. An online bulletin board setting provided participants with statistical information and authentic narratives about the occurrence and nonoccurrence of adverse events. Experiment 1 followed a single factorial design with 1, 2, or 4 narratives out of 10 reporting adverse events. Experiment 2 implemented a 2 (statistical risk 20% vs. 40%) × 2 (2/10 vs. 4/10 narratives reporting adverse events) × 2 (high vs. low richness) × 2 (high vs. low emotionality) between-subjects design. Dependent variables were perceived risk of side-effects and vaccination intentions. Results. Experiment 1 shows an inverse relation between the number of narratives reporting adverse-events and vaccination intentions, which was mediated by the perceived risk of vaccinating. Experiment 2 showed a stronger influence of the number of narratives than of the statistical risk information. High (vs. low) emotional narratives had a greater impact on the perceived risk, while richness had no effect. Implications. The number of narratives influences risk judgments can potentially override statistical information about risk.

Background music has been found to have beneficial, detrimental, or no effect on a variety of behavioral and psychological outcome measures. This article reports a meta-analysis that attempts to summarize the impact of background music. A global analysis shows a null effect, but a detailed examination of the studies that allow the calculation of effects sizes reveals that this null effect is most probably due to averaging out specific effects. In our analysis, the probability of detecting such specific effects was not very high as a result of the scarcity of studies that allowed the calculation of respective effect sizes. Nonetheless, we could identify several such cases: a comparison of studies that examined background music compared to no music indicates that background music disturbs the reading process, has some small detrimental effects on memory, but has a positive impact on emotional reactions and improves achievements in sports. A comparison of different types of background music reveals that the tempo of the music influences the tempo of activities that are performed while being exposed to background music. It is suggested that effort should be made to develop more specific theories about the impact of background music and to increase the methodological quality of relevant studies.

This paper examines conditions that do or do not lead to accurate judgments of frequency (JOF) and judgments of duration (JOD). In three experiments, duration and frequency of visually presented stimuli are varied orthogonally in a within-subjects design. Experiment 1 reveals anasymmetricjudgment pattern. JOFs reflected actual presentation frequency quite accurately and were unbiased by exposure duration. Conversely, JODs were almost insensitive to actual exposure duration and were systematically biased by presentation frequency. We show, however, that a tendency towards asymmetricjudgment pattern can be obtained by manipulating encoding conditions. Sustaining attention during encoding (Experiment 2) or enhancing richness of the encoded stimuli (Experiment 3) increases judgment sensitivity in JOD and yields biases in both directions (JOF biased by exposure duration, JOD biased by presentation frequency). The implications of these findings for underlying memory mechanisms are discussed.

Immunization rates are below the Global Immunization Vision and Strategy established by the World Health Organization. One reason for this are anti-vaccination activists, who use the Internet to disseminate their agenda, frequently by publishing narrative reports about alleged vaccine adverse events. In health communication, the use of narrative information has been shown to be effectively persuasive. Furthermore, persuasion research indicates that the credibility of an information source may serve as a cue to discount or augment the communicated message. Thus, the present study investigated the effect of source credibility on the biasing effect of narrative information regarding the perception of vaccination risks. 265 participants were provided with statistical information (20%) regarding the occurrence of vaccine adverse events after vaccination against a fictitious disease. This was followed by 20 personalized narratives from an online forum on vaccination experiences. The authors varied the relative frequency of narratives reporting vaccine adverse events (35% vs. 85%), narrative source credibility (anti-vaccination website vs. neutral health forum), and the credibility of the statistical information (reliable data vs. unreliable data vs. control) in a between-subjects design. Results showed a stable narrative bias on risk perception that was not affected by credibility cues. However, narratives from an anti-vaccination website generally led to lower perceptions of vaccination risks.

IntroductionReproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.RationaleThere is concern about the rate and predictors of reproducibility, but limited evidence. Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results. Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding and is the means of establishing reproducibility of a finding with new data. We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science.ResultsWe conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. There is no single standard for evaluating replication success. Here, we evaluated reproducibility using significance and P values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes. The mean effect size (r) of the replication effects (Mr = 0.197, SD = 0.257) was half the magnitude of the mean effect size of the original effects (Mr = 0.403, SD = 0.188), representing a substantial decline. Ninety-seven percent of original studies had significant results (P < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.ConclusionNo single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes. Moreover, correlational evidence is consistent with the conclusion that variation in the strength of initial evidence (such as original P value) was more predictive of replication success than variation in the characteristics of the teams conducting the research (such as experience and expertise). The latter factors certainly can influence replication success, but they did not appear to do so here. Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.

The RISK of an event generally relates to its expected severity and the perceived probability of its occurrence. In RISK research, however, there is no standard measure for subjective probability estimates. In this study, we compared five commonly used measurement formats--two rating scales, a visual analog scale, and two numeric measures--in terms of their ability to assess subjective probability judgments when objective probabilities are available. We varied the probabilities (low vs. moderate) and severity (low vs. high) of the events to be judged as well as the presentation mode of objective probabilities (sequential presentation of singular events vs. graphical presentation of aggregated information). We employed two complementary goodness-of-fit criteria: the correlation between objective and subjective probabilities (sensitivity), and the root mean square deviations of subjective probabilities from objective values (accuracy). The numeric formats generally outperformed all other measures. The severity of events had no effect on the performance. Generally, a rise in probability led to decreases in performance. This effect, however, depended on how the objective probabilities were encoded: pictographs ensured perfect information, which improved goodness of fit for all formats and diminished this negative effect on the performance. Differences in performance between scales are thus caused only in part by characteristics of the scales themselves-they also depend on the process of encoding. Consequently, researchers should take the source of probability information into account before selecting a measure.

This large-scale Internet-experiment tests whether vaccine-critical pages raise perceptions of the riskiness of vaccinations and alter vaccination intentions. We manipulated the information environment (vaccine-critical website, control, both) and the focus of search (on vaccination risks, omission risks, no focus). Our analyses reveal that accessing vaccine-critical websites for five to 10 minutes increases the perception of risk of vaccinating and decreases the perception of risk of omitting vaccinations as well as the intentions to vaccinate. In line with the ‘risk-as-feelings’ approach, the affect elicited by the vaccine-critical websites was positively related to changes in risk perception.

The topic of frequency processing and cognition plays a crucial role in several areas of psychological research and there exists a wealth of models and empirical results. At present, one of the main problems that hinders faster progress in the understanding of how frequencies of events are processed and how this influences cognition is that the exchange of information does not work adequately among the different areas. Some of the differences arose because researchers were interested in different questions, but other differences seem to be due to a missing exchange of ideas, models, and experimental results. This chapter first gives a brief overview about the research on how frequency processing affects cognition and then discusses briefly the content of the remaining chapters.