Search

A large amount of scientific literature in social and behavioural sciences bases their conclusions on one or more hypothesis tests. As such, it is important to obtain more knowledge about how researchers in social and behavioural sciences interpret quantities that result from such hypothesis tests, such as p-values and Bayes factors. In the present study, we will explore the relationship between obtained statistical evidence and the degree of belief or confidence that there is a positive effect in the population of interest. In particular, we are interested in the existence of a so-called cliff effect: A qualitative drop in the degree of belief that there is a positive effect around certain threshold values of statistical evidence. We will compare this relationship for p-values to the relationship for corresponding degrees of evidence quantified through Bayes factors, and we will examine whether this relationship is affected by two different modes of presentation.

We plan a collaborative research analysis of the original ‘many analysts’ study by Silberzahn et al.1 testing whether European soccer referees were skin-tone biased in giving red cards. Their study concluded that the answer has great model dependency, and in this sense is unclear. Auspurg and Brüderl2 re-analysed the methods and data and argued the answer is clearer when attending to causal inference and outliers. Researchers involved in the original study and follow-up agreed to an adversarial collaboration to better answer this pertinent research question and sources of variability in answers. They recruited other scholars to provide independent methodological and theoretical insights. The result is this study where together we jointly tackle the skin-tone bias hypothesis and the ‘parsing problem’ of identifying meaningful moderators that explain effect size dispersion across a multiverse of alternative specifications. We build on Auspurg and Brüderl’s proposal that analysts in the original study had very different interpretations of the conceptual research question, leading them to different empirical research questions, which in turn produced results that are not directly comparable. This pre-analysis plan describes our motivation, methods, and plan to contextualize our findings in larger discussions of race bias, multiverse analysis parsing, and subjectivity in science.

Effective grassroots governance is the foundation for the stability of state power, but the "empty busy" behavior of a few grassroots cadres has seriously deteriorated the local political ecology. In order to gain a deeper understanding of the mechanism of "empty busy" behavior of grassroots cadres, this paper, following the analysis of typological research and combined with publicly available data, classifies the "empty busy" behavior of grassroots cadres into eight types, including hedging and self-preservation, drifting with the stream, patchwork response, conforming old rulers, replacing targets, blame avoidance, trace doctrine, and self-waiver. It is found that the "empty busy" behavior of grassroots cadres is shaped by the risk society paradox, local management paradox, incentive intensity paradox and technical governance paradox, and the "empty busy" behavior of grassroots cadres needs to be corrected from the four-dimensional logic of incentive, restraint, deep care, and strict control. This study systematically describes the manifestation, causes and correction mechanism of the "empty busy" behavior of grassroots cadres, which enriches and expands the theoretical research on the performance behavior of grassroots cadres.

We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices; (4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.Wagenmakers and colleagues describe seven statistical procedures that increase transparency in data analysis. These procedures highlight common ground among data analysts from different schools and find inspiration in Merton's ethos of science.

The scientific method is predicated on transparency -- yet the pace at which transparent research practices are being adopted by the scientific community is slow. The replication crisis in psychology showed that published findings employing statistical inference are threatened by undetected errors, data manipulation, and data falsification. To mitigate these problems and bolster research credibility, open data and preregistration have increasingly been adopted in the natural and social sciences. While many political science and international relations journals have committed to implementing these reforms, the extent of open science practices is unknown. We bring large-scale text analysis and machine learning classifiers to bear on the question. Using population-level data -- 93,931 articles across the top 160 political science and IR journals between 2010 and 2021 -- we find that approximately 21% of all statistical inference papers have open data, and 5% of all experiments are preregistered. Despite this shortfall, the example of leading journals in the field shows that change is feasible and can be effected quickly.

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings. ispartof: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA vol:119 issue:44 ispartof: location:United States status: published

How do perceptions of one's relative economic status affect beliefs regarding gender roles? We conducted a 2019 survey experiment with approximately 2,000 adults in Nepal. Employing an established survey treatment called a priming experiment to subtly alter half of respondents' perceptions of their relative economic well-being, we ?find that increased feelings of relative deprivation make married women significantly less likely to support gender egalitarian perspectives. Women decrease their support for women making decisions over household expenditures, having equal control over household income, sharing household chores, and women working outside the home. A message randomly read to some women and designed to spur increased support for women's empowerment does little to alter beliefs regarding gender roles or to attenuate the effects of the relative deprivation prime. Despite the negative impacts on women's gender attitudes, however, we do not find a similar pattern among married men. The results underscore the deleterious effects that feelings of relative deprivation can have on women's own gender attitudes and provide a cautionary tale given trends toward greater economic inequality further exacerbated by the COVID-19 pandemic.

Sharing of key evidence on strategies for reducing resources consumption and lowering carbon footprint is essential to alleviating risks of increasing urbanization, population growth and looming climate change impacts. However, finding scientifically robust research and distilling knowledge to draw confident conclusions in a reasonable timeframe is challenging due to the sheer volume of rapidly accumulating research evidence. Primary evidence is aggregated in secondary studies, i.e. various types of reviews, including systematic reviews and meta-analyses. However, the number of secondary studies is also growing rapidly and their quality varies. This work presents the first systematic review of secondary studies which were claimed to be systematic reviews or meta-analyses of literature relevant to reducing carbon footprint of the built environment or its co-benefits (e.g., health and well-being). The resulting searchable and updatable database contains 131 reviews published between 2001 and early 2018. Importantly, the quality of the included reviews was assessed and categorized. The database is available via a dedicated website which includes interactive visualizations and filtering tools. In its current form, the interactive map and database can help discovery and appraisal of secondary evidence for decision-making and research use, supporting transition to low carbon future. This overview aimed to reveal trends and patterns, including gaps, in the publications of research syntheses relevant to sustainable built environment. The results showed that the number of reviews claiming to by systematic reviews or meta-analyses is rapidly increasing in recent years. However, the exact methodology used and the standard of adherence to methodological and reporting guidelines varies widely among reviews. The included reviews span different disciplines, such as environmental sciences, engineering, urban planning, public health, and social sciences. The first authors from USA, UK and Australia contributed the most relevant reviews. Health and well-being, along with closely related environment and nature, are currently the best synthesized subject areas, while there are still very few included reviews related to policy, economy or information technology, indicating synthesis gaps in these areas. Our analysis reveals the lack of standardized definitions of different types of research syntheses across disciplines, as well as poor reporting quality of research syntheses. We encourage detailed descriptions of methods, providing datasets and interactive visualizations as integral components of the future systematic reviews and evidence maps.

BACKGROUND. Experimental vignette research methods have been used to study a diverse range of theoretical and practical issues. Vignettes are designed to create hypothetical cultural/normative contexts for the study of variation in self-reported attitudes. A key problem in such research, however, is potential social desirability response bias. METHOD. A vignette experimental test of an hypothesis derived from a dual-process theory (the MODE framework initially developed by Fazio) linking explicit vs. implicit self-reported attitude measurement and social desirability response bias is reported here. RESULTS. The data show that measuring the social approval of a central vignette character explicitly results in greater social desirability responding than measuring such approval implicitly, supporting MODE theory. CONCLUSIONS. Vignette research methodologies provide a rich, flexible toolkit for studying many important social psychological topics, including issues of inequality and equity. However, researchers can and should design a measurement strategy that carefully manages inferences drawn in light of conditions likely to produce social desirability response bias.

We invite you to participate in a collaborative research effort among scholars with conflicting viewpoints to jointly devise and conduct a quantitative survey on free speech on campus.A recent study (Revers & Traunmüller 2020) claims to have found evidence for widespread support for restricting free speech on campus, experiences of conformity pressures, and self-censorship. The study attracted broad public interest along with critical reactions from other scholars. The criticism concerned issues of study design and item wording as well as the normative framing of the results. Some of the disagreement regarding the empirical findings seems to reflect differences in the researchers’ epistemic, methodological, and ideological backgrounds. This call and the proposed project are motivated by the desire to leverage this diversity of perspectives in a constructive way. Specifically, we are looking for scholars to join the group of skeptics who do not agree with the original study’s conclusions. In the form of a pre-registered adversarial collaboration, the group of skeptics along with the group of proponents work to consent on a questionnaire and analysis plan for a new quantitative survey that is to be conducted in the German university context (and potentially beyond).

We explore the promise of statistical reform by starting from the assumption that most researchers would endorse Mertons ethos of science as reflected in the four norms of communalism, universalism, disinterestedness, and organized skepticism. Translated to data analysis, these norms imply a need for transparency, a fair acknowledgement of uncertainty, and openness to alternative interpretations. We discuss seven statistical procedures, both old and new, that we believe can positively impact statistical practice in the social and behavioral sciences.

What are the causal pathways through which natural resources are linked to civil conflict? The most comprehensive answer to this question comes from Ross (2004), who conducted the first qualitative causal pathway analysis on this issue. Despite the study's prominence, its findings have never been replicated due to the challenge of re- coding thirteen hypotheses across thirteen cases. To overcome this, we conduct the first qualitative replication in Political Science to employ an original codebook, a team of coders, and inter-coder reliability checks. We find that 24% of Ross' codings fail to replicate, a large enough share to alter his core findings. Contrary to Ross, we find that resources affect conflict onset through the pathways of both greed and grievance. Additionally, we find that resources generally increase conflict intensity and duration. Our methods and approach can be broadly applied to future qualitative research, especially medium-N causal pathway analysis.

The household economy approach is used as an early warning tool to identify areas of intervention and priority groups to benefit from food security support. Adopted in 2012 in Burkina Faso, the application of this tool is still in the experimental phase. After having sketched the national context, from a qualitative survey and using data from targeting surveys, paper exposes questions posed by the development practitioners and realities of application of this tool on a project in Burkina Faso in the region of Boucle du Mouhoun. Implementation of HEA in the field shows a dysfunction. The analyzes reveal a part of methodological subjectivity, the weight of opinion leaders, acting as development brokers and the absence of a complaints committee, an extremely cumbersome mechanism that is unrealistic to initiate. While HEA is seen by communities as a breakthrough, the current trend for practitioners to fill in the gaps is towards increasing the complexity of targeting methods. The initiatives developed require practitioners to maintain a close presence on targeting committees, an unrealistic presence to be deployed in a change of scale in the method and also accompanied by greater bureaucratization of aid.

What is the effect of the COVID-19 pandemic on the 2020 U.S. presidential election? Guided by a pre-analysis plan, we estimate the effect of COVID-19 cases and deaths on the change in county-level voting for Donald Trump between 2016 and 2020. To account for potential confounders, we include a large number of COVID-19-related controls as well as demographic and socioeconomic variables. Moreover, we instrument the numbers of cases and deaths with the share of workers employed in meat-processing factories to sharpen our identification strategy. We find that COVID-19 cases negatively affected Trump's vote share. The estimated effect appears strongest in urban counties, in swing states, and in states that Trump won in 2016. A simple counterfactual analysis suggests that Trump would likely have won re-election if COVID-19 cases had been 5 percent lower. Our paper contributes to the literature of retrospective voting and demonstrates that voters hold leaders accountable for their (mis-)handling of negative shocks.

В современном российском обществе важную роль играет тема национальной государственной идеологии. Это потому, что национальная идея и идеология в целом выступают интегрирующей силой. Анализ проблематики духовной составляющей в традиционной идеологии русской идеи ведёт к обоснованию необходимости трансляции культурной традиции, накопленного поколениями исторического духовного опыта.In modern Russian society, the theme of national state ideology plays an important role. This is because the national idea and ideology as a whole act as an integrating force. An analysis of the problems of the spiritual component in the traditional ideology of the Russian idea leads to the justification of the need to translate the cultural tradition accumulated by generations of historical spiritual experience.

We report a number of irregularities in the replication dataset posted for LaCour and Green (Science, "When contact changes minds: An experiment on transmission of support for gay equality," 2014) that jointly suggest the dataset (LaCour 2014) was not collected as described. These irregularities include baseline outcome data that is statistically indistinguishable from a national survey and over-time changes that are unusually small and indistinguishable from perfectly normally distributed noise. Other elements of the dataset are inconsistent with patterns typical in randomized experiments and survey responses and/or inconsistent with the claimed design of the study. A straightforward procedure may generate these anomalies nearly exactly: for both studies reported in the paper, a random sample of the 2012 Cooperative Campaign Analysis Project (CCAP) form the baseline data and normally distributed noise are added to simulate follow-up waves.

The evidence-based community has championed the public registration of pre-analysis plans (PAPs) as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze a representative sample of 195 PAPs from the American Economic Association (AEA) and Evidence in Governance and Politics (EGAP) registration platforms to assess whether PAPs are sufficiently clear, precise and comprehensive to be able to achieve their objectives of preventing “fishing” and reducing the scope for post-hoc adjustment of research hypotheses. We also analyze a subset of 93 PAPs from projects that have resulted in publicly available papers to ascertain how faithfully they adhere to their pre-registered specifications and hypotheses. We find significant variation in the extent to which PAPs are accomplishing the goals they were designed to achieve

Replication and transparency are increasingly important in bolstering the credibility of political science research, yet open science tools are typically designed for experiments. For observational studies, current replication practice suffers from an important pathology: just as researchers can often "p-hack" their way to initial findings, it is often possible to "null hack" findings away through specification and case search. We propose an observational open science framework that consists of extending the original time series, independent data collection, pre-registration, multiple simultaneous replications, and collaborators with mixed incentives. We apply the approach to three studies on "irrelevant" events and voting behavior. Each study replicates well in some areas and poorly in others. Had we sought to debunk any of the three with ex post specification search, we could have done so. However, our approach required us to see the full, complicated picture. We conclude with suggestions for future refinements to our approach.

The importance of replication has been a major issue of discussion amongst political scientists for decades. However, in the last five years, the matter has gained greater traction, with many major journals adopting official standards for Data Access and Research Transparency (DA-RT). At the same time, scholars have suggested the pivotal role that replication might play in methods training for students. Unfortunately, these conversations have been somewhat limited, in that they place an emphasis on quantitative methods and training graduate students. This paper seeks to fill in this hole, offering commentary on the author’s experience introducing a qualitative replication project in an upper-division undergraduate course. The students in the course replicated Michael Ross’ (2004) influential paper on the resource curse, but the assignment framework can be adopted for a variety of topics and contexts.

How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that researchers have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a re-analysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.