Search

This paper reports findings from a crowdsourced replication. Eighty-five independent teams attempted a computational replication of results reported in an original study of policy preferences and immigration by fitting the same statistical models to the same data. The replication involved an experimental condition. Random assignment put participating teams into either the transparent group that received the original study and code, or the opaque group receiving only a methods section, rough results description and no code. The transparent group mostly verified the numerical results of the original study with the same sign and p-value threshold (95.7%), while the opaque group had less success (89.3%). Exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%), and the number of teams who verified at least 95% of all effects in all models they ran was 79.5% and 65.2% respectively. Therefore, the reliability we quantify depends on how reliability is defined, but most definitions suggest it would take a minimum of three independent replications to achieve reliability. Qualitative investigation of the teams’ workflows reveals many causes of error including mistakes and procedural variations. Although minor error across researchers is not surprising, we show this occurs where it is least expected in the case of computational reproduction. Even when we curate the results to boost ecological validity, the error remains large enough to undermine reliability between researchers to some extent. The presence of inter-researcher variability may explain some of the current “reliability crisis” in the social sciences because it may be undetected in all forms of research involving data analysis. The obvious implication of our study is more transparency. Broader implications are that researcher variability adds an additional meta-source of error that may not derive from conscious measurement or modeling decisions, and that replications cannot alone resolve this type of uncertainty.

In an era of mass migration, social scientists, populist parties and social movements raise concerns over the future of immigration-destination societies. What impacts does this have on policy and social solidarity? Comparative cross-national research, relying mostly on secondary data, has findings in different directions. There is a threat of selective model reporting and lack of replicability. The heterogeneity of countries obscures attempts to clearly define data-generating models. P-hacking and HARKing lurk among standard research practices in this area.This project employs crowdsourcing to address these issues. It draws on replication, deliberation, meta-analysis and harnessing the power of many minds at once. The Crowdsourced Replication Initiative carries two main goals, (a) to better investigate the linkage between immigration and social policy preferences across countries, and (b) to develop crowdsourcing as a social science method. The Executive Report provides short reviews of the area of social policy preferences and immigration, and the methods and impetus behind crowdsourcing plus a description of the entire project. Three main areas of findings will appear in three papers, that are registered as PAPs or in process.

We analyze the effects of formal recognition of foreign higher education on employment probabilities and earnings for newly arrived immigrants in Sweden. Prior research has found that immigrants have lower returns on education if it was acquired in the country of origin than if it was acquired in the host country. One reason for this is that foreign credentials work poorly as productivity signals and risk-averse employers avoid employees with credentials they do not fully understand. A formal recognition statement can help overcome this problem by providing credible information about the foreign education, thus reducing uncertainty. Data consists of immigrants who, within the first ten years of residence in Sweden, had their foreign degree formally recognized during 2007–2011. Using fixed effects regressions, we estimate the treatment effect of official recognition to be 4.4 percentage points higher probability of being employed, and 13.9 log points higher wage for those with employment. We also find considerable treatment effect heterogeneity across subcategories of immigrants from different regions of origin, with different reasons for immigration and who obtained recognition during different economic conditions. Our conclusions are that the mechanism of employer uncertainty is real, and that recognition does reduce it. But as the signal of foreign education becomes better, other mechanisms such as human capital transferability problems and quality differences, and the ability to use foreign human capital, become more salient, leading to heterogeneous effects.

In linking self-employment before and after migration, the often-cited home-country self-employment hypothesis states that immigrants who come from countries with large self-employment sectors are themselves more likely to have been self-employed and hence have a higher propensity for self-employment in their destination country. Using Swedish data, this study shows that the first part of the hypothesis, that origin-country average rates of self-employment can be used to approximate individual experience, is false; but the second part, the connection between self-employment before and after migration, is true if the measurement is done on the individual level. Migrants who have been self-employed before migration accumulate entrepreneurial human capital, making future self-employment a more desirable labor market alternative vis-a-vis wage employment. But because of migrant selection, this association cannot be captured by aggregate measures, and this is the reason why the home-country self-employment hypothesis, although intuitive, has underperformed in previous empirical tests. Fulltext published under Creative Commons Attribution 4.0 International License.https://creativecommons.org/licenses/by/4.0/