Search

To emulate the interactivity of in-person math instruction, we developed MathBot, a rule-based chatbot that explains math concepts, provides practice questions, and offers tailored feedback. We evaluated MathBot through three Amazon Mechanical Turk studies in which participants learned about arithmetic sequences. In the first study, we found that more than 40% of our participants indicated a preference for learning with MathBot over videos and written tutorials from Khan Academy. The second study measured learning gains, and found that MathBot produced comparable gains to Khan Academy videos and tutorials. We solicited feedback from users in those two studies to emulate a real-world development cycle, with some users finding the lesson too slow and others finding it too fast. We addressed these concerns in the third and main study by integrating a contextual bandit algorithm into MathBot to personalize the pace of the conversation, allowing the bandit to either insert extra practice problems or skip explanations. We randomized participants between two conditions in which actions were chosen uniformly at random (i.e., a randomized A/B experiment) or by the contextual bandit. We found that the bandit learned a similarly effective pedagogical policy to that learned by the randomized A/B experiment while incurring a lower cost of experimentation. Our findings suggest that personalized conversational agents are promising tools to complement existing online resources for math education, and that data-driven approaches such as contextual bandits are valuable tools for learning effective personalization.

Beliefs about the incidence of voter fraud inform how people view the trade-off between electoral integrity and voter accessibility. To better inform such beliefs about the rate of double voting, we develop and apply a method to estimate how many people voted twice in the 2012 presidential election. We estimate that about one in 4,000 voters cast two ballots, although an audit suggests that the true rate may be lower due to small errors in electronic vote records. We corroborate our estimates and extend our analysis using data from a subset of states that share social security numbers, making it easier to quantify who may have voted twice. For this subset of states, we find that one suggested strategy to reduce double voting—removing the registration with an earlier registration date when two share the same name and birthdate—could impede approximately 300 legitimate votes for each double vote prevented.

Jurisdictions across the country, including the federal government through its recently enacted First Step Act, have begun using statistical algorithms (also called “instruments”) to help determine an arrestee’s or an offender’s risk of reoffending. These risk assessment instruments (RAIs) might be used at a number of points in the criminal process, including at the front-end by judges to impose a sentence after conviction, at the back-end by parole boards to make decisions about prison release, or in between these two points by correctional authorities determining the optimal security and service arrangements for an offender. At the pretrial stage, RAIs might come into play at the time of the bail or pretrial detention determination by a judge, which usually takes place shortly after arrest. The increased use of RAIs in the criminal justice system has given rise to several criticisms. RAIs are said to be no more accurate than clinical assessments, racially biased, lacking in transparency and, because of their quantitative nature, dehumanizing. This chapter critically examines a number of these concerns. It also highlights how the law has, and should, respond to these issues.

On a variety of complex decision-making tasks, from doctors prescribing treatment to judges setting bail, machine learning algorithms have been shown to outperform expert human judgments. One complication, however, is that it is often difficult to anticipate the effects of algorithmic policies prior to deployment, as one generally cannot use historical data to directly observe what would have happened had the actions recommended by the algorithm been taken. A common strategy is to model potential outcomes for alternative decisions assuming that there are no unmeasured confounders (i.e., to assume ignorability). But if this ignorability assumption is violated, the predicted and actual effects of an algorithmic policy can diverge sharply. In this paper we present a flexible Bayesian approach to gauge the sensitivity of predicted policy outcomes to unmeasured confounders. In particular, and in contrast to past work, our modeling framework easily enables confounders to vary with the observed covariates. We demonstrate the efficacy of our method on a large dataset of judicial actions, in which one must decide whether defendants awaiting trial should be required to pay bail or can be released without payment.

It is widely thought that news organizations exhibit ideological bias, but rigorously quantifying such slant has proven methodologically challenging. Through a combination of machine learning and crowdsourcing techniques, we investigate the selection and framing of political issues in 15 major U.S. news outlets. Starting with 803,146 news stories published over 12 months, we first used supervised learning algorithms to identify the 14% of articles pertaining to political events. We then recruited 749 online human judges to classify a random subset of 10,950 of these political articles according to topic and ideological position. Our analysis yields an ideological ordering of outlets consistent with prior work. We find, however, that news outlets are considerably more similar than generally believed. Specifically, with the exception of political scandals, we find that major news organizations present topics in a largely non-partisan manner, casting neither Democrats nor Republicans in a particularly favorable or unfavorable light. Moreover, again with the exception of political scandals, there is little evidence of systematic differences in story selection, with all major news outlets covering a wide variety of topics with frequency largely unrelated to the outlet's ideological position. Finally, we find that news organizations express their ideological bias not by directly advocating for a preferred political party, but rather by disproportionately criticizing one side, a convention that further moderates overall differences.

There has been a marked shift towards learning and consuming information through video. Most academic research, however, is still distributed only in text form, as researchers often have limited time, resources, and incentives to create video versions of their work. To address this gap, we propose, deploy, and evaluate a scalable, end-to-end system for crowdsourcing the creation of short, 5-minute research videos based on academic papers. Doing so requires solving complex coordination and collaborative video production problems. To assist coordination, we designed a structured workflow that enables efficient delegation of tasks, while also motivating the crowd through a collaborative learning environment. To facilitate video production, we developed an online tool with which groups can make micro-audio recordings that are automatically stitched together to create a complete talk. We tested this approach with a group of volunteers recruited from 52 countries through an open call. This distributed crowd produced over 100 video talks in 12 languages based on papers from top-tier computer science conferences. The produced talks consistently received high ratings from a diverse group of non-experts and experts, including the authors of the original papers. These results indicate that our crowdsourcing approach is a promising method for producing high-quality research talks at scale, increasing the distribution and accessibility of scientific knowledge.

While most existing research knowledge is distributed in the form of papers, there has been a shift towards learning and consuming information through video. Limited time and resources, however, prevent individual researchers from making their work available in this format. Crowdsourcing this task is a promising alternative, but it requires solving complex coordination and collaborative video production problems. In this paper, we propose an end-to-end solution to crowdsource the creation of research videos. To assist coordination, we designed a structured workflow that enables efficient delegation of tasks, while also providing a collaborative learning environment to the crowd for motivation. To facilitate video production, we developed an online system through which groups can make micro audio recordings that are automatically stitched together to create a complete talk. We tested this approach with a group of volunteers recruited from 24 countries through an open call. This distributed crowd produced five video talks based on the best paper winners and nominees from WWW 2016. Evaluations from the authors of the papers and outside reviewers rated the talks "very good" (giving a median score of 4 out of 5). We further applied this method to translate these talks and produce 11 additional videos in Spanish, Romanian, and Catalan. These results suggest that our crowdsourcing approach has the potential to significantly increase learning and the accessibility of the scientific knowledge.

What explains the relative persistence of same-race romantic relationships? One possible explanation is structural–this phenomenon could reflect the fact that social interactions are already stratified along racial lines–while another attributes these patterns to individual-level preferences. We present novel evidence from an online dating community involving more than 250,000 people in the United States about the frequency with which individuals both express a preference for same-race romantic partners and act to choose same-race partners. Prior work suggests that political ideology is an important correlate of conservative attitudes about race in the United States, and we find that conservatives, including both men and women and blacks and whites, are much more likely than liberals to state a preference for same-race partners. Further, conservatives are not simply more selective in general; they are specifically selective with regard to race. Do these stated preferences predict real behaviors? In general, we find that stated preferences are a strong predictor of a behavioral preference for same-race partners, and that this pattern persists across ideological groups. At the same time, both men and women of all political persuasions act as if they prefer same-race relationships even when they claim not to. As a result, the gap between conservatives and liberals in revealed same-race preferences, while still substantial, is not as pronounced as their stated attitudes would suggest. We conclude by discussing some implications of our findings for the broader issues of racial homogamy and segregation.

In the course of conducting traffic stops, officers have discretion to search motorists for drugs, weapons, and other contraband. There is concern that these search decisions are prone to racial bias, but it has proven difficult to rigorously assess claims of discrimination. Here we develop a new statistical method --- the threshold test --- to test for racial discrimination in motor vehicle searches. We use geographic variation in stop outcomes to infer the effective race-specific standards of evidence that officers apply when deciding whom to search, an approach we formalize with a hierarchical Bayesian latent variable model. This technique mitigates the problems of omitted variables and infra-marginality associated with benchmark and outcome tests for discrimination. On a dataset of 4.5 million police stops in North Carolina, we find that the standard for searching black and Hispanic drivers is considerably lower than the standard for searching white and Asian drivers, a pattern that holds consistently across the 100 largest police departments in the state.

It is often asserted that friends and acquaintances have more similar beliefs and attitudes than do strangers; yet empirical studies disagree over exactly how much diversity of opinion exists within local social networks and, relatedly, how much awareness individuals have of their neighbors' views. This article reports results from a network survey, conducted on the Facebook social networking platform, in which participants were asked about their own political attitudes, as well as their beliefs about their friends' attitudes. Although considerable attitude similarity exists among friends, the results show that friends disagree more than they think they do. In particular, friends are typically unaware of their disagreements, even when they say they discuss the topic, suggesting that discussion is not the primary means by which friends infer each other's views on particular issues. Rather, it appears that respondents infer opinions in part by relying on stereotypes of their friends and in part by projecting their own views. The resulting gap between real and perceived agreement may have implications for the dynamics of political polarization and theories of social influence in general.

Internet search companies generate the majority of their multibillion dollar revenue from selling ad placement, often in the form of short, textual ads that appear next to web search results. Such sponsored search ads are typically sold via pay-per-click auctions in which advertisers bid for placement, but pay publishers (i.e., search companies) only when users click on their ads. As payments are made only when ads are clicked, publishers consider both the advertiser's click bid and also the ad's quality (i.e., the probability of it being clicked) when deciding which ads to display [Edelman et al. 2007].

Online publishing, social networks, and web search have dramatically lowered the costs of producing, distributing, and discovering news articles. Some scholars argue that such technological changes increase exposure to diverse perspectives, while others worry that they increase ideological segregation. We address the issue by examining webbrowsing histories for 50,000 US-located users who regularly read online news. We find that social networks and search engines are associated with an increase in the mean ideological distance between individuals. However, somewhat counterintuitively, these same channels also are associated with an increase in an individual's exposure to material from his or her less preferred side of the political spectrum. Finally, the vast majority of online news consumption is accounted for by individuals simply visiting the home pages of their favorite, typically mainstream, news outlets, tempering the consequences -- both positive and negative -- of recent technological changes. We thus uncover evidence for both sides of the debate, while also finding that the magnitude of the effects is relatively modest

Decision makers can benefit from the subjective judgment of experts. For example, estimates of disease prevalence are quite valuable, yet can be difficult to measure objectively. Useful features of mechanisms for aggregating expert opinions include the ability to: (1) incentivize participants to be truthful; (2) adjust for the fact that some experts are better informed than others; and (3) circumvent the need for objective, "ground truth" observations. Subsets of these properties are attainable by previous elicitation methods, including proper scoring rules, prediction markets, and the Bayesian truth serum. Our mechanism of collective revelation, however, is the first to simultaneously achieve all three. Furthermore, we introduce a general technique for constructing budget-balanced mechanisms-where no net payments are made to participants--that applies both to collective revelation and to past peer-prediction methods.

In sponsored search auctions advertisers typically pay a fixed amount per click that their advertisements receive. In particular, the advertiser and the publisher enter into a contract (e.g., the publisher displays the ad; the advertiser pays the publisher 10 cents per click), and each party's subjective value for such a contract depends on their estimated click-through rates (CTR) for the ad. Starting from this motivating example, we define and analyze a class of contract auctions that generalize the classical second price auction. As an application, we introduce impression-plus-click pricing for sponsored search, in which advertisers pay a fixed amount per impression plus an additional amount if their ad is clicked. Of note, when the advertiser's estimated CTR is higher than the publisher's estimated CTR, both parties find negative click payments advantageous, where the advertiser pays the publisher a premium for the impression but the publisher then pays the advertiser per click.