Search

Redundant or excessive information can sometimes lead people to lean on it unnecessarily. Certain experimental designs can sometimes bias results in the researcher's favor. And, sometimes, interesting effects are too small to be studied, practically, or are simply zero. We believe a confluence of these factors led to a recent paper (Isaac & Brough, 2014, JCR). This initial paper proposed a new means by which probability judgments can be led astray: the category size bias, by which an individual event coming from a large category is judged more likely to occur than an event coming from a small one. Our work shows that this effect may be due to instructional and mechanical confounds, rather than interesting psychology. We present eleven studies with over ten times the sample size of the original in support of our conclusion: We replicate three of the five original studies and reduce or eliminate the effect by resolving these methodological issues, even significantly reversing the bias in one case (Study 6). Studies 7-8c suggest the remaining two studies are false positives. We conclude with a discussion of the subtleties of instruction wording, the difficulties of correcting the record, and the importance of replication and open science. [ABSTRACT FROM AUTHOR]

This paper presents the results of a study that sheds new light on the shape of indifference curves in the Marschak-Machina triangle. The most important observation, obtained non-parametrically, concerns jumps in indifference curves at the triangle legs towards the triangle origin. These jumps, however, do not appear at the hypotenuse. The pattern observed suggests discontinuity in lottery valuation when the range of lottery outcomes changes and is best explained by decision-making models based on the psychological phenomenon of range dependence (Parducci, 1965; Cohen, 1992; Kontek & Lewandowski, 2018). Models founded on other psychological phenomena, e.g., discontinuity in decision weights (Kahneman & Tversky, 1979), cumulative probability weighting (Tversky & Kahneman, 1992), attention shifting (Birnbaum, 2008), overweighting of salient payoffs (Bordallo, Gennaioli & Shefrin, 2012), and treating stated probabilities as imperfect information (Viscusi, 1989), predict indifference curve shapes that differ from the one obtained in this study. [ABSTRACT FROM AUTHOR]

This paper explored how frames influence people's evaluation of others' probabilistic predictions in light of the outcomes of binary events. Most probabilistic predictions (e.g., "there is a 75% chance that Denver will win the Super Bowl") can be partitioned into two components: A qualitative component that describes the predicted outcome ("Denver will win the Super Bowl"), and a quantitative component that represents the chance of the outcome occurring ("75% chance"). Various logically equivalent variations of a single prediction can be created through different combinations of these components and their logical or numerical complements (e.g., "25% chance that Denver will lose the Super Bowl", "75% chance that Seattle will lose the Super Bowl"). Based on the outcome of the predicted event, these logically equivalent predictions can be categorized into two classes: Congruently framed predictions, in which the qualitative component matches the outcome, and incongruently framed predictions, in which it does not. Although the two classes of predictions are logically equivalent, we hypothesize that people would judge congruently framed predictions to be more accurate. The paper tested this hypothesis in seven experiments and found supporting evidence across a number of domains and experimental manipulations, and even when the congruently framed prediction was logically inferior. It also found that this effect held even for subjects who saw both congruently framed and incongruently framed versions of a prediction and judged the two to be logically equivalent. [ABSTRACT FROM AUTHOR]

Stastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community predictionmarket to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were asked to read the reports and state what probability they thought the reports implied for each forecast question. These were called imputed probabilities. Stastny and Lehner found that the prediction market was more accurate than the imputed probabilities and concluded that this was evidence that the prediction market was more accurate than the analysis reports. In a commentary, Mandel (2019) took exception to this interpretation. In a re-analysis of the data, Mandel found a very strong correlation between readers' personal and imputed probabilities. From this Mandel builds a case that the imputed probabilities are little more than a reflection of the readers' personal views; that they do not fairly reflect the contents of the analysis reports; and therefore, any accuracy results are spurious. This paper argues two points. First, the high correlation between imputed and personal probabilities was not evidence of substantial imputation bias. Rather it was the natural by-product of the fact that the imputed and personal probabilities were both forecasts of the same events. An additional analysis shows a much lower level of imputation bias that is consistent with the original results and interpretation. Second, the focus of Stastny and Lehner (2018) was on the reports as understood by readers. In this context, even if there was substantial imputation bias it would not invalidate accuracy results; it would instead provide a possible causal explanation of those results. [ABSTRACT FROM AUTHOR]

This paper summarizes an empirical comparison of the accuracy of forecasts included in analysis reports developed by professional intelligence analysts to comparable forecasts in a prediction market that has broad participation from across an intelligence community. To compare forecast accuracy, 99 event forecasts were extracted from qualitative descriptions found in 41 analysis reports and posted on the prediction market. Quantitative probabilities were imputed from the qualitative forecasts by asking seasoned professional analysts, who did not participate in the prediction market, to read the reports and to infer a quantitative probability based on what was written. These readers were also asked to provide their personal probabilities before and after reading the reports. There were two statistically significant results of particular interest. First, the primary result is that the prediction market forecasts were more accurate than the analysis reports. On average prediction market probabilities were 0.114 closer to ground truth than the analysis report probabilities. Second, in cases where analysts (readers) updated their personal probabilities in a direction opposite to what the reports implied, analysts tended to update their probabilities in the correct direction. This occurred even though, on average, reading the reports did not make readers more accurate. [ABSTRACT FROM AUTHOR]

Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods. [ABSTRACT FROM AUTHOR]

When decisions involve opting in or out of competition many decision makers will opt-in even when doing so leads to losses on average. In the current paper, we examine the generality of this effect in risky choices not involving competition. We found that re-framing a sure (certain) zero option as an option to observe the results of the other options without choosing would lead to increased consequential choice (i.e., increased selection of risky options rather than the zero option). Specifically, in two studies we compared the rate of consequential choice in a novel paradigm where decision makers decide to observe or to choose with consequence from a set of risky options (decisions-to-engage) to a full-feedback decisions-from-feedback paradigm where the choice set included a labeled sure zero option. Compared to decisions-from-feedback, participants were more likely to choose from mixed (risky) gambles with consequence (over a zero outcome) in decisions-to-engage. This occurred irrespective of whether doing so was advantageous (i.e., when choice led to monetary gains on average) or disadvantageous (i.e., when choice led to monetary losses on average), and when descriptions of the options outcomes and probabilities were provided (Study 2). These findings provide an important boundary condition for the positive effects of experience on the quality of choice, and suggest that decision makers' preference for agency can sometimes induce poorer choices. [ABSTRACT FROM AUTHOR]

In most previous studies of verbal probabilities, participants are asked to translate expressions such as possible and not certain into numeric probability values. This probabilistic translation approach can be contrasted with a novel which-outcome (WO) approach that focuses on the outcomes that people naturally associate with probability terms. The WO approach has revealed that, when given bell-shaped distributions of quantitative outcomes, people tend to associate certainty with minimum (unlikely) outcome magnitudes and possibility with (unlikely) maximal ones. The purpose of the present paper is to test the factors that foster these effects and the conditions in which they apply. Experiment 1 showed that the association of probability term and outcome was related to the association of scalar modifiers (i.e., it is certain that the battery will last at least..., it is possible that the battery will last up to...). Further, we tested whether this pattern was dependent on the frequency (e.g., increasing vs. decreasing distribution) or the nature of the outcomes presented (i.e., categorical vs. continuous). Results showed that despite being slightly affected by the shape of the distribution, participants continue to prefer to associate possible with maximum outcomes and certain with minimum outcomes. The final experiment provided a boundary condition to the effect, showing that it applies to verbal but not numerical probabilities. [ABSTRACT FROM AUTHOR]

Research on forecasting is effectively limited to forecasts that are expressed with clarity; which is to say that the forecasted event must be sufficiently well-defined so that it can be clearly resolved whether or not the event occurred and forecasts certainties are expressed as quantitative probabilities. When forecasts are expressed with clarity, then quantitative measures (scoring rules, calibration, discrimination, etc.) can be used to measure forecast accuracy, which in turn can be used to measure the comparative accuracy of different forecasting methods. Unfortunately most real world forecasts are not expressed clearly. This lack of clarity extends to both the description of the forecast event and to the use of vague language to express forecast certainty. It is thus difficult to assess the accuracy of most real world forecasts, and consequently the accuracy the methods used to generate real world forecasts. This paper addresses this deficiency by presenting an approach to measuring the accuracy of imprecise real world forecasts using the same quantitative metrics routinely used to measure the accuracy of well-defined forecasts. To demonstrate applicability, the inferred probability method is applied to measure the accuracy of forecasts in fourteen documents examining complex political domains. [ABSTRACT FROM AUTHOR]

The tempting fate effect is that the probability of a fateful outcome is deemed higher following an action that "tempts" the outcome than in the absence of such an action. In this paper we evaluate the hypothesis that the effect is due to an illusion of control induced by a causal framing of the situation. Causal frames require that the action make a difference to an outcome and that the action precedes the outcome. If an illusion of control modulates the reluctance to tempt fate, then actions that make a difference to well-being and that occur prior to the outcome should tempt fate most strongly. In Experiments 1-3 we varied whether the action makes a difference and the temporal order of action and outcome. In Experiment 4 we tested whether an action can tempt fate if all outcomes are negative. The results of all four experiments supported our hypothesis that the tempting fate effect depends on a causal construal that gives rise to a false sense of control. [ABSTRACT FROM AUTHOR]