Search

Expert judgment often involves estimating magnitudes, such as the frequency of deaths due to a pandemic. Three experiments (Ns = 902, 431, and 755, respectively) were conducted to examine the effect of outcome framing (e.g., half of a threatened group expected to survive vs. die), probability level (low vs. high), and probability format (verbal, numeric, or combined) on the estimated frequency of survivals/deaths. Each experiment found an interactive effect of frame and probability level, which supported the hypothesis that forecasted outcomes received by participants were implicitly quantified as lower bounds (i.e., "at least half"). Responding in a manner consistent with a lower‐bound "at least" interpretation was unrelated to incoherence (Experiments 1 and 2) and positively related to numeracy (Experiments 1 and 3), verbal reasoning (Experiment 3), and actively open‐minded thinking (Experiments 2 and 3). The correlational results indicate that implicit lower bounding is an aspect of linguistic inference and not a cognitive error. Implications for research on framing effects are discussed. [ABSTRACT FROM AUTHOR]

For the last half-century, the U.S. and Allied Intelligence Community (IC) has sought to minimize the ostensibly detrimental effects of cognitive biases on intelligence practice. The dominant approach has been to develop structured analytic techniques (SATs), teach them to analysts in brief training sessions, provide the means to use SATs on the job, and hope they work. The SAT approach, however, suffers from severe conceptual problems and a paucity of support from scientific research. For example, a highly promoted SAT—the analysis of competing hypotheses—was shown in several recent studies to either not improve judgment quality or to make it worse. This article recaps the key problems with the SAT approach and sketches some alternative interventions. At the core of these proposals is the idea that intelligence agencies should be focused broadly on improving intelligence and not narrowly on minimizing bias. While the latter contributes to achieving the former, overemphasis on bias minimization could inadvertently bias agencies toward a singular form of intervention, blinding then from potentially more effective interventions. Two lines of alternative intervention are sketched. The first line focuses on postanalytic statistical optimization methods such as recalibration and performance-weighted aggregation of analysts' judgments. The second line focuses on a broad human augmentation program to optimize human cognition through better sleep, exercise, nutrition (including nootropic compounds), and biometric tracking. Both lines of effort would require substantial scientific investment by the IC to examine risks and efficacy. [ABSTRACT FROM AUTHOR]

The intelligence community uses "structured analytic techniques" to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)--a technique designed to reduce "confirmation bias." Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH-trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.

Experts often communicate probabilities verbally (e.g., unlikely) rather than numerically (e.g., 25% chance). Although criticism has focused on the vagueness of verbal probabilities, less attention has been given to the potential unintended, biasing effects of verbal probabilities in communicating probabilities to decision-makers. In four experiments (Ns = 201, 439, 435, 696), we showed that probability format (i.e., verbal vs. numeric) influenced participants' inferences and decisions following a hypothetical financial expert's forecast. We observed a format effect for low probability forecasts: verbal probabilities were interpreted more pessimistically than numeric equivalents. We attributed the difference to directionality, a linguistic property that biases attention toward an outcome. In the high-probability conditions, the directionality of verbal and numeric probabilities aligned (both were positive), whereas they differed in the low-probability conditions (verbal probabilities were more negative). Participants inferred recommendations congruent with the communicated direction and these inferences mediated the effect of probability format on decisions. [ABSTRACT FROM AUTHOR]

Meta‐information is information about information that can be used as cues to guide judgments and decisions. Three types of meta‐information that are routinely used in intelligence analysis are source reliability, information credibility, and classification level. The first two cues are intended to speak to information quality (in particular, the probability that the information is accurate), and classification level is intended to describe the information's security sensitivity. Two experiments involving professional intelligence analysts (N = 25 and 27, respectively) manipulated meta‐information in a 6 (source reliability) × 6 (information credibility) × 2 (classification) repeated‐measures design. Ten additional items were retested to measure intra‐individual reliability. Analysts judged the probability of information accuracy based on its meta‐informational profile. In both experiments, the judged probability of information accuracy was sensitive to ordinal position on the scales and the directionality of linguistic terms used to anchor the levels of the two scales. Directionality led analysts to group the first three levels of each scale in a positive group and the fourth and fifth levels in a negative group, with the neutral term "cannot be judged" falling between these groups. Critically, as reliability and credibility cue inconsistency increased, there was a corresponding decrease in intra‐analyst reliability, interanalyst agreement, and effective cue utilization. Neither experiment found a significant effect of classification on probability judgments. [ABSTRACT FROM AUTHOR]

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also communicated using terms such as low, moderate, or high to describe the analyst's confidence level. However, little research has examined how intelligence professionals interpret these terms and whether they prefer them to numeric uncertainty quantifiers. In two experiments (N = 481 and 624, respectively), uncertainty communication preferences of expert (n = 41 intelligence analysts in Experiment 1) and nonexpert intelligence consumers were elicited. We examined which format participants judged to be more informative and simpler to process. We further tested whether participants treated verbal probability and confidence terms as independent constructs and whether participants provided coherent numeric probability translations of verbal probabilities. Results showed that although most nonexperts favored the numeric format, experts were about equally split, and most participants in both samples regarded the numeric format as more informative. Experts and nonexperts consistently conflated probability and confidence. For instance, confidence intervals inferred from verbal confidence terms had a greater effect on the location of the estimate than the width of the estimate, contrary to normative expectation. Approximately one‐fourth of experts and over one‐half of nonexperts provided incoherent numeric probability translations for the terms likely and unlikely when the elicitation of best estimates and lower and upper bounds were briefly spaced by intervening tasks. [ABSTRACT FROM AUTHOR]

As the US-China great power competition intensifies, public opinion polling may help gauge internal drivers of foreign policy decision-making. Using Pew Research Center data, the authors analyzed how American and Chinese respondents viewed their own and each other's countries between 2008–2016. They further examined how American attitudes towards China varied by political affiliation between 2008–2019. Both Americans and the Chinese displayed ingroup bias (i.e. rating their own country more positively than the other) and viewed China as a challenger to US hegemony. However, while the Chinese exhibited higher levels of ingroup bias overall, there was no evidence of increasing bias over time. Meanwhile, Americans showed increasing ingroup bias, primarily due to their souring evaluations of China, a tendency that was strongest among Republicans. [ABSTRACT FROM AUTHOR]

Coherent judgment is a cardinal feature of rational cognition. Six experiments revealed systematic violations of coherence in probability judgment in which participants assigned different probabilities to mathematically equiprobable events. Experiments 1-5 revealed a "strict refocusing effect": Compared to an occurrence frame, a non-occurrence frame resulted in higher estimates if base-rate evidence favored occurrence, lower estimates if evidence favored non-occurrence, and similar estimates if evidence supported indifference. Moreover, Experiments 5 and 6 revealed a "pessimistic bias" in which the less favorable of two equiprobable events was assigned greater probability. The findings support a Representational and Assessment Processes account (RAP) in which subjective probability is influenced by the perceived compatibility between representations of focal events and representations of evidence.