Search

Purpose: Evaluation of the added value of Dietary Approaches to Stop Hypertension (DASH) and Mediterranean diet scores on the prediction model of the World Health Organization (WHO) to predict 10-year cardiovascular disease (CVD) mortality using the Golestan Cohort Study data. Methods: A total of 44,648 participants (25,268 women and 18,531 men) were included in the final analysis. To assess the external validity of the non-laboratory risk model of WHO, the Area Under the Curve (AUC) and calibration plot methods were used. The multivariate Cox proportional hazards regression analysis was used to evaluate the association of 10-year CVD mortality risk with DASH and Mediterranean scores and their components. The added value of each significant variables was evaluated by the concordance C-statistic and integrated discrimination improvement (IDI). Statistical significance was defined as p-value < 0.05. Results: DASH and Mediterranean diet scores were not significant predictors of 10-year CVD mortality in both genders (p > 0.05). However, sodium and total vegetable in both genders and added sugar in women were significant predictors for 10-year stroke mortality (p < 0.05). Sodium intake in women and monounsaturated fatty acid (MUFA) to saturated fatty acid (SFA) ratio in men had significant associations with 10-year mortality of myocardial infarction/coronary heart disease (MI/CHD). Calculation of IDI showed that none of the evaluated nutritional indices/variables could significantly improve the WHO model performance and predictive ability. Conclusion: Inclusion of DASH and Mediterranean diet scores and their components did not improve WHO risk prediction model performance and predictive ability to predict 10-year CVD mortality. [ABSTRACT FROM AUTHOR]

Background Carbapenem-resistant Enterobacterales (CRE) bloodstream infections (BSIs) are a major threat to patients. To date, data on risk factors have been limited, with low internal and external validity. In this multicentre study, risk factors for CRE BSI were determined by comparison with two control groups: patients with carbapenem-susceptible Enterobacterales (CSE) BSI, and patients without Enterobacterales infection (uninfected patients). Methods A multicentre, case-control-control study was nested in a European prospective cohort study on CRE (EURECA). CRE BSI:CSE BSI matching was 1:1, CRE BSI:Uninfected patients matching was 1:3, based on hospital, ward and length of stay. Conditional logistic regression was applied. Results From March 2016 to November 2018, 73 CRE BSIs, 73 CSE BSIs and 219 uninfected patients were included from 18 European hospitals. For CRE versus CSE BSI, previous CRE colonization/infection [incidence rate ratio (IRR) 7.32; 95% CI 1.65–32.38) increased the risk. For CRE versus uninfected controls, independent risk factors included: older age (IRR 1.03; 95% CI 1.01–1.06), patient referral (long-term care facility: IRR 7.19; 95% CI 1.51–34.24; acute care hospital: IRR 5.26; 95% CI 1.61–17.11), previous colonization/infection with other MDR organisms (MDROs) (IRR 9.71; 95% CI 2.33–40.56), haemodialysis (IRR 8.59; 95% CI 1.82–40.53), invasive procedures (IRR 5.66; 95% CI 2.11–15.16), and β-lactam/β-lactamase inhibitor combinations (IRR 3.92; 95% CI 1.68–9.13) or third/fourth generation cephalosporin (IRR 2.75; 95% CI 1.06–7.11) exposure within 3 months before enrolment. Conclusions Evidence of previous CRE colonization/infection was a major risk factor for carbapenem resistance among Enterobacterales BSI. Compared with uninfected patients, evidence of previous MDRO colonization/infection and healthcare exposure were important risk factors for CRE BSI. Targeted screening, infection prevention and antimicrobial stewardship should focus on these high-risk patients. [ABSTRACT FROM AUTHOR]

The objectives of our study were 1) to examine whether the force-velocity-power (FvP) outcomes in the concentric and eccentric phases of flywheel (FW) squats differ among sports disciplines and 2) to investigate the association between FvP outcome variables and two key sport-performance indicators: countermovement jump height (CMJ) and change of direction (CoD) time involving 90° or 180° turns. Tests were performed by 469 athletes from five different sport disciplines and physical education students. Our results showed that FvP outcomes, when measured during the eccentric phase of the FW squat, effectively differed between athletes from different sports. However, during the concentric phase of the squat, only for the slope variable differences between sports were found. Contrary to our hypothesis, there were no statistically significant correlations between FW squat FvP outcomes and CMJ or CoD test results. These results suggest that FvP outcomes derived from FW squats may lack external validity and cannot be reliably used as a predictor of athletes' jumping and agility performance. The paper discusses possible reasons for the larger differences between sports in eccentric FvP outcomes, as well as the absence of correlations between FvP outcomes and functional tests. [ABSTRACT FROM AUTHOR]

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations. [ABSTRACT FROM AUTHOR]

In the DSM-5 Alternative Model of Personality Disorders (AMPD), psychopathy is marked by the presence of attention seeking, low anxiousness, and lack of social withdrawal, along with traits from the domains of Antagonism and Disinhibition. The triarchic model of psychopathy (TriPM) posits three biobehaviorally based traits underlying it: disinhibition, meanness, and boldness. The current study directly compared relations for measures of the two models with the broad dimensions of externalizing, internalizing, and positive adjustment. Participants (1,678 adults) were surveyed regarding maladaptive personality traits, clinical symptoms, and positive adjustment features. The TriPM model explained more variance than the AMPD in substance use, positive adjustment, and empathy, whereas the AMPD model explained more variance in internalizing symptoms. In addition, AMPD Antagonism and the Psychopathy Specifier diverged from TriPM Meanness and Boldness in their associations with some specific outcomes. Overall, our study provides evidence for complementarity of the two models in characterizing the multifaceted nature of psychopathy. [ABSTRACT FROM AUTHOR]

AbstractObjective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems. [ABSTRACT FROM AUTHOR]

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology. [ABSTRACT FROM AUTHOR]

ObjectiveMethodResultsConclusionTo test the predictive accuracy and generalisability of a personalised advantage index (PAI) model designed to support treatment selection for Post-Traumatic Stress Disorder (PTSD).A PAI model developed by Deisenhofer et al. (2018) was used to predict treatment outcomes in a statistically independent dataset including archival records for N = 152 patients with PTSD who accessed either trauma-focussed cognitive behavioural therapy or eye movement desensitisation and reprocessing in routine care. Outcomes were compared between patients who received their PAI-indicated optimal treatment versus those who received their suboptimal treatment.The model did not yield treatment specific predictions and patients who had received their PAI-indicated optimal treatment did not have better treatment outcomes in this external validation sample.This PAI model did not generalise to an external validation sample. [ABSTRACT FROM AUTHOR]

Scholars have made considerable strides in evaluating and improving the external validity of experimental research. However, little attention has been paid to a crucial aspect of external validity – the topic of study. Researchers frequently develop a general theory and hypotheses (e.g., about policy attitudes), then conduct a study on a specific topic (e.g., environmental attitudes). Yet, the results may vary depending on the topic chosen. In this paper, we develop the idea of topic sampling – rather than studying a single topic, we randomly sample many topics from a defined population. As an application, we combine topic sampling with a classic survey experiment design on partisan cues. Using a hierarchical model, we efficiently estimate the effect of partisan cues for each policy, showing that the size of the effect varies considerably, and predictably, across policies. We conclude with advice on implementing our approach and using it to improve theory testing. [ABSTRACT FROM AUTHOR]

In this article, we review the ways in which replication has been and could be featured in consumer behavior, using Journal of Consumer Research as a specific setting. We present a framework for thinking about the generalizability of research findings and differentiate various potential benefits that replication can have for understanding variability in consumer research findings. We then define four different types of replications, describe how researchers can use these approaches to produce distinct benefits, and give guidance regarding conducting, interpreting, and the potential contributions of these different types of replications. We conclude with a discussion of various ways in which replication could be more fully integrated into different phases of the scientific research process, taking into account the contribution necessary for publication. In particular, we identify opportunities to incorporate independent replication into original papers, to increase the replication-based contribution in papers that build on prior work, and to use systematic replication in conjunction with meta-analysis to synthesize and confirm conclusions from a mature research literature. More fully integrating replication into scientific practice can yield a new equilibrium, in which replication is routine, typically consistent with previous results, and recognized as necessary for establishing an empirical generalization. [ABSTRACT FROM AUTHOR]

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts. [ABSTRACT FROM AUTHOR]

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail. [ABSTRACT FROM AUTHOR]

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator. [ABSTRACT FROM AUTHOR]

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three‐round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty‐four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus‐based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity. [ABSTRACT FROM AUTHOR]

Objectives Agriculture entered the discussions about safety climates late, despite being one of the most hazardous industries. We recently developed a safety climate scale in Bordeaux vineyards, for which we provided good evidence of reliability and construct validity (Grimbuhler and Viel 2019). In this study, we aimed to establish the external validity of this safety climate scale with the help of an independent national sample of vineyard professionals. Population and methods We approached vineyard managers and operators during compulsory training and certification procedures for pesticide-related activities. Trainees giving informed consent for participation in the study were asked to complete a safety climate questionnaire at the start and end of a training session. In total, 406 vineyard managers or operators completed the questionnaire at the start of the study, 37 of whom declined to complete the questionnaire at the end of the training session, leaving 369 subjects available for pretraining/post-training comparisons. Statistical comparisons were based on t -tests and mixed models for repeated measures. Results A mean safety climate score of 82.91 (SD: 9.06) was obtained in the initial survey in the Bordeaux region, whereas the safety climate score was estimated at 83.78 (SD 10.39) in this nationwide survey (P = 0.23). A significant increase was observed after the training course, for the safety climate score (7.5%, P < 10–15) and for each of its 7 dimensions (P < 10–4 or less), in both univariate and multivariate analyses. However, the magnitude of these increases varied with dimension, ranging from 2.4% for rules and best practices to 15.5% for communication and feedback. Conclusions External validity was demonstrated by transferability and sensitivity to intervention. This safety climate scale can now be considered to provide a good inference of the safety culture, with a meaning generalizable across vineyards. [ABSTRACT FROM AUTHOR]

Generalizability of trial-based cost-effectiveness estimates to real-world target populations is important for decision making. In the context of independent aggregate time-to-event baseline and relative effects data, complex hazards can make modeling of data for use in economic evaluation challenging. Our article provides an overview of methods that can be used to apply trial-derived relative treatment effects to external real-world baselines when faced with complex hazards and follows with a motivating example. Approaches for applying trial-derived relative effects to real-world baselines are presented in the context of complex hazards. Appropriate methods are applied in a cost-effectiveness analysis using data from a previously published study assessing the real-world cost-effectiveness of a treatment for carcinoma of the head and neck as a motivating example. Lack of common hazards between the trial and target real-world population, a complex baseline hazard function, and nonproportional relative effects made the use of flexible models necessary to adequately estimate survival. Assuming common distributions between trial and real-world reference survival substantially affected survival and cost-effectiveness estimates. Modeling time-dependent vs proportional relative effects affected estimates to a lesser extent, dependent on assumptions used in cost-effectiveness modeling. Appropriately capturing reference treatment survival when attempting to generalize trial-derived relative treatment effects to real-world target populations can have important impacts on cost-effectiveness estimates. A balance between model complexity and adequacy for decision making should be considered where multiple data sources with complex hazards are being evaluated. • Generalizability is an important consideration for decision making. • Approaches used for applying trial-derived treatment effects to real-world target population baseline risks can substantially affect cost-effectiveness estimates. • Model complexity in light of requirements for informed decision making should be considered when generalizing cost-effectiveness estimates. [ABSTRACT FROM AUTHOR]

Considerations related to generalization of a study's findings are often interconnected to researchers' judgements regarding the 'quality' of the methodology and methodological pluralism. Too often, researchers consider generalization as only possible with respect to quantitative studies with large numbers of randomly selected participants (statistical generalization). Recently, Levitt suggests qualitative research can be generalized to phenomenon. Like others, she differentiates the type of generalization typically identified with quantitative research, which requires large samples and randomization so that findings can be generalized to the population, with the type of generalization that can be achieved with qualitative research. In turn, Levitt proposes the concept of qualitative generalization, an idea suggested by other researchers. However, her assertions about generalization in qualitative research resonate with the assertions of Thomas and Baas about generalizability in Q methodology. Q methodology offers a unique blend of qualitative and quantitative principles to study subjectivity and includes factor analysis. Yet Q findings are about developing theories, much like qualitative research. Q researchers may discuss substantive inference, connecting generalizations about phenomenon to a population, rather than statistical inference about a population. We discuss acceptance of two types of generalization within and their importance in relation to research equity. [ABSTRACT FROM AUTHOR]

Matching crop varieties to their target use context and user preferences is a challenge faced by many plant breeding programs serving smallholder agriculture. Numerous participatory approaches proposed by CGIAR and other research teams over the last four decades have attempted to capture farmers' priorities/preferences and crop variety field performance in representative growing environments through experimental trials with higher external validity. Yet none have overcome the challenges of scalability, data validity and reliability, and difficulties in capturing socio-economic and environmental heterogeneity. Building on the strengths of these attempts, we developed a new data-generation approach, called triadic comparison of technology options (tricot). Tricot is a decentralized experimental approach supported by crowdsourced citizen science. In this article, we review the development, validation, and evolution of the tricot approach, through our own research results and reviewing the literature in which tricot approaches have been successfully applied. The first results indicated that tricot-aggregated farmer-led assessments contained information with adequate validity and that reliability could be achieved with a large sample. Costs were lower than current participatory approaches. Scaling the tricot approach into a large on-farm testing network successfully registered specific climatic effects of crop variety performance in representative growing environments. Tricot's recent application in plant breeding networks in relation to decision-making has (i) advanced plant breeding lines recognizing socio-economic heterogeneity, and (ii) identified consumers' preferences and market demands, generating alternative breeding design priorities. We review lessons learned from tricot applications that have enabled a large scaling effort, which should lead to stronger decision-making in crop improvement and increased use of improved varieties in smallholder agriculture. [ABSTRACT FROM AUTHOR]

The purpose of this study was to explore the validity of evidence for self-efficacy to regulate physical activity scale (SERPA) measurement using an exploratory latent variable approach. The objectives were to explore the dimensionality, temporal invariance, and external validity of scores produced by the SERPA, a modified version of the barriers self-efficacy scale. Data (Nbaseline= 461 and N30 days post-baseline = 427) from the Well-Being and Physical Activity Study (ClinicalTrials.gov, identifier: NCT03194854), which deployed the Fun For Wellness intervention, were analyzed. A two-dimensional factor structure explained responses to the SERPA at baseline. There was evidence for at least partial strict temporal measurement invariance for this two-dimensional structure. There was evidence for external validity of the SERPA measurement as evidenced by group assignment (i.e. the intervention) exerting a direct effect on the proposed two-dimensional structure of latent self-efficacy to regulate physical activity in adults with obesity at 30 days post-baseline. [ABSTRACT FROM AUTHOR]

Roberts and colleagues focus on two aspects of racial inequality in psychological research, namely an alleged underrepresentation of racial minorities and the effects attributed to this state of affairs. My comment focuses only on one aspect, namely the assumed consequences of the lack of diversity in subject populations. Representativeness of samples is essential in survey research or applied research that examines whether a particular intervention will work for a particular population. Representativeness or diversity is not necessary in theory-testing research, where we attempt to establish laws of causality. Because theories typically apply to all of humanity, all members of humanity (even American undergraduates) are suitable for assessing the validity of theoretical hypotheses. Admittedly, the assumption that a theory applies to all of humanity is also a hypothesis that can be tested. However, to test it, we need theoretical hypotheses about specific moderating variables. Supporting a theory with a racially diverse sample does not make conclusions more valid than support from a nondiverse sample. In fact, cause-effect conclusions based on a diverse sample might not be valid for any member of that sample. [ABSTRACT FROM AUTHOR]

While much of the causal inference literature has focused on addressing internal validity biases, both internal and external validity are necessary for unbiased estimates in a target population of interest. However, few generalizability approaches exist for estimating causal quantities in a target population that is not well‐represented by a randomized study but is reflected when additionally incorporating observational data. To generalize to a target population represented by a union of these data, we propose a novel class of conditional cross‐design synthesis estimators that combine randomized and observational data, while addressing their estimates' respective biases—lack of overlap and unmeasured confounding. These methods enable estimating the causal effect of managed care plans on health care spending among Medicaid beneficiaries in New York City, which requires obtaining estimates for the 7% of beneficiaries randomized to a plan and 93% who choose a plan, who do not resemble randomized beneficiaries. Our new estimators include outcome regression, propensity weighting, and double robust approaches. All use the covariate overlap between the randomized and observational data to remove potential unmeasured confounding bias. Applying these methods, we find substantial heterogeneity in spending effects across managed care plans. This has major implications for our understanding of Medicaid, where this heterogeneity has previously been hidden. Additionally, we demonstrate that unmeasured confounding rather than lack of overlap poses a larger concern in this setting. [ABSTRACT FROM AUTHOR]

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings. [ABSTRACT FROM AUTHOR]

Having a representative population in randomized clinical trials (RCTs) improves external validity and generalizability of trial results. There are limited data examining differences between RCT-enrolled and real-world populations in bloodstream infections (BSI). We conducted a scoping review aiming to review studies assessing generalizability of BSI RCT populations, to identify sub-groups that have been systematically under-represented and to explore approaches to improve external validity of future RCTs. MEDLINE, Embase, and Cochrane Library databases were searched for terms related to external validity or generalizability, BSI, and clinical trials in papers published up to 1 August 2023. Studies comparing enrolled versus nonenrolled patients, or papers discussing external validity or generalizability in the context of BSI RCTs were included. Sixteen papers were included in the final review. Five compared RCT-enrolled and nonenrolled participants from the same source population. There were significant differences between the two groups in all studies, with nonenrolled patients having a greater comorbidity burden and consistently worse outcomes including mortality. We identified several barriers to improving generalizability of RCT populations and outlined potential approaches to reduce these barriers, such as alternative/simplified consent processes, streamlining eligibility criteria and follow-up procedures, quota-based sampling techniques, and ensuring diversity in site and study team selection. Study cohorts in BSI RCTs are not representative of the general BSI patient population. As we increasingly adopt large pragmatic trials in infectious diseases, it is important to recognize the importance of maximizing generalizability to ensure that our research findings are of direct relevance to our patients. [ABSTRACT FROM AUTHOR]

Interest has increased in the recent literature on characterizing psychopathology dimensionally in hierarchical models. One dimension of psychopathology that has received considerable attention is externalizing. Although extensively studied and well-characterized in late adolescents and adults, delineation of the externalizing spectrum in youth has lagged behind. As a complement to structural analyses of externalizing, in this study, we use quantitative genetic analyses of twin data to adjudicate among alternative models of youth externalizing that differ in granularity. Specifically, we compared model fit, estimates of genetic and environmental influences on the externalizing dimension, and the average, variability, and precision of genetic and environmental influences on individual symptoms due to the externalizing dimension, specific symptom dimensions, and unique etiological influences. Given that none of these criteria are definitive on their own, we looked to the confluence of these criteria to exclude particular models while highlighting others as leading contenders. We analyzed parent-report data on 38 externalizing symptoms from a population-representative, ethnically diverse sample of 883 youth twin pairs (51% female), who were on average 8.5 years old. Although models including an externalizing composite and attention deficit hyperactivity disorder, oppositional defiant disorder, and conduct disorder diagnoses and symptom dimensions showed similar heritability to latent variable models of externalizing, models that included latent dimensions of externalizing and more fine-grained symptom dimensions fit better and were more balanced in the magnitude of genetic and environmental influences on individual symptoms due to the externalizing dimension and specific symptom dimensions. Pending replication, these more granular and elaborated model(s) can be useful for advancing research on causes and outcomes of youth externalizing and its fine-grained specific components. General Scientific Summary: This study used data on 883 twin pairs to adjudicate among alternative models of youth externalizing psychopathology. We found that models containing more fine-grained symptom factors (e.g., inattention, impulsivity, hyperactivity, irritability/negative affect, defiant behavior, aggression, and rule breaking) better captured the patterns of genetic and environmental influences on externalizing symptoms than models containing coarser attention deficit hyperactivity disorder, oppositional defiant disorder, and conduct disorder symptom factors. Our results show the advantages of more granular and elaborated models for understanding the classification and causes of youth externalizing. [ABSTRACT FROM AUTHOR]

Individuals who are Asian or Asian American, Black or African American, Native American or American Indian or Alaska Native, Native Hawaiian or Pacific Islander, and Hispanic or Latino (i.e., presently considered racial ethnic minoritized groups in the USA) lacked equal access to resources for mitigating risk during COVID-19, which highlighted public health disparities and exacerbated inequities rooted in structural racism that have contributed to many injustices, such as failing public school systems and unsafe neighborhoods. Minoritized groups are also vulnerable to climate change wherein the most severe harms disproportionately fall upon underserved communities. While systemic changes are needed to address these pervasive syndemic conditions, immediate efforts involve examining strategies to promote equitable health and well-being-which served as the impetus for this study. We conducted a descriptive analysis on the prevalence of culturally tailored interventions and reporting of sample characteristics among 885 programs with evaluations published from 2010 to 2021 and recorded in the Blueprints for Healthy Youth Development registry. Inferential analyses also examined (1) reporting time trends and (2) the relationship between study quality (i.e., strong methods, beneficial effects) and culturally tailored programs and racial ethnic enrollment. Two percent of programs were developed for Black or African American youth, and 4% targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% collapsed across race or categorized race with ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees were Hispanic or Latino. Reporting has not improved, and there was no relationship between high-quality studies and programs developed for racial ethnic youth, or samples with high proportions of racial ethnic enrollees. Research gaps on racial ethnic groups call for clear reporting and better representation to reduce disparities and improve the utility of interventions. [ABSTRACT FROM AUTHOR]

Researchers have been concerned with internal and external validity for decades, and the discussion continues. The present proposal is that there are less important and more important senses in which one can interpret internal and external validity, and these can be integrated with a taxonomy that includes theoretical, auxiliary, statistical, and inferential assumptions. The integration sheds light on recent exchanges in the literature on validity and suggests that the vaunted internal–external validity trade-off is false for more important senses of internal and external validity; internal and external validity increase or decrease together when there is an emphasis on underlying theories. Finally, the integration implies the desirability of some changes in typical research advice and practice. [ABSTRACT FROM AUTHOR]

The power to use force is a defining characteristic of policing, one that is accompanied by a responsibility to exercise these powers in the circumstances deemed necessary. This study analyzes data from four policing agencies to predict the likelihood of an officer drawing and pointing their firearm at a use of force incident. Findings suggest that situational factors were important in influencing whether an officer may draw and point their firearm. However, a priming effect, in which officers were more likely to draw their firearms when dispatched to an incident, may also be present. The rate that officers drew and pointed their firearms varied between jurisdictions, as did the nature of the incidents. Caution should be exercised in generalizing the results of single-site studies on police use of force, or introducing research into policy beyond the jurisdiction in which it was performed. [ABSTRACT FROM AUTHOR]

In this investigation, we tested hypotheses concerning how external validity, in relation to leadership and teamwork, was affected as participants moved from organizational to academic settings. Participants consisted of working business students (N = 159) from two countries, Peru and the United States, who adopted leader/teammate roles across settings. Results indicated that (a) transactional leadership and teamwork behavior demonstrated in organizational contexts were predictive of similar behavior in academic contexts, (b) the cultural setting of the study moderates the carry over effect of teamwork and leadership behavior from organizations to laboratories, and (c) for several leadership and teamwork behaviors, role identity and self-awareness incrementally added to the prediction of similar behaviors in academic contexts. We discuss the implications of our findings for enhancing the external validity of laboratory studies in applied psychology and for instruction of teamwork and leadership in academe. [ABSTRACT FROM AUTHOR]

An increasing number of studies call into question the external validity of social preference games. In this paper, we show that these games have a low correlation with single pro-social behaviors in the field, but this correlation can be substantially increased by aggregating behaviors to reduce measurement error. We tracked people's daily pro-social behaviors for 14 days using a day reconstruction method; the same people played three different social preference games on seven different occasions. We show that, as more pro-social behaviors and game rounds are aggregated, the games become much better predictors of pro-sociality. This predictive power is further increased by using statistical methods designed to better account for measurement error. These findings suggest that social preference games capture important underlying dispositions of real-world pro-sociality, and they can be successfully used to predict aggregated pro-social inclinations. This has crucial implications for the external validity and applicability of economic games. [ABSTRACT FROM AUTHOR]

When effect modifiers influence the decision to participate in randomized trials, generalizing causal effect estimates to an external target population requires the knowledge of two scores - the propensity score for receiving treatment and the sampling score for trial participation. While the former score is known due to randomization, the latter score is usually unknown and estimated from data. Under unconfounded trial participation, we characterize the asymptotic efficiency bounds for estimating two causal estimands - the population average treatment effect and the average treatment effect among the non-participants - and examine the role of the scores. We also study semiparametric efficient estimators that directly balance the weighted trial sample toward the target population, and illustrate their operating characteristics via simulations. [ABSTRACT FROM AUTHOR]

Policing scholars frequently use surveys to understand officer attitudes and behavioral intentions. Yet, it is difficult to gain access to one – let alone multiple – agencies. Thus, officer surveys often reflect views in a single department, making it unclear how generalizable the findings are. For the present study, we conducted an exploratory review of articles published in 16 criminology and policing journals from 2000 to 2017. We identified 600 studies that involved surveying one or more samples of police officers. From this list, we set out to determine: (1) how often authors administered their surveys to more than one sample, and (2) when surveys were administered to more than one sample, how often were results consistent across samples? We found eighty-seven (14.5%) articles that involved collecting survey data from multiple agencies, though only 29 (4.8% overall, 33.3% of multi-agencies studies) met our inclusion criteria. Importantly, only 15 studies could be analyzed as some authors no longer had data, could not share data, or did not response to our emails. Results were fully consistent across samples in just one published study. In the other studies, findings partially replicated – though sometimes results were in the opposite direction across departments. Thus, replication is critical before policy is created from single-agency surveys. [ABSTRACT FROM AUTHOR]

In attempting to draw bigger conclusions, researchers in psychology open their labs to more diverse groups of people. Yet even the most far-reaching theories must be tested with specific stimuli, materials, and methodology. To the extent that a study's stimuli are familiar beyond the lab to groups of people writ large, an experiment is said to have mundane realism —a type of external validity. We propose that an experiment's stimuli will vary in their relevance to each individual participant (such as how much they consume the stimuli outside the lab) and can be assessed using a tool: reality checks. We found that accounting for a study's mundane realism, at the individual level, significantly altered a study's results—which we found to be the case in testing well-established findings in psychology and behavioral economics. Our work suggests that measuring mundane realism (in addition to creating it) is a useful way of testing effects in psychology among the participants for whom the studies' scenarios and decisions will matter most outside of the lab. [ABSTRACT FROM AUTHOR]

Epidemiologic researchers generalizing or transporting effect estimates from a study to a target population must account for effect-measure modifiers (EMMs) on the scale of interest. However, little attention is paid to how the EMMs required may vary depending on the mathematical nuances of each effect measure. We defined 2 types of EMMs: a marginal EMM, where the effect on the scale of interest differs across levels of a variable, and a conditional EMM, where the effect differs conditional on other variables associated with the outcome. These types define 3 classes of variables: class 1 (conditional EMM), class 2 (marginal but not conditional EMM), and class 3 (neither marginal nor conditional EMM). Class 1 variables are necessary to achieve a valid estimate of the RD in a target population, while an RR requires class 1 and class 2 and an OR requires classes 1, 2, and 3 (i.e. all variables associated with the outcome). This does not mean that fewer variables are required for an externally valid RD (because variables may not modify effects on all scales), but it does suggest that researchers should consider the scale of the effect measure when identifying an EMM necessary for an externally valid treatment effect estimate. [ABSTRACT FROM AUTHOR]

This article reviews recent advances in causal inference relevant to sociology. We focus on a selective subset of contributions aligning with four broad topics: causal effect identification and estimation in general, causal effect heterogeneity, causal effect mediation, and temporal and spatial interference. We describe how machine learning, as an estimation strategy, can be effectively combined with causal inference, which has been traditionally concerned with identification. The incorporation of machine learning in causal inference enables researchers to better address potential biases in estimating causal effects and uncover heterogeneous causal effects. Uncovering sources of effect heterogeneity is key for generalizing to populations beyond those under study. While sociology has long emphasized the importance of causal mechanisms, historical and life-cycle variation, and social contexts involving network interactions, recent conceptual and computational advances facilitate more principled estimation of causal effects under these settings. We encourage sociologists to incorporate these insights into their empirical research. [ABSTRACT FROM AUTHOR]

This letter responds to the article by Yvonne S. Yang et al. (2024) on the relationship between social isolation, loneliness, and stress-related gene expression, published in Brain, Behavior, and Immunity. The authors' recruitment methods and use of genome-wide transcriptional profiling to assess the Conserved Transcriptional Response to Adversity (CTRA) are highly commendable, offering a comprehensive understanding of how social isolation impacts immune gene expression. However, we raise concerns about the small sample size, demographic limitations, cross-sectional design, and lack of correction for multiple comparisons, which may affect the study's generalizability and validity. Despite these limitations, the study provides valuable insights, paving the way for future research to explore the complex interactions between social isolation, loneliness, and health. We recommend that future studies address these limitations to enhance the robustness of findings and the development of effective interventions. [ABSTRACT FROM AUTHOR]

An important component of multi-criteria decision analysis (MCDA) in the public sector is the elicitation and aggregation of preference data collected via surveys into the relative importance of the criteria for the decision at hand. These aggregated preference data, usually in the form of mean weights on the criteria, are intended to represent the preferences of the relevant population overall. However, random sampling is often not feasible for public-sector MCDA for logistical reasons, including the expense involved in identifying and recruiting participants. Instead, non-random sampling methods such as convenience, purposive or snowball sampling are widely used. Nonetheless, provided the preference data collected are sufficiently 'cohesive' in terms of the extent to which the weights of the individuals belonging to the various exogenously defined groups in the sample are similar, non-random sampling can still produce externally valid aggregate preference data. We explain a method for measuring cohesiveness using the Kemeny and Hellinger distance measures, which involve measuring the 'distance' of participants' weights (and the corresponding rankings of the criteria) from each other, within and between the groups respectively. As an illustration, these distance measures are applied to data from a MCDA to rank non-communicable diseases according to their overall burden to society. We conclude that the method is useful for evaluating the external validity of preference data obtained from non-random sampling. [ABSTRACT FROM AUTHOR]

Objective: The OVHIPEC‐1 trial (Phase III randomised clinical trial for stage III ovarian carcinoma randomising between interval cytoreductive surgery with or without hyperthermic intraperitoneal chemotherapy) showed improved survival when interval cytoreductive surgery (CRS) was combined with hyperthermic intraperitoneal chemotherapy in patients with stage III epithelial ovarian cancer (EOC). The authors compared the control arm of the trial with a real‐world population treated in the Netherlands during the same period to explore generalizability of the trial results. Methods: For this nationwide comparative cohort study, all patients with EOC undergoing interval CRS between 2007 and 2016 were identified from the Netherlands Cancer Registry if they fulfilled the eligibility criteria of OVHIPEC‐1 (n = 1376). Patient and treatment characteristics, and overall survival (OS) were compared between trial and real‐world populations. Results: Age, comorbidity, BRCA status, histologic subtype, and residual disease were similar in trial and real‐world patients. Trial patients had a better performance status, higher socioeconomic status, and underwent bowel surgery more often. In a real‐world setting, patients more often received more than six cycles. The difference in OS between the trial and the real‐world populations was not statistically significant (unadjusted hazard ratio, 1.09 [95% confidence interval, 0.87–1.37]; P = 0.44). Conclusion: Despite differences in patient characteristics, OS of patients treated in the control arm of OVHIPEC‐1 was similar to patients treated outside the trial. The trial population accurately represents real‐world patients with stage III EOC undergoing interval CRS in terms of outcome. Synopsis: Despite differences in patient and treatment characteristics, overall survival of patients in the control arm of OVHIPEC‐1 was similar to patients treated outside the trial. [ABSTRACT FROM AUTHOR]

Ecologists often rely on randomized control trials (RCTs) to quantify causal relationships in nature. Many of our foundational insights of ecological phenomena can be traced back to well‐designed experiments, and RCTs continue to provide valuable insights today. Although RCTs are often regarded as the "gold standard" for causal inference, it is important to recognize that they too rely on a set of causal assumptions that must be justified and met by the researcher to draw valid causal conclusions. We use key ecological examples to show how biases such as confounding, overcontrol, and collider bias can occur in experimental setups. In tandem, we highlight how such biases can be removed through the application of the structural causal model (SCM) framework. The SCM framework visualizes the causal structure of a system or process under study using directed acyclic graphs (DAGs) and subsequently applies a set of graphical rules to remove bias from both observational and experimental data. We show how DAGs can be applied across ecological experimental studies to ensure proper study design and statistical analysis, leading to more accurate causal estimates drawn from experimental data. Although causal conclusions drawn from RCTs are often taken at face value, ecologists are increasingly becoming aware that experimental approaches must be carefully designed and analyzed to avoid potential biases. By applying DAGs as a visual and conceptual tool, experimental ecologists can increasingly meet the causal assumptions required for valid causal inference. [ABSTRACT FROM AUTHOR]

When individuals self-select (or are selected) into a study based on factors that influence the outcome, conclusions may not generalize to the full population. To compensate for this, results may be adjusted, for example, by standardization on the set of common causes of participation and outcome. Although such standardization is useful in some contexts, the common causes of participation and outcome may in practice not be fully observed. Instead, the researcher may have access to one or several variables related to the common causes, that is, to proxies for the common causes. This article defines and examines different types of proxy variables and shows how these can be used to obtain generalizable study results. First of all, the researcher may exploit proxies that influence only participation or outcome but which still allow for perfect generalizability by rendering participation and outcome conditionally independent. Further, generalizability can be achieved by leveraging 2 proxies, one of which is allowed to influence participation and one of which is allowed to influence the outcome, even if participation and outcome do not become independent conditional on these. Finally, approximate generalizability may be obtained by exploiting a single proxy that does not itself influence participation or outcome. [ABSTRACT FROM AUTHOR]

Introduction: Reports regarding the external validity of randomized controlled trials (RCTs) are scarce. We aimed to assess the population external validity of an investigator-initiated RCT on the duration of antibiotics for the treatment of Gram-negative bacteremia by comparing patients included in the RCT to patients that were not included in the trial. Methods: Hospitalized patients with Gram-negative bacteremia were recruited into an RCT and randomized to receive 7 or 14 days of covering antibiotic therapy in Israel and Italy from 2013 to 2017. In a concomitant observational study, RCT participants were compared with patients who fulfilled the inclusion criteria but were not included in the trial due to participation in other trials, discharge before approached by researchers, refusal to participate, or unwillingness of the treating physician to allow participants' recruitment. Results: Six hundred and four RCT patients were compared with 613 nonincluded patients. Almost 50% of nonincluded patients (288/613) were dependent on others for activities of daily living at baseline compared to 37.7% of RCT participants (228/604). Dementia was nearly 2-fold more frequent in nonincluded patients than those included (5.9% [36/613] versus 3.6% [22/604], p = 0.07). Patients who were not included in the RCT were more likely to acquire their infection in the hospital (53.3% [327/613] versus 29.1% [176/604], p < 0.001). The primary composite outcome of mortality, clinical failure, readmissions, or extended hospitalization at 90 days occurred in 353 of 613 nonincluded patients (57.6%) compared to 299 of 604 RCT participants (49.6%), p = 0.005. However, on multivariate analysis noninclusion in the RCT was not an independent risk factor for clinical failure and mortality. Conclusions: RCTs, even with broad eligibility criteria, do not represent the whole spectrum of patients and leave out a population with more severe illness for whom the evidence is lacking. [ABSTRACT FROM AUTHOR]

Our primary objective was to determine the proportion of trials that report the number of patients assessed for eligibility before randomization. We performed the systematic retrieval and analysis of all phase II, III, and IV RCTs published between 2013 and 2015 in four high-impact-factor journals in the field of clinical oncology. Among 456 RCTs reviewed, 236 trials (51.8%) reported the number of patients assessed for eligibility. Among the 236 trials that reported the entire enrollment process, the reasons for patient exclusion could be found in 184 trials (78%). A flow diagram was presented in 452 trials (99.1%), and 98 trials (21.5%) included a discussion on generalizability. Reporting the parameters of external validity in medical oncology RCTs is challenging. Improving adherence to the 2010 CONSORT guidelines concerning the enrollment process could help clinicians and health policymakers establish to whom trial results apply. [ABSTRACT FROM AUTHOR]

The dictator game has become a celebrated workhorse of experimental economics and social psychology. In the standard version of the game an individual is given a sum of money and must choose how to split this money between themselves and some other individual. In a variant of the game the individual must split the money between themselves and a charitable cause. This charity version of the dictator game has now been used in well over fifty studies and has provided critical insight on the motives behind giving. It also provides a simple tool that policy makers and practitioners can use to test the effect of interventions. In this paper we explain the different ways in which charity dictator games can and have been used. We also look at the external validity of charity dictator games and discuss the research questions that can be appropriately studied using them. [ABSTRACT FROM AUTHOR]

Evaluative conditioning (EC) research investigates changes in the evaluation of a stimulus after co-occurrence with an affective stimulus. To explain the motivation behind this research, this review begins with an overview of the history of EC research, followed by a summary of the state of the art with respect to three key questions. First, how should EC procedures be used to influence evaluation? We provide a guide based on evidence concerning the functional properties of EC effects. Second, how does the EC effect occur? We discuss the possible mediating cognitive processes and their automaticity. Third, are EC effects ubiquitous outside the lab? We discuss the evidence for the external validity of EC research. We conclude that the most important open questions pertain to the relevance of EC to everyday life and to the level of control that characterizes the processes that mediate the EC effect after people notice the stimulus co-occurrence. [ABSTRACT FROM AUTHOR]

There is growing concern about the extent to which economic games played in the laboratory generalize to social behaviors outside the lab. Here, we show that it is possible to make a game much more predictive of field behavior by bringing contextual elements from the field to the lab. We report three experiments where we present the same participants with different versions of the dictator game and with two different field situations. The games are designed to include elements that make them progressively more similar to the field. We find a dramatic increase in lab-field correlations as contextual elements are incorporated, which has wide-ranging implications for experiments on economic decision making. [ABSTRACT FROM AUTHOR]

Critical reviews of a test's measurement validity are valuable scientific contributions, yet even strong reviews can be undermined by subtle problems in how evidence is compiled and presented to readers. First, if discussions of poor reporting practices by a test's users are interwoven with discussions about validity support for the test itself, readers can be inadvertently misled into impressions of the latter which are improperly conflated with the former. Second, test reviewers should give at least as much careful attention to a test's external validity as to its structural validity; test reviewers who prioritize factor analysis and internal consistency at the expense of discriminant and convergent validity can inadvertently mislead readers into perceptions of a test which are more negative or more positive than is warranted by the evidence overall. In this commentary, we aim to help test evaluators in crafting critical investigations of measurement validity. We use Higgins et al.'s (2024) review of the Reading the Mind in the Eyes Test (RMET; Baron-Cohen et al., 2001) as a basis for discussion. We argue that their otherwise impressive review went astray in the two ways described above. After considering both the psychometric evidence that Higgins et al. (2024) provided and the external validity evidence that they did not provide, we conclude that their recommendations that the RMET should be abandoned, and that most prior research findings based on it should be reassessed or disregarded, are unwarranted. • The Reading the Mind in the Eyes Test (RMET) has substantial external validity and recommendations to abandon are more negative than is warranted. • We make recommendations for scholars conducting measurement reviews of specific tests. [ABSTRACT FROM AUTHOR]

Background: Sex-specific analysis and reporting may allow a better understanding of intervention effects and can support the decision-making process. Well-conducted systematic reviews (SRs), like those carried out by the Cochrane Collaboration, provide clinical responses transparently and stress gaps of knowledge. This study aimed to describe the extent to which sex is analysed and reported in a cross-section of Cochrane SRs of interventions, and assess the association with the gender of main authorships. Methods: We searched SRs published during 2018 within the Cochrane Database of Systematic Reviews. An investigator appraised the sex-related analysis and reporting across sections of SRs and collected data on gender and country of affiliation of the review first and last authors, and a second checked for accuracy. We conducted descriptive statistics and bivariate logistic regression to explore the association between the gender of the authors and sex-related analysis and reporting. Results: Six hundred and ten Cochrane SRs were identified. After removing those that met no eligibility criteria, 516 reviews of interventions were included. Fifty-six reviews included sex-related reporting in the abstract, 90 considered sex in their design, 380 provided sex-disaggregated descriptive data, 142 reported main outcomes or performed subgroup analyses by sex, and 76 discussed the potential impact of sex or the lack of such on the interpretations of findings. Women represented 53.1 and 42.2% of first and last authorships, respectively. Women authors (in first and last position) had a higher possibility to report sex in at least one of the review sections (OR 2.05; CI 95% 1.12–3.75, P=0.020) than having none. Conclusions: Sex consideration amongst Cochrane SRs was frequently missing. Structured guidance to sex-related analysis and reporting is needed to enhance the external validity of findings. Likewise, including gender diversity within the research workforce and relevant authorship positions may foster equity in the evidence generated. [ABSTRACT FROM AUTHOR]

To compare the outcomes of real-world patients who would have been eligible for asthma biologics to those who would not have been eligible. We used data from the OptumLabs Data Warehouse (OLDW) to categorize patients into eligible and ineligible groups based on clinical trials (n = 19 trials) used for Food and Drug Administration (FDA) approval. We then compared the change in the number of asthma exacerbations before and after biological initiation between the two groups. The percentage of people who would have been eligible for asthma biologic clinical trials ranged from 0–10.2%. The eligible group had a greater reduction in number of asthma exacerbations compared to the ineligible group based on eligibility criteria from 1 omalizumab trial (1.52, 95% CI 1.25, 1.8 in eligible vs. 0.47, 95% CI 0.43, 0.52 in ineligible) and from 1 dupilumab trial (1.6, 95% CI 0.92, 2.28 in eligible vs. 0.52, 95% CI 0.38, 0.65 ineligible). Notably, 15 of the 19 trials had fewer than 11 eligible people, limiting additional comparisons. Fewer than 1 in 10 people in the United States treated with asthma biologics would have been eligible to participate in the trial for the biologic they used. Where comparisons could be made, trial eligible people have a greater reduction in exacerbations. Supplemental data for this article is available online at https://doi.org/10.1080/02770903.2021.2010749. [ABSTRACT FROM AUTHOR]

Within-study comparisons (WSCs) compare quasi-experimental results to an experimental benchmark and assess the extent to which quasi-experiments (QEs) sacrifice internal validity for external validity. WSCs inform decisions about when to use experiments or QEs, as well as methodological decisions (e.g., covariate selection) about how to implement QEs when they are necessary or preferred. We review the methodological literature on WSCs and present the findings from a WSC of a school voucher program as an example of the use of WSCs to inform policy debates involving the third sector. Prior literature and our results suggest that QE evaluations of third-sector institutions can produce low levels of bias, but only if analytic models include key proxy variables for voluntary self-selection into programs. [ABSTRACT FROM AUTHOR]

Background: There is increasing use of sodium glucose co-transporter 2 (SGLT2) inhibitors to treat diabetes. Since trials apply specific entry and exclusion criteria to ensure internal validity, comparisons of trial populations with nationally representative samples can inform the applicability of study findings to practice. Objective: To compare individuals with diabetes from a nationally representative sample to patients who underwent randomization in the EMPA-REG trial. A secondary aim was to characterize what proportion of individuals prescribed an SGLT2 inhibitor in a nationally representative sample would have been included in the EMPA-REG trial. Design: Retrospective cross-sectional study. Participants: Adults with diabetes who took part in the National Health and Nutrition Examination Survey (NHANES) between 2011–2014 (primary analysis corresponding to EMPA-REG enrollment) and 2015–2018 (secondary analysis corresponding to contemporary sample). Main Measures: The primary outcome was a comparison of demographic (age, sex, ethnicity, and pregnancy status), clinical (comorbidities and medication use), examination (weight, body mass index, and systolic and diastolic blood pressure), and laboratory (hgba1c, low- and high-density lipoprotein cholesterol, triglycerides, and estimated glomerular filtration rate) characteristics of NHANES respondents versus EMPA-REG trial participants. The secondary outcome was the proportion of NHANES respondents who had been prescribed an SGLT2 inhibitor that would have met inclusion criteria for the EMPA-REG trial. Key Results: There were 655 and 48 respondents, representing a weighted sample of 21,849,775 and 1,062,573 individuals, included in the primary and secondary analyses, respectively. Overall, 7.6% (95% CI 4.8–10.6%) of 2011–2014 NHANES respondents would have met all EMPA-REG trial inclusion criteria. NHANES respondents and EMPA-REG participants differed across demographic, clinical, examination, and laboratory domains. Of NHANES respondents from 2015 to 2018 who were prescribed an SGLT2 inhibitor, 10.6% (95% CI <1–24.7%) would have met all inclusion criteria for the EMPA-REG trial. Conclusions: The EMPA-REG population differed from a nationally representative sample, which could affect generalizability. [ABSTRACT FROM AUTHOR]

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods—including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation—that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations. [ABSTRACT FROM AUTHOR]