Search

In this article, we review the ways in which replication has been and could be featured in consumer behavior, using Journal of Consumer Research as a specific setting. We present a framework for thinking about the generalizability of research findings and differentiate various potential benefits that replication can have for understanding variability in consumer research findings. We then define four different types of replications, describe how researchers can use these approaches to produce distinct benefits, and give guidance regarding conducting, interpreting, and the potential contributions of these different types of replications. We conclude with a discussion of various ways in which replication could be more fully integrated into different phases of the scientific research process, taking into account the contribution necessary for publication. In particular, we identify opportunities to incorporate independent replication into original papers, to increase the replication-based contribution in papers that build on prior work, and to use systematic replication in conjunction with meta-analysis to synthesize and confirm conclusions from a mature research literature. More fully integrating replication into scientific practice can yield a new equilibrium, in which replication is routine, typically consistent with previous results, and recognized as necessary for establishing an empirical generalization. [ABSTRACT FROM AUTHOR]

Interventional researchers face many design challenges when assessing intervention implementation in real-world settings. Intervention implementation requires holding fast on internal validity needs while incorporating external validity considerations (such as uptake by diverse subpopulations, acceptability, cost, and sustainability). Quasi-experimental designs (QEDs) are increasingly employed to achieve a balance between internal and external validity. Although these designs are often referred to and summarized in terms of logistical benefits, there is still uncertainty about (a) selecting from among various QEDs and (b) developing strategies to strengthen the internal and external validity of QEDs. We focus here on commonly used QEDs (prepost designs with nonequivalent control groups, interrupted time series, and stepped-wedge designs) and discuss several variants that maximize internal and external validity at the design, execution and implementation, and analysis stages.

Purpose: Evaluation of the added value of Dietary Approaches to Stop Hypertension (DASH) and Mediterranean diet scores on the prediction model of the World Health Organization (WHO) to predict 10-year cardiovascular disease (CVD) mortality using the Golestan Cohort Study data. Methods: A total of 44,648 participants (25,268 women and 18,531 men) were included in the final analysis. To assess the external validity of the non-laboratory risk model of WHO, the Area Under the Curve (AUC) and calibration plot methods were used. The multivariate Cox proportional hazards regression analysis was used to evaluate the association of 10-year CVD mortality risk with DASH and Mediterranean scores and their components. The added value of each significant variables was evaluated by the concordance C-statistic and integrated discrimination improvement (IDI). Statistical significance was defined as p-value < 0.05. Results: DASH and Mediterranean diet scores were not significant predictors of 10-year CVD mortality in both genders (p > 0.05). However, sodium and total vegetable in both genders and added sugar in women were significant predictors for 10-year stroke mortality (p < 0.05). Sodium intake in women and monounsaturated fatty acid (MUFA) to saturated fatty acid (SFA) ratio in men had significant associations with 10-year mortality of myocardial infarction/coronary heart disease (MI/CHD). Calculation of IDI showed that none of the evaluated nutritional indices/variables could significantly improve the WHO model performance and predictive ability. Conclusion: Inclusion of DASH and Mediterranean diet scores and their components did not improve WHO risk prediction model performance and predictive ability to predict 10-year CVD mortality. [ABSTRACT FROM AUTHOR]

Technological and social developments foster the implementation of digital and hybrid live-communication. The COVID-19 pandemic only accelerated the digitalization processes. Experiments are the proper data collection methods to investigate the effects of digital live-communication. However, there are challenges of conducting experimental research on digital and hybrid live-communication during business-tobusiness events. In these research notes, some of those challenges were pointed out with the aim of supporting future researchers who work on the effects of digital and hybrid live-communication. In particular the balance between internal and external validity has to be considered when conducting experiments. Digital tools can contribute to accurate data collection and analysis. Experiments need to consider ethical aspects. [ABSTRACT FROM AUTHOR]

Background/Objectives: An adapted version of an online pictorial food frequency questionnaire (FFQ), VioScreen-Allergy, assesses total dietary intake and intake of allergens and foods in the maternal diet index (MDI), linked to offspring allergy. This study assessed intermethod reliability, test–retest reliability, and external validity of the VioScreen-Allergy. Methods: Females of childbearing age were recruited at Denver Health and Children's Hospital, Colorado, USA, and were asked to complete four 24 h recalls and two VioScreen-Allergy FFQs over the course of a month. All those with at least two 24 h dietary recalls and both VioScreen-Allergy assessments were analyzed. Energy-adjusted and non-adjusted linear mixed models (1) compared MDI scores and intake of nutrients and allergens as measures of intermethod reliability; (2) evaluated VioScreen-Allergy test–retest reliability as differences between repeated measurements; and (3) assessed external validity by modeling associations between VioScreen-Allergy-derived intake of beta-carotene and orange vegetables and Veggie Meter®-assessed skin carotenoids. Bonferroni corrections controlled multiple comparisons within the assessment. Results: Of 53 participants enrolled, 25 demographically dissimilar participants were included in the analysis. There were no significant differences between 24 h recall and VioScreen-Allergy mean intakes of macronutrients, micronutrients, allergens, or MDI, except for Vitamin C, niacin, and cashew allergen protein. There were no significant differences between repeated measurements of VioScreen-Allergy, either energy-adjusted or unadjusted. Both beta-carotene and orange vegetable servings were significantly associated with Veggie Meter®. Conclusions: Although non-significance could have been due to low power, clinical as well as statistical assessments of intermethod reliability, test–retest reliability, and external validity suggest that VioScreen-Allergy has reasonable utility for trials assessing food allergens and MDI in the context of overall intake. The VioScreen questionnaire can also be used in future studies to assess macro- and micronutrient intake. Additional validation studies assessing different portion sizes and foods eaten by infants and young children are currently undergoing. [ABSTRACT FROM AUTHOR]

A popular interpretation of "evidence‐based" decision‐making is "rely (only) on the rigorous evidence" (RORE) via "systematic" reviews that: use objective protocols to generating the potentially relevant papers from the literature; then filter those to retain only the small subset that provide impact estimates regarded as "rigorous"; and summarize only those estimates. I use two sets of cross‐country impact estimates—on wage gains for migrants and private school learning gains—to illustrate this seemingly attractive approach is both empirically and conceptually unsound. First, the cross‐country variation in the rigorous estimates of impact is very large, which implies the average(s) from a systematic review is of little predictive use. In both empirical examples the "systematic review of the rigorous estimates" approach leads to worse predictions of impact across countries than the naïve use of country‐specific ordinary least squares estimates. Second, I contrast a systematic review—RORE approach with an "understanding" approach—which seeks to encompass all of the available evidence into coherent understandings in forming judgments. In both examples the notion that the impact effects are constant across countries—"external validity"—is easily rejected. Insisting on privileged reliance on "rigorous" estimates in making context‐specific decisions is logically incoherent and deeply anti‐scientific. [ABSTRACT FROM AUTHOR]

Randomized controlled trials (RCT's) allow researchers to estimate causal effects in an experimental sample with minimal identifying assumptions. However, to generalize or transport a causal effect from an RCT to a target population, researchers must adjust for a set of treatment effect moderators. In practice, it is impossible to know whether the set of moderators has been properly accounted for. I propose a two parameter sensitivity analysis for generalizing or transporting experimental results using weighted estimators. The contributions in the article are threefold. First, I show that the sensitivity parameters are scale-invariant and standardized, and introduce an estimation approach for researchers to account for both bias in their estimates from omitting a moderator, as well as potential changes to their inference. Second, I propose several tools researchers can use to perform sensitivity analysis: (1) numerical measures to summarize the uncertainty in an estimated effect to omitted moderators; (2) graphical summary tools to visualize the sensitivity in estimated effects; and (3) a formal benchmarking approach for researchers to estimate potential sensitivity parameter values using existing data. Finally, I demonstrate that the proposed framework can be easily extended to the class of doubly robust, augmented weighted estimators. [ABSTRACT FROM AUTHOR]

Background Carbapenem-resistant Enterobacterales (CRE) bloodstream infections (BSIs) are a major threat to patients. To date, data on risk factors have been limited, with low internal and external validity. In this multicentre study, risk factors for CRE BSI were determined by comparison with two control groups: patients with carbapenem-susceptible Enterobacterales (CSE) BSI, and patients without Enterobacterales infection (uninfected patients). Methods A multicentre, case-control-control study was nested in a European prospective cohort study on CRE (EURECA). CRE BSI:CSE BSI matching was 1:1, CRE BSI:Uninfected patients matching was 1:3, based on hospital, ward and length of stay. Conditional logistic regression was applied. Results From March 2016 to November 2018, 73 CRE BSIs, 73 CSE BSIs and 219 uninfected patients were included from 18 European hospitals. For CRE versus CSE BSI, previous CRE colonization/infection [incidence rate ratio (IRR) 7.32; 95% CI 1.65–32.38) increased the risk. For CRE versus uninfected controls, independent risk factors included: older age (IRR 1.03; 95% CI 1.01–1.06), patient referral (long-term care facility: IRR 7.19; 95% CI 1.51–34.24; acute care hospital: IRR 5.26; 95% CI 1.61–17.11), previous colonization/infection with other MDR organisms (MDROs) (IRR 9.71; 95% CI 2.33–40.56), haemodialysis (IRR 8.59; 95% CI 1.82–40.53), invasive procedures (IRR 5.66; 95% CI 2.11–15.16), and β-lactam/β-lactamase inhibitor combinations (IRR 3.92; 95% CI 1.68–9.13) or third/fourth generation cephalosporin (IRR 2.75; 95% CI 1.06–7.11) exposure within 3 months before enrolment. Conclusions Evidence of previous CRE colonization/infection was a major risk factor for carbapenem resistance among Enterobacterales BSI. Compared with uninfected patients, evidence of previous MDRO colonization/infection and healthcare exposure were important risk factors for CRE BSI. Targeted screening, infection prevention and antimicrobial stewardship should focus on these high-risk patients. [ABSTRACT FROM AUTHOR]

Policymakers and educational researchers have grown increasingly interested in the extent to which study results generalize across different groups of students. Current generalization research in education has largely focused on the compositional similarity among students based on a set of observable characteristics. However, generalization is defined differently across various disciplines. While the concept of compositional similarity is prominent in causal research, generalization among the statistical learning community refers to the extent to which a model produces accurate predictions across samples and populations. The purpose of this study is to assess the extent to which concepts related to contextual generalization (based on compositional similarity) are associated with the ideas related to model generalization (based on accuracy of prediction). We use observational data from the Programme for International Student Assessment (PISA) 2015 wave as a case study to examine the conditions under which contextual and model generalization are aligned. We assess the correlations between statistical measures that quantify compositional similarity and prediction accuracy and discuss the implications for generalization research. [ABSTRACT FROM AUTHOR]

The objectives of our study were 1) to examine whether the force-velocity-power (FvP) outcomes in the concentric and eccentric phases of flywheel (FW) squats differ among sports disciplines and 2) to investigate the association between FvP outcome variables and two key sport-performance indicators: countermovement jump height (CMJ) and change of direction (CoD) time involving 90° or 180° turns. Tests were performed by 469 athletes from five different sport disciplines and physical education students. Our results showed that FvP outcomes, when measured during the eccentric phase of the FW squat, effectively differed between athletes from different sports. However, during the concentric phase of the squat, only for the slope variable differences between sports were found. Contrary to our hypothesis, there were no statistically significant correlations between FW squat FvP outcomes and CMJ or CoD test results. These results suggest that FvP outcomes derived from FW squats may lack external validity and cannot be reliably used as a predictor of athletes' jumping and agility performance. The paper discusses possible reasons for the larger differences between sports in eccentric FvP outcomes, as well as the absence of correlations between FvP outcomes and functional tests. [ABSTRACT FROM AUTHOR]

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations. [ABSTRACT FROM AUTHOR]

In the DSM-5 Alternative Model of Personality Disorders (AMPD), psychopathy is marked by the presence of attention seeking, low anxiousness, and lack of social withdrawal, along with traits from the domains of Antagonism and Disinhibition. The triarchic model of psychopathy (TriPM) posits three biobehaviorally based traits underlying it: disinhibition, meanness, and boldness. The current study directly compared relations for measures of the two models with the broad dimensions of externalizing, internalizing, and positive adjustment. Participants (1,678 adults) were surveyed regarding maladaptive personality traits, clinical symptoms, and positive adjustment features. The TriPM model explained more variance than the AMPD in substance use, positive adjustment, and empathy, whereas the AMPD model explained more variance in internalizing symptoms. In addition, AMPD Antagonism and the Psychopathy Specifier diverged from TriPM Meanness and Boldness in their associations with some specific outcomes. Overall, our study provides evidence for complementarity of the two models in characterizing the multifaceted nature of psychopathy. [ABSTRACT FROM AUTHOR]

AbstractObjective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems. [ABSTRACT FROM AUTHOR]

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology. [ABSTRACT FROM AUTHOR]

We present a new approach to establish an empirical overview of farmers' risk preferences and the characteristics associated with these preferences. We rely on an Individual Participant Data Meta-analysis whereby we identify studies eliciting risk preferences through self-assessments and Holt and Laury lotteries, and construct and analyse a unique dataset of 5,157 farmers from 19 studies in 13 European countries. Our results reveal significant heterogeneity in risk preferences across elicitation methods, within and across studies, risk domains and farm and farmer characteristics. European farmers are on average risk-averse when risk preferences are elicited by lotteries, and on average risk-neutral when elicited by self-assessments. Beyond differences in the average risk aversion, there are distributional differences pointing to a limited convergence between elicitation methods and a larger explanatory power of farm and farmer characteristics to determine risk preferences elicited with self-assessments compared to lotteries. [ABSTRACT FROM AUTHOR]

This article reviews recent advances in causal inference relevant to sociology. We focus on a selective subset of contributions aligning with four broad topics: causal effect identification and estimation in general, causal effect heterogeneity, causal effect mediation, and temporal and spatial interference. We describe how machine learning, as an estimation strategy, can be effectively combined with causal inference, which has been traditionally concerned with identification. The incorporation of machine learning in causal inference enables researchers to better address potential biases in estimating causal effects and uncover heterogeneous causal effects. Uncovering sources of effect heterogeneity is key for generalizing to populations beyond those under study. While sociology has long emphasized the importance of causal mechanisms, historical and life-cycle variation, and social contexts involving network interactions, recent conceptual and computational advances facilitate more principled estimation of causal effects under these settings. We encourage sociologists to incorporate these insights into their empirical research.

Background: The 6-minute walking test (6WT) has previously shown to be reliable and valid outcome measure in patients undergoing surgery for degenerative lumbar disorders (DLD). A role of 6WT in conservatively treated patients undergoing epidural steroid injection (ESI) remains unclear. Methods: About 50 patients with DLD, scheduled for ESI were assessed by the smartphone-based 6WT and common paper-based patient-reported outcome measures (PROMs), including the Core Outcome Measures Index [COMI] back, Oswestry Disability Index (ODI) and Short Form Survey (SF-12). Pearson correlation coefficient (PCC) was used to define the relationship between 6WT and PROMs. Reliability of the 6WT was determined by intraclass correlation coefficient (ICC). Age- and sex-adjusted objective functional impairment (OFI) z-scores were calculated for each patient. Results: A total of 50 patients (mean age 52 years, SD 13; 25% female), including 35 patients (70%) with lumbar disc herniation and 15 patients (30%) with lumbar spinal stenosis were included. The mean 6-minute walking distance (6WD) was 454.1 m (SD 89.1); the age- and sex-adjusted mean OFI z-score was −2.1 (SD 4.0). A total of 17 (34%) patients had mild, 8 (16%) moderate, and 4 (8%) severe OFI, while 21 (42%) had 6WT results within the normal population range (no OFI). The PCCs between the 6WD and VAS back pain were r=−0.30, ODI r=−0.43, COMI back r=−0.36, and PCS-12 r=0.51 (all p

Introduction: The Scoliosis Research Society-30 (SRS-30) is a questionnaire originally developed from the SRS-22r questionnaire and is used to evaluate adolescent idiopathic scoliosis (AIS). It comprised questions on five domains: function, pain, self-image, mental health, and satisfaction, with seven additional questions related to postoperative aspects. In addition to the original English version, translations in multiple languages have been effectively applied. Herein, we evaluated the internal consistency and external validity of the Japanese version of the SRS-30 for AIS patients. Methods: Among the 30 questions in SRS-30, the eight additional questions from SRS-22r were translated and back-translated to create a Japanese version of the SRS-30. This translated questionnaire was then used to survey patients with AIS who underwent corrective fusion surgery one year postoperatively. The internal consistency of the responses was evaluated using the Cronbach α coefficient. Additionally, the Spearman correlation analyses were conducted to assess the correlation between the scores obtained from the SRS-30 Japanese version and SRS-22r and the Oswestry Disability Index (ODI) for the overall scale and the five domains. Results: A total of 81 cases (eight males and 73 females; mean age at surgery 14.4 years) were enrolled. The mean preoperative Cobb angle was 51.0°. The Cronbach α coefficient for the overall SRS-30 was 0.861, indicating high internal consistency, while the coefficients for each domain were as follows: function/activity, 0.697; pain, 0.405; self-image/appearance, 0.776; mental health, 0.845; and satisfaction, 0.559. The SRS-30 total score significantly correlated with the SRS-22r total (r=0.945, P

Researchers and practitioners in marketing, economics, and public policy often use preference elicitation tasks to forecast real-world behaviors. These tasks typically ask a series of similarly structured questions. The authors posit that every time a respondent answers an additional elicitation question, two things happen: (1) they provide information about some parameter(s) of interest, such as their time preference or the partworth for a product attribute, and (2) the respondent increasingly "adapts" to the task—that is, using task-specific decision processes specialized for this task that may or may not apply to other tasks. Importantly, adaptation comes at the cost of potential mismatch between the task-specific decision process and real-world processes that generate the target behaviors, such that asking more questions can reduce external validity. The authors used mouse and eye tracking to trace decision processes in time preference measurement and conjoint choice tasks. Respondents increasingly relied on task-specific decision processes as more questions were asked, leading to reduced external validity for both related tasks and real-world behaviors. Importantly, the external validity of measured preferences peaked after as few as seven questions in both types of tasks. When measuring preferences, less can be more. [ABSTRACT FROM AUTHOR]

Homeopathy is a system of therapeutics that treats disease with highly diluted substances based on the Law of Similars, which holds that “like cures like.” Despite widespread use, homeopathy lacks a comprehensive and robust evidence base.We examined the state of homeopathic clinical research by critically assessing the overall quality of peer-reviewed, recently published, English-language, homeopathic clinical research in terms of internal, external, and model validity using standard and homeopathic-specific instruments. Further, we convened an international panel of nine experts in research methods and homeopathy to identify gaps in homeopathic research and prioritize areas for future study.We reviewed 99 clinical research studies targeting a wide range of populations and conditions. Studies were conducted in Western and Asian countries, with the largest number (30 percent) conducted in India. Of the 99 studies reviewed, 85 were controlled trials; 79 of these were randomized. There were many areas where the quality of the studies could be improved. About two-thirds of the 85 controlled trials had either high (42 percent) or unclear (24 percent) risk of bias according to internationally recognized standards for internal validity. Of the 14 observational (cohort) studies, over one-third did not control for important confounders in the outcome analyses. Regarding external validity, adherence was reported in less than a third of studies (n=31). Forty percent of studies (79 % of observational studies) did not report on safety. Regarding model validity, fewer than two-thirds of the studies were consistent with homeopathic principles.Our expert panel was mixed on whether the homeopathic research literature was missing important populations and/or conditions, and they suggested a variety of priority areas. Panelists also expressed a variety of opinions about the types of homeopathy that should be prioritized for future study but also noted that since homeopathic practice differs by country, each country may have different priorities.Panelists agreed with the findings of the literature review that the research literature was at least somewhat deficient in all three types of validity. Although our assessment of validity was [by necessity] based only on what was reported, this assessment suggests the need for both better reporting and higher quality research. They recommended the use of reporting guidelines to improve all types of validity, the identification of exemplar studies to help guide researchers to improve internal validity, and, given the limitations of the instruments available to measure external and model validity, that these instruments be validated and configured to provide summary scores. Finally, substantial discussion addressed the need to bring more research expertise into homeopathic studies, both by better training homeopathic researchers and by collaborating with experienced conventional medicine research groups.

BackgroundReporting retention data is critical to determining the soundness of a study's conclusions (internal validity) and broader generalizability (external validity). Although selective attrition can lead to overestimates of effects, biased conclusions, or overly expansive generalizations, retention rates are not reported in many longitudinal studies.MethodsWe examined multiple child- and family-level factors potentially associated with retention in a longitudinal study of younger siblings of children with autism spectrum disorder (ASD; n = 304) or typical development (n = 163). The sample was followed from the first year of life to 36 months of age, for up to 7 visits.ResultsOf the 467 infant siblings who were consented and participated in at least one research visit, 397 (85.0%) were retained to study completion at 36 months. Retention rates did not differ by familial risk group (ASD-risk vs. Low-risk), sex, race, ethnicity, age at enrollment, number of children in the family, maternal employment, marital status, or parent concerns about the child at enrollment. A stepwise regression model identified 4 variables that, together, provided the most parsimonious predictive model of study retention: maternal education, maternal age at child's birth, travel distance to the study site, and diagnostic outcome classification at the final study visit.ConclusionsThe retained and not-retained groups did not differ on most demographic and clinical variables, suggesting few threats to internal and external validity. The significantly higher rate of retention of children diagnosed with ASD (95%) than typically developing children (83%) may, however, present biases when studying recurrence risk. We conclude by describing engagement and tracking methods that can be used to maximize retention in longitudinal studies of children at risk of ASD.

Background Randomized clinical trials (RCTs) might not be representative of the real-world population because of unreasonable exclusion criteria. We sought to determine which groups of patients are excluded from RCTs that included lipid-lowering therapy. Methods and Results We retrieved all trials from the Cholesterol Treatment Trialists Collaboration and systematically searched for large (≥1000 participants) lipid-lowering therapy RCTs, defined as statins, ezetimibe, and PCSK9 inhibitors. We predefined groups: older adults (>70 or >75 years), women, non-Whites, chronic kidney failure, heart failure, immunosuppression, cancer, dementia, treated thyroid disease, chronic obstructive pulmonary disease, mental illness, atrial fibrillation, multimorbidity (≥2 chronic diseases), and polypharmacy. We counted the number of RCTs excluding patients of the predefined groups and meta-analyzed the prevalence of included patients to obtain pooled estimates with a random-effects model. We included 42 RCTs (298 605 patients). Eighty-one percent of trials excluded patients with severe and 76% those with moderate kidney failure. Seventy-one percent of trials excluded groups of women, 64% excluded patients with moderate to severe heart failure, 64% those with immunosuppressant conditions, 48% those with cancer, 29% those with dementia, and 29% of trials excluded older adults. The pooled prevalence for patients >70 years of age was 25% (95% CI, 0%-49%), 11% (3%-18%) for >75 years of age, and 51% (38%-63%) for multimorbidity. Conclusions The majority of lipid-lowering therapy trials excluded patients with common diseases, such as moderate-to-severe kidney disease or heart failure or with immunosuppression. Underrepresenting certain populations, including women and older adults, might lead to limited transportability of study results and uncertainty on possible side-effects and efficacy in these groups. Future trials should promote diversity in the recruitment strategies and improve equity in cardiovascular research. Registration URL: ClinicalTrials.gov; Unique Identifier: CRD42021253909.

ObjectiveMethodResultsConclusionTo test the predictive accuracy and generalisability of a personalised advantage index (PAI) model designed to support treatment selection for Post-Traumatic Stress Disorder (PTSD).A PAI model developed by Deisenhofer et al. (2018) was used to predict treatment outcomes in a statistically independent dataset including archival records for N = 152 patients with PTSD who accessed either trauma-focussed cognitive behavioural therapy or eye movement desensitisation and reprocessing in routine care. Outcomes were compared between patients who received their PAI-indicated optimal treatment versus those who received their suboptimal treatment.The model did not yield treatment specific predictions and patients who had received their PAI-indicated optimal treatment did not have better treatment outcomes in this external validation sample.This PAI model did not generalise to an external validation sample. [ABSTRACT FROM AUTHOR]

Scholars have made considerable strides in evaluating and improving the external validity of experimental research. However, little attention has been paid to a crucial aspect of external validity – the topic of study. Researchers frequently develop a general theory and hypotheses (e.g., about policy attitudes), then conduct a study on a specific topic (e.g., environmental attitudes). Yet, the results may vary depending on the topic chosen. In this paper, we develop the idea of topic sampling – rather than studying a single topic, we randomly sample many topics from a defined population. As an application, we combine topic sampling with a classic survey experiment design on partisan cues. Using a hierarchical model, we efficiently estimate the effect of partisan cues for each policy, showing that the size of the effect varies considerably, and predictably, across policies. We conclude with advice on implementing our approach and using it to improve theory testing. [ABSTRACT FROM AUTHOR]

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts. [ABSTRACT FROM AUTHOR]

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail. [ABSTRACT FROM AUTHOR]

Researchers and practitioners are increasingly embracing systems approaches to deal with the complexity of public service delivery and policy evaluation. However, there is little agreement on what exactly constitutes a systems approach, conceptually or methodologically. We review and critically synthesize systems literature from the fields of health, education, and infrastructure. We argue that the common theoretical core of systems approaches is the idea that multi-dimensional complementarities between a policy and other aspects of the policy context are the first-order problem of policy design and evaluation. We distinguish between macro-systems approaches, which focus on the collective coherence of a set of policies or institutions, and micro-systems approaches, which focus on how a single policy interacts with the context in which it operates. We develop a typology of micro-systems approaches and discuss their relationship to standard impact evaluation methods as well as to work in external validity, implementation science, and complexity theory. [ABSTRACT FROM AUTHOR]

Predictive models for Clostridioides difficile infection can identify high-risk patients and aid clinicians in preventing infection. Issues of generalizability regarding current predictive models have been acknowledged but, to the authors' knowledge, have never been quantified. C. difficile infection, severity and recurrence predictive models were created using multi-variate logistic regression through case–control sampling from an urban safety-net hospital. Models were validated using five-fold cross-validation, and inverse probability weights (IPW) based on two different catchment area definitions were used to improve external validity. Akaike Information Criterion (AIC), area under the receiver operating characteristic curve (AUROC), and sensitivity and specificity with bootstrapped confidence intervals (CI) were used to assess and compare model fit and performance. Changes in performance before and after weighting were small across all models, although differences were more apparent after weighting the recurrence model (AUROC values of 0.78, 0.76 and 0.71 for the unweighted and two weighted models, respectively). Overall, the infection model performed the best (AUROC 0.82, 95% CI 0.78–0.85), followed by the recurrence model (AUROC 0.78, 95% CI 0.69–0.86) and then the severity model (AUROC 0.70, 95% CI 0.63–0.78). The performance of the models after weighting did not change drastically, suggesting that the models predicting C. difficile infection, severity and recurrence may not be impacted by patient selection factors. However, other researchers may wish to consider addressing these catchment forces using IPW. [ABSTRACT FROM AUTHOR]

How informative are treatment effects estimated in one region or time period for another region or time? In this article, I derive bounds on the average treatment effect in a context of interest using experimental evidence from another context. The bounds are based on (a) the information identified about treatment effect heterogeneity due to unobservables in the experiment and (b) using differences in outcome distributions across contexts to learn about differences in distributions of unobservables. Empirically, using data from a pair of remedial education experiments carried out in India, I show the bounds are able to recover average treatment effects in one location using results from the other while the benchmark method cannot. [ABSTRACT FROM AUTHOR]

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods - including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation - that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations.

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator. [ABSTRACT FROM AUTHOR]

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three‐round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty‐four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus‐based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity. [ABSTRACT FROM AUTHOR]

Objectives Agriculture entered the discussions about safety climates late, despite being one of the most hazardous industries. We recently developed a safety climate scale in Bordeaux vineyards, for which we provided good evidence of reliability and construct validity (Grimbuhler and Viel 2019). In this study, we aimed to establish the external validity of this safety climate scale with the help of an independent national sample of vineyard professionals. Population and methods We approached vineyard managers and operators during compulsory training and certification procedures for pesticide-related activities. Trainees giving informed consent for participation in the study were asked to complete a safety climate questionnaire at the start and end of a training session. In total, 406 vineyard managers or operators completed the questionnaire at the start of the study, 37 of whom declined to complete the questionnaire at the end of the training session, leaving 369 subjects available for pretraining/post-training comparisons. Statistical comparisons were based on t -tests and mixed models for repeated measures. Results A mean safety climate score of 82.91 (SD: 9.06) was obtained in the initial survey in the Bordeaux region, whereas the safety climate score was estimated at 83.78 (SD 10.39) in this nationwide survey (P = 0.23). A significant increase was observed after the training course, for the safety climate score (7.5%, P < 10–15) and for each of its 7 dimensions (P < 10–4 or less), in both univariate and multivariate analyses. However, the magnitude of these increases varied with dimension, ranging from 2.4% for rules and best practices to 15.5% for communication and feedback. Conclusions External validity was demonstrated by transferability and sensitivity to intervention. This safety climate scale can now be considered to provide a good inference of the safety culture, with a meaning generalizable across vineyards. [ABSTRACT FROM AUTHOR]

Background: In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research. Methods: In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States. Results: The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations. Conclusions: The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings. [ABSTRACT FROM AUTHOR]

Generalizability of trial-based cost-effectiveness estimates to real-world target populations is important for decision making. In the context of independent aggregate time-to-event baseline and relative effects data, complex hazards can make modeling of data for use in economic evaluation challenging. Our article provides an overview of methods that can be used to apply trial-derived relative treatment effects to external real-world baselines when faced with complex hazards and follows with a motivating example. Approaches for applying trial-derived relative effects to real-world baselines are presented in the context of complex hazards. Appropriate methods are applied in a cost-effectiveness analysis using data from a previously published study assessing the real-world cost-effectiveness of a treatment for carcinoma of the head and neck as a motivating example. Lack of common hazards between the trial and target real-world population, a complex baseline hazard function, and nonproportional relative effects made the use of flexible models necessary to adequately estimate survival. Assuming common distributions between trial and real-world reference survival substantially affected survival and cost-effectiveness estimates. Modeling time-dependent vs proportional relative effects affected estimates to a lesser extent, dependent on assumptions used in cost-effectiveness modeling. Appropriately capturing reference treatment survival when attempting to generalize trial-derived relative treatment effects to real-world target populations can have important impacts on cost-effectiveness estimates. A balance between model complexity and adequacy for decision making should be considered where multiple data sources with complex hazards are being evaluated. • Generalizability is an important consideration for decision making. • Approaches used for applying trial-derived treatment effects to real-world target population baseline risks can substantially affect cost-effectiveness estimates. • Model complexity in light of requirements for informed decision making should be considered when generalizing cost-effectiveness estimates. [ABSTRACT FROM AUTHOR]

Considerations related to generalization of a study's findings are often interconnected to researchers' judgements regarding the 'quality' of the methodology and methodological pluralism. Too often, researchers consider generalization as only possible with respect to quantitative studies with large numbers of randomly selected participants (statistical generalization). Recently, Levitt suggests qualitative research can be generalized to phenomenon. Like others, she differentiates the type of generalization typically identified with quantitative research, which requires large samples and randomization so that findings can be generalized to the population, with the type of generalization that can be achieved with qualitative research. In turn, Levitt proposes the concept of qualitative generalization, an idea suggested by other researchers. However, her assertions about generalization in qualitative research resonate with the assertions of Thomas and Baas about generalizability in Q methodology. Q methodology offers a unique blend of qualitative and quantitative principles to study subjectivity and includes factor analysis. Yet Q findings are about developing theories, much like qualitative research. Q researchers may discuss substantive inference, connecting generalizations about phenomenon to a population, rather than statistical inference about a population. We discuss acceptance of two types of generalization within and their importance in relation to research equity. [ABSTRACT FROM AUTHOR]

Matching crop varieties to their target use context and user preferences is a challenge faced by many plant breeding programs serving smallholder agriculture. Numerous participatory approaches proposed by CGIAR and other research teams over the last four decades have attempted to capture farmers' priorities/preferences and crop variety field performance in representative growing environments through experimental trials with higher external validity. Yet none have overcome the challenges of scalability, data validity and reliability, and difficulties in capturing socio-economic and environmental heterogeneity. Building on the strengths of these attempts, we developed a new data-generation approach, called triadic comparison of technology options (tricot). Tricot is a decentralized experimental approach supported by crowdsourced citizen science. In this article, we review the development, validation, and evolution of the tricot approach, through our own research results and reviewing the literature in which tricot approaches have been successfully applied. The first results indicated that tricot-aggregated farmer-led assessments contained information with adequate validity and that reliability could be achieved with a large sample. Costs were lower than current participatory approaches. Scaling the tricot approach into a large on-farm testing network successfully registered specific climatic effects of crop variety performance in representative growing environments. Tricot's recent application in plant breeding networks in relation to decision-making has (i) advanced plant breeding lines recognizing socio-economic heterogeneity, and (ii) identified consumers' preferences and market demands, generating alternative breeding design priorities. We review lessons learned from tricot applications that have enabled a large scaling effort, which should lead to stronger decision-making in crop improvement and increased use of improved varieties in smallholder agriculture. [ABSTRACT FROM AUTHOR]

The purpose of this study was to explore the validity of evidence for self-efficacy to regulate physical activity scale (SERPA) measurement using an exploratory latent variable approach. The objectives were to explore the dimensionality, temporal invariance, and external validity of scores produced by the SERPA, a modified version of the barriers self-efficacy scale. Data (Nbaseline= 461 and N30 days post-baseline = 427) from the Well-Being and Physical Activity Study (ClinicalTrials.gov, identifier: NCT03194854), which deployed the Fun For Wellness intervention, were analyzed. A two-dimensional factor structure explained responses to the SERPA at baseline. There was evidence for at least partial strict temporal measurement invariance for this two-dimensional structure. There was evidence for external validity of the SERPA measurement as evidenced by group assignment (i.e. the intervention) exerting a direct effect on the proposed two-dimensional structure of latent self-efficacy to regulate physical activity in adults with obesity at 30 days post-baseline. [ABSTRACT FROM AUTHOR]

Obesity is a pandemic that disproportionately affects children from vulnerable populations in the USA. Current treatment approaches in primary care settings in the USA have been reported to be insufficient at managing pediatric obesity, primarily due to implementation challenges for healthcare systems and barriers for families. While the literature has examined the efficacy of pediatric obesity interventions focused on internal validity, it lacks sufficient reporting and analysis of external validity necessary for successful translation to primary care settings. We conducted a systematic review of the primary-care-setting literature from January 2007 to March 2020 on family-based pediatric weight management interventions in both English and/or Spanish for children ages 6–12 years in the USA using the Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework. A literature search, using PRISMA guidelines, was conducted in January 2022 using the following electronic databases: Medline Ovid, Embase, and Cochrane Library. 22 270 records were screened, and 376 articles were reviewed in full. 184 studies were included. The most commonly reported dimensions of the RE-AIM framework were Reach (65%), Efficacy/Effectiveness (64%), and Adoption (64%), while Implementation (47%) and Maintenance (42%) were less often reported. The prevalence of reporting RE-AIM construct indicators ranged greatly, from 1% to 100%. This systematic review underscores the need for more focus on external validity to guide the development, implementation, and dissemination of future pediatric obesity interventions based in primary care settings. It also suggests conducting additional research on sustainable financing for pediatric obesity interventions. [ABSTRACT FROM AUTHOR]

We address a concern about the external validity of an experiment conducted with university students related to the representativeness of the sampled population in particular. We do so by conducting large-scale (partly) incentivized online surveys of students at a Japanese university and of a sample of Japanese adults to measure individual characteristics such as cognitive ability, mentalizing skills, preferences for risk and distribution, and personality traits. While significant differences between these two samples are observed in many of these characteristics, the correlational structures among these characteristics are very similar in the two samples. [ABSTRACT FROM AUTHOR]

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods—lasso regression and Bayesian Additive Regression Trees (BART)—using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites. [ABSTRACT FROM AUTHOR]

Background: In this study, we aimed to establish the external validity of the cutoff values for frailty identification using both the 15-item and 12-item Questionnaire for Medical Checkups of the Old-Old (QMCOO-15 and QMCOO-12) and the frailty screening index (FSI). Method: This cross-sectional study was conducted in Takasaki, Gunma Prefecture, Japan. Data from 114 participants aged ≥65 years were analyzed. The frailty assessment was based on the Revised Japanese version of the Cardiovascular Health Study (J-CHS) criteria. The QMCOO-15, QMCOO-12, and FSI were also examined. The cutoff value for frailty discrimination using these three scales was set at 3/4. The frailty discrimination performance of the three scales was validated using receiver operating characteristic (ROC) analysis, sensitivity, and specificity. Results: The average age of the participants was 76.4 years, and there were 79 female participants (69.3 %). According to the J-CHS criteria, nine participants (7.9 %) were identified as frail. The areas under the ROC curves were 0.878 for QMCOO-15, 0.853 for QMCOO-12, and 0.912 for FSI. The QMCOO-15 exhibited the highest sensitivity (0.778), whereas the FSI exhibited the highest specificity (0.943). The FSI also achieved the highest accuracy of 0.912. Conclusions: This study revealed the external validity of frailty discrimination using the QMCOO-15, QMCOO-12, and FSI based on the J-CHS frailty assessment criteria. ROC analysis showed that all three tools exhibited good frailty discrimination performance. Hence, we demonstrated the validity of frailty assessments using questionnaires that did not involve physical measurements.

Gruntkowski and Martinez examined the impact of factors such as perceived risk and perceived usefulness on German consumers’ intention to purchase groceries online once the COVID-19 pandemic had subsided. They also compared consumer perceptions before and during the COVID-19 outbreak. This comment shows that Gruntkowski and Martinez’s research suffers from a number of problems, the most important of which is the use of an unrepresentative sample. They should therefore have refrained from generalizing their findings to the German population.

Roberts and colleagues focus on two aspects of racial inequality in psychological research, namely an alleged underrepresentation of racial minorities and the effects attributed to this state of affairs. My comment focuses only on one aspect, namely the assumed consequences of the lack of diversity in subject populations. Representativeness of samples is essential in survey research or applied research that examines whether a particular intervention will work for a particular population. Representativeness or diversity is not necessary in theory-testing research, where we attempt to establish laws of causality. Because theories typically apply to all of humanity, all members of humanity (even American undergraduates) are suitable for assessing the validity of theoretical hypotheses. Admittedly, the assumption that a theory applies to all of humanity is also a hypothesis that can be tested. However, to test it, we need theoretical hypotheses about specific moderating variables. Supporting a theory with a racially diverse sample does not make conclusions more valid than support from a nondiverse sample. In fact, cause-effect conclusions based on a diverse sample might not be valid for any member of that sample. [ABSTRACT FROM AUTHOR]