Search

Digital health research often relies on case vignettes (descriptions of fictitious or real patients) to navigate ethical and practical challenges. Despite their utility, the quality and lack of standardization of these vignettes has often been criticized, especially in studies on symptom-assessment applications (SAAs) and self-triage decision-making. To address this, our paper introduces a method to refine an existing set of vignettes, drawing on principles from classical test theory. First, we removed any vignette with an item difficulty of zero and an item-total correlation below zero. Second, we stratified the remaining vignettes to reflect the natural base rates of symptoms that SAAs are typically approached with, selecting those vignettes with the highest item-total correlation in each quota. Although this two-step procedure reduced the size of the original vignette set by 40%, comparing self-triage performance on the reduced and the original vignette sets, we found a strong correlation (r = 0.747 to r = 0.997, p <.001). This indicates that using our refinement method helps identifying vignettes with high predictive power of an agent's self-triage performance while simultaneously increasing cost-efficiency of vignette-based evaluation studies. This might ultimately lead to higher research quality and more reliable results. [ABSTRACT FROM AUTHOR]

Policymakers and educational researchers have grown increasingly interested in the extent to which study results generalize across different groups of students. Current generalization research in education has largely focused on the compositional similarity among students based on a set of observable characteristics. However, generalization is defined differently across various disciplines. While the concept of compositional similarity is prominent in causal research, generalization among the statistical learning community refers to the extent to which a model produces accurate predictions across samples and populations. The purpose of this study is to assess the extent to which concepts related to contextual generalization (based on compositional similarity) are associated with the ideas related to model generalization (based on accuracy of prediction). We use observational data from the Programme for International Student Assessment (PISA) 2015 wave as a case study to examine the conditions under which contextual and model generalization are aligned. We assess the correlations between statistical measures that quantify compositional similarity and prediction accuracy and discuss the implications for generalization research. [ABSTRACT FROM AUTHOR]

AbstractObjective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems. [ABSTRACT FROM AUTHOR]

This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent mixed-methods study design. The study used the Yes/No ratings as the baseline method in two rounds of ratings, while differentiating the two methods by incorporating item maps and an Ordered Item Booklet, each of which is an integral tool of the Mapmark and the Bookmark methods. The results showed that the internal validity evidence is similar across both methods, especially after Round 2 ratings. When procedural validity evidence was considered, however, a preference emerged for the method where panelists conducted the initial ratings unbeknownst to the empirical item difficulty information, and then such information was provided on an item map as part of the Round 1 feedback. The findings highlight the importance of evaluating both internal and procedural validity evidence when considering standard-setting methods. [ABSTRACT FROM AUTHOR]

BackgroundTime to symptom resolution measures were used in outpatient coronavirus disease 2019 (COVID-19) treatment trials without prior validation.MethodsACTIV-2/A5401 trial participants completed a COVID-19 diary assessing 13 targeted symptoms and global experience (overall COVID-19 symptoms, return to pre-COVID-19 health) daily for 29 days. We evaluated concordance of time to sustained (2 days) resolution of all targeted symptoms (TSR) with resolution of overall symptoms and return to health in participants receiving placebo.ResultsThe analysis included 77 high-risk and 81 standard-risk participants with overall median 6 days of symptoms at entry and median age 47 years, 50% female, 82% white, and 31% Hispanic/Latino. Correlation between TSR and resolution of overall symptoms was 0.80 and 0.68, and TSR and return to health, 0.66 and 0.57 for high- and standard-risk groups, respectively. Of the high- and standard-risk participants, 61% and 79%, respectively, achieved targeted symptom resolution, of which 47% and 43%, respectively, reported symptom recurrence. Requiring >2 days to define sustained resolution reduced the frequency of recurrences.ConclusionsThere was good internal consistency between TSR and COVID-19-specific global outcomes, supporting TSR as a trial end point. Requiring >2 days of symptom resolution better addresses natural symptom fluctuations but must be balanced against the potential influence of non-COVID-19 symptoms.Clinical trials registrationNCT04518410.

BackgroundReporting retention data is critical to determining the soundness of a study's conclusions (internal validity) and broader generalizability (external validity). Although selective attrition can lead to overestimates of effects, biased conclusions, or overly expansive generalizations, retention rates are not reported in many longitudinal studies.MethodsWe examined multiple child- and family-level factors potentially associated with retention in a longitudinal study of younger siblings of children with autism spectrum disorder (ASD; n = 304) or typical development (n = 163). The sample was followed from the first year of life to 36 months of age, for up to 7 visits.ResultsOf the 467 infant siblings who were consented and participated in at least one research visit, 397 (85.0%) were retained to study completion at 36 months. Retention rates did not differ by familial risk group (ASD-risk vs. Low-risk), sex, race, ethnicity, age at enrollment, number of children in the family, maternal employment, marital status, or parent concerns about the child at enrollment. A stepwise regression model identified 4 variables that, together, provided the most parsimonious predictive model of study retention: maternal education, maternal age at child's birth, travel distance to the study site, and diagnostic outcome classification at the final study visit.ConclusionsThe retained and not-retained groups did not differ on most demographic and clinical variables, suggesting few threats to internal and external validity. The significantly higher rate of retention of children diagnosed with ASD (95%) than typically developing children (83%) may, however, present biases when studying recurrence risk. We conclude by describing engagement and tracking methods that can be used to maximize retention in longitudinal studies of children at risk of ASD.

Abstract: We report psychometric analyses of an adapted version of the measure of individualism–collectivism (IC) and its horizontal–vertical (HV) manifestations (Singelis et al., 1995) for Brazilian samples. We used a large sample (N = 1,669) split into an exploratory and two cross-validation samples. Using confirmatory factor analysis and a new metaheuristic machine learning approach inspired by ant colony foraging, we found poor psychometric properties. We developed and compared three short scales, which showed poor reliability and poor factor structures. Considering widely used short scales of this measure, our results suggest that key items may be understood differently by Brazilians, and we urge caution in interpreting individualism–collectivism scores with Brazilian samples. We discuss issues for further developing cultural orientation scales to measure individual differences and the utility of an advanced metaheuristic data-driven approach for psychometric research.

The Implicit Association Test (IAT, Greenwald et al., J. Pers. Soc. Psychol., 74, 1998, 1464) is a popular instrument for measuring attitudes and (stereotypical) biases. Greenwald et al. (Behav. Res. Methods, 54, 2021, 1161) proposed a concrete method for validating IAT stimuli: appropriate stimuli should be familiar and easy to classify – translating to rapid (response times <800 ms) and accurate (error < 10%) participant responses. We conducted three analyses to explore the theoretical and practical utility of these proposed validation criteria. We first applied the proposed validation criteria to the data of 15 IATs that were available via Project Implicit. A bootstrap approach with 10,000 'experiments' of 100 participants showed that 5.85% of stimuli were reliably valid (i.e., we are more than 95% confident that a stimulus will also be valid in a new sample of 18‐ to 25–year‐old US participants). Most stimuli (78.44%) could not be reliably validated, indicating a less than 5% certainty in the outcome of stimulus (in)validity for a new sample of participants. We then explored how stimulus validity differs across IATs. Results show that only some stimuli are consistently (in)valid. Most stimuli show between‐IAT variances, which indicate that stimulus validity differs across IAT contexts. In the final analysis, we explored the effect of stimulus type (images, nouns, names, adjectives) on stimulus validity. Stimulus type was a significant predictor of stimulus validity. Although images attain the highest stimulus validity, raw data show large differences within stimulus types. Together, the results indicate a need for revised validation criteria. We finish with practical recommendations for stimulus selection and (post‐hoc) stimulus validation. [ABSTRACT FROM AUTHOR]

Correlation does not imply causation and psychologists' causal inference training often focuses on the conclusion that therefore experiments are needed—without much consideration for the causal inference frameworks used elsewhere. This leaves researchers ill‐equipped to solve inferential problems that they encounter in their work, leading to mistaken conclusions and incoherent statistical analyses. For a more systematic approach to causal inference, this article provides brief introductions to the potential outcomes framework—the "lingua franca" of causal inference—and to directed acyclic graphs, a graphical notation that makes it easier to systematically reason about complex causal situations. I then discuss two issues that may be of interest to researchers in social and personality psychology who think that formalized causal inference is of little relevance to their work. First, posttreatment bias: In various common scenarios (noncompliance, mediation analysis, missing data), researchers may analyze data from experimental studies in a manner that results in internally invalid conclusions, despite randomization. Second, tests of incremental validity: Routine practices in personality psychology suggest that they may be conducted for at least two different reasons (to demonstrate the non‐redundancy of new scales, to support causal conclusions) without being particularly suited for either purpose. Taking causal inference seriously is challenging; it reveals assumptions that may make many uncomfortable. However, ultimately it is a necessary step to ensure the validity of psychological research. [ABSTRACT FROM AUTHOR]

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example. [ABSTRACT FROM AUTHOR]

We report psychometric analyses of an adapted version of the measure of individualism–collectivism (IC) and its horizontal–vertical (HV) manifestations (Singelis et al., 1995) for Brazilian samples. We used a large sample (N = 1,669) split into an exploratory and two cross-validation samples. Using confirmatory factor analysis and a new metaheuristic machine learning approach inspired by ant colony foraging, we found poor psychometric properties. We developed and compared three short scales, which showed poor reliability and poor factor structures. Considering widely used short scales of this measure, our results suggest that key items may be understood differently by Brazilians, and we urge caution in interpreting individualism–collectivism scores with Brazilian samples. We discuss issues for further developing cultural orientation scales to measure individual differences and the utility of an advanced metaheuristic data-driven approach for psychometric research. [ABSTRACT FROM AUTHOR]

Meta‐analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst‐case" form of publication bias. Robust estimation methods accommodate non‐normal effects, small meta‐analyses, and clustered estimates. As we illustrate by re‐analyzing published meta‐analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta). [ABSTRACT FROM AUTHOR]

Active learning is a popular approach to teaching and learning that has gained traction through research on STEM educational improvement. There have been numerous university- and national/international-level efforts focused on transitioning courses from the lecture method to active learning. However, despite these large-scale changes, the active learning literature has not been assessed on its methodological rigor to ensure instructional recommendations are rooted in rigorous research studies. The purpose of the present review was to determine areas of strengths and areas in need of improvement and to provide specific recommendations on how to continue or improve active learning research to strengthen the respective literature base and increase confidence in results. We assessed the articles included in the Freeman et al. (PNAS, 111:8410–8415, 2014) meta-analysis as well as a random sample of more recent active learning articles (2015–2022) on 12 internal validity controls (i.e., control procedure used to prevent a threat to the internal validity of a study). Results indicated that there were high percentages of articles that did not meet each internal validity control. In fact, no articles from the Freeman et al. meta-analysis and no sampled 2015–2022 articles met each of the 12 internal validity controls. Therefore, the active learning literature contains numerous internal validity control issues that need to be addressed if we are to determine the extent to which active learning interventions are effective and if there are any boundary conditions for when particular active learning interventions are or are not effective. [ABSTRACT FROM AUTHOR]

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings. [ABSTRACT FROM AUTHOR]

Practitioners frequently use single-case data for decision-making related to behavioral programming and progress monitoring. Visual analysis is an important and primary tool for reporting results of graphed single-case data because it provides immediate, contextualized information. Criticisms exist concerning the objectivity and reliability of the visual analysis process. When practitioners are equipped with knowledge about single-case designs, including threats and safeguards to internal validity, they can make technically accurate conclusions and reliable data-based decisions with relative ease. This paper summarizes single-case experimental design and considerations for professionals to improve the accuracy and reliability of judgments made from single-case data. This paper can also help practitioners to appropriately incorporate single-case research design applications in their practice. [ABSTRACT FROM AUTHOR]

Critical appraisal scales play an important role in evaluating methodological rigor (MR) of between-groups and single-case designs (SCDs). For intervention research this forms an essential basis for ascertaining the strength of evidence. Yet, few such scales provide classifications that take into account the differential weighting of items contributing to internal validity. This study aimed to develop an algorithm derived from the Risk of Bias in N-of-1 Trials (RoBiNT) Scale to classify MR and risk of bias magnitude in SCDs. The algorithm was applied to 46 SCD experiments. Two experiments (4%) were classified as Very High MR, 14 (30%) as High, 5 (11%) as Moderate, 2 (4%) as Fair, 2 (4%) as Low, and 21 (46%) as Very Low. These proportions were comparable to the What Works Clearinghouse classifications: 13 (28%) met standards, 8 (17%) met standards with reservations, and 25 (54%) did not meet standards. There was strong association between the two classification systems. [ABSTRACT FROM AUTHOR]

Researchers have been concerned with internal and external validity for decades, and the discussion continues. The present proposal is that there are less important and more important senses in which one can interpret internal and external validity, and these can be integrated with a taxonomy that includes theoretical, auxiliary, statistical, and inferential assumptions. The integration sheds light on recent exchanges in the literature on validity and suggests that the vaunted internal–external validity trade-off is false for more important senses of internal and external validity; internal and external validity increase or decrease together when there is an emphasis on underlying theories. Finally, the integration implies the desirability of some changes in typical research advice and practice. [ABSTRACT FROM AUTHOR]

To be ethical, clinical trials must exhibit a favorable risk‐benefit balance at the time of their initiation. However, in some cases, the expected value of a study decreases while the study is ongoing, due to developments outside of the study itself, such as findings from other studies or an otherwise shifting evidence base. While such situations are acknowledged in the research community, they have not received sufficient attention, given the high costs of uninformative studies, both in material and human capital. In addition, the Covid‐19 pandemic has exposed serious shortcomings with current approaches to monitoring studies for continued relevance and value. In this article, with reference to a case study from the Covid‐19 pandemic, we identify and describe the importance and challenge of ensuring that clinical trials continue to exhibit scientific relevance and value once initiated. We explore the ethical dynamics of these situations and identify unresolved issues. While more empirical work is needed to ensure that proposed solutions to the issues are evidence based, we offer some provisional considerations that amount to a framework for approaching these challenging situations. [ABSTRACT FROM AUTHOR]

We approach attrition in feld experiments with baseline data as an identifcation problem in a panel model. A systematic review of the literature indicates that there is no consensus on how to test for attrition bias. We establish identifying assumptions for treatment effects for both the respondent subpopulation and the study population. We propose randomization and regression-based procedures to test their sharp implications. We then relate our proposed tests to current empirical practice, and demonstrate that the most commonly used test can over-reject internal validity. Simulations and applications further support the relevance of our analysis.

Homeopathy is a system of therapeutics that treats disease with highly diluted substances based on the Law of Similars, which holds that “like cures like.” Despite widespread use, homeopathy lacks a comprehensive and robust evidence base.We examined the state of homeopathic clinical research by critically assessing the overall quality of peer-reviewed, recently published, English-language, homeopathic clinical research in terms of internal, external, and model validity using standard and homeopathic-specific instruments. Further, we convened an international panel of nine experts in research methods and homeopathy to identify gaps in homeopathic research and prioritize areas for future study.We reviewed 99 clinical research studies targeting a wide range of populations and conditions. Studies were conducted in Western and Asian countries, with the largest number (30 percent) conducted in India. Of the 99 studies reviewed, 85 were controlled trials; 79 of these were randomized. There were many areas where the quality of the studies could be improved. About two-thirds of the 85 controlled trials had either high (42 percent) or unclear (24 percent) risk of bias according to internationally recognized standards for internal validity. Of the 14 observational (cohort) studies, over one-third did not control for important confounders in the outcome analyses. Regarding external validity, adherence was reported in less than a third of studies (n=31). Forty percent of studies (79 % of observational studies) did not report on safety. Regarding model validity, fewer than two-thirds of the studies were consistent with homeopathic principles.Our expert panel was mixed on whether the homeopathic research literature was missing important populations and/or conditions, and they suggested a variety of priority areas. Panelists also expressed a variety of opinions about the types of homeopathy that should be prioritized for future study but also noted that since homeopathic practice differs by country, each country may have different priorities.Panelists agreed with the findings of the literature review that the research literature was at least somewhat deficient in all three types of validity. Although our assessment of validity was [by necessity] based only on what was reported, this assessment suggests the need for both better reporting and higher quality research. They recommended the use of reporting guidelines to improve all types of validity, the identification of exemplar studies to help guide researchers to improve internal validity, and, given the limitations of the instruments available to measure external and model validity, that these instruments be validated and configured to provide summary scores. Finally, substantial discussion addressed the need to bring more research expertise into homeopathic studies, both by better training homeopathic researchers and by collaborating with experienced conventional medicine research groups.

Copyright of Zeitschrift für Erziehungswissenschaft is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Transparency in research reporting is crucial for evaluating the reproducibility and validity of research, including potential confounding factors (internal validity) and generalizability (external validity). Here, we focus on visual stimuli—stimuli routinely used to elicit mental processes and behaviors—as a case study to systematically assess and evaluate current practices in reporting visual characteristics, including display setup, stimulus size, luminance/color, and contrast. Our first study scrutinized recent publications (N = 360) in leading journals in neuroscience and psychology—spanning vision, cognitive, clinical, developmental, and social/personality psychology. The second study examined recent publications (N = 114) on visual attentional bias in clinical samples, involving tasks known to be sensitive to visual properties. Analyzing the full text and supplemental materials of these articles, the two studies reveal a systematic lapse in current practices of reporting characteristics of visual stimuli. This reporting failure was not due to authors making visual materials available online, which was rare (<20%) and could not replace the reporting of visual characteristics. Failure to report stimulus properties hinders efforts to build cumulative science: (a) direct replications become challenging if not impossible; (b) internal validity may be compromised; and (c) generalizability across stimulus properties is prematurely assumed, and its evaluation is precluded in the first place. Our findings have immediate implications for journal policies on reporting practices, urging for explicit emphasis on transparent reporting of stimulus properties, particularly when perceptual components are involved. To assist in this effort, we provide templates for reporting study setup, visual displays, and visual stimuli. Public Significance Statement: Numerous editorials, writing guides, author guidelines, and manifestos for reproducible science have focused on the importance of transparency in data, code, and research materials. We identify a pervasive failure to report properties of visual stimuli in research in psychology and neuroscience—which may undermine research reproducibility and validity—and outline potential solutions. [ABSTRACT FROM AUTHOR]

Many reviews referred to as 'systematic reviews' in ecology are not consistent with best practice in that they generally lack appropriate critical appraisal of included studies. This limitation is particularly important in applied ecology, where there have been increasing calls for more systematic reviews to guide decision making. To identify the available critical appraisal tools (CATs) and hierarchies of evidence available for ecology studies, we systematically searched for: studies that described the development and/or examination of tools to assess the potential methodological bias in studies of ecology; and the tools used to assess potential methodological bias of included studies in ecological systematic reviews. We identified 680 reviews labelled as 'systematic reviews' in ecology, however only 4.0% performed critical appraisal of the included studies. Three hierarchies of evidence and 23 CATs were identified, and assessed as lacking independent development, validity and reliability testing, and/or completeness. The authors of the reviews that included critical appraisal have appropriately identified the need to move reviews in ecology in the direction of this higher level of evidence, and have taken applied ecology further in the direction of evidence‐based practice. However, we identified shortcomings in these approaches when compared with best practice, and conclude that new tools are needed that reflect a range of questions posed in ecology. Through increasing the availability of such tools, the strength of evidence provided by systematic reviews in ecology would improve. [ABSTRACT FROM AUTHOR]

In their introductory marketing, management, and social psychology courses, undergraduates learn that correlation coefficients provide weak evidence for causal conclusions. Nonetheless, researchers conclude causally from correlation coefficients by drawing causal arrows in their structural equation models (SEMs). Although most researchers avoid describing their findings in causal language, obligatory recommendations for applying those findings insert causation. Researchers' standard rejoinder to validity challenges is "the critics have ignored theory's role in rendering our SEM internally and externally valid". To evaluate this rejoinder, we explore SEMs based on comprehensive underlying theories and as stand-alone and testable context-specific theories that blend previously published hypotheses and findings. In most cases, the rejoinder is unconvincing. [ABSTRACT FROM AUTHOR]

Urban ecological studies have the potential to expand our understanding of socioecological systems beyond that of an individual city or region. Cross‐comparative empirical work and synthesis are imperative to develop a general urban ecological theory. This can be achieved only if studies are replicable and generalizable. Transparency in methods reporting facilitates generalizability and replicability by documenting the decisions scientists make during the various steps of research design; this is particularly true for sampling design and selection because of their impact on both internal and external validity and the potential to unintentionally introduce bias. Three interdependent aspects of sample design are study sample selection (e.g., specific organisms, soils, or water), sample specification (measurement of specific variable of interest), and site selection (locations sampled). Of these, documentation of site selection—the where component of sample design—is underrepresented in the urban ecology literature. Using a stratified random sample of 158 papers from 12 major urban ecology journals, we investigated how researchers selected study sites in urban ecosystems and evaluated whether their site selection methods were transparent. We extracted data from these papers using a 50‐question, theory‐based questionnaire and a multiple‐reviewer approach. Our sample represented almost 45 years of urban ecology research across 40 different countries. We found that more than 80% of the papers we read were not transparent in their site selection methodology. We do not believe site selection methods are replicable for 70% of the papers read. Key weaknesses include incomplete descriptions of populations and sampling frames, urban gradients, sample selection methods, and property access. Low transparency in reporting the where methodology limits urban ecologists' ability to assess the internal and external validity of studies' findings and to replicate published studies; it also limits the generalizability of existing studies. The challenges of low transparency are particularly relevant in urban ecology, a field where standard protocols for site selection and delineation are still being developed. These limitations interfere with the fields' ability to build theory and inform policy. We conclude by offering a set of recommendations to increase transparency, replicability, and generalizability. [ABSTRACT FROM AUTHOR]

BACKGROUND: This study aimed to determine the face, convergent validity, internal consistency, and stability reliability of the impact on participation and autonomy-Persian version (IPA-p) scale for using among Iranian patients with type 2 diabetes. MATERIALS AND METHODS: Trained experts interviewed 227 type 2 diabetes patients who were registered in out-patient Diabetes Clinic of Ali-Ibn Abi-Talib Hospital, (Rafsanjan, Southeast Iran from May 2018 to February 2019) and their relatives to assess the face and convergent validity, internal consistency, and stability reliability of the IPA-p scale. A checklist was used to collect demographic information and also to record expert’s̛̛̛̛̛̛ points of view about the scale to assess face validity. Internal consistency was measured using Cronbach’s alpha, and stability was assessed using interclass correlation coefficients (ICCs). Test–retest method was used to detect the reliability of the questionnaire. Respondents completed the IPA-p scale on the two occasions with an interval of 30-45 days. RESULTS: In relation to convergent validity, the confirmatory model showed an acceptable fit and the scale had a highly convergent validity. Exploratory factor analysis showed that the IPA-p scale has a ten-factor structure that explained 77.42% of the variance. Cronbach’s alpha between the mean IPA-p scores achieved on the two occasions ranged from 0.65 to 0.92. Test–retest ICCs for the ten domains were between 0.64 and 0.81. CONCLUSIONS: The IPA-p questionnaire can be a relatively valid and reliable instrument for assessing self-reported participation among Iranian type 2 diabetes patients. However, some improvement is needed to make it fully suitable for using among Persian-language diabetic patients.

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example.

We approach attrition in field experiments with baseline data as an identification problem in a panel model. A systematic review of the literature indicates that there is no consensus on how to test for attrition bias. We establish identifying assumptions for treatment effects for both the respondent subpopulation and the study population, and propose procedures to test their sharp implications. We then relate our proposed tests to current empirical practice, and demonstrate that the most commonly used test in the literature is not a test of internal validity in general. We illustrate the relevance of our analysis using several empirical applications.

Abstract Background Reporting retention data is critical to determining the soundness of a study's conclusions (internal validity) and broader generalizability (external validity). Although selective attrition can lead to overestimates of effects, biased conclusions, or overly expansive generalizations, retention rates are not reported in many longitudinal studies. Methods We examined multiple child‐ and family‐level factors potentially associated with retention in a longitudinal study of younger siblings of children with autism spectrum disorder (ASD; n = 304) or typical development (n = 163). The sample was followed from the first year of life to 36 months of age, for up to 7 visits. Results Of the 467 infant siblings who were consented and participated in at least one research visit, 397 (85.0%) were retained to study completion at 36 months. Retention rates did not differ by familial risk group (ASD‐risk vs. Low‐risk), sex, race, ethnicity, age at enrollment, number of children in the family, maternal employment, marital status, or parent concerns about the child at enrollment. A stepwise regression model identified 4 variables that, together, provided the most parsimonious predictive model of study retention: maternal education, maternal age at child's birth, travel distance to the study site, and diagnostic outcome classification at the final study visit. Conclusions The retained and not‐retained groups did not differ on most demographic and clinical variables, suggesting few threats to internal and external validity. The significantly higher rate of retention of children diagnosed with ASD (95%) than typically developing children (83%) may, however, present biases when studying recurrence risk. We conclude by describing engagement and tracking methods that can be used to maximize retention in longitudinal studies of children at risk of ASD.

Abstract Urban ecological studies have the potential to expand our understanding of socioecological systems beyond that of an individual city or region. Cross‐comparative empirical work and synthesis are imperative to develop a general urban ecological theory. This can be achieved only if studies are replicable and generalizable. Transparency in methods reporting facilitates generalizability and replicability by documenting the decisions scientists make during the various steps of research design; this is particularly true for sampling design and selection because of their impact on both internal and external validity and the potential to unintentionally introduce bias. Three interdependent aspects of sample design are study sample selection (e.g., specific organisms, soils, or water), sample specification (measurement of specific variable of interest), and site selection (locations sampled). Of these, documentation of site selection—the where component of sample design—is underrepresented in the urban ecology literature. Using a stratified random sample of 158 papers from 12 major urban ecology journals, we investigated how researchers selected study sites in urban ecosystems and evaluated whether their site selection methods were transparent. We extracted data from these papers using a 50‐question, theory‐based questionnaire and a multiple‐reviewer approach. Our sample represented almost 45 years of urban ecology research across 40 different countries. We found that more than 80% of the papers we read were not transparent in their site selection methodology. We do not believe site selection methods are replicable for 70% of the papers read. Key weaknesses include incomplete descriptions of populations and sampling frames, urban gradients, sample selection methods, and property access. Low transparency in reporting the where methodology limits urban ecologists' ability to assess the internal and external validity of studies' findings and to replicate published studies; it also limits the generalizability of existing studies. The challenges of low transparency are particularly relevant in urban ecology, a field where standard protocols for site selection and delineation are still being developed. These limitations interfere with the fields' ability to build theory and inform policy. We conclude by offering a set of recommendations to increase transparency, replicability, and generalizability.

Philosophers have debated at length the epistemological status of scientific thought experiments. I contend that the literature on this topic still lacks a common conceptual framework, a lacuna that produces radical disagreement among the participants in this debate. To remedy this problem, I suggest focusing on the distinction between the internal and the external validity of an experiment, which is also crucial for thought experiments. I then develop an account of both kinds of validity in the context of thought experiments. I show that we can naturally conceptualise internal validity in terms of games of make-believe. Then, I argue that external validity is best defined as accurate representation of a target system. Finally, I turn back to the current debate on thought experiments and show that my diarchic account provides a general framework that can be shared by the competing philosophical views, as well as a fruitful guide for their reconciliation. [ABSTRACT FROM AUTHOR]

Objective: To introduce a novel analytical approach for randomized controlled trials that are underpowered because of low participant enrollment or engagement.Data Sources: Reanalysis of data for 805 patients randomized as part of a pilot complex care intervention in 2015-2016 in a large delivery system. In the pilot randomized trial, only 64.6% of patients assigned to the intervention group participated.Study Design: A case study and simulation. The "Distillation Method" capitalizes on the frequently observed correlation between the probability of subjects' participation or engagement in the intervention and the magnitude of benefit they experience. The novel method involves three stages: first, it uses baseline covariates to generate predicted probabilities of participation. Next, these are used to produce nested subsamples of the randomized intervention and control groups that are more concentrated with subjects who were likely to participate/engage. Finally, for the outcomes of interest, standard statistical methods are used to re-evaluate intervention effectiveness in these concentrated subsets.Data Extraction Methods: We assembled secondary data on patients who were randomized to the pilot intervention for one year prior to randomization and two follow-up years. Data included program enrollment status, membership data, demographics, utilization, costs, and clinical data.Principal Findings: Using baseline covariates only, Generalized Boosted Regression Models predicting program enrollment performed well (AUC 0.884). We then distilled the full randomized sample to increasing levels of concentration and reanalyzed program outcomes. We found statistically significant differences in outpatient utilization and emergency department utilization (both follow-up years), and in total costs (follow-up year two only) at select levels of population concentration.Conclusions: By offering an internally valid analytic framework, the Distillation Method can increase the power to detect effects by redefining the estimand to subpopulations with higher enrollment probabilities and stronger average treatment effects while maintaining the original randomization. [ABSTRACT FROM AUTHOR]

Within-study comparisons (WSCs) compare quasi-experimental results to an experimental benchmark and assess the extent to which quasi-experiments (QEs) sacrifice internal validity for external validity. WSCs inform decisions about when to use experiments or QEs, as well as methodological decisions (e.g., covariate selection) about how to implement QEs when they are necessary or preferred. We review the methodological literature on WSCs and present the findings from a WSC of a school voucher program as an example of the use of WSCs to inform policy debates involving the third sector. Prior literature and our results suggest that QE evaluations of third-sector institutions can produce low levels of bias, but only if analytic models include key proxy variables for voluntary self-selection into programs. [ABSTRACT FROM AUTHOR]

Copyright of Gazi University Journal of Gazi Educational Faculty (GUJGEF) is the property of Gazi University Journal of Gazi Educational Faculty (GUJGEF) and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract In our article ‘European agroforestry has no unequivocal effect on biodiversity: a time-cumulative meta-analysis’ (BMC Ecology and Evolution, 2021) we synthesize the effect of agroforestry on biodiversity. Boinot et al. (BMC Ecology and Evolution, 2022) criticise our approach arguing that our definitions of agroforestry and biodiversity are too narrow; that we use inappropriate control sites for primary studies lacking distance to the treatment sites; that there are too few studies for a meta-analysis in silvoarable systems; and that local practice should be emphasized. We agree on the importance of local practices and that more studies would improve the robustness of our meta-analytical results. However, our conclusions are robust to removing studies criticised for inappropriate controls. We also recognize the problem of different definitions of agroforestry and using species richness as sole proxy for biodiversity. We appreciate being given the opportunity to clarify our results and to trigger future discussions about definitions and the interpretation of results from meta-analysis.

Abstract The internal validity of conclusions about effectiveness or impact in systematic reviews, and of decisions based on them, depends on risk of bias assessments being conducted appropriately. However, a random sample of 50 recently-published articles claiming to be quantitative environmental systematic reviews found 64% did not include any risk of bias assessment, whilst nearly all that did omitted key sources of bias. Other limitations included lack of transparency, conflation of quality constructs, and incomplete application of risk of bias assessments to the data synthesis. This paper addresses deficiencies in risk of bias assessments by highlighting core principles that are required for risk of bias assessments to be fit-for-purpose, and presenting a framework based on these principles to guide review teams on conducting risk of bias assessments appropriately and consistently. The core principles require that risk of bias assessments be Focused, Extensive, Applied and Transparent (FEAT). These principles support risk of bias assessments, appraisal of risk of bias tools, and the development of new tools. The framework follows a Plan-Conduct-Apply-Report approach covering all stages of risk of bias assessment. The scope of this paper is comparative quantitative environmental systematic reviews which address PICO or PECO-type questions including, but not limited to, topic areas such as environmental management, conservation, ecosystem restoration, and analyses of environmental interventions, exposures, impacts and risks.

Abstract Introduction The Swedish version of the patient-reported Clinical Outcomes in Routine Evaluation Outcome Measures (CORE-OM) has demonstrated high reliability and acceptable convergent validity in explanatory factor analyses. However, the fundamental scale properties have not yet been validated according to item response theory. The aim of this study was to analyze the measurement properties of the Swedish CORE-OM in a cohort of psychiatric out-patients with depression and anxiety in a multicultural area and to explore combinations of items based on shorter versions of the scale (CORE-10, CORE-6D) to improve measurement properties. Methods Data from CORE-OM assessments of 337 patients were analyzed using Rasch analysis. The patients had a mean age of 30 ± 14 years, the majority were women (72%). Requirements for measurement properties were checked: overall model fit, item fit residuals, targeting, internal consistency, differential item functioning and thresholds. Sensitivity to change was also analyzed. Results The CORE-OM showed high internal consistency (person separation index = 0.947) and adequate targeting, but there was overall model misfit (item trait interaction χ2 = 917.53, p