Search

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c‐for‐benefit), and propose model‐based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model‐based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R. [ABSTRACT FROM AUTHOR]

We propose to combine the benefits of flexible parametric survival modeling and regularization to improve risk prediction modeling in the context of time‐to‐event data. Thereto, we introduce ridge, lasso, elastic net, and group lasso penalties for both log hazard and log cumulative hazard models. The log (cumulative) hazard in these models is represented by a flexible function of time that may depend on the covariates (i.e., covariate effects may be time‐varying). We show that the optimization problem for the proposed models can be formulated as a convex optimization problem and provide a user‐friendly R implementation for model fitting and penalty parameter selection based on cross‐validation. Simulation study results show the advantage of regularization in terms of increased out‐of‐sample prediction accuracy and improved calibration and discrimination of predicted survival probabilities, especially when sample size was relatively small with respect to model complexity. An applied example illustrates the proposed methods. In summary, our work provides both a foundation for and an easily accessible implementation of regularized parametric survival modeling and suggests that it improves out‐of‐sample prediction performance. [ABSTRACT FROM AUTHOR]

Competing Interests: Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare support from the Swiss National Science Foundation, National Institutes of Health, and European Union's Horizon 2020 research and innovation programme for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

As the scientific research community along with healthcare professionals and decision makers around the world fight tirelessly against the coronavirus disease 2019 (COVID-19) pandemic, the need for comparative effectiveness research (CER) on preventive and therapeutic interventions for COVID-19 is immense. Randomized controlled trials markedly under-represent the frail and complex patients seen in routine care, and they do not typically have data on long-term treatment effects. The increasing availability of electronic health records (EHRs) for clinical research offers the opportunity to generate timely real-world evidence reflective of routine care for optimal management of COVID-19. However, there are many potential threats to the validity of CER based on EHR data that are not originally generated for research purposes. To ensure unbiased and robust results, we need high-quality healthcare databases, rigorous study designs, and proper implementation of appropriate statistical methods. We aimed to describe opportunities and challenges in EHR-based CER for COVID-19-related questions and to introduce best practices in pharmacoepidemiology to minimize potential biases. We structured our discussion into the following topics: (1) study population identification based on exposure status; (2) ascertainment of outcomes; (3) common biases and potential solutions; and (iv) data operational challenges specific to COVID-19 CER using EHRs. We provide structured guidance for the proper conduct and appraisal of drug and vaccine effectiveness and safety research using EHR data for the pandemic. This paper is endorsed by the International Society for Pharmacoepidemiology (ISPE)., (© 2022 The Authors. Clinical Pharmacology & Therapeutics published by Wiley Periodicals LLC on behalf of American Society for Clinical Pharmacology and Therapeutics.)

Background: Most pregnancy-related deaths in low and middle income countries occur around the time of birth and are avoidable with timely care. This study aimed to develop a prognostic model to identify women at risk of intrapartum-related perinatal deaths in low-resourced settings, by (1) external validation of an existing prediction model, and subsequently (2) development of a novel model., Methods: A prospective cohort study was conducted among pregnant women who presented consecutively for delivery at the maternity unit of Zanzibar's tertiary hospital, Mnazi Mmoja Hospital, the Republic of Tanzania between October 2017 and May 2018. Candidate predictors of perinatal deaths included maternal and foetal characteristics obtained from routine history and physical examination at the time of admission to the labour ward. The outcomes were intrapartum stillbirths and neonatal death before hospital discharge. An existing stillbirth prediction model with six predictors from Nigeria was applied to the Zanzibar cohort to assess its discrimination and calibration performance. Subsequently, a new prediction model was developed using multivariable logistic regression. Model performance was evaluated through internal validation and corrected for overfitting using bootstrapping methods., Findings: 5747 mother-baby pairs were analysed. The existing model showed poor discrimination performance (c-statistic 0·57). The new model included 15 clinical predictors and showed promising discriminative and calibration performance after internal validation (optimism adjusted c-statistic of 0·78, optimism adjusted calibration slope =0·94)., Interpretation: The new model consisted of predictors easily obtained through history-taking and physical examination at the time of admission to the labour ward. It had good performance in predicting risk of perinatal death in women admitted in labour wards. Therefore, it has the potential to assist skilled birth attendance to triage women for appropriate management during labour. Before routine implementation, external validation and usefulness should be determined in future studies., Funding: The study received funding from Laerdal Foundation, Otto Kranendonk Fund and UMC Global Health Fellowship. TD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050)., Competing Interests: NH received funding from Laerdal Foundation and Otto Kranendonk Fund for the described work. TD reports consulting fees from Biogen, unrelated to this work., (© 2022 The Author(s).)

Objectives: Missing data is a common problem during the development, evaluation, and implementation of prediction models. Although machine learning (ML) methods are often said to be capable of circumventing missing data, it is unclear how these methods are used in medical research. We aim to find out if and how well prediction model studies using machine learning report on their handling of missing data., Study Design and Setting: We systematically searched the literature on published papers between 2018 and 2019 about primary studies developing and/or validating clinical prediction models using any supervised ML methodology across medical fields. From the retrieved studies information about the amount and nature (e.g. missing completely at random, potential reasons for missingness) of missing data and the way they were handled were extracted., Results: We identified 152 machine learning-based clinical prediction model studies. A substantial amount of these 152 papers did not report anything on missing data (n = 56/152). A majority (n = 96/152) reported details on the handling of missing data (e.g., methods used), though many of these (n = 46/96) did not report the amount of the missingness in the data. In these 96 papers the authors only sometimes reported possible reasons for missingness (n = 7/96) and information about missing data mechanisms (n = 8/96). The most common approach for handling missing data was deletion (n = 65/96), mostly via complete-case analysis (CCA) (n = 43/96). Very few studies used multiple imputation (n = 8/96) or built-in mechanisms such as surrogate splits (n = 7/96) that directly address missing data during the development, validation, or implementation of the prediction model., Conclusion: Though missing values are highly common in any type of medical research and certainly in the research based on routine healthcare data, a majority of the prediction model studies using machine learning does not report sufficient information on the presence and handling of missing data. Strategies in which patient data are simply omitted are unfortunately the most often used methods, even though it is generally advised against and well known that it likely causes bias and loss of analytical power in prediction model development and in the predictive accuracy estimates. Prediction model researchers should be much more aware of alternative methodologies to address missing data., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Introduction: Causal methods have been adopted and adapted across health disciplines, particularly for the analysis of single studies. However, the sample sizes necessary to best inform decision-making are often not attainable with single studies, making pooled individual-level data analysis invaluable for public health efforts. Researchers commonly implement causal methods prevailing in their home disciplines, and how these are selected, evaluated, implemented and reported may vary widely. To our knowledge, no article has yet evaluated trends in the implementation and reporting of causal methods in studies leveraging individual-level data pooled from several studies. We undertake this review to uncover patterns in the implementation and reporting of causal methods used across disciplines in research focused on health outcomes. We will investigate variations in methods to infer causality used across disciplines, time and geography and identify gaps in reporting of methods to inform the development of reporting standards and the conversation required to effect change., Methods and Analysis: We will search four databases (EBSCO, Embase, PubMed, Web of Science) using a search strategy developed with librarians from three universities (Heidelberg University, Harvard University, and University of California, San Francisco). The search strategy includes terms such as 'pool*', 'harmoniz*', 'cohort*', 'observational', variations on 'individual-level data'. Four reviewers will independently screen articles using Covidence and extract data from included articles. The extracted data will be analysed descriptively in tables and graphically to reveal the pattern in methods implementation and reporting. This protocol has been registered with PROSPERO (CRD42020143148)., Ethics and Dissemination: No ethical approval was required as only publicly available data were used. The results will be submitted as a manuscript to a peer-reviewed journal, disseminated in conferences if relevant, and published as part of doctoral dissertations in Global Health at the Heidelberg University Hospital., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Introduction: Pooling (or combining) and analysing observational, longitudinal data at the individual level facilitates inference through increased sample sizes, allowing for joint estimation of study- and individual-level exposure variables, and better enabling the assessment of rare exposures and diseases. Empirical studies leveraging such methods when randomization is unethical or impractical have grown in the health sciences in recent years. The adoption of so-called "causal" methods to account for both/either measured and/or unmeasured confounders is an important addition to the methodological toolkit for understanding the distribution, progression, and consequences of infectious diseases (IDs) and interventions on IDs. In the face of the Covid-19 pandemic and in the absence of systematic randomization of exposures or interventions, the value of these methods is even more apparent. Yet to our knowledge, no studies have assessed how causal methods involving pooling individual-level, observational, longitudinal data are being applied in ID-related research. In this systematic review, we assess how these methods are used and reported in ID-related research over the last 10 years. Findings will facilitate evaluation of trends of causal methods for ID research and lead to concrete recommendations for how to apply these methods where gaps in methodological rigor are identified., Methods and Analysis: We will apply MeSH and text terms to identify relevant studies from EBSCO (Academic Search Complete, Business Source Premier, CINAHL, EconLit with Full Text, PsychINFO), EMBASE, PubMed, and Web of Science. Eligible studies are those that apply causal methods to account for confounding when assessing the effects of an intervention or exposure on an ID-related outcome using pooled, individual-level data from 2 or more longitudinal, observational studies. Titles, abstracts, and full-text articles, will be independently screened by two reviewers using Covidence software. Discrepancies will be resolved by a third reviewer. This systematic review protocol has been registered with PROSPERO (CRD42020204104)., Competing Interests: No authors have competing interests.

Introduction: Uncomplicated urinary tract infection (UTI) in women is a common reason to present in general practice and is usually treated with antibiotics to reduce symptom severity and duration. Results of recent clinical trials indicate that non-antibiotic treatment approaches can also be effective. However, it remains unclear which patients would benefit from antibiotic treatment and which can effectively and safely be treated without antibiotics. This systematic review and meta-analysis aims to estimate the effect of treatment strategies to reduce antibiotic use in comparison with immediate antibiotic treatment and to identify prognostic factors and moderators of treatment effects. A further aim is to identify subgroups of patients benefiting from a specific therapy., Methods and Analysis: A systematic literature search will be performed to identify randomised controlled trials which investigated the effect of treatment strategies to reduce antibiotic use in female adults with uncomplicated UTI compared with immediate antibiotic treatment. Therefore, the primary outcome of the meta-analysis is incomplete recovery. Anonymised individual patient data (IPD) will be collected. Aggregate data will be used for pairwise comparisons of treatment strategies using meta-analysis models with random effects accounting for potential between-study heterogeneity. Potential effect moderators will be explored in meta-regressions. For IPD, generalised linear mixed models will be used, which may be adjusted for baseline characteristics. Interactions of baseline variables with treatment effects will be explored. These models will be used to assess direct comparisons of treatment, but might be extended to networks., Ethics and Dissemination: The local institutional review and ethics board judged the project a secondary analysis of existing anonymous data which meet the criteria for waiver of ethics review. Dissemination of the results will be via published scientific papers and presentations. Key messages will be promoted for example, via social media or press releases., Prospero Registration Number: CRD42019125804., Competing Interests: Competing interests: IG, GF, TF, EH, MM are involved in the following studies: ‘REGATTA—reducing antibiotic use for uncomplicated urinary tract infection in general practice by treatment with uva-ursi’. Afshar K, Fleischmann N, Schmiemann G, Bleidorn J, Hummers-Pradier E, Friede T, Wegscheider K, Moore M, Gagyor I. 'Reducing antibiotic use for uncomplicated urinary tract infection in general practice by treatment with uva-ursi (REGATTA)—a double-blind, randomised, controlled comparative effectiveness trial'. BMC Complement Altern Med. 2018 Jul 3;18(1):203. doi: 10.1186/s12906-018-2266-x). IV was involved in the study: ‘Vik I, Bollestad M, Grude N, Bærheim A, Damsgaard E, Neumark T, Bjerrum L, Cordoba G, Olsen IC, Lindbæk M. Ibuprofen versus pivmecillinam for uncomplicated urinary tract infection in women—a double-blind, randomised non-inferiority trial'. PLoS Med 15;5:e1002569. Doi.org/10.1371/journal.pmed.1002569. MM, ADH and PL are coauthors of the study: ‘Moore M, Trill J, Simpson C, Webley F, Radford M, Stanton L, Maishman T, Glanopoulou A, Flower A, Eyles C, Willcox M, Hay AD, van der Werf E, Gibbons S, Lewith G, Little P, Griffiths G. Uva-ursi extract and ibuprofen as alternative treatments for uncomplicated urinary tract infection in women (ATAFUTI): a factorial randomised trial. Clinical Microbiology and Infection'. Doi.org/10.1016/j.cmi.2019.01.011. SH was involved in the study with the reference number 11. PL and MM were involved in the study with the reference number 12. SF and TM were involved in the study with the reference number 13. IG, EH were involved in the studies with the reference numbers 14 and 15. AK was involved in the study with the reference number 16., (© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Introduction: Timing of cord clamping and other cord management strategies may improve outcomes at preterm birth. However, it is unclear whether benefits apply to all preterm subgroups. Previous and current trials compare various policies, including time-based or physiology-based deferred cord clamping, and cord milking. Individual participant data (IPD) enable exploration of different strategies within subgroups. Network meta-analysis (NMA) enables comparison and ranking of all available interventions using a combination of direct and indirect comparisons., Objectives: (1) To evaluate the effectiveness of cord management strategies for preterm infants on neonatal mortality and morbidity overall and for different participant characteristics using IPD meta-analysis. (2) To evaluate and rank the effect of different cord management strategies for preterm births on mortality and other key outcomes using NMA., Methods and Analysis: Systematic searches of Medline, Embase, clinical trial registries, and other sources for all ongoing and completed randomised controlled trials comparing cord management strategies at preterm birth (before 37 weeks' gestation) have been completed up to 13 February 2019, but will be updated regularly to include additional trials. IPD will be sought for all trials; aggregate summary data will be included where IPD are unavailable. First, deferred clamping and cord milking will be compared with immediate clamping in pairwise IPD meta-analyses. The primary outcome will be death prior to hospital discharge. Effect differences will be explored for prespecified participant subgroups. Second, all identified cord management strategies will be compared and ranked in an IPD NMA for the primary outcome and the key secondary outcomes. Treatment effect differences by participant characteristics will be identified. Inconsistency and heterogeneity will be explored., Ethics and Dissemination: Ethics approval for this project has been granted by the University of Sydney Human Research Ethics Committee (2018/886). Results will be relevant to clinicians, guideline developers and policy-makers, and will be disseminated via publications, presentations and media releases., Registration Number: Australian New Zealand Clinical Trials Registry (ANZCTR) (ACTRN12619001305112) and International Prospective Register of Systematic Reviews (PROSPERO, CRD42019136640)., Competing Interests: Competing interests: LD, ACK, CDPM, ED, HR, JK, JM, JJ, KF, OA, SH, VS, VD, WE-N and WT-M are Chief Investigators for potentially eligible trials., (© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY. Published by BMJ.)

Objectives: With the increasing interest in personalised medicine, the use of subgroup analyses is likely to increase. Subgroup analyses are challenging and often misused, possibly leading to false interpretations of the effect. It remains unclear to what extent key organisations warn for such pitfalls and translate current methodological research to detect these effects into research guidelines. The aim of this scoping review is to determine and evaluate the current guidance used by organisations for exploring, confirming and interpreting subgroup effects., Design: Scoping review., Eligibility Criteria: We identified four types of key stakeholder organisations: industry, health technology assessment organisations (HTA), academic/non-profit research organisations and regulatory bodies. After literature search and expert consultation, we identified international and national organisations of each type. For each organisation that was identified, we searched for official research guidance documents and contacted the organisation for additional guidance., Results: Twenty-seven (45%) of the 60 organisations that we included had relevant research guidance documents. We observed large differences between organisation types: 18% (n=2) of the industry organisations, 64% (n=9) of the HTA organisations, 38% (n=8) of academic/non-profit research organisations and 57% (n=8) of regulatory bodies provided guidance documents. The majority of the documents (n=33, 63%) mentioned one or more challenges in subgroup analyses, such as false positive findings or ecological bias with variations across the organisation types. Statistical recommendations were less common (n=19, 37%) and often limited to a formal test of interaction., Conclusions: Almost half of the organisations included in this scoping review provided guidance on subgroup effect research in their guidelines. However, there were large differences between organisations in the amount and level of detail of their guidance. Effort is required to translate and integrate research findings on subgroup analysis to practical guidelines for decision making and to reduce the differences between organisations and organisation types., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Essentials The widely recommended CHA2DS2-VASc shows conflicting results in contemporary validation studies. We performed a systematic review and meta-analysis of 19 studies validating CHA2DS2-VASc. There was high heterogeneity in stroke risks for different CHA2DS2-VASc scores. This was not explained by differences between setting of care, or by performing meta-regression., Summary: Background The CHA2DS2-VASc decision rule is widely recommended for estimating stroke risk in patients with atrial fibrillation (AF), although validation studies show ambiguous and conflicting results. Objectives To: (i) review existing studies validating CHA2DS2-VASc in AF patients who are not (yet) anticoagulated; (ii) meta-analyze estimates of stroke risk per score; and (iii) explore sources of heterogeneity across the validation studies. Methods We performed a systematic literature review and random effects meta-analysis of studies externally validating CHA2DS2-VASc in AF patients not receiving anticoagulants. To explore between-study heterogeneity in stroke risk, we stratified studies to the clinical setting in which patient enrollment started, and performed meta-regression. Results In total, 19 studies were evaluated, with over two million person-years of follow-up. In studies recruiting AF patients in hospitals, stroke risks for scores of 0, 1 and 2 were 0.4% (approximate 95% prediction interval [PI] 0.2-3.2%), 1.2% (95% PI 0.1-3.8%), and 2.2% (95% PI 0.03-7.8%), respectively. These were consistently higher than those in studies recruiting patients from the open general population, with risks of 0.2% (95% PI 0.0-0.9%), 0.7% (95% PI 0.3-1.2%) and 1.5% (95% PI 0.4-3.3%) for scores of 0, 1, and 2, respectively. Heterogeneity, as reflected by the wide PIs, could not be fully explained by meta-regression. Conclusions Studies validating CHA2DS2-VASc show high heterogeneity in predicted stroke risks for different scores., (© 2017 International Society on Thrombosis and Haemostasis.)

Objective: Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods., Study Design and Setting: We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration., Results: There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations., Conclusion: The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods., (Copyright © 2012 Elsevier Inc. All rights reserved.)