Search

Abstract Background Although aggregate data (AD) from randomised clinical trials (RCTs) are used in the majority of network meta-analyses (NMAs), other study designs (e.g., cohort studies and other non-randomised studies, NRS) can be informative about relative treatment effects. The individual participant data (IPD) of the study, when available, are preferred to AD for adjusting for important participant characteristics and to better handle heterogeneity and inconsistency in the network. Results We developed the R package crossnma to perform cross-format (IPD and AD) and cross-design (RCT and NRS) NMA and network meta-regression (NMR). The models are implemented as Bayesian three-level hierarchical models using Just Another Gibbs Sampler (JAGS) software within the R environment. The R package crossnma includes functions to automatically create the JAGS model, reformat the data (based on user input), assess convergence and summarize the results. We demonstrate the workflow within crossnma by using a network of six trials comparing four treatments. Conclusions The R package crossnma enables the user to perform NMA and NMR with different data types in a Bayesian framework and facilitates the inclusion of all types of evidence recognising differences in risk of bias.

Background: Although aggregate data (AD) from randomised clinical trials (RCTs) are used in the majority of network meta-analyses (NMAs), other study designs (e.g., cohort studies and other non-randomised studies, NRS) can be informative about relative treatment effects. The individual participant data (IPD) of the study, when available, are preferred to AD for adjusting for important participant characteristics and to better handle heterogeneity and inconsistency in the network. Results: We developed the R package crossnma to perform cross-format (IPD and AD) and cross-design (RCT and NRS) NMA and network meta-regression (NMR). The models are implemented as Bayesian three-level hierarchical models using Just Another Gibbs Sampler (JAGS) software within the R environment. The R package crossnma includes functions to automatically create the JAGS model, reformat the data (based on user input), assess convergence and summarize the results. We demonstrate the workflow within crossnma by using a network of six trials comparing four treatments. Conclusions: The R package crossnma enables the user to perform NMA and NMR with different data types in a Bayesian framework and facilitates the inclusion of all types of evidence recognising differences in risk of bias. [ABSTRACT FROM AUTHOR]

To estimate causal effects, analysts performing observational studies in health settings utilize several strategies to mitigate bias due to confounding by indication. There are two broad classes of approaches for these purposes: use of confounders and instrumental variables (IVs). Because such approaches are largely characterized by untestable assumptions, analysts must operate under an indefinite paradigm that these methods will work imperfectly. In this tutorial, we formalize a set of general principles and heuristics for estimating causal effects in the two approaches when the assumptions are potentially violated. This crucially requires reframing the process of observational studies as hypothesizing potential scenarios where the estimates from one approach are less inconsistent than the other. While most of our discussion of methodology centers around the linear setting, we touch upon complexities in non-linear settings and flexible procedures such as target minimum loss-based estimation and double machine learning. To demonstrate the application of our principles, we investigate the use of donepezil off-label for mild cognitive impairment. We compare and contrast results from confounder and IV methods, traditional and flexible, within our analysis and to a similar observational study and clinical trial.

Abstract To estimate causal effects, analysts performing observational studies in health settings utilize several strategies to mitigate bias due to confounding by indication. There are two broad classes of approaches for these purposes: use of confounders and instrumental variables (IVs). Because such approaches are largely characterized by untestable assumptions, analysts must operate under an indefinite paradigm that these methods will work imperfectly. In this tutorial, we formalize a set of general principles and heuristics for estimating causal effects in the two approaches when the assumptions are potentially violated. This crucially requires reframing the process of observational studies as hypothesizing potential scenarios where the estimates from one approach are less inconsistent than the other. While most of our discussion of methodology centers around the linear setting, we touch upon complexities in non-linear settings and flexible procedures such as target minimum loss-based estimation and double machine learning. To demonstrate the application of our principles, we investigate the use of donepezil off-label for mild cognitive impairment. We compare and contrast results from confounder and IV methods, traditional and flexible, within our analysis and to a similar observational study and clinical trial.

Abstract Background This meta-epidemiological study aimed to assess methodological quality of a sample of contemporary non-randomised clinical studies of clinical interventions. Methods This was a cross-sectional study of observational studies published between January 1, 2012 and December 31, 2018. Studies were identified in PubMed using search terms ‘association’, ‘observational,’ ‘non-randomised’ ‘comparative effectiveness’ within titles or abstracts. Each study was appraised against 35 quality criteria by two authors independently, with each criterion rated fully, partially or not satisfied. These quality criteria were grouped into 6 categories: justification for observational design (n = 2); minimisation of bias in study design and data collection (n = 11); use of appropriate methods to create comparable groups (n = 6); appropriate adjustment of observed effects (n = 5); validation of observed effects (n = 9); and authors interpretations (n = 2). Results Of 50 unique studies, 49 (98%) were published in two US general medical journals. No study fully satisfied all applicable criteria; the mean (+/−SD) proportion of applicable criteria fully satisfied across all studies was 72% (+/− 10%). The categories of quality criteria demonstrating the lowest proportions of fully satisfied criteria were measures used to adjust observed effects (criteria 20, 23, 24) and validate observed effects (criteria 25, 27, 33). Criteria associated with ≤50% of full satisfaction across studies, where applicable, comprised: imputation methods to account for missing data (50%); justification for not performing an RCT (42%); interaction analyses in identifying independent prognostic factors potentially influencing intervention effects (42%); use of statistical correction to minimise type 1 error in multiple outcome analyses (33%); clinically significant effect sizes (30%); residual bias analyses for unmeasured or unknown confounders (14%); and falsification tests for residual confounding (8%). The proportions of fully satisfied criteria did not change over time. Conclusions Recently published observational studies fail to fully satisfy more than one in four quality criteria. Criteria that were not or only partially satisfied were identified which serve as remediable targets for researchers and journal editors.

Background: This meta-epidemiological study aimed to assess methodological quality of a sample of contemporary non-randomised clinical studies of clinical interventions. Methods: This was a cross-sectional study of observational studies published between January 1, 2012 and December 31, 2018. Studies were identified in PubMed using search terms 'association', 'observational,' 'non-randomised' 'comparative effectiveness' within titles or abstracts. Each study was appraised against 35 quality criteria by two authors independently, with each criterion rated fully, partially or not satisfied. These quality criteria were grouped into 6 categories: justification for observational design (n = 2); minimisation of bias in study design and data collection (n = 11); use of appropriate methods to create comparable groups (n = 6); appropriate adjustment of observed effects (n = 5); validation of observed effects (n = 9); and authors interpretations (n = 2). Results: Of 50 unique studies, 49 (98%) were published in two US general medical journals. No study fully satisfied all applicable criteria; the mean (+/−SD) proportion of applicable criteria fully satisfied across all studies was 72% (+/− 10%). The categories of quality criteria demonstrating the lowest proportions of fully satisfied criteria were measures used to adjust observed effects (criteria 20, 23, 24) and validate observed effects (criteria 25, 27, 33). Criteria associated with ≤50% of full satisfaction across studies, where applicable, comprised: imputation methods to account for missing data (50%); justification for not performing an RCT (42%); interaction analyses in identifying independent prognostic factors potentially influencing intervention effects (42%); use of statistical correction to minimise type 1 error in multiple outcome analyses (33%); clinically significant effect sizes (30%); residual bias analyses for unmeasured or unknown confounders (14%); and falsification tests for residual confounding (8%). The proportions of fully satisfied criteria did not change over time. Conclusions: Recently published observational studies fail to fully satisfy more than one in four quality criteria. Criteria that were not or only partially satisfied were identified which serve as remediable targets for researchers and journal editors. [ABSTRACT FROM AUTHOR]

Abstract Background Immortal time bias is common in observational studies but is typically described for pharmacoepidemiology studies where there is a delay between cohort entry and treatment initiation. Methods This study used the Clinical Practice Research Datalink (CPRD) and linked national mortality data in England from 2000 to 2019 to investigate immortal time bias for a specific life-long condition, intellectual disability. Life expectancy (Chiang’s abridged life table approach) was compared for 33,867 exposed and 980,586 unexposed individuals aged 10+ years using five methods: (1) treating immortal time as observation time; (2) excluding time before date of first exposure diagnosis; (3) matching cohort entry to first exposure diagnosis; (4) excluding time before proxy date of inputting first exposure diagnosis (by the physician); and (5) treating exposure as a time-dependent measure. Results When not considered in the design or analysis (Method 1), immortal time bias led to disproportionately high life expectancy for the exposed population during the first calendar period (additional years expected to live: 2000–2004: 65.6 [95% CI: 63.6,67.6]) compared to the later calendar periods (2005–2009: 59.9 [58.8,60.9]; 2010–2014: 58.0 [57.1,58.9]; 2015–2019: 58.2 [56.8,59.7]). Date of entry of diagnosis (Method 4) was unreliable in this CPRD cohort. The final methods (Method 2, 3 and 5) appeared to solve the main theoretical problem but residual bias may have remained. Conclusions We conclude that immortal time bias is a significant issue for studies of life-long conditions that use electronic health record data and requires careful consideration of how clinical diagnoses are entered onto electronic health record systems.

Abstract Background During the COVID-19 pandemic, the scientific world is in urgent need for new evidence on the treatment of COVID patients. The reporting quality is crucial for transparent scientific publication. Concerns of data integrity, methodology and transparency were raised. Here, we assessed the adherence of observational studies comparing treatments of COVID 19 to the STROBE checklist in 2020. Methods Design: We performed a retrospective, cross-sectional study. Setting: We conducted a systematic literature search in the Medline database. This study was performed at the RWTH Aachen University Hospital, Department of Anaesthesiology Participants: We extracted all observational studies on the treatment of COVID-19 patients from the year 2020. Main outcome measures: The adherence of each publication to the STROBE checklist items was analysed. The journals’ impact factor (IF), the country of origin, the kind of investigated treatment and the month of publication were assessed. Results We analysed 147 observational studies and found a mean adherence of 45.6% to the STROBE checklist items. The percentage adherence per publication correlated significantly with the journals’ IF (point estimate for the difference between 1st and 4th quartile 11.07%, 95% CI 5.12 to 17.02, p

Abstract Background Individual-patient data meta-analysis (IPD-MA) is an increasingly popular approach because of its analytical benefits. IPD-MA of observational studies must overcome the problem of confounding, otherwise biased estimates of treatment effect may be obtained. One approach to reducing confounding bias could be the use of propensity score matching (PSM). IPD-MA can be considered as two-stage clustered data (patients within studies) and propensity score matching can be implemented within studies, across studies, and combining both. Methods This article focuses on implementation of four PSM-based approaches for the analysis of data structure that exploit IPD-MA in two ways: (i) estimation of propensity score model using single-level or random-effects logistic regression; and (ii) matching of propensity scores (PS) across studies, within studies or preferential-within studies. We investigated the performance of these approaches through a simulation study, which considers an IPD-MA that examined the success of different treatments for multidrug-resistant tuberculosis (MDR-TB). The simulation parameters were varied according to three treatment prevalences (according to studies, 50% and 30%), three levels of heterogeneity between studies (low, moderate and high) and three levels of pooled odds ratio (1, 1.5, 3). Results All approaches showed greater biases at the higher levels of heterogeneity regardless of the choices of treatment prevalences. However, matching of propensity scores using within-study and preferential-within study reported better performance compared to matching across studies when treatment prevalence varied across-studies. For fixed prevalences, a random-effect propensity score model to estimate propensity scores followed by matching of propensity scores across-studies achieved lower biases compared to other PSM-based approaches. Conclusions Propensity score matching has wide application in health research while only limited literature is available on the implementation of PSM methods in IPD-MA, and until now methodological performance of PSM methods have not been examined. We believe, this work offers an intuition to the applied researcher for the choice of the PSM-based approaches.

Abstract Background Observational studies are increasingly being used to provide supplementary evidence in addition to Randomized Control Trials (RCTs) because they provide a scale and diversity of participants and outcomes that would be infeasible in an RCT. Additionally, they more closely reflect the settings in which the studied interventions will be applied in the future. Well-established propensity-score-based methods exist to overcome the challenges of working with observational data to estimate causal effects. These methods also provide quality assurance diagnostics to evaluate the degree to which bias has been removed and the estimates can be trusted. In large medical datasets it is common to find the same underlying health condition being treated with a variety of distinct drugs or drug combinations. Conventional methods require a manual iterative workflow, making them scale poorly to studies with many intervention arms. In such situations, automated causal inference methods that are compatible with traditional propensity-score-based workflows are highly desirable. Methods We introduce an automated causal inference method BCAUS, that features a deep-neural-network-based propensity model that is trained with a loss which penalizes both the incorrect prediction of the assigned treatment as well as the degree of imbalance between the inverse probability weighted covariates. The network is trained end-to-end by dynamically adjusting the loss term for each training batch such that the relative contributions from the two loss components are held fixed. Trained BCAUS models can be used in conjunction with traditional propensity-score-based methods to estimate causal treatment effects. Results We tested BCAUS on the semi-synthetic Infant Health & Development Program dataset with a single intervention arm, and a real-world observational study of diabetes interventions with over 100,000 individuals spread across more than a hundred intervention arms. When compared against other recently proposed automated causal inference methods, BCAUS had competitive accuracy for estimating synthetic treatment effects and provided highly concordant estimates on the real-world dataset but was an order-of-magnitude faster. Conclusions BCAUS is directly compatible with trusted protocols to estimate treatment effects and diagnose the quality of those estimates, while making the established approaches automatically scalable to an arbitrary number of simultaneous intervention arms without any need for manual iteration.

Background: Immortal time bias is common in observational studies but is typically described for pharmacoepidemiology studies where there is a delay between cohort entry and treatment initiation.Methods: This study used the Clinical Practice Research Datalink (CPRD) and linked national mortality data in England from 2000 to 2019 to investigate immortal time bias for a specific life-long condition, intellectual disability. Life expectancy (Chiang's abridged life table approach) was compared for 33,867 exposed and 980,586 unexposed individuals aged 10+ years using five methods: (1) treating immortal time as observation time; (2) excluding time before date of first exposure diagnosis; (3) matching cohort entry to first exposure diagnosis; (4) excluding time before proxy date of inputting first exposure diagnosis (by the physician); and (5) treating exposure as a time-dependent measure.Results: When not considered in the design or analysis (Method 1), immortal time bias led to disproportionately high life expectancy for the exposed population during the first calendar period (additional years expected to live: 2000-2004: 65.6 [95% CI: 63.6,67.6]) compared to the later calendar periods (2005-2009: 59.9 [58.8,60.9]; 2010-2014: 58.0 [57.1,58.9]; 2015-2019: 58.2 [56.8,59.7]). Date of entry of diagnosis (Method 4) was unreliable in this CPRD cohort. The final methods (Method 2, 3 and 5) appeared to solve the main theoretical problem but residual bias may have remained.Conclusions: We conclude that immortal time bias is a significant issue for studies of life-long conditions that use electronic health record data and requires careful consideration of how clinical diagnoses are entered onto electronic health record systems. [ABSTRACT FROM AUTHOR]

Abstract Background Randomized controlled trials are considered the gold standard to evaluate causal associations, whereas assessing causality in observational studies is challenging. Methods We applied Hill’s Criteria, counterfactual reasoning, and causal diagrams to evaluate a potentially causal relationship between an exposure and outcome in three published observational studies: a) one burden of disease cohort study to determine the association between type 2 diabetes and herpes zoster, b) one post-authorization safety cohort study to assess the effect of AS04-HPV-16/18 vaccine on the risk of autoimmune diseases, and c) one matched case-control study to evaluate the effectiveness of a rotavirus vaccine in preventing hospitalization for rotavirus gastroenteritis. Results Among the 9 Hill’s criteria, 8 (Strength, Consistency, Specificity, Temporality, Plausibility, Coherence, Analogy, Experiment) were considered as met for study c, 3 (Temporality, Plausibility, Coherence) for study a, and 2 (Temporary, Plausibility) for study b. For counterfactual reasoning criteria, exchangeability, the most critical assumption, could not be tested. Using these tools, we concluded that causality was very unlikely in study b, unlikely in study a, and very likely in study c. Directed acyclic graphs provided complementary visual structures that identified confounding bias and helped determine the most accurate design and analysis to assess causality. Conclusions Based on our assessment we found causal Hill’s criteria and counterfactual thinking valuable in determining some level of certainty about causality in observational studies. Application of causal inference frameworks should be considered in designing and interpreting observational studies.

Background: During the COVID-19 pandemic, the scientific world is in urgent need for new evidence on the treatment of COVID patients. The reporting quality is crucial for transparent scientific publication. Concerns of data integrity, methodology and transparency were raised. Here, we assessed the adherence of observational studies comparing treatments of COVID 19 to the STROBE checklist in 2020.Methods: Design: We performed a retrospective, cross-sectional study.Setting: We conducted a systematic literature search in the Medline database. This study was performed at the RWTH Aachen University Hospital, Department of Anaesthesiology Participants: We extracted all observational studies on the treatment of COVID-19 patients from the year 2020.Main Outcome Measures: The adherence of each publication to the STROBE checklist items was analysed. The journals' impact factor (IF), the country of origin, the kind of investigated treatment and the month of publication were assessed.Results: We analysed 147 observational studies and found a mean adherence of 45.6% to the STROBE checklist items. The percentage adherence per publication correlated significantly with the journals' IF (point estimate for the difference between 1st and 4th quartile 11.07%, 95% CI 5.12 to 17.02, p < 0.001). U.S. American authors gained significantly higher adherence to the checklist than Chinese authors, mean difference 9.10% (SD 2.85%, p = 0.023).Conclusions: We conclude a poor reporting quality of observational studies on the treatment of COVID-19 throughout the year 2020. A considerable improvement is mandatory. [ABSTRACT FROM AUTHOR]

Abstract Backgrounds Master of public health (MPH) plays an important role in Chinese medical education, and the dissertations is an important part of MPH education. In MPH dissertations, most are observational studies. Compared with randomized controlled trial (RCT), observational studies are more prone to information bias. So, the reporting of the observational studies should be transparent and standard. But, no research on evaluating the reporting quality of the MPH dissertation has been found. Methods A systematic literature search was performed in the Wanfang database from January 1, 2014 to May 31, 2019. The Strengthening the Reporting of Observation Studies in Epidemiology (STROBE) statement was adopted to evaluate the reporting quality of the selected studies. Articles that met the following criteria were selected: (1) observational studies, including cross-sectional studies, case-control studies, and cohort studies; (2) original articles; (3) studies on humans, including both adults and children. Results The Median of compliance to individual STROBE items was 74.79%. The mean (standard deviation) of STROBE score was 14.29 (1.84). Five items/sub-items were 100% reported (“reported” and “partly reported” were combined): background, objectives, study design, report numbers of individuals at each stage, and key result. Fifteen items/sub-items were reported by 75% or more. Reporting of methods and results was often omitted: missing data (6.67%), sensitivity analyses (3.63%), flow diagram (15.15%), and absolute risk (0%). Logistic regression analysis indicated that cohort studies (OR = 3.41, 95% CI = 1.27–9.16), funding support (OR = 4.37, 95% CI = 1.27–9.16) and more published papers during postgraduate period (OR = 3.46, 95% CI = 1.40–8.60) were related to high reporting quality. Conclusion In short, the reporting quality of observational studies in MPH’s dissertations in China is suboptimal. However, it’s necessary to improve the reporting of method and results sections. We recommend that authors should be stricter to adhere STROBE statement when conducting observational studies.

Abstract Background Missing data in covariates can result in biased estimates and loss of power to detect associations. It can also lead to other challenges in time-to-event analyses including the handling of time-varying effects of covariates, selection of covariates and their flexible modelling. This review aims to describe how researchers approach time-to-event analyses with missing data. Methods Medline and Embase were searched for observational time-to-event studies in oncology published from January 2012 to January 2018. The review focused on proportional hazards models or extended Cox models. We investigated the extent and reporting of missing data and how it was addressed in the analysis. Covariate modelling and selection, and assessment of the proportional hazards assumption were also investigated, alongside the treatment of missing data in these procedures. Results 148 studies were included. The mean proportion of individuals with missingness in any covariate was 32%. 53% of studies used complete-case analysis, and 22% used multiple imputation. In total, 14% of studies stated an assumption concerning missing data and only 34% stated missingness as a limitation. The proportional hazards assumption was checked in 28% of studies, of which, 17% did not state the assessment method. 58% of 144 multivariable models stated their covariate selection procedure with use of a pre-selected set of covariates being the most popular followed by stepwise methods and univariable analyses. Of 69 studies that included continuous covariates, 81% did not assess the appropriateness of the functional form. Conclusion While guidelines for handling missing data in epidemiological studies are in place, this review indicates that few report implementing recommendations in practice. Although missing data are present in many studies, we found that few state clearly how they handled it or the assumptions they have made. Easy-to-implement but potentially biased approaches such as complete-case analysis are most commonly used despite these relying on strong assumptions and where often more appropriate methods should be employed. Authors should be encouraged to follow existing guidelines to address missing data, and increased levels of expectation from journals and editors could be used to improve practice.

Abstract Background Among different investigators studying the same exposures and outcomes, there may be a lack of consensus about potential confounders that should be considered as matching, adjustment, or stratification variables in observational studies. Concerns have been raised that confounding factors may affect the results obtained for the alcohol-ischemic heart disease relationship, as well as their consistency and reproducibility across different studies. Therefore, we assessed how confounders are defined, operationalized, and discussed across individual studies evaluating the impact of alcohol on ischemic heart disease risk. Methods For observational studies included in a recent alcohol-ischemic heart disease meta-analysis, we identified all variables adjusted, matched, or stratified for in the largest reported multivariate model (i.e. potential confounders). We recorded how the variables were measured and grouped them into higher-level confounder domains. Abstracts and Discussion sections were then assessed to determine whether authors considered confounding when interpreting their study findings. Results 85 of 87 (97.7%) studies reported multivariate analyses for an alcohol-ischemic heart disease relationship. The most common higher-level confounder domains included were smoking (79, 92.9%), age (74, 87.1%), and BMI, height, and/or weight (57, 67.1%). However, no two models adjusted, matched, or stratified for the same higher-level confounder domains. Most (74/87, 85.1%) articles mentioned or alluded to “confounding” in their Abstract or Discussion sections, but only one stated that their main findings were likely to be affected by residual confounding. There were five (5/87, 5.7%) authors that explicitly asked for caution when interpreting results. Conclusion There is large variation in the confounders considered across observational studies evaluating the impact of alcohol on ischemic heart disease risk and almost all studies spuriously ignore or eventually dismiss confounding in their conclusions. Given that study results and interpretations may be affected by the mix of potential confounders included within multivariate models, efforts are necessary to standardize approaches for selecting and accounting for confounders in observational studies.

Abstract Background In the data pipeline from the data collection process to the planned statistical analyses, initial data analysis (IDA) typically takes place between the end of the data collection and do not touch the research questions. A systematic process for IDA and clear reporting of the findings would help to understand the potential shortcomings of a dataset, such as missing values, or subgroups with small sample sizes, or shortcomings in the collection process, and to evaluate the impact of these shortcomings on the research results. A clear reporting of findings is also relevant when making datasets available to other researchers. Initial data analyses can provide valuable insights into the suitability of a data set for a future research study. Our aim was to describe the practice of reporting of initial data analyses in observational studies in five highly ranked medical journals with focus on data cleaning, screening, and reporting of findings which led to a potential change in the analysis plan. Methods This review was carried out using systematic search strategies with eligibility criteria for articles to be reviewed. A total of 25 papers about observational studies were selected from five medical journals published in 2018. Each paper was reviewed by two reviewers and IDA statements were further discussed by all authors. The consensus was reported. Results IDA statements were reported in the methods, results, discussion, and supplement of papers. Ten out of 25 papers (40%) included a statement about data cleaning. Data screening statements were included in all articles, and 18 (72%) indicated the methods used to describe them. Item missingness was reported in 11 papers (44%), unit missingness in 15 papers (60%). Eleven papers (44%) mentioned some changes in the analysis plan. Reported changes referred to missing data treatment, unexpected values, population heterogeneity and aspects related to variable distributions or data properties. Conclusion Reporting of initial data analyses were sparse, and statements on IDA were located throughout the research articles. There is a lack of systematic reporting of IDA. We conclude the article with recommendations on how to overcome shortcomings in the practice of IDA reporting in observational studies.

Background: Individual-patient data meta-analysis (IPD-MA) is an increasingly popular approach because of its analytical benefits. IPD-MA of observational studies must overcome the problem of confounding, otherwise biased estimates of treatment effect may be obtained. One approach to reducing confounding bias could be the use of propensity score matching (PSM). IPD-MA can be considered as two-stage clustered data (patients within studies) and propensity score matching can be implemented within studies, across studies, and combining both.Methods: This article focuses on implementation of four PSM-based approaches for the analysis of data structure that exploit IPD-MA in two ways: (i) estimation of propensity score model using single-level or random-effects logistic regression; and (ii) matching of propensity scores (PS) across studies, within studies or preferential-within studies. We investigated the performance of these approaches through a simulation study, which considers an IPD-MA that examined the success of different treatments for multidrug-resistant tuberculosis (MDR-TB). The simulation parameters were varied according to three treatment prevalences (according to studies, 50% and 30%), three levels of heterogeneity between studies (low, moderate and high) and three levels of pooled odds ratio (1, 1.5, 3).Results: All approaches showed greater biases at the higher levels of heterogeneity regardless of the choices of treatment prevalences. However, matching of propensity scores using within-study and preferential-within study reported better performance compared to matching across studies when treatment prevalence varied across-studies. For fixed prevalences, a random-effect propensity score model to estimate propensity scores followed by matching of propensity scores across-studies achieved lower biases compared to other PSM-based approaches.Conclusions: Propensity score matching has wide application in health research while only limited literature is available on the implementation of PSM methods in IPD-MA, and until now methodological performance of PSM methods have not been examined. We believe, this work offers an intuition to the applied researcher for the choice of the PSM-based approaches. [ABSTRACT FROM AUTHOR]

Background: Observational studies are increasingly being used to provide supplementary evidence in addition to Randomized Control Trials (RCTs) because they provide a scale and diversity of participants and outcomes that would be infeasible in an RCT. Additionally, they more closely reflect the settings in which the studied interventions will be applied in the future. Well-established propensity-score-based methods exist to overcome the challenges of working with observational data to estimate causal effects. These methods also provide quality assurance diagnostics to evaluate the degree to which bias has been removed and the estimates can be trusted. In large medical datasets it is common to find the same underlying health condition being treated with a variety of distinct drugs or drug combinations. Conventional methods require a manual iterative workflow, making them scale poorly to studies with many intervention arms. In such situations, automated causal inference methods that are compatible with traditional propensity-score-based workflows are highly desirable.Methods: We introduce an automated causal inference method BCAUS, that features a deep-neural-network-based propensity model that is trained with a loss which penalizes both the incorrect prediction of the assigned treatment as well as the degree of imbalance between the inverse probability weighted covariates. The network is trained end-to-end by dynamically adjusting the loss term for each training batch such that the relative contributions from the two loss components are held fixed. Trained BCAUS models can be used in conjunction with traditional propensity-score-based methods to estimate causal treatment effects.Results: We tested BCAUS on the semi-synthetic Infant Health & Development Program dataset with a single intervention arm, and a real-world observational study of diabetes interventions with over 100,000 individuals spread across more than a hundred intervention arms. When compared against other recently proposed automated causal inference methods, BCAUS had competitive accuracy for estimating synthetic treatment effects and provided highly concordant estimates on the real-world dataset but was an order-of-magnitude faster.Conclusions: BCAUS is directly compatible with trusted protocols to estimate treatment effects and diagnose the quality of those estimates, while making the established approaches automatically scalable to an arbitrary number of simultaneous intervention arms without any need for manual iteration. [ABSTRACT FROM AUTHOR]

Abstract Background The availability of clinical and therapeutic data drawn from medical records and administrative databases has entailed new opportunities for clinical and epidemiologic research. However, these databases present inherent limitations which may render them prone to new biases. We aimed to conduct a structured review of biases specific to observational clinical studies based on secondary databases, and to propose strategies for the mitigation of those biases. Methods Scoping review of the scientific literature published during the period 2000–2018 through an automated search of MEDLINE, EMBASE and Web of Science, supplemented with manually cross-checking of reference lists. We included opinion essays, methodological reviews, analyses or simulation studies, as well as letters to the editor or retractions, the principal objective of which was to highlight the existence of some type of bias in pharmacoepidemiologic studies using secondary databases. Results A total of 117 articles were included. An increasing trend in the number of publications concerning the potential limitations of secondary databases was observed over time and across medical research disciplines. Confounding was the most reported category of bias (63.2% of articles), followed by selection and measurement biases (47.0% and 46.2% respectively). Confounding by indication (32.5%), unmeasured/residual confounding (28.2%), outcome misclassification (28.2%) and “immortal time” bias (25.6%) were the subcategories most frequently mentioned. Conclusions Suboptimal use of secondary databases in pharmacoepidemiologic studies has introduced biases in the studies, which may have led to erroneous conclusions. Methods to mitigate biases are available and must be considered in the design, analysis and interpretation phases of studies using these data sources.

Background: Randomized controlled trials are considered the gold standard to evaluate causal associations, whereas assessing causality in observational studies is challenging.Methods: We applied Hill's Criteria, counterfactual reasoning, and causal diagrams to evaluate a potentially causal relationship between an exposure and outcome in three published observational studies: a) one burden of disease cohort study to determine the association between type 2 diabetes and herpes zoster, b) one post-authorization safety cohort study to assess the effect of AS04-HPV-16/18 vaccine on the risk of autoimmune diseases, and c) one matched case-control study to evaluate the effectiveness of a rotavirus vaccine in preventing hospitalization for rotavirus gastroenteritis.Results: Among the 9 Hill's criteria, 8 (Strength, Consistency, Specificity, Temporality, Plausibility, Coherence, Analogy, Experiment) were considered as met for study c, 3 (Temporality, Plausibility, Coherence) for study a, and 2 (Temporary, Plausibility) for study b. For counterfactual reasoning criteria, exchangeability, the most critical assumption, could not be tested. Using these tools, we concluded that causality was very unlikely in study b, unlikely in study a, and very likely in study c. Directed acyclic graphs provided complementary visual structures that identified confounding bias and helped determine the most accurate design and analysis to assess causality.Conclusions: Based on our assessment we found causal Hill's criteria and counterfactual thinking valuable in determining some level of certainty about causality in observational studies. Application of causal inference frameworks should be considered in designing and interpreting observational studies. [ABSTRACT FROM AUTHOR]

Abstract Background Research waste is estimated to be very common, but assessments of its prevalence and scope are rare. As an example, we assessed research waste in clinical research on calcium intake (assessing study design and endpoint type) and vitamin D supplementation (assessing endpoint type). Methods We examined 404 randomised controlled trials (RCTs) and observational studies of calcium intake (diet or supplements) and bone mineral density (BMD) or fracture, and 547 RCTs of vitamin D supplements, and assessed the proportion of studies that used surrogate or clinical endpoints. For studies with BMD or fracture as an endpoint, we estimated when the ‘tipping’ point occurred indicating the need for RCTs with fracture as an endpoint (based on cumulative meta-analyses of BMD RCTs, and chronological review of observational studies), and whether each study published at least 5y after the tipping point was novel, added new clinical knowledge or was research waste. Results Observational studies of calcium intake and BMD or fracture outnumbered RCTs by 3.3–4.5 times. For both calcium intake and vitamin D supplements, studies using surrogate endpoints outnumbered studies using clinical endpoints by 1.6–3 times. Of 41 RCT publications of calcium intake and BMD or fracture published at least 5y after the tipping point in 1994, we considered that 19 (46%) lacked novelty, another 13 (32%) added no new clinical knowledge, and 30 (73%) were research waste. Of 204 observational study publications of calcium intake and BMD or fracture, 197 (96%) lacked novelty, another 5 (2%) added no new clinical knowledge, and 202 (99%) were research waste. Of 39 RCTs of vitamin D supplementation and BMD or fracture published at least 5y after the tipping point in 1999, 14 (36%) lacked novelty, another 13 (33%) added no new clinical knowledge, and 27 (69%) were research waste. Conclusions A high proportion of studies of calcium intake since 2000 (95%) and trials of vitamin D supplements since 2005 (69%) on BMD or fracture represent research waste.

Abstract Background The data from individual observational studies included in meta-analyses of drug effects are collected either from ad hoc methods (i.e. “primary data”) or databases that were established for non-research purposes (i.e. “secondary data”). The use of secondary sources may be prone to measurement bias and confounding due to over-the-counter and out-of-pocket drug consumption, or non-adherence to treatment. In fact, it has been noted that failing to consider the origin of the data as a potential cause of heterogeneity may change the conclusions of a meta-analysis. We aimed to assess to what extent the origin of data is explored as a source of heterogeneity in meta-analyses of observational studies. Methods We searched for meta-analyses of drugs effects published between 2012 and 2018 in general and internal medicine journals with an impact factor > 15. We evaluated, when reported, the type of data source (primary vs secondary) used in the individual observational studies included in each meta-analysis, and the exposure- and outcome-related variables included in sensitivity, subgroup or meta-regression analyses. Results We found 217 articles, 23 of which fulfilled our eligibility criteria. Eight meta-analyses (8/23, 34.8%) reported the source of data. Three meta-analyses (3/23, 13.0%) included the method of outcome assessment as a variable in the analysis of heterogeneity, and only one compared and discussed the results considering the different sources of data (primary vs secondary). Conclusions In meta-analyses of drug effects published in seven high impact general medicine journals, the origin of the data, either primary or secondary, is underexplored as a source of heterogeneity.

Abstract Background In pharmacoepidemiology, the prescription preference-based instrumental variables (IV) are often used with linear models to solve the endogeneity due to unobserved confounders even when the outcome and the endogenous treatment are dichotomous variables. Using this instrumental variable, we proceed by Monte-Carlo simulations to compare the IV-based generalized method of moment (IV-GMM) and the two-stage residual inclusion (2SRI) method in this context. Methods We established the formula allowing us to compute the instrument’s strength and the confounding level in the context of logistic regression models. We then varied the instrument’s strength and the confounding level to cover a large range of scenarios in the simulation study. We also explore two prescription preference-based instruments. Results We found that the 2SRI is less biased than the other methods and yields satisfactory confidence intervals. The proportion of previous patients of the same physician who were prescribed the treatment of interest displayed a good performance as a proxy of the physician’s preference instrument. Conclusions This work shows that when analysing real data with dichotomous outcome and exposure, appropriate 2SRI estimation could be used in presence of unmeasured confounding.

Abstract Background Systematic reviews and meta-analyses of observational studies are frequently performed, but no widely accepted guidance is available at present. We performed a systematic scoping review of published methodological recommendations on how to systematically review and meta-analyse observational studies. Methods We searched online databases and websites and contacted experts in the field to locate potentially eligible articles. We included articles that provided any type of recommendation on how to conduct systematic reviews and meta-analyses of observational studies. We extracted and summarised recommendations on pre-defined key items: protocol development, research question, search strategy, study eligibility, data extraction, dealing with different study designs, risk of bias assessment, publication bias, heterogeneity, statistical analysis. We summarised recommendations by key item, identifying areas of agreement and disagreement as well as areas where recommendations were missing or scarce. Results The searches identified 2461 articles of which 93 were eligible. Many recommendations for reviews and meta-analyses of observational studies were transferred from guidance developed for reviews and meta-analyses of RCTs. Although there was substantial agreement in some methodological areas there was also considerable disagreement on how evidence synthesis of observational studies should be conducted. Conflicting recommendations were seen on topics such as the inclusion of different study designs in systematic reviews and meta-analyses, the use of quality scales to assess the risk of bias, and the choice of model (e.g. fixed vs. random effects) for meta-analysis. Conclusion There is a need for sound methodological guidance on how to conduct systematic reviews and meta-analyses of observational studies, which critically considers areas in which there are conflicting recommendations.

Background: Missing data in covariates can result in biased estimates and loss of power to detect associations. It can also lead to other challenges in time-to-event analyses including the handling of time-varying effects of covariates, selection of covariates and their flexible modelling. This review aims to describe how researchers approach time-to-event analyses with missing data.Methods: Medline and Embase were searched for observational time-to-event studies in oncology published from January 2012 to January 2018. The review focused on proportional hazards models or extended Cox models. We investigated the extent and reporting of missing data and how it was addressed in the analysis. Covariate modelling and selection, and assessment of the proportional hazards assumption were also investigated, alongside the treatment of missing data in these procedures.Results: 148 studies were included. The mean proportion of individuals with missingness in any covariate was 32%. 53% of studies used complete-case analysis, and 22% used multiple imputation. In total, 14% of studies stated an assumption concerning missing data and only 34% stated missingness as a limitation. The proportional hazards assumption was checked in 28% of studies, of which, 17% did not state the assessment method. 58% of 144 multivariable models stated their covariate selection procedure with use of a pre-selected set of covariates being the most popular followed by stepwise methods and univariable analyses. Of 69 studies that included continuous covariates, 81% did not assess the appropriateness of the functional form.Conclusion: While guidelines for handling missing data in epidemiological studies are in place, this review indicates that few report implementing recommendations in practice. Although missing data are present in many studies, we found that few state clearly how they handled it or the assumptions they have made. Easy-to-implement but potentially biased approaches such as complete-case analysis are most commonly used despite these relying on strong assumptions and where often more appropriate methods should be employed. Authors should be encouraged to follow existing guidelines to address missing data, and increased levels of expectation from journals and editors could be used to improve practice. [ABSTRACT FROM AUTHOR]

Background: In the data pipeline from the data collection process to the planned statistical analyses, initial data analysis (IDA) typically takes place between the end of the data collection and do not touch the research questions. A systematic process for IDA and clear reporting of the findings would help to understand the potential shortcomings of a dataset, such as missing values, or subgroups with small sample sizes, or shortcomings in the collection process, and to evaluate the impact of these shortcomings on the research results. A clear reporting of findings is also relevant when making datasets available to other researchers. Initial data analyses can provide valuable insights into the suitability of a data set for a future research study. Our aim was to describe the practice of reporting of initial data analyses in observational studies in five highly ranked medical journals with focus on data cleaning, screening, and reporting of findings which led to a potential change in the analysis plan.Methods: This review was carried out using systematic search strategies with eligibility criteria for articles to be reviewed. A total of 25 papers about observational studies were selected from five medical journals published in 2018. Each paper was reviewed by two reviewers and IDA statements were further discussed by all authors. The consensus was reported.Results: IDA statements were reported in the methods, results, discussion, and supplement of papers. Ten out of 25 papers (40%) included a statement about data cleaning. Data screening statements were included in all articles, and 18 (72%) indicated the methods used to describe them. Item missingness was reported in 11 papers (44%), unit missingness in 15 papers (60%). Eleven papers (44%) mentioned some changes in the analysis plan. Reported changes referred to missing data treatment, unexpected values, population heterogeneity and aspects related to variable distributions or data properties.Conclusion: Reporting of initial data analyses were sparse, and statements on IDA were located throughout the research articles. There is a lack of systematic reporting of IDA. We conclude the article with recommendations on how to overcome shortcomings in the practice of IDA reporting in observational studies. [ABSTRACT FROM AUTHOR]

Abstract Background Observational studies are widely used in pharmacoepidemiology. Several designs can be used, in particular self-controlled designs (case-crossover and self-controlled case series). These designs offer the advantage of controlling for time-invariant confounders, which may not be collected in electronic healthcare databases. They are particularly useful in pharmacoepidemiology involving healthcare database. To be valid, they require the presence of some characteristics (key validity assumptions), and in such situations, these designs should be preferred. We aimed at describing the appropriate use and reporting of the key validity assumptions in self-controlled design studies. Methods Articles published between January 2011 and December 2014, and describing a self-controlled study design involving electronic healthcare databases were retrieved. The appropriate use (fulfilment of key assumptions) was studied in terms of major (abrupt onset event, rare or recurrent event, and intermittent exposure) and minor assumptions (those for which the design can be adapted). Results Among the 107 articles describing a self-controlled design, 35/53 (66%) case-crossover studies, and 48/55 (87%) self-controlled case series fulfilled the major validity assumptions for use of the design; 4/35 and 14/48 respectively did not fulfill the minor assumptions. Overall, 31/53 (58%) case-crossover studies and 34/55 (62%) self-controlled case series fulfilled both major and minor assumptions. The reporting of the methodology or the results was appropriate, except for power calculation. Conclusions Self-controlled designs were not appropriately used in34% and 13% of the articles we reviewed that described a case-crossover or a self-controlled case series design, respectively. We encourage better use of these designs in situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted.

Background: The availability of clinical and therapeutic data drawn from medical records and administrative databases has entailed new opportunities for clinical and epidemiologic research. However, these databases present inherent limitations which may render them prone to new biases. We aimed to conduct a structured review of biases specific to observational clinical studies based on secondary databases, and to propose strategies for the mitigation of those biases. Methods: Scoping review of the scientific literature published during the period 2000-2018 through an automated search of MEDLINE, EMBASE and Web of Science, supplemented with manually cross-checking of reference lists. We included opinion essays, methodological reviews, analyses or simulation studies, as well as letters to the editor or retractions, the principal objective of which was to highlight the existence of some type of bias in pharmacoepidemiologic studies using secondary databases. Results: A total of 117 articles were included. An increasing trend in the number of publications concerning the potential limitations of secondary databases was observed over time and across medical research disciplines. Confounding was the most reported category of bias (63.2% of articles), followed by selection and measurement biases (47.0% and 46.2% respectively). Confounding by indication (32.5%), unmeasured/residual confounding (28.2%), outcome misclassification (28.2%) and "immortal time" bias (25.6%) were the subcategories most frequently mentioned. Conclusions: Suboptimal use of secondary databases in pharmacoepidemiologic studies has introduced biases in the studies, which may have led to erroneous conclusions. Methods to mitigate biases are available and must be considered in the design, analysis and interpretation phases of studies using these data sources. [ABSTRACT FROM AUTHOR]

Abstract Background Observational post-marketing assessment studies often involve evaluating the effect of a rare treatment on a time-to-event outcome, through the estimation of a marginal hazard ratio. Propensity score (PS) methods are the most used methods to estimate marginal effect of an exposure in observational studies. However there is paucity of data concerning their performance in a context of low prevalence of exposure. Methods We conducted an extensive series of Monte Carlo simulations to examine the performance of the two preferred PS methods, known as PS-matching and PS-weighting to estimate marginal hazard ratios, through various scenarios. Results We found that both PS-weighting and PS-matching could be biased when estimating the marginal effect of rare exposure. The less biased results were obtained with estimators of average treatment effect in the treated population (ATT), in comparison with estimators of average treatment effect in the overall population (ATE). Among ATT estimators, PS-weighting using ATT weights outperformed PS-matching. These results are illustrated using a real observational study. Conclusions When clinical objectives are focused on the treated population, applied researchers are encouraged to estimate ATT with PS-weighting for studying the relative effect of a rare treatment on time-to-event outcomes.

Background: Observational studies are widely used in pharmacoepidemiology. Several designs can be used, in particular self-controlled designs (case-crossover and self-controlled case series). These designs offer the advantage of controlling for time-invariant confounders, which may not be collected in electronic healthcare databases. They are particularly useful in pharmacoepidemiology involving healthcare database. To be valid, they require the presence of some characteristics (key validity assumptions), and in such situations, these designs should be preferred. We aimed at describing the appropriate use and reporting of the key validity assumptions in self-controlled design studies.Methods: Articles published between January 2011 and December 2014, and describing a self-controlled study design involving electronic healthcare databases were retrieved. The appropriate use (fulfilment of key assumptions) was studied in terms of major (abrupt onset event, rare or recurrent event, and intermittent exposure) and minor assumptions (those for which the design can be adapted).Results: Among the 107 articles describing a self-controlled design, 35/53 (66%) case-crossover studies, and 48/55 (87%) self-controlled case series fulfilled the major validity assumptions for use of the design; 4/35 and 14/48 respectively did not fulfill the minor assumptions. Overall, 31/53 (58%) case-crossover studies and 34/55 (62%) self-controlled case series fulfilled both major and minor assumptions. The reporting of the methodology or the results was appropriate, except for power calculation.Conclusions: Self-controlled designs were not appropriately used in34% and 13% of the articles we reviewed that described a case-crossover or a self-controlled case series design, respectively. We encourage better use of these designs in situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted. [ABSTRACT FROM AUTHOR]

Background: Observational post-marketing assessment studies often involve evaluating the effect of a rare treatment on a time-to-event outcome, through the estimation of a marginal hazard ratio. Propensity score (PS) methods are the most used methods to estimate marginal effect of an exposure in observational studies. However there is paucity of data concerning their performance in a context of low prevalence of exposure.Methods: We conducted an extensive series of Monte Carlo simulations to examine the performance of the two preferred PS methods, known as PS-matching and PS-weighting to estimate marginal hazard ratios, through various scenarios.Results: We found that both PS-weighting and PS-matching could be biased when estimating the marginal effect of rare exposure. The less biased results were obtained with estimators of average treatment effect in the treated population (ATT), in comparison with estimators of average treatment effect in the overall population (ATE). Among ATT estimators, PS-weighting using ATT weights outperformed PS-matching. These results are illustrated using a real observational study.Conclusions: When clinical objectives are focused on the treated population, applied researchers are encouraged to estimate ATT with PS-weighting for studying the relative effect of a rare treatment on time-to-event outcomes. [ABSTRACT FROM AUTHOR]

Background: Propensity score methods have become a popular tool for reducing selection bias in making causal inference from observational studies in medical research. Propensity score matching, a key component of propensity score methods, normally matches units based on the distance between point estimates of the propensity scores. The problem with this technique is that it is difficult to establish a sensible criterion to evaluate the closeness of matched units without knowing estimation errors of the propensity scores. Methods: The present study introduces interval matching using bootstrap confidence intervals for accommodating estimation errors of propensity scores. In interval matching, if the confidence interval of a unit in the treatment group overlaps with that of one or more units in the comparison group, they are considered as matched units. Results: The procedure of interval matching is illustrated in an empirical example using a real-life dataset from the Nursing Home Compare, a national survey conducted by the Centers for Medicare and Medicaid Services. The empirical example provided promising evidence that interval matching reduced more selection bias than did commonly used matching methods including the rival method, caliper matching. Interval matching's approach methodologically sounds more meaningful than its competing matching methods because interval matching develop a more "scientific" criterion for matching units using confidence intervals. Conclusions: Interval matching is a promisingly better alternative tool for reducing selection bias in making causal inference from observational studies, especially useful in secondary data analysis on national databases such as the Centers for Medicare and Medicaid Services data. [ABSTRACT FROM AUTHOR]

Background: Propensity score methods have become a popular tool for reducing selection bias in making causal inference from observational studies in medical research. Propensity score matching, a key component of propensity score methods, normally matches units based on the distance between point estimates of the propensity scores. The problem with this technique is that it is difficult to establish a sensible criterion to evaluate the closeness of matched units without knowing estimation errors of the propensity scores. Methods: The present study introduces interval matching using bootstrap confidence intervals for accommodating estimation errors of propensity scores. In interval matching, if the confidence interval of a unit in the treatment group overlaps with that of one or more units in the comparison group, they are considered as matched units. Results: The procedure of interval matching is illustrated in an empirical example using a real-life dataset from the Nursing Home Compare, a national survey conducted by the Centers for Medicare and Medicaid Services. The empirical example provided promising evidence that interval matching reduced more selection bias than did commonly used matching methods including the rival method, caliper matching. Interval matching’s approach methodologically sounds more meaningful than its competing matching methods because interval matching develop a more “scientific” criterion for matching units using confidence intervals. Conclusions: Interval matching is a promisingly better alternative tool for reducing selection bias in making causal inference from observational studies, especially useful in secondary data analysis on national databases such as the Centers for Medicare and Medicaid Services data. [ABSTRACT FROM AUTHOR]

Background: Despite their close relationship, clinical research and medical care have become separated by clear boundaries. The purpose of clinical research is to generate generalizable knowledge useful for future patients, whereas medical care aims to promote the well-being of individual patients. The evolution towards patient-centered medicine and patient-oriented research, and the gradual standardization of medicine are contributing to closer ties between clinical research and medical practice. But the integration of both activities requires addressing important ethical and methodological challenges. Discussion: From an ethical perspective, clinical research should evolve from a position of paternalistic beneficence to a situation in which the principle of non-maleficence and patient autonomy predominate. The progressive adoption of "patient-oriented informed consent", "patient equipoise", and "altruism-based research", and the application of risk-based ethical oversight, in which the level of regulatory scrutiny is adapted to the potential risk for patients, are crucial steps to achieve the integration between research and care. From a methodological standpoint, careful and systematic observations should have greater relevance in clinical research, and experiments should be embedded into usual clinical practice. Clinical research should focus on individuals through the development of patient-oriented research. In a complementary way, the integration of experiments into medical practice through the systematic application of "point of care research" could help to generate knowledge for the individuals and for the populations. Summary: The integration of clinical research and medical care will require researchers, clinicians, health care managers, and patients to reevaluate the way they understand both activities. The development of an integrated learning health care system will contribute to generating and applying clinically relevant medical knowledge, producing benefits for present and future patients. [ABSTRACT FROM AUTHOR]

Background The tendency towards publication bias is greater for observational studies than for randomized clinical trials. Several statistical methods have been developed to test the publication bias. However, almost all existing methods exhibit rather low power or have inappropriate type I error rates. Methods We propose a modified regression method, which used a smoothed variance to estimate the precision of a study, to test for publication bias in meta-analyses of observational studies. A comprehensive simulation study is carried out, and a real-world example is considered. Results The simulation results indicate that the performance of tests varies with the number of included studies, level of heterogeneity, event rates, and sample size ratio between two groups. Neither the existing tests nor the newly developed method is particularly powerful in all simulation scenarios. However, our proposed method has a more robust performance across different settings. In the presence of heterogeneity, the arcsine-Thompson test is a suitable alternative, and Peters' test can be considered as a complementary method when mild or no heterogeneity is present. Conclusions Several factors should be taken into consideration when employing asymmetry tests for publication bias. Based on our simulation results, we provide a concise table to show the appropriate use of regression methods to test for publication bias based on our simulation results. [ABSTRACT FROM AUTHOR]

Background: Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza. Methods: Cohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics. Results: Authors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11). Conclusions: Differences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews. [ABSTRACT FROM AUTHOR]