Search

ObjectiveWe sought to apply natural language processing to the task of automatic risk of bias assessment in preclinical literature, which could speed the process of systematic review, provide information to guide research improvement activity, and support translation from preclinical to clinical research.Materials and MethodsWe use 7,840 full-text publications describing animal experiments with yes/no annotations for five risk of bias items. We implement a series of models including baselines (support vector machine, logistic regression, random forest), neural models (convolutional neural network, recurrent neural network with attention, hierarchical neural network) and models using BERT with two strategies (document chunk pooling and sentence extraction). We tune hyperparameters to obtain the highest F1 scores for each risk of bias item on the validation set and compare evaluation results on the test set to our previous regular expression approach.ResultsThe F1 scores of best models on test set are 82.0% for random allocation, 81.6% for blinded assessment of outcome, 82.6% for conflict of interests, 91.4% for compliance with animal welfare regulations and 46.6% for reporting animals excluded from analysis. Our models significantly outperform regular expressions for four risk of bias items.ConclusionFor random allocation, blinded assessment of outcome, conflict of interests and animal exclusions, neural models achieve good performance, and for animal welfare regulations, BERT model with sentence extraction strategy works better.

Background Natural language processing could assist multiple tasks in systematic reviews to reduce workflow, including the extraction of PICO elements such as study populations, interventions, comparators and outcomes. The PICO framework provides a basis for the retrieval and selection for inclusion of evidence relevant to a specific systematic review question, and automatic approaches to PICO extraction have been developed particularly for reviews of clinical trial findings. Considering the difference between preclinical animal studies and clinical trials, developing separate approaches is necessary. Facilitating preclinical systematic reviews will inform the translation from preclinical to clinical research. Methods We randomly selected 400 abstracts from the PubMed Central Open Access database which described in vivo animal research and manually annotated these with PICO phrases for Species, Strain, methods of Induction of disease model, Intervention, Comparator and Outcome. We developed a two-stage workflow for preclinical PICO extraction. Firstly we fine-tuned BERT with different pre-trained modules for PICO sentence classification. Then, after removing the text irrelevant to PICO features, we explored LSTM-, CRF- and BERT-based models for PICO entity recognition. We also explored a self-training approach because of the small training corpus. Results For PICO sentence classification, BERT models using all pre-trained modules achieved an F1 score of over 80%, and models pre-trained on PubMed abstracts achieved the highest F1 of 85%. For PICO entity recognition, fine-tuning BERT pre-trained on PubMed abstracts achieved an overall F1 of 71% and satisfactory F1 for Species (98%), Strain (70%), Intervention (70%) and Outcome (67%). The score of Induction and Comparator is less satisfactory, but F1 of Comparator can be improved to 50% by applying self-training. Conclusions Our study indicates that of the approaches tested, BERT pre-trained on PubMed abstracts is the best for both PICO sentence classification and PICO entity recognition in the preclinical abstracts. Self-training yields better performance for identifying comparators and strains.

IntroductionMotor neuron disease (MND) is a rapidly fatal neurodegenerative disease. Despite decades of research and clinical trials there remains no cure and only one globally approved drug, riluzole, which prolongs survival by 2–3 months. Recent improved mechanistic understanding of MND heralds a new translational era with many potential targets being identified that are ripe for clinical trials. Motor Neuron Disease Systematic Multi-Arm Adaptive Randomised Trial (MND-SMART) aims to evaluate the efficacy of drugs efficiently and definitively in a multi-arm, multi-stage, adaptive trial. The first two drugs selected for evaluation in MND-SMART are trazodone and memantine.Methods and analysisInitially, up to 531 participants (177/arm) will be randomised 1:1:1 to oral liquid trazodone, memantine and placebo. The coprimary outcome measures are the Amyotrophic Lateral Sclerosis Functional Rating Scale Revised (ALSFRS-R) and survival. Comparisons will be conducted in four stages. The decision to continue randomising to arms after each stage will be made by the Trial Steering Committee who receive recommendations from the Independent Data Monitoring Committee. The primary analysis of ALSFRS-R will be conducted when 150 participants/arm, excluding long survivors, have completed 18 months of treatment; if positive the survival effect will be inferentially analysed when 113 deaths have been observed in the placebo group. The trial design ensures that other promising drugs can be added for evaluation in planned trial adaptations. Using this novel trial design reduces time, cost and number of participants required to definitively (phase III) evaluate drugs and reduces exposure of participants to potentially ineffective treatments.Ethics and disseminationMND-SMART was approved by the West of Scotland Research Ethics Committee on 2 October 2019. (REC reference: 19/WS/0123) Results of the study will be submitted for publication in a peer-reviewed journal and a summary provided to participants.Trial registration numbersEuropean Clinical Trials Registry (2019-000099-41); NCT04302870.

Based on previous case reports and disease-based cohorts, a minority of patients with cerebral small vessel disease (cSVD) have a monogenic cause, with many also manifesting extra-cerebral phenotypes. We investigated the frequency, penetrance, and phenotype associations of rare variants in cSVD genes in UK Biobank (UKB), a large population-based study.We used a systematic review of previous literature and ClinVar to identify putative pathogenic rare variants in CTSA, TREX1, HTRA1, COL4A1/2. We mapped phenotypes previously attributed to these variants (phenotypes-of-interest) to disease coding systems used in UKB’s linked health data from UK hospital admissions, death records and primary care. Among 199,313 exome-sequenced UKB participants, we assessed: the proportion of participants carrying ≥1 variant(s); phenotype-of-interest penetrance; and the association between variant carrier status and phenotypes-of-interest using a binary (any phenotype present/absent) and phenotype burden (linear score of the number of phenotypes a participant possessed) approach.Among UKB participants, 0.5% had ≥1 variant(s) in studied genes. Using hospital admission and death records, 4-20% of variant carriers per gene had an associated phenotype. This increased to 7-55% when including primary care records. Only COL4A1 variant carrier-status was significantly associated with having ≥1 phenotype-of-interest and a higher phenotype score (OR=1.29, p=0.006).While putative pathogenic rare variants in monogenic cSVD genes occur in 1:200 people in the UKB population, only around half of variant carriers have a relevant disease phenotype recorded in their linked health data. We could not replicate most previously reported gene-phenotype associations, suggesting lower penetrance rates, overestimated pathogenicity and/or limited statistical power.

Researchers, institutions, funders, and publishers are considering how to improve research culture and quality, but no single part of the research ecosystem can effect change on its own. The UK Reproducibility Network (UKRN) was established to facilitate the necessary coordination. Its experience can inform the establishment of like-minded networks around the world to drive positive change.

Background Meta-analysis of preclinical data is used to evaluate the consistency of findings and to inform the design and conduct of future studies. Unlike clinical meta-analysis, preclinical data often involve many heterogeneous studies reporting outcomes from a small number of animals. Here, we review the methodological challenges in preclinical meta-analysis in estimating and explaining heterogeneity in treatment effects. Methods Assuming aggregate-level data, we focus on two topics: (1) estimation of heterogeneity using commonly used methods in preclinical meta-analysis: method of moments (DerSimonian and Laird; DL), maximum likelihood (restricted maximum likelihood; REML) and Bayesian approach; (2) comparison of univariate versus multivariable meta-regression for adjusting estimated treatment effects for heterogeneity. Using data from a systematic review on the efficacy of interleukin-1 receptor antagonist in animals with stroke, we compare these methods, and explore the impact of multiple covariates on the treatment effects. Results We observed that the three methods for estimating heterogeneity yielded similar estimates for the overall effect, but different estimates for between-study variability. The proportion of heterogeneity explained by a covariate is estimated larger using REML and the Bayesian method as compared with DL. Multivariable meta-regression explains more heterogeneity than univariate meta-regression. Conclusions Our findings highlight the importance of careful selection of the estimation method and the use of multivariable meta-regression to explain heterogeneity. There was no difference between REML and the Bayesian method and both methods are recommended over DL. Multiple meta-regression is worthwhile to explain heterogeneity by more than one variable, reducing more variability than any univariate models and increasing the explained proportion of heterogeneity.

Background Hospital-based studies have reported variable associations between outcome after spontaneous intracerebral hemorrhage and peri-hematomal edema volume. Aims In a community-based study, we aimed to investigate the existence, strength, direction, and independence of associations between intracerebral hemorrhage and peri-hematomal edema volumes on diagnostic brain CT and one-year functional outcome and long-term survival. Methods We identified all adults, resident in Lothian, diagnosed with first-ever, symptomatic spontaneous intracerebral hemorrhage between June 2010 and May 2013 in a community-based, prospective inception cohort study. We defined regions of interest manually and used a semi-automated approach to measure intracerebral hemorrhage volume, peri-hematomal edema volume, and the sum of these measurements (total lesion volume) on first diagnostic brain CT performed at ≤3 days after symptom onset. The primary outcome was death or dependence (scores 3–6 on the modified Rankin Scale) at one-year after intracerebral hemorrhage. Results Two hundred ninety-two (85%) of 342 patients (median age 77.5 y, IQR 68–83, 186 (54%) female, median time from onset to CT 6.5 h (IQR 2.9–21.7)) were dead or dependent one year after intracerebral hemorrhage. Peri-hematomal edema and intracerebral hemorrhage volumes were colinear ( R2 = 0.77). In models using both intracerebral hemorrhage and peri-hematomal edema, 10 mL increments in intracerebral hemorrhage (adjusted odds ratio (aOR) 1.72 (95% CI 1.08–2.87); p = 0.029) but not peri-hematomal edema volume (aOR 0.92 (0.63–1.45); p = 0.69) were independently associated with one-year death or dependence. 10 mL increments in total lesion volume were independently associated with one-year death or dependence (aOR 1.24 (1.11–1.42); p = 0.0004). Conclusion Total volume of intracerebral hemorrhage and peri-hematomal edema, and intracerebral hemorrhage volume alone on diagnostic brain CT, undertaken at three days or sooner, are independently associated with death or dependence one-year after intracerebral hemorrhage, but peri-hematomal edema volume is not. Data access statement Anonymized summary data may be requested from the corresponding author.

Background Here, we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies. The aim is to achieve a high-performing algorithm comparable to human screening that can reduce human resources required for carrying out this step of a systematic review. Methods We applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross-validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis). Results ML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using the assigned inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm. Conclusions This work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews with different inclusion prevalence levels, but represents a promising approach to integrating human decisions and automation in systematic review methodology. Electronic supplementary material The online version of this article (10.1186/s13643-019-0942-7) contains supplementary material, which is available to authorized users.

While high risk of failure is an inherent part of developing innovative therapies, it can be reduced by adherence to evidence-based rigorous research practices. Numerous analyses conducted to date have clearly identified measures that need to be taken to improve research rigor. Supported through the European Union's Innovative Medicines Initiative, the EQIPD consortium has developed a novel preclinical research quality system that can be applied in both public and private sectors and is free for anyone to use. The EQIPD Quality System was designed to be suited to boost innovation by ensuring the generation of robust and reliable preclinical data while being lean, effective and not becoming a burden that could negatively impact the freedom to explore scientific questions. EQIPD defines research quality as the extent to which research data are fit for their intended use. Fitness, in this context, is defined by the stakeholders, who are the scientists directly involved in the research, but also their funders, sponsors, publishers, research tool manufacturers and collaboration partners such as peers in a multi-site research project. The essence of the EQIPD Quality System is the set of 18 core requirements that can be addressed flexibly, according to user-specific needs and following a user-defined trajectory. The EQIPD Quality System proposes guidance on expectations for quality-related measures, defines criteria for adequate processes (i.e., performance standards) and provides examples of how such measures can be developed and implemented. However, it does not prescribe any pre-determined solutions. EQIPD has also developed tools (for optional use) to support users in implementing the system and assessment services for those research units that successfully implement the quality system and seek formal accreditation. Building upon the feedback from users and continuous improvement, a sustainable EQIPD Quality System will ultimately serve the entire community of scientists conducting non-regulated preclinical research, by helping them generate reliable data that are fit for their intended use.

The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research., A meta-analysis study of the variability in phenotypic outcomes in both control and experimental animal models of ischaemic stroke provides novel perspectives in which heterogeneity can be embraced to improve the reproducibility and translation of preclinical studies.

Transparency in reporting benefits scientific communication on many levels. While specific needs and expectations vary across fields, the effective interpretation and use of research findings relies on the availability of core information about research materials, study design, data, and experimental and analytical methods. For preclinical research, transparency in reporting is a key focus in response to concerns of replication failure. The inconsistent reporting of key elements of experimental and analytical design, alongside ambiguous description of reagents and lack of access to underlying data and code, has been shown to impair replication (1) and raise doubt about the robustness of results (2, 3). In response to early concerns about replication of published results, funders, publishers, and other stakeholders have called for improvements in reporting transparency (4⇓⇓–7). Several initiatives ensued, including journal policies and joint efforts by journals, funders, and other stakeholders (8⇓–10). One of these initiatives, the Transparency and Openness Promotion (TOP) guidelines (11), outlines a policy framework at the journal level that over 1,000 journals and publishers have adopted. The National Academies have focused on reproducibility and replicability* challenges through several recent initiatives leading to consensus reports, including Reproducibility and Replicability in Science (12), Open Science by Design: Realizing a Vision for 21st Century Research (13), and Fostering Integrity in Research (14). Each of these reports concludes that lack of reporting transparency is one factor which contributes to these systemic problems. Building on these findings, the National Academies convened a public workshop in September 2019 titled “Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting.” The workshop was designed to discuss the current state of transparency in reporting biomedical research and to explore the possibility of improving the harmonization of guidelines across journals and funding agencies. During this workshop, we provided … [↵][1]1To whom correspondence may be addressed: Email: Malcolm.Macleod{at}ed.ac.uk (correspondence regarding the development and evaluation of the guideline) or mellor.david{at}gmail.com (for further information about implementation and stewardship). [1]: #xref-corresp-1-1

Introduction The initiation and conduct of randomised clinical trials are complicated by multiple barriers, including delays in obtaining regulatory approvals. Quantitative data on the extent of the delays due to national or local review in randomised clinical trials is scarce. Materials and methods We assessed the times needed to obtain regulatory approval and to initiate a trial site for an academic, EU-funded, phase III, randomised clinical trial of pharmacological prevention of complications in patients with acute stroke in over 80 sites in nine European countries. The primary outcome was the time from the first submission to a regulatory authority to initiation of a trial site. Secondary outcomes included time needed to complete each individual preparatory requirement and the number of patients recruited by each site in the first 6 and 12 months. Results The median time from the first submission to a regulatory authority to initiation of a trial site was 784 days (IQR: 586–1102). The single most time-consuming step was the conclusion of a clinical trial agreement between the national coordinator and the trial site, which took a median of 194 days (IQR: 93–293). A longer time to site initiation was associated with a lower patient recruitment rate in the first six months after initiation (B = –0.002; p = 0.02). Discussion Conclusion In this EU-funded clinical trial, approximately 26 months were needed to initiate a trial site for patient recruitment. The conclusion of a contract with a trial site was the most time-consuming activity. To simplify and speed up the process, we suggest that the level of detail of contracts for academic trials should be proportional to the risks and commercial interests of these trials.

Preclinical research is a vital step in the drug discovery pipeline and more generally in helping to better understand human disease aetiology and its management. Systematic reviews (SRs) can be powerful in summarising and appraising this evidence concerning a specific research question, to highlight areas of improvements, areas for further research and areas where evidence may be sufficient to take forward to other research domains, for instance clinical trial. Guidance and tools for preclinical research synthesis remain limited despite their clear utility. We aimed to create an online end-to-end platform primarily for conducting SRs of preclinical studies, that was flexible enough to support a wide variety of experimental designs, was adaptable to different research questions, would allow users to adopt emerging automated tools and support them during their review process using best practice. In this article, we introduce the Systematic Review Facility (https://syrf.org.uk), which was launched in 2016 and designed to support primarily preclinical SRs from small independent projects to large, crowdsourced projects. We discuss the architecture of the app and its features, including the opportunity to collaborate easily, to efficiently manage projects, to screen and annotate studies for important features (metadata), to extract outcome data into a secure database, and tailor these steps to each project. We introduce how we are working to leverage the use of automation tools and allow the integration of these services to accelerate and automate steps in the systematic review workflow.

Amyotrophic lateral sclerosis is a progressive and devastating neurodegenerative disease. Despite decades of clinical trials, effective disease-modifying drugs remain scarce. To understand the challenges of trial design and delivery, we performed a systematic review of Phase II, Phase II/III and Phase III amyotrophic lateral sclerosis clinical drug trials on trial registries and PubMed between 2008 and 2019. We identified 125 trials, investigating 76 drugs and recruiting more than 15 000 people with amyotrophic lateral sclerosis. About 90% of trials used traditional fixed designs. The limitations in understanding of disease biology, outcome measures, resources and barriers to trial participation in a rapidly progressive, disabling and heterogenous disease hindered timely and definitive evaluation of drugs in two-arm trials. Innovative trial designs, especially adaptive platform trials may offer significant efficiency gains to this end. We propose a flexible and scalable multi-arm, multi-stage trial platform where opportunities to participate in a clinical trial can become the default for people with amyotrophic lateral sclerosis., Wong et al. reviewed historical approaches to clinical trials in amyotrophic lateral sclerosis. Lack of sensitive outcome measures, limitations in resources and barriers to trial participation were challenges for timely and definitive evaluation of drugs in two-arm trials. They concluded that future trials should be more flexible, scalable and efficient., Graphical Abstract Graphical Abstract

In an effort to better utilize published evidence obtained from animal experiments, systematic reviews of preclinical studies are increasingly more common—along with the methods and tools to appraise them (e.g., SYstematic Review Center for Laboratory animal Experimentation [SYRCLE’s] risk of bias tool). We performed a cross-sectional study of a sample of recent preclinical systematic reviews (2015–2018) and examined a range of epidemiological characteristics and used a 46-item checklist to assess reporting details. We identified 442 reviews published across 43 countries in 23 different disease domains that used 26 animal species. Reporting of key details to ensure transparency and reproducibility was inconsistent across reviews and within article sections. Items were most completely reported in the title, introduction, and results sections of the reviews, while least reported in the methods and discussion sections. Less than half of reviews reported that a risk of bias assessment for internal and external validity was undertaken, and none reported methods for evaluating construct validity. Our results demonstrate that a considerable number of preclinical systematic reviews investigating diverse topics have been conducted; however, their quality of reporting is inconsistent. Our study provides the justification and evidence to inform the development of guidelines for conducting and reporting preclinical systematic reviews., A cross sectional study of a sample of recent preclinical systematic reviews reveals deficiencies in reporting and provides the justification and evidence to inform the development of specific guidelines for conducting and reporting preclinical systematic reviews.

Metaresearch is a scientific field involving the study of research itself. It has been applied to clinical trials since the 1980s,1 but has only become an emerging discipline over the last decade in the preclinical field. The primary tool of metaresearch is the systematic review, which uses predefined methods to provide a transparent and comprehensive summary of the evidence relating to a research question. A systematic review is defined as ‘a review that uses explicit, systematic methods to collate and synthesize findings of studies that address a clearly formulated question’.2 Systematic reviews allow for evaluation of methods and comprehensiveness of reporting, to assay likelihood of reproducibility and potential for translatability to subsequent domains of research and can investigate the impact of incentives on primary research. This, in turn, allows for a more rigorous understanding of what makes research reliable, and how research can be improved,3 while driving evidence-based decisions for future research.4 5 Systematic reviews typically comprise several steps. Before beginning a systematic review, it is recommended that the author team develop a protocol, which defines the research question and the methods that will be used to conduct, analyse and report the findings of the review. The research question determines the resources required to complete the review, the broader the question and the larger the field the more resource intensive a review will be. The search strategy is developed to identify as much potentially relevant literature as possible, often involving searching of multiple databases. Following database searches, deduplication (if searching multiple database with overlapping coverage), the unique search results are screened for inclusion or exclusion. Full-text retrieval may be conducted before or after title and abstract citation screening. Metadata including information regarding reported study quality and design, and outcome data are then extracted from the included …

The reproducibility of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets reproducibility has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that are irreproducible. An alternative approach is to, instead, deliberately introduce heterogeneity; known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischaemic stroke. First, by quantifying inter-individual variability across control groups we illustrate that the samount of heterogeneity in disease-state (infarct volume) differs according to methodological approach, for example, in disease-induction methods and disease models. We argue that such methods may improve reproducibility by creating diverse and representative distribution of baseline disease-state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e. low inter-individual variability), as well as those where there is high inter-individual variability in response; for these latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimise variability in phenotypic outcomes, we can motivate the shift towards heterogenization and improve both the reproducibility and generalizability of preclinical research.

Background Anemia is common in neurocritically ill patients. Considering the limited clinical evidence in this population, preclinical data may provide some understanding of the potential impact of anemia and of red blood cell transfusion in these patients. We aim to estimate the association between different transfusion strategies and neurobehavioral outcome in animal models. Methods We will conduct a systematic review of comparative studies of red blood cell transfusion strategies using animal models of traumatic brain injury, ischemic stroke or cerebral hemorrhage. We will search MEDLINE, EMBASE, and Web of Science databases for eligible studies from inception onwards. Two independent reviewers will perform study selection and data extraction. We will report our results in a descriptive synthesis focusing on characteristics of included studies, reported outcomes, risk of bias, and construct validity. Our primary outcome is the neurological function (neurobehavioral performance) and our secondary outcomes include mortality, infarct size, intracranial pressure, cerebral perfusion pressure, cerebral blood flow, and brain tissue oxygen tension. If appropriate, we will also perform a quantitative synthesis and pool results using random-effect models. Heterogeneity will be expressed with I2 statistics. Subgroup analyses are planned according to animal model characteristics, co-interventions, and risks of bias. Discussion Our study is aligned with the efforts to better understand the level of evidence on the impact of red blood cell transfusion strategies from preclinical studies in animal models of acute brain injury and the potential translation of information from the preclinical to the clinical research field. Systematic review registration PROSPERO CRD42018086662.

Assessment of outcome in preclinical studies of vascular cognitive impairment (VCI) is heterogenous. Through an ARUK Scottish Network supported questionnaire and workshop (mostly UK-based researchers), we aimed to determine underlying variability and what could be implemented to overcome identified challenges. Twelve UK VCI research centres were identified and invited to complete a questionnaire and attend a one-day workshop. Questionnaire responses demonstrated agreement that outcome assessments in VCI preclinical research vary by group and even those common across groups, may be performed differently. From the workshop, six themes were discussed: issues with preclinical models, reasons for choosing functional assessments, issues in interpretation of functional assessments, describing and reporting functional outcome assessments, sharing resources and expertise, and standardization of outcomes. Eight consensus points emerged demonstrating broadly that the chosen assessment should reflect the deficit being measured, and therefore that one assessment does not suit all models; guidance/standardisation on recording VCI outcome reporting is needed and that uniformity would be aided by a platform to share expertise, material, protocols and procedures thus reducing heterogeneity and so increasing potential for collaboration, comparison and replication. As a result of the workshop, UK wide consensus statements were agreed and future priorities for preclinical research identified.

Therapeutic hypothermia was a mainstay of severe traumatic brain injury (TBI) management for half a century. Recent trials have suggested that its effect on long-term functional outcome is neutral or negative, despite apparently promising pre-clinical data. Systematic review and meta-analysis is a useful tool to collate experimental data and investigate the basis of its conclusions. We searched three online databases to identify studies testing systemic hypothermia as monotherapy for treatment of animals subjected to a TBI. Data pertaining to TBI paradigm, animal subjects, and hypothermia management were extracted as well as those relating to risk of bias. We pooled outcome data where sufficient numbers allowed and investigated heterogeneity in neurobehavioral outcomes using multi-variate meta-regression. We identified 90 publications reporting 272 experiments testing hypothermia in animals subject to TBI. The subjects were mostly small animals, with well-established models predominating. Target temperature was comparable to clinical trial data but treatment was initiated very early. Study quality was low and there was some evidence of publication bias. Delay to treatment, comorbidity, and blinded outcome assessment appeared to predict neurobehavioral outcome on multi-variate meta-regression. Therapeutic hypothermia appears to be an efficacious treatment in experimental TBI, which differs from the clinical evidence. The pre-clinical literature showed limitations in quality and design and these both appeared to affect neurobehavioral experiment outcome. These should be acknowledged when designing and interpreting pre-clinical TBI studies in the future.

Over the last two decades, awareness of the negative repercussions of flaws in the planning, conduct and reporting of preclinical research involving experimental animals has been growing. Several initiatives have set out to increase transparency and internal validity of preclinical studies, mostly publishing expert consensus and experience. While many of the points raised in these various guidelines are identical or similar, they differ in detail and rigour. Most of them focus on reporting, only few of them cover the planning and conduct of studies. The aim of this systematic review is to identify existing experimental design, conduct, analysis and reporting guidelines relating to preclinical animal research. A systematic search in PubMed, Embase and Web of Science retrieved 13 863 unique results. After screening these on title and abstract, 613 papers entered the full-text assessment stage, from which 60 papers were retained. From these, we extracted unique 58 recommendations on the planning, conduct and reporting of preclinical animal studies. Sample size calculations, adequate statistical methods, concealed and randomised allocation of animals to treatment, blinded outcome assessment and recording of animal flow through the experiment were recommended in more than half of the publications. While we consider these recommendations to be valuable, there is a striking lack of experimental evidence on their importance and relative effect on experiments and effect sizes.

Supplemental Digital Content is Available in the Text., We report a systematic review and meta-analysis of studies that assessed the antinociceptive efficacy of cannabinoids, cannabis-based medicines, and endocannabinoid system modulators on pain-associated behavioural outcomes in animal models of pathological or injury-related persistent pain. In April 2019, we systematically searched 3 online databases and used crowd science and machine learning to identify studies for inclusion. We calculated a standardised mean difference effect size for each comparison and performed a random-effects meta-analysis. We assessed the impact of study design characteristics and reporting of mitigations to reduce the risk of bias. We meta-analysed 374 studies in which 171 interventions were assessed for antinociceptive efficacy in rodent models of pathological or injury-related pain. Most experiments were conducted in male animals (86%). Antinociceptive efficacy was most frequently measured by attenuation of hypersensitivity to evoked limb withdrawal. Selective cannabinoid type 1, cannabinoid type 2, nonselective cannabinoid receptor agonists (including delta-9-tetrahydrocannabinol) and peroxisome proliferator-activated receptor-alpha agonists (predominantly palmitoylethanolamide) significantly attenuated pain-associated behaviours in a broad range of inflammatory and neuropathic pain models. Fatty acid amide hydrolase inhibitors, monoacylglycerol lipase inhibitors, and cannabidiol significantly attenuated pain-associated behaviours in neuropathic pain models but yielded mixed results in inflammatory pain models. The reporting of criteria to reduce the risk of bias was low; therefore, the studies have an unclear risk of bias. The value of future studies could be enhanced by improving the reporting of methodological criteria, the clinical relevance of the models, and behavioural assessments. Notwithstanding, the evidence supports the hypothesis of cannabinoid-induced analgesia.

Research synthesis is the process of bringing together findings and attributes from different publications, for example, to give a more complete description of phenomena than is usually possible in a single work. We bring the Research Synthesis Series to BMC Biology to promote meta-analyses, other research syntheses including meta-research studies, and research synthesis methodologies in biology, facilitating their dissemination to broader communities. ispartof: BMC BIOLOGY vol:18 issue:1 ispartof: location:England status: published

There is a pressing need to increase the rigor of research in the life and biomedical sciences. To address this issue, we propose that communities of 'rigor champions' be established to campaign for reforms of the research culture that has led to shortcomings in rigor. These communities of rigor champions would also assist in the development and adoption of a comprehensive educational platform that would teach the principles of rigorous science to researchers at all career stages.

Background Elderly patients are at high risk of complications after stroke, such as infections and fever. The occurrence of these complications has been associated with an increased risk of death or dependency. Hypothesis: Prevention of aspiration, infections, or fever with metoclopramide, ceftriaxone, paracetamol, or any combination of these in the first four days after stroke onset will improve functional outcome at 90 days in elderly patients with acute stroke. Design International, 3 × 2-factorial, randomised-controlled, open-label clinical trial with blinded outcome assessment (PROBE) in 3800 patients aged 66 years or older with acute ischaemic stroke or intracerebral haemorrhage and an NIHSS score ≥ 6. Patients will be randomly allocated to any combination of oral, rectal, or intravenous metoclopramide (10 mg thrice daily); intravenous ceftriaxone (2000 mg once daily); oral, rectal, or intravenous paracetamol (1000 mg four times daily); or usual care, started within 24 h after symptom onset and continued for four days or until complete recovery or discharge from hospital, if earlier. Outcome: The primary outcome measure is the score on the modified Rankin Scale at 90 days (± 14 days), as analysed with multiple regression. Summary: This trial will provide evidence for a simple, safe and generally available treatment strategy that may reduce the burden of death or disability in patients with stroke at very low costs. Planning: First patient included in May 2016; final follow-up of the last patient by April 2020. Registration: ISRCTN, ISRCTN82217627, https://doi.org/10.1186/ISRCTN82217627

Helping every scientist to improve is more effective than ferreting out a few frauds. Helping every scientist to improve is more effective than ferreting out a few frauds.

There is ongoing debate regarding the robustness and credibility of published scientific research. We argue that these issues stem from two broad causal mechanisms: the cognitive biases of researchers and the incentive structures within which researchers operate. The UK Reproducibility Network (UKRN) is working with researchers, institutions, funders, publishers, and other stakeholders to address these issues.

Stroke is the second highest cause of death globally and a leading cause of disability, with an increasing incidence in developing countries. Ischaemic stroke caused by arterial occlusion is responsible for the majority of strokes. Management focuses on rapid reperfusion with intravenous thrombolysis and endovascular thrombectomy, which both reduce disability but are time-critical. Accordingly, improving the system of care to reduce treatment delays is key to maximizing the benefits of reperfusion therapies. Intravenous thrombolysis reduces disability when administered within 4.5 h of the onset of stroke. Thrombolysis also benefits selected patients with evidence from perfusion imaging of salvageable brain tissue for up to 9 h and in patients who awake with stroke symptoms. Endovascular thrombectomy reduces disability in a broad group of patients with large vessel occlusion when performed within 6 h of stroke onset and in patients selected by perfusion imaging up to 24 h following stroke onset. Secondary prevention of ischaemic stroke shares many common elements with cardiovascular risk management in other fields, including blood pressure control, cholesterol management and antithrombotic medications. Other preventative interventions are tailored to the mechanism of stroke, such as anticoagulation for atrial fibrillation and carotid endarterectomy for severe symptomatic carotid artery stenosis.

Reproducible science requires transparent reporting. The ARRIVE guidelines were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved.Here we introduce ARRIVE 2019. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise the items and split the guidelines into two sets, the ARRIVE Essential 10, which constitute the minimum requirement, and the Recommended Set, which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers to verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document that serves 1) to explain the rationale behind each item in the guidelines, 2) to clarify key concepts and 3) to provide illustrative examples. We aim through these changes to help ensure that researchers, reviewers and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

Improving the reproducibility of biomedical research is a major challenge. Transparent and accurate reporting are vital to this process; it allows readers to assess the reliability of the findings, and repeat or build upon the work of other researchers. The NC3Rs developed the ARRIVE guidelines in 2010 to help authors and journals identify the minimum information necessary to report in publications describing in vivo experiments.Despite widespread endorsement by the scientific community, the impact of the ARRIVE guidelines on the transparency of reporting in animal research publications has been limited. We have revised the ARRIVE guidelines to update them and facilitate their use in practice. The revised guidelines are published alongside this paper. This Explanation and Elaboration document was developed as part of the revision. It provides further information about each of the 21 items in ARRIVE 2019, including the rationale and supporting evidence for their inclusion in the guidelines, elaboration of details to report, and examples of good reporting from the published literature.

ObjectiveTo determine whether and to what degree bias and underestimated variability undermine the predictive value of preclinical research for clinical translation.MethodsWe investigated experimental spinal cord injury (SCI) studies for outcome heterogeneity and the impact of bias. Data from 549 preclinical SCI studies including 9,535 animals were analyzed with meta-regression to assess the effect of various study characteristics and the quality of neurologic recovery.ResultsOverall, the included interventions reported a neurobehavioral outcome improvement of 26.3% (95% confidence interval 24.3–28.4). Response to treatment was dependent on experimental modeling paradigms (neurobehavioral score, site of injury, and animal species). Applying multiple outcome measures was consistently associated with smaller effect sizes compared with studies applying only 1 outcome measure. More than half of the studies (51.2%) did not report blinded assessment, constituting a likely source of evaluation bias, with an overstated effect size of 7.2%. Assessment of publication bias, which extrapolates to identify likely missing data, suggested that between 2% and 41% of experiments remain unpublished. Inclusion of these theoretical missing studies suggested an overestimation of efficacy, reducing the effect sizes by between 0.9% and 14.3%.ConclusionsWe provide empirical evidence of prevalent bias in the design and reporting of experimental SCI studies, resulting in overestimation of the effectiveness. Bias compromises the internal validity and jeopardizes the successful translation of SCI therapies from the bench to bedside.

We report a systematic review and meta-analysis of research using animal models of chemotherapy-induced peripheral neuropathy (CIPN). We systematically searched 5 online databases in September 2012 and updated the search in November 2015 using machine learning and text mining to reduce the screening for inclusion workload and improve accuracy. For each comparison, we calculated a standardised mean difference (SMD) effect size, and then combined effects in a random-effects meta-analysis. We assessed the impact of study design factors and reporting of measures to reduce risks of bias. We present power analyses for the most frequently reported behavioural tests; 337 publications were included. Most studies (84%) used male animals only. The most frequently reported outcome measure was evoked limb withdrawal in response to mechanical monofilaments. There was modest reporting of measures to reduce risks of bias. The number of animals required to obtain 80% power with a significance level of 0.05 varied substantially across behavioural tests. In this comprehensive summary of the use of animal models of CIPN, we have identified areas in which the value of preclinical CIPN studies might be increased. Using both sexes of animals in the modelling of CIPN, ensuring that outcome measures align with those most relevant in the clinic, and the animal’s pain contextualised ethology will likely improve external validity. Measures to reduce risk of bias should be employed to increase the internal validity of studies. Different outcome measures have different statistical power, and this can refine our approaches in the modelling of CIPN., This systematic review assesses limitations in the experimental design of studies modelling peripheral neuropathy induced by chemotherapy. The data allow the authors to provide guidance on experimental design to improve the robustness and validity of future studies., Author summary Many frequently used and effective cancer chemotherapies can cause a disabling side effect that features pain, numbness, tingling, and sensitivity to cold and heat in the extremities known as chemotherapy-induced peripheral neuropathy (CIPN). There are currently no effective therapies to treat or prevent this condition, and animal models have been developed to address this. It is important that experiments using animal models of CIPN are robust and valid if they are to effectively help patients. We used a systematic approach to identify all 337 studies that have been published describing the use of animal models of CIPN. We were able to identify that many studies are imperfect in their experimental design, use only male animals, and assess outcomes with limited relevance to the human condition. Based on a meta-analysis, we provide guidance to the CIPN animal modelling community to guide future experiments that may improve their utility and validity.

Schultz et al.1 concluded that tetrabenazine (TBZ) use is not associated with depression or suicidality. This analysis of cross-sectional observational data from the Enroll-HD dataset contradicts prospective experimental evidence.2 This alone should caution against the authors' conclusion,1 but we have other concerns. Principally, 2 sources of bias may have distorted the observed associations. First, only individuals who tolerated TBZ are captured as users—those who abandoned treatment (e.g., due to depression/suicidality) are not counted. This survival bias would serve to induce a spurious negative association between TBZ use and depression/suicidality (i.e., a bias in the direction of an apparent protective effect). Second, antidepressant use may result from TBZ use (if this causes depression) and from depression itself (from other causes). Adjusting for this, as in this study,1 may introduce collider bias, and—again—a spurious negative association. These problems are compounded by the cross-sectional nature of the analysis, which precludes definition of incident depression/suicidality, despite prospective data being available within Enroll-HD. The conclusion that TBZ may be safe for patients with Huntington disease with a history of depression/suicidality is flawed and potentially dangerous. We welcome use of the Enroll-HD dataset, and CHDI Foundation funds an independent statistical standing committee to provide support and expertise (see supplemental full-length version, [links.lww.com/WNL/A825][1]). [1]: http://links.lww.com/WNL/A825

Any given research claim can be made with a degree of confidence that a phenomenon is present, with an estimate of the precision of the observed effects and a prediction of the extent to which the findings might hold true under different experimental or real-world conditions. In some situations, the certainty and precision obtained from a single study are sufficient reliably to inform future research decisions. However, in other situations greater certainty is required. This might be the case where a substantial research investment is planned, a pivotal claim is to be made or the launch of a clinical trial programme is being considered. Under these circumstances, some form of summary of findings across studies may be helpful.Summary estimates can describe findings from exploratory (observational) or hypothesis testing experiments, but importantly, the creation of such summaries is, in itself, observational rather than experimental research. The process is therefore particularly at risk from selective identification of literature to be included, and this can be addressed using systematic search strategies and pre-specified criteria for inclusion and exclusion against which possible contributing data will be assessed. This characterises a systematic review (in contrast to nonsystematic or narrative reviews). In meta-analysis, there is an attempt to provide a quantitative summary of such research findings.

IntroductionWe assessed whether modest systemic cooling started within 6 hours of symptom onset improves functional outcome at three months in awake patients with acute ischaemic stroke.Patients and methodsIn this European randomised open-label clinical trial with blinded outcome assessment, adult patients with acute ischaemic stroke were randomised to cooling to a target body temperature of 34.0–35.0°C, started within 6 h after stroke onset and maintained for 12 or 24 h , versus standard treatment. The primary outcome was the score on the modified Rankin Scale at 91 days, as analysed with ordinal logistic regression.ResultsThe trial was stopped after inclusion of 98 of the originally intended 1500 patients because of slow recruitment and cessation of funding. Forty-nine patients were randomised to hypothermia versus 49 to standard treatment. Four patients were lost to follow-up. Of patients randomised to hypothermia, 15 (31%) achieved the predefined cooling targets. The primary outcome did not differ between the groups (odds ratio for good outcome, 1.01; 95% confidence interval, 0.48–2.13; p = 0.97). The number of patients with one or more serious adverse events did not differ between groups (relative risk, 1.22; 95% confidence interval, 0.65–1.94; p = 0.52).DiscussionIn this trial, cooling to a target of 34.0–35.0°C and maintaining this for 12 or 24 h was not feasible in the majority of patients. The final sample was underpowered to detect clinically relevant differences in outcomes.ConclusionBefore new trials are launched, the feasibility of cooling needs to be improved.

In recent years there has been increasing concern about the rigor of laboratory research. Here we present the protocol for a study comparing the completeness of reporting of in vivo and in vitro research carried in Nature Publication Group journals before and after the introduction of a change in editorial policy (the introduction of a set of guidelines for reporting); and in similar research published in other journals in the same periods. Electronic supplementary material The online version of this article (doi:10.1007/s11192-016-1964-8) contains supplementary material, which is available to authorized users.