Search

Abstract Background The covid-19 pandemic has highlighted the role of living systematic reviews. The speed of evidence generated during the covid-19 pandemic accentuated the challenges of managing high volumes of research literature. Methods In this article, we summarise the characteristics of ongoing living systematic reviews on covid-19, and we follow a life cycle approach to describe key steps in a living systematic review. Results We identified 97 living systematic reviews on covid-19, published up to 7th November 2022, which focused mostly on the effects of pharmacological interventions (n = 46, 47%) or the prevalence of associated conditions or risk factors (n = 30, 31%). The scopes of several reviews overlapped considerably. Most living systematic reviews included both observational and randomised study designs (n = 45, 46%). Only one-third of the reviews has been updated at least once (n = 34, 35%). We address practical aspects of living systematic reviews including how to judge whether to start a living systematic review, methods for study identification and selection, data extraction and evaluation, and give recommendations at each step, drawing from our own experience. We also discuss when it is time to stop and how to publish updates. Conclusions Methods to improve the efficiency of searching, study selection, and data extraction using machine learning technologies are being developed, their performance and applicability, particularly for reviews based on observational study designs should improve, and ways of publishing living systematic reviews and their updates will continue to evolve. Finally, knowing when to end a living systematic review is as important as knowing when to start.

Abstract Background The COVID-19 pandemic has led to an unprecedented amount of scientific publications, growing at a pace never seen before. Multiple living systematic reviews have been developed to assist professionals with up-to-date and trustworthy health information, but it is increasingly challenging for systematic reviewers to keep up with the evidence in electronic databases. We aimed to investigate deep learning-based machine learning algorithms to classify COVID-19-related publications to help scale up the epidemiological curation process. Methods In this retrospective study, five different pre-trained deep learning-based language models were fine-tuned on a dataset of 6365 publications manually classified into two classes, three subclasses, and 22 sub-subclasses relevant for epidemiological triage purposes. In a k-fold cross-validation setting, each standalone model was assessed on a classification task and compared against an ensemble, which takes the standalone model predictions as input and uses different strategies to infer the optimal article class. A ranking task was also considered, in which the model outputs a ranked list of sub-subclasses associated with the article. Results The ensemble model significantly outperformed the standalone classifiers, achieving a F1-score of 89.2 at the class level of the classification task. The difference between the standalone and ensemble models increases at the sub-subclass level, where the ensemble reaches a micro F1-score of 70% against 67% for the best-performing standalone model. For the ranking task, the ensemble obtained the highest recall@3, with a performance of 89%. Using an unanimity voting rule, the ensemble can provide predictions with higher confidence on a subset of the data, achieving detection of original papers with a F1-score up to 97% on a subset of 80% of the collection instead of 93% on the whole dataset. Conclusion This study shows the potential of using deep learning language models to perform triage of COVID-19 references efficiently and support epidemiological curation and review. The ensemble consistently and significantly outperforms any standalone model. Fine-tuning the voting strategy thresholds is an interesting alternative to annotate a subset with higher predictive confidence.