Back to Search
Start Over
Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error
- Source :
- Systematic Reviews, Vol 8, Iss 1, Pp 1-12 (2019), Bannach-Brown, A, Przybyła, P, Thomas, J, Rice, A S C, Ananiadou, S, Liao, J & Macleod, M R 2019, ' Machine learning algorithms for systematic review : Reducing workload in a preclinical review of animal studies and reducing human screening error ', Systematic Reviews, vol. 8, no. 1, 23 . https://doi.org/10.1186/s13643-019-0942-7, Bannach-brown, A, Przybyła, P, Thomas, J, Rice, A S C, Ananiadou, S, Liao, J & Macleod, M R 2019, ' Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error ', Systematic Reviews, vol. 8, no. 1, 23 . https://doi.org/10.1186/s13643-019-0942-7, Systematic Reviews
- Publication Year :
- 2019
- Publisher :
- BMC, 2019.
-
Abstract
- Background Here, we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies. The aim is to achieve a high-performing algorithm comparable to human screening that can reduce human resources required for carrying out this step of a systematic review. Methods We applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross-validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis). Results ML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using the assigned inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm. Conclusions This work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews with different inclusion prevalence levels, but represents a promising approach to integrating human decisions and automation in systematic review methodology. Electronic supplementary material The online version of this article (10.1186/s13643-019-0942-7) contains supplementary material, which is available to authorized users.
- Subjects :
- Computer science
Human error
Systematic review methodology
lcsh:Medicine
Workload
Machine learning
computer.software_genre
Sensitivity and Specificity
Cross-validation
Set (abstract data type)
03 medical and health sciences
0302 clinical medicine
Medicine, General & Internal
SEARCH FILTER
General & Internal Medicine
Manchester Institute of Biotechnology
Feature (machine learning)
Animals
Humans
030212 general & internal medicine
Sensitivity (control systems)
Citation screening
11 Medical and Health Sciences
030304 developmental biology
Automation tools
0303 health sciences
Depressive Disorder
Science & Technology
business.industry
lcsh:R
Methodology
Analysis of human error
ResearchInstitutes_Networks_Beacons/manchester_institute_of_biotechnology
Systematic review
Bibliometrics
Models, Animal
Artificial intelligence
business
computer
Algorithm
Life Sciences & Biomedicine
Algorithms
Systematic Reviews as Topic
Subjects
Details
- Language :
- English
- ISSN :
- 20464053
- Volume :
- 8
- Issue :
- 1
- Database :
- OpenAIRE
- Journal :
- Systematic Reviews
- Accession number :
- edsair.doi.dedup.....98839b9a04df9f8ef93086cb77709141
- Full Text :
- https://doi.org/10.1186/s13643-019-0942-7