Back to Search
Start Over
An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes
- Source :
- BMC Medical Research Methodology, BMC Medical Research Methodology, Vol 20, Iss 1, Pp 1-14 (2020)
- Publication Year :
- 2020
- Publisher :
- Springer Science and Business Media LLC, 2020.
-
Abstract
- Background Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings. Methods Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented. Results Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%). Conclusions The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.
- Subjects :
- Prioritization
Artificial intelligence
medicine.medical_specialty
Epidemiology
Health Informatics
Rapid reviews
Efficiency
Time saving
Machine Learning
Automation
03 medical and health sciences
0302 clinical medicine
Systematic reviews
Natural language processing
Interquartile range
Included study
medicine
Humans
Mass Screening
Prospective Studies
030212 general & internal medicine
lcsh:R5-920
Recall
business.industry
030503 health policy & services
Systematic review
Emergency medicine
Time savings
True recall
lcsh:Medicine (General)
0305 other medical science
business
Algorithms
Research Article
Subjects
Details
- ISSN :
- 14712288
- Volume :
- 20
- Database :
- OpenAIRE
- Journal :
- BMC Medical Research Methodology
- Accession number :
- edsair.doi.dedup.....e4835bbf5a3e425f318b85391fb4e03e