The added value of an artificial intelligence system in assisting radiologists on indeterminate BI-RADS 0 mammograms

To investigate the value of an artificial intelligence (AI) system in assisting radiologists to improve the assessment accuracy of BI-RADS 0 cases in mammograms. We included 34,654 consecutive digital mammography studies, collected between January 2011 and January 2019, among which, 1088 cases from 1010 unique patients with initial BI-RADS 0 assessment who were recalled during 2 years of follow-up were used in this study. Two mid-level radiologists retrospectively re-assessed these BI-RADS 0 cases with the assistance of an AI system developed by us previously. In addition, four entry-level radiologists were split into two groups to cross-read 80 cases with and without the AI. Diagnostic performance was evaluated using the follow-up diagnosis or biopsy results as the reference standard. Of the 1088 cases, 626 were actually normal (BI-RADS 1 and no recall required). Assisted by the AI system, 351 (56%) and 362 (58%) normal cases were correctly identified by the two mid-level radiologists hence can be avoided for unnecessary follow-ups. However, they would have missed 12 (10 invasive cancers and 2 ductal carcinoma in situ cancers) and 6 (invasive cancers) malignant lesions respectively as a result. These missed lesions were not highly malignant tumors. The inter-rater reliability of entry-level radiologists increased from 0.20 to 0.30 (p