Search

As the role of artificial intelligence (AI) in clinical practice evolves, governance structures oversee the implementation, maintenance, and monitoring of clinical AI algorithms to enhance quality, manage resources, and ensure patient safety. In this article, a framework is established for the infrastructure required for clinical AI implementation and presents a road map for governance. The road map answers four key questions: Who decides which tools to implement? What factors should be considered when assessing an application for implementation? How should applications be implemented in clinical practice? Finally, how should tools be monitored and maintained after clinical implementation? Among the many challenges for the implementation of AI in clinical practice, devising flexible governance structures that can quickly adapt to a changing environment will be essential to ensure quality patient care and practice improvement objectives.

Over the past decade, technological advances have provided new tools for radiologists. However, the effect of these technological advances on radiologist workload and detecting pathologies needs to be assessed.The purpose of this study is to assess the workload, including non-interpretative tasks, associated with Computed Tomography Angiogram (CTA) of Aorta exams performed in the Emergency Department (ED) over a 10-year period and their relationship to detection of aortic pathology.This is a retrospective analysis of CTAs of Aorta performed on adults with suspected acute aortic pathology within the ED at an academic level I quaternary care hospital from January 1, 2005, through December 31, 2015. Data assessed included (1) Interpretive tasks: total number of images, number of reformat series, number of radiology reports with positive aortic pathologies; and (2) Non-interpretative tasks: recommendations and documentation of verbal communication with requesting providers. Statistical analyses were performed to assess temporal trends of variables. P values less than 0.05 are considered significant.A total of 4368 examinations (mean age: 69.8, M/F: 56.8%/43.2%) were performed. Studies per year increased significantly from 2005 (n = 278) to 2007 (n = 445), but not significantly after. The number of images and reformat series per scan increased from 487 to 2819 and 6.4 to 13.7, respectively (both P-value0.01). The proportion of exams with aortic findings did not significantly change (28.1% in 2005 and 24.9% in 2015). However, The proportions of exams with verbal communication increased from 9.3% to 24.6% and with recommendations from 1.8% to 28.9% (both P-value0.01).During a 10-year period, CTAs performed in the ED for suspected aortic pathology were associated with a significant increase in images created, reformat series generated, recommendations, and verbal communications with ordering providers without a concomitant increase in the rate of aortic pathologies. To completely capture the complexities of CTA workloads, non-interpretive tasks such as radiologist recommendations and verbal communications should also be included.

Artificial intelligence is already innovating in the provision of neurologic care. This review explores key artificial intelligence concepts; their application to neurologic diagnosis, prognosis, and treatment; and challenges that await their broader adoption. The development of new diagnostic biomarkers, individualization of prognostic information, and improved access to treatment are among the plethora of possibilities. These advances, however, reflect only the tip of the iceberg for the ways in which artificial intelligence may transform neurologic care in the future.

Background Adrenal masses are common, but radiology reporting and recommendations for management can be variable. Purpose To create a machine learning algorithm to segment adrenal glands on contrast-enhanced CT images and classify glands as normal or mass-containing and to assess algorithm performance. Materials and Methods This retrospective study included two groups of contrast-enhanced abdominal CT examinations (development data set and secondary test set). Adrenal glands in the development data set were manually segmented by radiologists. Images in both the development data set and the secondary test set were manually classified as normal or mass-containing. Deep learning segmentation and classification models were trained on the development data set and evaluated on both data sets. Segmentation performance was evaluated with use of the Dice similarity coefficient (DSC), and classification performance with use of sensitivity and specificity. Results The development data set contained 274 CT examinations (251 patients; median age, 61 years; 133 women), and the secondary test set contained 991 CT examinations (991 patients; median age, 62 years; 578 women). The median model DSC on the development test set was 0.80 (IQR, 0.78-0.89) for normal glands and 0.84 (IQR, 0.79-0.90) for adrenal masses. On the development reader set, the median interreader DSC was 0.89 (IQR, 0.78-0.93) for normal glands and 0.89 (IQR, 0.85-0.97) for adrenal masses. Interreader DSC for radiologist manual segmentation did not differ from automated machine segmentation (

Acute ischemic stroke can be subtle to detect on non-contrast computed tomography imaging. We show that a novel artificial intelligence model significantly improves the performance of physicians, including ED physicians, neurologists and radiologists, in identifying and quantifying the volume of acute ischemic stroke lesions. This model may lead to improved clinical decision-making for stroke patients.

ImportanceEarly detection of pneumothorax, most often via chest radiography, can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care.ObjectiveTo compare the accuracy of an AI model vs consensus thoracic radiologist interpretations in detecting any pneumothorax (incorporating both nontension and tension pneumothorax) and tension pneumothorax.Design, Setting, and ParticipantsThis diagnostic study was a retrospective standalone performance assessment using a data set of 1000 chest radiographs captured between June 1, 2015, and May 31, 2021. The radiographs were obtained from patients aged at least 18 years at 4 hospitals in the Mass General Brigham hospital network in the United States. Included radiographs were selected using 2 strategies from all chest radiography performed at the hospitals, including inpatient and outpatient. The first strategy identified consecutive radiographs with pneumothorax through a manual review of radiology reports, and the second strategy identified consecutive radiographs with tension pneumothorax using natural language processing. For both strategies, negative radiographs were selected by taking the next negative radiograph acquired from the same radiography machine as each positive radiograph. The final data set was an amalgamation of these processes. Each radiograph was interpreted independently by up to 3 radiologists to establish consensus ground-truth interpretations. Each radiograph was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax. This study was conducted between July and October 2021, with the primary analysis performed between October and November 2021.Main Outcomes and MeasuresThe primary end points were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary end points were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax.ResultsThe final analysis included radiographs from 985 patients (mean [SD] age, 60.8 [19.0] years; 436 [44.3%] female patients), including 307 patients with nontension pneumothorax, 128 patients with tension pneumothorax, and 550 patients without pneumothorax. The AI model detected any pneumothorax with an AUC of 0.979 (95% CI, 0.970-0.987), sensitivity of 94.3% (95% CI, 92.0%-96.3%), and specificity of 92.0% (95% CI, 89.6%-94.2%) and tension pneumothorax with an AUC of 0.987 (95% CI, 0.980-0.992), sensitivity of 94.5% (95% CI, 90.6%-97.7%), and specificity of 95.3% (95% CI, 93.9%-96.6%).Conclusions and RelevanceThese findings suggest that the assessed AI model accurately detected pneumothorax and tension pneumothorax in this chest radiograph data set. The model’s use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.

The efficient and accurate interpretation of radiologic images is paramount.To evaluate whether a deep learning-based artificial intelligence (AI) engine used concurrently can improve reader performance and efficiency in interpreting chest radiograph abnormalities.This multicenter cohort study was conducted from April to November 2021 and involved radiologists, including attending radiologists, thoracic radiology fellows, and residents, who independently participated in 2 observer performance test sessions. The sessions included a reading session with AI and a session without AI, in a randomized crossover manner with a 4-week washout period in between. The AI produced a heat map and the image-level probability of the presence of the referrable lesion. The data used were collected at 2 quaternary academic hospitals in Boston, Massachusetts: Beth Israel Deaconess Medical Center (The Medical Information Mart for Intensive Care Chest X-Ray [MIMIC-CXR]) and Massachusetts General Hospital (MGH).The ground truths for the labels were created via consensual reading by 2 thoracic radiologists. Each reader documented their findings in a customized report template, in which the 4 target chest radiograph findings and the reader confidence of the presence of each finding was recorded. The time taken for reporting each chest radiograph was also recorded. Sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) were calculated for each target finding.A total of 6 radiologists (2 attending radiologists, 2 thoracic radiology fellows, and 2 residents) participated in the study. The study involved a total of 497 frontal chest radiographs-247 from the MIMIC-CXR data set (demographic data for patients were not available) and 250 chest radiographs from MGH (mean [SD] age, 63 [16] years; 133 men [53.2%])-from adult patients with and without 4 target findings (pneumonia, nodule, pneumothorax, and pleural effusion). The target findings were found in 351 of 497 chest radiographs. The AI was associated with higher sensitivity for all findings compared with the readers (nodule, 0.816 [95% CI, 0.732-0.882] vs 0.567 [95% CI, 0.524-0.611]; pneumonia, 0.887 [95% CI, 0.834-0.928] vs 0.673 [95% CI, 0.632-0.714]; pleural effusion, 0.872 [95% CI, 0.808-0.921] vs 0.889 [95% CI, 0.862-0.917]; pneumothorax, 0.988 [95% CI, 0.932-1.000] vs 0.792 [95% CI, 0.756-0.827]). AI-aided interpretation was associated with significantly improved reader sensitivities for all target findings, without negative impacts on the specificity. Overall, the AUROCs of readers improved for all 4 target findings, with significant improvements in detection of pneumothorax and nodule. The reporting time with AI was 10% lower than without AI (40.8 vs 36.9 seconds; difference, 3.9 seconds; 95% CI, 2.9-5.2 seconds; P .001).These findings suggest that AI-aided interpretation was associated with improved reader performance and efficiency for identifying major thoracic findings on a chest radiograph.

Non-contrast head CT (NCCT) is extremely insensitive for early (2 > 0.98). When this 150 CT test set was expanded to include a total of 364 CT scans with a more heterogeneous distribution of infarct locations (94 stroke-negative, 270 stroke-positive mixed territory infarcts), model sensitivity was 97%, specificity 99%, for detection of infarcts larger than the 70 mL volume threshold used for patient selection in several major randomized controlled trials of thrombectomy treatment.

ImportanceEarly detection of pneumothorax, most often on chest radiograph (CXR), can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care.ObjectiveThis study aimed to compare the accuracy of an AI model (Annalise Enterprise) to consensus thoracic radiologist interpretations in detecting (1) pneumothorax (incorporating both non-tension and tension pneumothorax) and (2) tension pneumothorax.DesignA retrospective standalone performance assessment was conducted on a dataset of 1,000 CXR cases.SettingThe cases were obtained from four hospitals in the United States.ParticipantsThe cases were obtained from patients aged 18 years or older. They were selected using two strategies from all CXRs performed at the hospitals including inpatients and outpatients. The first strategy identified consecutive pneumothorax cases through a manual review of radiology reports and the second strategy identified consecutive tension pneumothorax cases using natural language processing. For both strategies, negative cases were selected by taking the next negative case acquired from the same x-ray machine. The final dataset was an amalgamation of these processes.MethodsEach case was interpreted independently by up to three radiologists to establish consensus ground truth interpretations. Each case was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax.Main OutcomeThe primary endpoints were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary endpoints were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax at predefined operating points.ResultsModel inference was successfully performed in 307 non-tension pneumothorax, 128 tension pneumothorax and 550 negative cases. The AI model detected pneumothorax with AUC of 0.979 (94.3% sensitivity, 92.0% specificity) and tension pneumothorax with AUC of 0.987 (94.5% sensitivity, 95.3% specificity).Conclusions and RelevanceThe assessed AI model accurately detected pneumothorax and tension pneumothorax on this CXR dataset. Its use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.Key PointsQuestionDoes a commercial artificial intelligence model accurately detect simple and tension pneumothorax on chest x-ray?FindingsThis retrospective study used 1,000 chest x-rays from four hospitals in the United States to compare artificial intelligence model outputs to consensus thoracic radiologist interpretations. The model detected pneumothorax (incorporating both simple and tension pneumothorax) with area under the curve (AUC) of 0.979 and tension pneumothorax with AUC of 0.987. The sensitivity and specificity were 94.3% and 92.0% respectively for pneumothorax, and 94.5% and 95.3% for tension pneumothorax.MeaningThis artificial intelligence model could assist radiologists through its accurate detection of pneumothorax.

To assess feasibility of automated segmentation and measurement of tracheal collapsibility for detecting tracheomalacia on inspiratory and expiratory chest CT images.Our study included 123 patients (age 67 ± 11 years; female: male 69:54) who underwent clinically indicated chest CT examinations in both inspiration and expiration phases. A thoracic radiologist measured anteroposterior length of trachea in inspiration and expiration phase image at the level of maximum collapsibility or aortic arch (in absence of luminal change). Separately, another investigator separately processed the inspiratory and expiratory DICOM CT images with Airway Segmentation component of a commercial COPD software (IntelliSpace Portal, Philips Healthcare). Upon segmentation, the software automatically estimated average lumen diameter (in mm) and lumen area (sq.mm) both along the entire length of trachea and at the level of aortic arch. Data were analyzed with independent t-tests and area under the receiver operating characteristic curve (AUC).Of the 123 patients, 48 patients had tracheomalacia and 75 patients did not. Ratios of inspiration to expiration phases average lumen area and lumen diameter from the length of trachea had the highest AUC of 0.93 (95% CI = 0.88-0.97) for differentiating presence and absence of tracheomalacia. A decrease of ≥25% in average lumen diameter had sensitivity of 82% and specificity of 87% for detecting tracheomalacia. A decrease of ≥40% in the average lumen area had sensitivity and specificity of 86% for detecting tracheomalacia.Automatic segmentation and measurement of tracheal dimension over the entire tracheal length is more accurate than a single-level measurement for detecting tracheomalacia.

A standardized objective evaluation method is needed to compare machine learning (ML) algorithms as these tools become available for clinical use. Therefore, we designed, built, and tested an evaluation pipeline with the goal of normalizing performance measurement of independently developed algorithms, using a common test dataset of our clinical imaging. Three vendor applications for detecting solid, part-solid, and groundglass lung nodules in chest CT examinations were assessed in this retrospective study using our data-preprocessing and algorithm assessment chain. The pipeline included tools for image cohort creation and de-identification; report and image annotation for ground-truth labeling; server partitioning to receive vendor “black box” algorithms and to enable model testing on our internal clinical data (100 chest CTs with 243 nodules) from within our security firewall; model validation and result visualization; and performance assessment calculating algorithm recall, precision, and receiver operating characteristic curves (ROC). Algorithm true positives, false positives, false negatives, recall, and precision for detecting lung nodules were as follows: Vendor-1 (194, 23, 49, 0.80, 0.89); Vendor-2 (182, 270, 61, 0.75, 0.40); Vendor-3 (75, 120, 168, 0.32, 0.39). The AUCs for detection of solid (0.61–0.74), groundglass (0.66–0.86) and part-solid (0.52–0.86) nodules varied between the three vendors. Our ML model validation pipeline enabled testing of multi-vendor algorithms within the institutional firewall. Wide variations in algorithm performance for detection as well as classification of lung nodules justifies the premise for a standardized objective ML algorithm evaluation process.

We evaluated and compared performance of an acute pulmonary embolism (PE) triaging artificial intelligence (PE-AI) model in suboptimal and optimal CT pulmonary angiography (CTPA).In an IRB approved, retrospective study we identified 104 consecutive, suboptimal CTPA which were deemed as suboptimal for PE evaluation in radiology reports due to motion, artifacts and/or inadequate contrast enhancement. We enriched this dataset, with additional 226 optimal CTPA (over same timeframe as suboptimal CTPA) with and without PE. Two thoracic radiologists (ground truth) independently reviewed all 330 CTPA for adequacy (to assess PE down to distal segmental level), reason for suboptimal CTPA (artifacts or poor contrast enhancement), as well as for presence and location of PE. CT values (HU) were measured in the main pulmonary artery. Same attributes were assessed in 80 patients who had repeat or follow-up CTPA following suboptimal CTPA. All CTPA were processed with the PE-AI (Aidoc).Among 104 suboptimal CTPA (mean age ± standard deviation 56 ± 15 years), 18/104 (17%) were misclassified as suboptimal for PE evaluation in their radiology reports but relabeled as optimal on ground truth evaluation. Of 226 optimal CTPA, 47 (21%) were reclassified as suboptimal CTPA. PEs were present in 97/330 CTPA. PE-AI had similar performance on suboptimal CTPA (sensitivity 100%; specificity 89%; AUC 0.89, 95% CI 0.80-0.98) and optimal CTPA (sensitivity 96%; specificity 92%; AUC 0.87, 95% CI 0.81-0.93).Suboptimal CTPA examinations do not impair the performance of PE-AI triage model; AI retains clinically meaningful sensitivity and high specificity regardless of diagnostic quality.

To assess key trends, strengths, and gaps in validation studies of the Food and Drug Administration (FDA)-regulated imaging-based artificial intelligence/machine learning (AI/ML) algorithms.We audited publicly available details of regulated AI/ML algorithms in imaging from 2008 until April 2021. We reviewed 127 regulated software (118 AI/ML) to classify information related to their parent company, subspecialty, body area and specific anatomy type, imaging modality, date of FDA clearance, indications for use, target pathology (such as trauma) and findings (such as fracture), technique (CAD triage, CAD detection and/or characterization, CAD acquisition or improvement, and image processing/quantification), product performance, presence, type, strength and availability of clinical validation data. Pertaining to validation data, where available, we recorded the number of patients or studies included, sensitivity, specificity, accuracy, and/or receiver operating characteristic area under the curve, along with information on ground-truthing of use-cases. Data were analyzed with pivot tables and charts for descriptive statistics and trends.We noted an increasing number of FDA-regulated AI/ML from 2008 to 2021. Seventeen (17/118) regulated AI/ML algorithms posted no validation claims or data. Just 9/118 reviewed AI/ML algorithms had a validation dataset sizes of over 1000 patients. The most common type of AI/ML included image processing/quantification (IPQ; n = 59/118), and triage (CADt; n = 27/118). Brain, breast, and lungs dominated the targeted body regions of interest.Insufficient public information on validation datasets in several FDA-regulated AI/ML algorithms makes it difficult to justify clinical applications since their generalizability and presence of bias cannot be inferred.

Background: Missed findings in chest X-ray interpretation are common and can have serious consequences. Methods: Our study included 2407 chest radiographs (CXRs) acquired at three Indian and five US sites. To identify CXRs reported as normal, we used a proprietary radiology report search engine based on natural language processing (mPower, Nuance). Two thoracic radiologists reviewed all CXRs and recorded the presence and clinical significance of abnormal findings on a 5-point scale (1—not important; 5—critical importance). All CXRs were processed with the AI model (Qure.ai) and outputs were recorded for the presence of findings. Data were analyzed to obtain area under the ROC curve (AUC). Results: Of 410 CXRs (410/2407, 18.9%) with unreported/missed findings, 312 (312/410, 76.1%) findings were clinically important: pulmonary nodules (n = 157), consolidation (60), linear opacities (37), mediastinal widening (21), hilar enlargement (17), pleural effusions (11), rib fractures (6) and pneumothoraces (3). AI detected 69 missed findings (69/131, 53%) with an AUC of up to 0.935. The AI model was generalizable across different sites, geographic locations, patient genders and age groups. Conclusion: A substantial number of important CXR findings are missed; the AI model can help to identify and reduce the frequency of important missed findings in a generalizable manner.

Purpose: To improve and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. Materials and Methods: A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from four test sets, including 3 from the United States (patients hospitalized at an academic medical center (N=154), patients hospitalized at a community hospital (N=113), and outpatients (N=108)) and 1 from Brazil (patients at an academic medical center emergency department (N=303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson r). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. Results: Tuning the deep learning model with outpatient data improved model performance in two United States hospitalized patient datasets (r=0.88 and r=0.90, compared to baseline r=0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (r=0.86 and r=0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. Conclusions: Performance of a deep learning-based model that extracts a COVID-19 severity score on CXRs improved using training data from a different patient cohort (outpatient versus hospitalized) and generalized across multiple populations.