Search

Artificial intelligence (AI) tools are rapidly being developed for radiology and other clinical areas. These tools have the potential to dramatically change clinical practice; however, for these tools to be usable and function as intended, they must be integrated into existing radiology systems. In a collaborative effort between the Radiological Society of North America, radiologists, and imaging-focused vendors, the Imaging AI in Practice (IAIP) demonstrations were developed to show how AI tools can generate, consume, and present results throughout the radiology workflow in a simulated clinical environment. The IAIP demonstrations highlight the critical importance of semantic and interoperability standards, as well as orchestration profiles for successful clinical integration of radiology AI tools. Keywords: Computer Applications-General (Informatics), Technology Assessment © RSNA, 2021.

The clinical deployment of artificial intelligence (AI) applications in medical imaging is perhaps the greatest challenge facing radiology in the next decade. One of the main obstacles to the incorporation of automated AI-based decision-making tools in medicine is the failure of models to generalize when deployed across institutions with heterogeneous populations and imaging protocols. The most well-understood pitfall in developing these AI models is overfitting, which has, in part, been overcome by optimizing training protocols. However, overfitting is not the only obstacle to the success and generalizability of AI. Underspecification is also a serious impediment that requires conceptual understanding and correction. It is well known that a single AI pipeline, with prescribed training and testing sets, can produce several models with various levels of generalizability. Underspecification defines the inability of the pipeline to identify whether these models have embedded the structure of the underlying system by using a test set independent of, but distributed identically, to the training set. An underspecified pipeline is unable to assess the degree to which the models will be generalizable. Stress testing is a known tool in AI that can limit underspecification and, importantly, assure broad generalizability of AI models. However, the application of stress tests is new in radiologic applications. This report describes the concept of underspecification from a radiologist perspective, discusses stress testing as a specific strategy to overcome underspecification, and explains how stress tests could be designed in radiology-by modifying medical images or stratifying testing datasets. In the upcoming years, stress tests should become in radiology the standard that crash tests have become in the automotive industry. Keywords: Computer Applications-General, Informatics, Computer-aided Diagnosis © RSNA, 2021., Competing Interests: Disclosures of conflicts of interest: T.E. No relevant relationships. L.H.S. Consultant for Merck and Regeneron (member, DSMB and endpoint analysis committees). F.Z.M. No relevant relationships. L.D. No relevant relationships., (2021 by the Radiological Society of North America, Inc.)