Search

Background: Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging., Objectives: This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients., Materials and Methods: In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process., Results: Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided., Discussion: AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow., Conclusions: Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)

Purpose: A critical gap in the adoption of genomic medicine into medical practice is the need for the rigorous evaluation of the utility of genomic medicine interventions., Methods: The Implementing Genomics in Practice Pragmatic Trials Network (IGNITE PTN) was formed in 2018 to measure the clinical utility and cost-effectiveness of genomic medicine interventions, to assess approaches for real-world application of genomic medicine in diverse clinical settings, and to produce generalizable knowledge on clinical trials using genomic interventions. Five clinical sites and a coordinating center evaluated trial proposals and developed working groups to enable their implementation., Results: Two pragmatic clinical trials (PCTs) have been initiated, one evaluating genetic risk APOL1 variants in African Americans in the management of their hypertension, and the other to evaluate the use of pharmacogenetic testing for medications to manage acute and chronic pain as well as depression., Conclusion: IGNITE PTN is a network that carries out PCTs in genomic medicine; it is focused on diversity and inclusion of underrepresented minority trial participants; it uses electronic health records and clinical decision support to deliver the interventions. IGNITE PTN will develop the evidence to support (or oppose) the adoption of genomic medicine interventions by patients, providers, and payers.

The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS., (© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Objective: Clinical decision support (CDS) tools have been shown to reduce inappropriate imaging orders. We hypothesized that CDS may be especially effective for house staff physicians who are prone to overuse of resources., Materials and Methods: Our hospital implemented CDS for CT and MRI orders in the emergency department with scores based on the American College of Radiology's Appropriateness Criteria (range, 1-9; higher scores represent more-appropriate orders). Data on CT and MRI orders from April 2013 through June 2016 were categorized as pre-CDS or baseline, post-CDS period 1 (i.e., intervention with active feedback for scores of ≤ 4), and post-CDS period 2 (i.e., intervention with active feedback for scores of ≤ 6). Segmented regression analysis with interrupted time series data estimated changes in scores stratified by house staff and non-house staff. Generalized linear models further estimated the modifying effect of the house staff variable., Results: Mean scores were 6.2, 6.2, and 6.7 in the pre-CDS, post-CDS 1, and post-CDS 2 periods, respectively (p < 0.05). In the segmented regression analysis, mean scores significantly (p < 0.05) increased when comparing pre-CDS versus post-CDS 2 periods for both house staff (baseline increase, 0.41; 95% CI, 0.17-0.64) and non-house staff (baseline increase, 0.58; 95% CI, 0.34-0.81), showing no differences in effect between the cohorts. The generalized linear model showed significantly higher scores, particularly in the post-CDS 2 period compared with the pre-CDS period (0.44 increase in scores; p < 0.05). The house staff variable did not significantly change estimates in the post-CDS 2 period., Conclusion: Implementation of active CDS increased overall scores of CT and MRI orders. However, there was no significant difference in effect on scores between house staff and non-house staff.

Background: Older adults are at risk for inadequate emergency department (ED) pain care. Unrelieved acute pain is associated with poor outcomes. Clinical decision support systems (CDSS) hold promise to improve patient care, but CDSS quality varies widely, particularly when usability evaluation is not employed., Objective: To conduct an iterative usability and redesign process of a novel geriatric abdominal pain care CDSS. We hypothesized this process would result in the creation of more usable and favorable pain care interventions., Methods: Thirteen emergency physicians familiar with the Electronic Health Record (EHR) in use at the study site were recruited. Over a 10-week period, 17 1-hour usability test sessions were conducted across 3 rounds of testing. Participants were given 3 patient scenarios and provided simulated clinical care using the EHR, while interacting with the CDSS interventions. Quantitative System Usability Scores (SUS), favorability scores and qualitative narrative feedback were collected for each session. Using a multi-step review process by an interdisciplinary team, positive and negative usability issues in effectiveness, efficiency, and satisfaction were considered, prioritized and incorporated in the iterative redesign process of the CDSS. Video analysis was used to determine the appropriateness of the CDS appearances during simulated clinical care., Results: Over the 3 rounds of usability evaluations and subsequent redesign processes, mean SUS progressively improved from 74.8 to 81.2 to 88.9; mean favorability scores improved from 3.23 to 4.29 (1 worst, 5 best). Video analysis revealed that, in the course of the iterative redesign processes, rates of physicians' acknowledgment of CDS interventions increased, however most rates of desired actions by physicians (such as more frequent pain score updates) decreased., Conclusion: The iterative usability redesign process was instrumental in improving the usability of the CDSS; if implemented in practice, it could improve geriatric pain care. The usability evaluation process led to improved acknowledgement and favorability. Incorporating usability testing when designing CDSS interventions for studies may be effective to enhance clinician use.

Background: Dissemination and adoption of clinical decision support (CDS) tools is a major initiative of the Affordable Care Act's Meaningful Use program. Adoption of CDS tools is multipronged with personal, organizational, and clinical settings factoring into the successful utilization rates. Specifically, the diffusion of innovation theory implies that 'early adopters' are more inclined to use CDS tools and younger physicians tend to be ranked in this category., Objective: This study examined the differences in adoption of CDS tools across providers' training level., Participants: From November 2010 to 2011, 168 residents and attendings from an academic medical institution were enrolled into a randomized controlled trial., Intervention: The intervention arm had access to the CDS tool through the electronic health record (EHR) system during strep and pneumonia patient visits., Main Measures: The EHR system recorded details on how intervention arm interacted with the CDS tool including acceptance of the initial CDS alert, completion of risk-score calculators and the signing of medication order sets. Using the EHR data, the study performed bivariate tests and general estimating equation (GEE) modeling to examine the differences in adoption of the CDS tool across residents and attendings., Key Results: The completion rates of the CDS calculator and medication order sets were higher amongst first year residents compared to all other training levels. Attendings were the less likely to accept the initial step of the CDS tool (29.3%) or complete the medication order sets (22.4%) that guided their prescription decisions, resulting in attendings ordering more antibiotics (37.1%) during an CDS encounter compared to residents., Conclusion: There is variation in adoption of CDS tools across training levels. Attendings tended to accept the tool less but ordered more medications. CDS tools should be tailored to clinicians' training levels.

Objective: To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools., Materials and Methods: Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders., Results: The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively)., Discussion: Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering., Conclusions: These results highlight the importance of workflow integration and flexibility for CDS success.