Search

Background:  Clinical Decision Support (CDS) tools have a mixed record of effectiveness, often due to inadequate alignment with clinical workflows and poor usability. While there is a consensus that usability testing methods address these issues, in practice, usability testing is generally only used for selected projects (such as funded research studies). There is a critical need for CDS operations to apply usability testing to all CDS implementations., Objectives:  In this State of the Art/Best Practice paper, we share challenges with scaling usability in health care operations and alternative methods and CDS governance structures to enable usability testing as a routine practice., Methods:  We coalesce our experience and results of applying guerilla in situ usability testing to over 20 projects in a 1-year period with the proposed solution., Results:  We demonstrate the feasibility of adopting " guerilla in situ usability testing" in operations and their effectiveness in incorporating user feedback and improving design., Conclusion:  Although some methodological rigor was relaxed to accommodate operational speed, the benefits outweighed the limitations. Broader adoption of usability testing may transform CDS implementation and improve health outcomes., Competing Interests: E.O. and N.M. are the cofounders and have equity in Phrase Health, a CDS analytics company. They are the Investigators on an R42 grant with Phrase Health from the National Library of Medicine (NLM) and the National Center for Advancing Translational Science (NCATS). Both of them receive salary support from the NLM and NCATS but no direct revenue from Phrase Health. Other authors have nothing to disclose., (Thieme. All rights reserved.)

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record., Competing Interests: None declared., (Thieme. All rights reserved.)

Background: Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process., Objectives: There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD., Methods: A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t -tests., Results: Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9-57%; p  = 0.03). Time on task did not vary significantly., Conclusion: UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design., Competing Interests: A.B.C. reports personal fees from American Medical Informatics Association, outside the submitted work. N.M. reports equity in Phrase Health, a clinical decision support analytics company. E.W.O. reports other from Phrase Health, outside the submitted work. J.B. has nothing to disclose. M.R. has nothing to disclose., (Georg Thieme Verlag KG Stuttgart · New York.)

Clinical decision support (CDS) systems delivered through the electronic health record are an important element of quality and safety initiatives within a health care system. However, managing a large CDS knowledge base can be an overwhelming task for informatics teams. Additionally, it can be difficult for these informatics teams to communicate their goals with external operational stakeholders and define concrete steps for improvement. We aimed to develop a maturity model that describes a roadmap toward organizational functions and processes that help health care systems use CDS more effectively to drive better outcomes. We developed a maturity model for CDS operations through discussions with health care leaders at 80 organizations, iterative model development by four clinical informaticists, and subsequent review with 19 health care organizations. We ceased iterations when feedback from three organizations did not result in any changes to the model. The proposed CDS maturity model includes three main "pillars": "Content Creation," "Analytics and Reporting," and "Governance and Management." Each pillar contains five levels-advancing along each pillar provides CDS teams a deeper understanding of the processes CDS systems are intended to improve. A "roof" represents the CDS functions that become attainable after advancing along each of the pillars. Organizations are not required to advance in order and can develop in one pillar separately from another. However, we hypothesize that optimal deployment of preceding levels and advancing in tandem along the pillars increase the value of organizational investment in higher levels of CDS maturity. In addition to describing the maturity model and its development, we also provide three case studies of health care organizations using the model for self-assessment and determine next steps in CDS development., Competing Interests: M.Z. and M.C.T. have equity in and are employees of Phrase Health, a health care governance and analytics company. E.W.O., N.M., and D.F. have equity in Phrase Health, a health care governance and analytics company. They receive no direct revenue. A.O.W., E.S., and J.S. declare no competing interests., (Georg Thieme Verlag KG Stuttgart · New York.)

Background: Surveillance for surgical site infections (SSIs) after ambulatory surgery in children requires a detailed manual chart review to assess criteria defined by the National Health and Safety Network (NHSN). Electronic health records (EHRs) impose an inefficient search process where infection preventionists must manually review every postsurgical encounter (< 30 days). Using text mining and business intelligence software, we developed an information foraging application, the SSI Workbench, to visually present which postsurgical encounters included SSI-related terms and synonyms, antibiotic, and culture orders., Objective: This article compares the Workbench and EHR on four dimensions: (1) effectiveness, (2) efficiency, (3) workload, and (4) usability., Methods: Comparative usability test of Workbench and EHR. Objective test metrics are time per case, encounters reviewed per case, time per encounter, and retrieval of information meeting NHSN definitions. Subjective measures are cognitive load using the National Aeronautics and Space Administration (NASA) Task Load Index (NASA TLX), and a questionnaire on system usability and utility., Results: Eight infection preventionists participated in the test. There was no difference in effectiveness as subjects retrieved information from all cases, using both systems, to meet the NHSN criteria. There was no difference in efficiency in time per case between the Workbench and EHR (8.58 vs. 7.39 minutes, p  = 0.36). However, with the Workbench subjects opened fewer encounters per case (3.0 vs. 7.5, p  = 0.002), spent more time per encounter (2.23 vs. 0.92 minutes, p  = 0.002), rated the Workbench lower in cognitive load (NASA TLX, 24 vs. 33, p  = 0.02), and significantly higher in measures of usability., Conclusion: Compared with the EHR, the Workbench was more usable, short, and reduced cognitive load. In overall efficiency, the Workbench did not save time, but demonstrated a shift from between-encounter foraging to within-encounter foraging and was rated as significantly more efficient. Our results suggest that infection surveillance can be better supported by systems applying information foraging theory., Competing Interests: None., (Georg Thieme Verlag KG Stuttgart · New York.)

Background and Objective: Pediatric in-hospital cardiac arrest most commonly occurs in the pediatric intensive care unit (PICU) and is frequently preceded by early warning signs of clinical deterioration. In this study, we describe the implementation and evaluation of criteria to identify high-risk patients from a paper-based checklist into a clinical decision support (CDS) tool in the electronic health record (EHR)., Materials and Methods: The validated paper-based tool was first adapted by PICU clinicians and clinical informaticians and then integrated into clinical workflow following best practices for CDS design. A vendor-based rule engine was utilized. Littenberg's assessment framework helped guide the overall evaluation. Preliminary testing took place in EHR development environments with more rigorous evaluation, testing, and feedback completed in the live production environment. To verify data quality of the CDS rule engine, a retrospective Structured Query Language (SQL) data query was also created. As a process metric, preparedness was measured in pre- and postimplementation surveys., Results: The system was deployed, evaluating approximately 340 unique patients monthly across 4 clinical teams. The verification against retrospective SQL of 15-minute intervals over a 30-day period revealed no missing triggered intervals and demonstrated 99.3% concordance of positive triggers. Preparedness showed improvements across multiple domains to our a priori goal of 90%., Conclusion: We describe the successful adaptation and implementation of a real-time CDS tool to identify PICU patients at risk of deterioration. Prospective multicenter evaluation of the tool's effectiveness on clinical outcomes is necessary before broader implementation can be recommended., Competing Interests: None., (Georg Thieme Verlag KG Stuttgart · New York.)

Background: Outpatient providers often do not receive discharge summaries from acute care providers prior to follow-up visits. These outpatient providers may use the after-visit summaries (AVS) that are given to patients to obtain clinical information. It is unclear how effectively AVS support care coordination between clinicians., Objectives: Goals for this effort include: (1) developing usability heuristics that may be applied both for assessment and to guide generation of medical documents in general, (2) conducting a heuristic evaluation to assess the use of AVS for communication between clinicians, and (3) providing recommendations for generating AVS that effectively support both patient/caregiver use and care coordination., Methods: We created a 17-item heuristic evaluation instrument for assessing usability of medical documents. Eight experts used the instrument to assess each of four simulated AVS. The simulations were created using examples from two hospitals and two pediatric patient cases developed by the National Institute of Standards and Technology., Results: Experts identified 224 unique usability problems ranging in severity from mild to catastrophic. Content issues (e.g., missing medical history, marital status of a 2-year-old) were rated as most severe, but widespread formatting and structural problems (e.g., inconsistent indentation, fonts, and headings; confusing ordering of information) were so distracting that they significantly reduced readers' ability to efficiently use the documents. Overall, issues in the AVS from Hospital 2 were more severe than those in the AVS from Hospital 1., Conclusion: The new instrument allowed for quick, inexpensive evaluations of AVS. Usability issues such as unnecessary information, poor organization, missing information, and inconsistent formatting make it hard for patients, caregivers, and clinicians to use the AVS. The heuristics in the new instrument may be used as guidance to adapt electronic health record systems so that they generate more useful and usable medical documents., Competing Interests: None., (Georg Thieme Verlag KG Stuttgart · New York.)