Search

AIMS To evaluate the performance of the WATCH-DM risk score, a clinical risk score for heart failure (HF), in patients with dysglycaemia and in combination with natriuretic peptides (NPs). METHODS AND RESULTS Adults with diabetes/pre-diabetes free of HF at baseline from four cohort studies (ARIC, CHS, FHS, and MESA) were included. The machine learning- [WATCH-DM(ml)] and integer-based [WATCH-DM(i)] scores were used to estimate the 5-year risk of incident HF. Discrimination was assessed by Harrell's concordance index (C-index) and calibration by the Greenwood-Nam-D'Agostino (GND) statistic. Improvement in model performance with the addition of NP levels was assessed by C-index and continuous net reclassification improvement (NRI). Of the 8938 participants included, 3554 (39.8%) had diabetes and 432 (4.8%) developed HF within 5 years. The WATCH-DM(ml) and WATCH-DM(i) scores demonstrated high discrimination for predicting HF risk among individuals with dysglycaemia (C-indices = 0.80 and 0.71, respectively), with no evidence of miscalibration (GND P ≥0.10). The C-index of elevated NP levels alone for predicting incident HF among individuals with dysglycaemia was significantly higher among participants with low/intermediate (

Introduction: The WATCH-DM score can predict risk of heart failure (HF) in patients with diabetes. Hypothesis: Addition of natriuretic peptide (NP) levels will improve WATCH-DM performance in individuals with dysglycemia. Methods: Adults with diabetes/pre-diabetes free of HF at baseline from 4 cohort studies (ARIC, CHS, FHS, and MESA) were included. The integer- [WATCH-DM(i)] and machine learning-based [WATCH-DM(ml)] scores were used to estimate the 5-year risk of incident HF. Discrimination was assessed by Harrell's concordance index (C-index) and calibration by the Greenwood-Nam-D'Agostino (GND) statistic. Improvement in model performance with the addition of NP-levels was assessed by C-index, Brier score, and continuous net reclassification improvement (NRI). Results: Of the 8,938 participants included, 3,554 (39.8%) had diabetes and 432 (4.8%) developed HF within 5-years. Among 5,384 (60.2%) participants with pre-diabetes, 647 (12.0%) developed incident HF. The WATCH-DM(ml) and (i) scores demonstrated high discrimination for predicting HF risk in diabetes (C-indices=0.76 and 0.69), pre-diabetes (0.83 and 0.72), and overall cohort (0.80 and 0.71), respectively, with no evidence of miscalibration (GND=P >0.10). A greater improvement in C-index was observed with the addition of NP-levels at lower WATCH-DM(i) scores with degradation of risk discrimination at higher scores (Fig. A). Calibration was also improved with addition of NP-levels at lower compared to higher WATCH-DM(i) scores (Fig. B). A greater improvement in reclassification was observed by combing WATCH-DM(i) score with selected NP-levels assessment in low (score Conclusions: The WATCH-DM risk score can accurately predict incident HF risk in community-based individuals with dysglycemia. The addition of NP-levels improves risk prediction among adults with low/intermediate but not high HF risk.

ObjectiveWe sought to demonstrate applicability of user stories, progressively elaborated by testable acceptance criteria, as lightweight requirements for agile development of clinical decision support (CDS).Materials and MethodsUser stories employed the template: As a [type of user], I want [some goal] so that [some reason]. From the “so that” section, CDS benefit measures were derived. Detailed acceptance criteria were elaborated through ensuing conversations. We estimated user story size with “story points,” and depicted multiple user stories with a use case diagram or feature breakdown structure. Large user stories were split to fit into 2-week iterations.ResultsOne example user story was: As a rheumatologist, I want to be advised if my patient with rheumatoid arthritis is not on a disease-modifying anti-rheumatic drug (DMARD), so that they receive optimal therapy and can experience symptom improvement. This yielded a process measure (DMARD use), and an outcome measure (Clinical Disease Activity Index). Following implementation, the DMARD nonuse rate decreased from 3.7% to 1.4%. Patients with a high Clinical Disease Activity Index improved from 13.7% to 7%. For a thromboembolism prevention CDS project, diagrams organized multiple user stories.DiscussionUser stories written in the clinician’s voice aid CDS governance and lead naturally to measures of CDS effectiveness. Estimation of relative story size helps plan CDS delivery dates. User stories prove to be practical even on larger projects.ConclusionsUser stories concisely communicate the who, what, and why of a CDS request, and serve as lightweight requirements for agile development to meet the demand for increasingly diverse CDS.

Objective Determine whether women and men differ in volunteering to join a Research Recruitment Registry when invited to participate via an electronic patient portal without human bias. Materials and Methods Under-representation of women and other demographic groups in clinical research studies could be due either to invitation bias (explicit or implicit) during screening and recruitment or by lower rates of deciding to participate when offered. By making an invitation to participate in a Research Recruitment Registry available to all patients accessing our patient portal, regardless of demographics, we sought to remove implicit bias in offering participation and thus independently assess agreement rates. Results Women were represented in the Research Recruitment Registry slightly more than their proportion of all portal users (n = 194 775). Controlling for age, race, ethnicity, portal use, chronic disease burden, and other questionnaire use, women were statistically more likely to agree to join the Registry than men (odds ratio 1.17, 95% CI, 1.12–1.21). In contrast, Black males, Hispanics (of both sexes), and particularly Asians (both sexes) had low participation-to-population ratios; this under-representation persisted in the multivariable regression model. Discussion This supports the view that historical under-representation of women in clinical studies is likely due, at least in part, to implicit bias in offering participation. Distinguishing the mechanism for under-representation could help in designing strategies to improve study representation, leading to more effective evidence-based recommendations. Conclusion Patient portals offer an attractive option for minimizing bias and encouraging broader, more representative participation in clinical research.

Background Professional society guidelines are emerging for cardiovascular care in cancer patients. However, it is not yet clear how effectively the cancer survivor population is screened and treated for cardiomyopathy in contemporary clinical practice. As electronic health records (EHRs) are now widely used in clinical practice, we tested the hypothesis that an EHR-based cardio-oncology registry can address these questions. Objective The aim of this study was to develop an EHR-based pragmatic cardio-oncology registry and, as proof of principle, to investigate care gaps in the cardiovascular care of cancer patients. Methods We generated a programmatically deidentified, real-time EHR-based cardio-oncology registry from all patients in our institutional Cancer Population Registry (N=8275, 2011-2017). We investigated: (1) left ventricular ejection fraction (LVEF) assessment before and after treatment with potentially cardiotoxic agents; and (2) guideline-directed medical therapy (GDMT) for left ventricular dysfunction (LVD), defined as LVEF Results Rapid development of an EHR-based cardio-oncology registry was feasible. Identification of tests and outcomes was similar using the EHR-based cardio-oncology registry and manual chart abstraction (100% sensitivity and 83% specificity for LVD). LVEF was documented prior to initiation of cancer therapy in 19.8% of patients. Prevalence of postchemotherapy LVD and HFrEF was relatively low (9.4% and 2.5%, respectively). Among patients with postchemotherapy LVD or HFrEF, those referred to cardiology had a significantly higher prescription rate of a GDMT. Conclusions EHR data can efficiently populate a real-time, pragmatic cardio-oncology registry as a byproduct of clinical care for health care delivery investigations.

BACKGROUND Professional society guidelines are emerging for cardiovascular care in cancer patients. However, it is not yet clear how effectively the cancer survivor population is screened and treated for cardiomyopathy in contemporary clinical practice. As electronic health records (EHRs) are now widely used in clinical practice, we tested the hypothesis that an EHR-based cardio-oncology registry can address these questions. OBJECTIVE The aim of this study was to develop an EHR-based pragmatic cardio-oncology registry and, as proof of principle, to investigate care gaps in the cardiovascular care of cancer patients. METHODS We generated a programmatically deidentified, real-time EHR-based cardio-oncology registry from all patients in our institutional Cancer Population Registry (N=8275, 2011-2017). We investigated: (1) left ventricular ejection fraction (LVEF) assessment before and after treatment with potentially cardiotoxic agents; and (2) guideline-directed medical therapy (GDMT) for left ventricular dysfunction (LVD), defined as LVEF RESULTS Rapid development of an EHR-based cardio-oncology registry was feasible. Identification of tests and outcomes was similar using the EHR-based cardio-oncology registry and manual chart abstraction (100% sensitivity and 83% specificity for LVD). LVEF was documented prior to initiation of cancer therapy in 19.8% of patients. Prevalence of postchemotherapy LVD and HFrEF was relatively low (9.4% and 2.5%, respectively). Among patients with postchemotherapy LVD or HFrEF, those referred to cardiology had a significantly higher prescription rate of a GDMT. CONCLUSIONS EHR data can efficiently populate a real-time, pragmatic cardio-oncology registry as a byproduct of clinical care for health care delivery investigations.

OBJECTIVE To develop and validate a novel, machine learning–derived model to predict the risk of heart failure (HF) among patients with type 2 diabetes mellitus (T2DM). RESEARCH DESIGN AND METHODS Using data from 8,756 patients free at baseline of HF, with RESULTS Over a median follow-up of 4.9 years, 319 patients (3.6%) developed incident HF. The RSF models demonstrated better discrimination than the best performing Cox-based method (C-index 0.77 [95% CI 0.75–0.80] vs. 0.73 [0.70–0.76] respectively) and had acceptable calibration (Hosmer-Lemeshow statistic χ2 = 9.63, P = 0.29) in the internal validation data set. From the identified predictors, an integer-based risk score for 5-year HF incidence was created: the WATCH-DM (Weight [BMI], Age, hyperTension, Creatinine, HDL-C, Diabetes control [fasting plasma glucose], QRS Duration, MI, and CABG) risk score. Each 1-unit increment in the risk score was associated with a 24% higher relative risk of HF within 5 years. The cumulative 5-year incidence of HF increased in a graded fashion from 1.1% in quintile 1 (WATCH-DM score ≤7) to 17.4% in quintile 5 (WATCH-DM score ≥14). In the external validation cohort, the RSF-based risk prediction model and the WATCH-DM risk score performed well with good discrimination (C-index = 0.74 and 0.70, respectively), acceptable calibration (P ≥0.20 for both), and broad risk stratification (5-year HF risk range from 2.5 to 18.7% across quintiles 1–5). CONCLUSIONS We developed and validated a novel, machine learning–derived risk score that integrates readily available clinical, laboratory, and electrocardiographic variables to predict the risk of HF among outpatients with T2DM.

Background Defining clinical conditions from electronic health record (EHR) data underpins population health activities, clinical decision support, and analytics. In an EHR, defining a condition commonly employs a diagnosis value set or “grouper.” For constructing value sets, Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT) offers high clinical fidelity, a hierarchical ontology, and wide implementation in EHRs as the standard interoperability vocabulary for problems. Objective This article demonstrates a practical approach to defining conditions with combinations of SNOMED CT concept hierarchies, and evaluates sharing of definitions for clinical and analytic uses. Methods We constructed diagnosis value sets for EHR patient registries using SNOMED CT concept hierarchies combined with Boolean logic, and shared them for clinical decision support, reporting, and analytic purposes. Results A total of 125 condition-defining “standard” SNOMED CT diagnosis value sets were created within our EHR. The median number of SNOMED CT concept hierarchies needed was only 2 (25th–75th percentiles: 1–5). Each value set, when compiled as an EHR diagnosis grouper, was associated with a median of 22 International Classification of Diseases (ICD)-9 and ICD-10 codes (25th–75th percentiles: 8–85) and yielded a median of 155 clinical terms available for selection by clinicians in the EHR (25th–75th percentiles: 63–976). Sharing of standard groupers for population health, clinical decision support, and analytic uses was high, including 57 patient registries (with 362 uses of standard groupers), 132 clinical decision support records, 190 rules, 124 EHR reports, 125 diagnosis dimension slicers for self-service analytics, and 111 clinical quality measure calculations. Identical SNOMED CT definitions were created in an EHR-agnostic tool enabling application across disparate organizations and EHRs. Conclusion SNOMED CT-based diagnosis value sets are simple to develop, concise, understandable to clinicians, useful in the EHR and for analytics, and shareable. Developing curated SNOMED CT hierarchy-based condition definitions for public use could accelerate cross-organizational population health efforts, “smarter” EHR feature configuration, and clinical–translational research employing EHR-derived data.

Constructing multi-site specialty registries typically proves time-consuming. Electronic health record (EHR) data collected during clinical care affords a pragmatic approach to accelerating registry implementation. Heart failure with preserved ejection fraction (HFpEF) is an increasingly common and morbid condition. Building a multi-site registry for HFpEF proved feasible using EHR data coded in standard terminologies (SNOMED CT, LOINC) and shared via Health Information Exchanges.

Background: Defining clinical phenotypes from electronic health record (EHR)–derived data proves crucial for clinical decision support, population health endeavors, and translational research. EHR diagnoses now commonly draw from a finely grained clinical terminology—either native SNOMED CT or a vendor-supplied terminology mapped to SNOMED CT concepts as the standard for EHR interoperability. Accordingly, electronic clinical quality measures (eCQMs) increasingly define clinical phenotypes with SNOMED CT value sets. The work of creating and maintaining list-based value sets proves daunting, as does insuring that their contents accurately represent the clinically intended condition. Objective: The goal of the research was to compare an intensional (concept hierarchy-based) versus extensional (list-based) value set approach to defining clinical phenotypes using SNOMED CT–encoded data from EHRs by evaluating value set conciseness, time to create, and completeness. Methods: Starting from published Centers for Medicare and Medicaid Services (CMS) high-priority eCQMs, we selected 10 clinical conditions referenced by those eCQMs. For each, the published SNOMED CT list-based (extensional) value set was downloaded from the Value Set Authority Center (VSAC). Ten corresponding SNOMED CT hierarchy-based intensional value sets for the same conditions were identified within our EHR. From each hierarchy-based intensional value set, an exactly equivalent full extensional value set was derived enumerating all included descendant SNOMED CT concepts. Comparisons were then made between (1) VSAC-downloaded list-based (extensional) value sets, (2) corresponding hierarchy-based intensional value sets for the same conditions, and (3) derived list-based (extensional) value sets exactly equivalent to the hierarchy-based intensional value sets. Value set conciseness was assessed by the number of SNOMED CT concepts needed for definition. Time to construct the value sets for local use was measured. Value set completeness was assessed by comparing contents of the downloaded extensional versus intensional value sets. Two measures of content completeness were made: for individual SNOMED CT concepts and for the mapped diagnosis clinical terms available for selection within the EHR by clinicians. Results: The 10 hierarchy-based intensional value sets proved far simpler and faster to construct than exactly equivalent derived extensional value set lists, requiring a median 3 versus 78 concepts to define and 5 versus 37 minutes to build. The hierarchy-based intensional value sets also proved more complete: in comparison, the 10 downloaded 2018 extensional value sets contained a median of just 35% of the intensional value sets’ SNOMED CT concepts and 65% of mapped EHR clinical terms. Conclusions: In the EHR era, defining conditions preferentially should employ SNOMED CT concept hierarchy-based (intensional) value sets rather than extensional lists. By doing so, clinical guideline and eCQM authors can more readily engage specialists in vetting condition subtypes to include and exclude, and streamline broad EHR implementation of condition-specific decision support promoting guideline adherence for patient benefit.

Even the most innovative healthcare technologies provide patient benefits only when adopted by clinicians and/or patients in actual practice. Yet realizing optimal positive impact from a new technology for the widest range of individuals who would benefit remains elusive. In software and new product development, iterative rapid-cycle “agile” methods more rapidly provide value, mitigate failure risks, and adapt to customer feedback. Co-development between builders and customers is a key agile principle. But how does one accomplish co-development with busy clinicians? In this paper, we discuss four practical agile co-development practices found helpful clinically: (1) User stories for lightweight requirements; (2) Time-boxed development for collaborative design and prompt course correction; (3) Automated acceptance test driven development, with clinician-vetted specifications; and (4) Monitoring of clinician interactions after release, for rapid-cycle product adaptation and evolution. In the coming wave of innovation in healthcare apps ushered in by open APIs to EHRs, learning rapidly what new product features work well for clinicians and patients will become even more crucial.

Designing effective Clinical Decision Support (CDS) tools in an Electronic Health Record (EHR) can prove challenging, due to complex real-world scenarios and newly-discovered requirements. Deploying new CDS tools shares much in common with new product development, where "agile" principles and practices consistently prove effective. Agile methods can thus prove helpful on CDS projects, including time-boxed "sprints" and lightweight requirements gathering with User Stories. Modeling CDS behavior promotes unambiguous shared understanding of desired behavior, but risks analysis paralysis: an Agile Modeling approach can foster effective rapid-cycle CDS design and optimization. The agile practice of automated testing for test-driven design and regression testing can be applied to CDS development using open-source tools. Ongoing monitoring of CDS behavior once released to production can identify anomalies and prompt rapid-cycle redesign to further enhance CDS effectiveness. The tutorial participant will learn about these topics in interactive didactic sessions, with time for practicing the techniques taught.

Transitioning to value-based care makes new demands on understanding and managing patient risk for a variety of adverse outcomes in multiple conditions. Optimizing use of finite healthcare resources then proves challenging, and would benefit from a data-driven approach. Modelling the "risk triangle" paradigm of disease management as a state diagram within the electronic health record helps bring clinical situational awareness and tailored decision support interventions to individual patients at the point-of-care, while automatically capturing new types of state duration and transition sequence data across the whole population. Such data can iteratively inform improving risk prediction models.

BACKGROUND Defining clinical phenotypes from electronic health record (EHR)–derived data proves crucial for clinical decision support, population health endeavors, and translational research. EHR diagnoses now commonly draw from a finely grained clinical terminology—either native SNOMED CT or a vendor-supplied terminology mapped to SNOMED CT concepts as the standard for EHR interoperability. Accordingly, electronic clinical quality measures (eCQMs) increasingly define clinical phenotypes with SNOMED CT value sets. The work of creating and maintaining list-based value sets proves daunting, as does insuring that their contents accurately represent the clinically intended condition. OBJECTIVE The goal of the research was to compare an intensional (concept hierarchy-based) versus extensional (list-based) value set approach to defining clinical phenotypes using SNOMED CT–encoded data from EHRs by evaluating value set conciseness, time to create, and completeness. METHODS Starting from published Centers for Medicare and Medicaid Services (CMS) high-priority eCQMs, we selected 10 clinical conditions referenced by those eCQMs. For each, the published SNOMED CT list-based (extensional) value set was downloaded from the Value Set Authority Center (VSAC). Ten corresponding SNOMED CT hierarchy-based intensional value sets for the same conditions were identified within our EHR. From each hierarchy-based intensional value set, an exactly equivalent full extensional value set was derived enumerating all included descendant SNOMED CT concepts. Comparisons were then made between (1) VSAC-downloaded list-based (extensional) value sets, (2) corresponding hierarchy-based intensional value sets for the same conditions, and (3) derived list-based (extensional) value sets exactly equivalent to the hierarchy-based intensional value sets. Value set conciseness was assessed by the number of SNOMED CT concepts needed for definition. Time to construct the value sets for local use was measured. Value set completeness was assessed by comparing contents of the downloaded extensional versus intensional value sets. Two measures of content completeness were made: for individual SNOMED CT concepts and for the mapped diagnosis clinical terms available for selection within the EHR by clinicians. RESULTS The 10 hierarchy-based intensional value sets proved far simpler and faster to construct than exactly equivalent derived extensional value set lists, requiring a median 3 versus 78 concepts to define and 5 versus 37 minutes to build. The hierarchy-based intensional value sets also proved more complete: in comparison, the 10 downloaded 2018 extensional value sets contained a median of just 35% of the intensional value sets’ SNOMED CT concepts and 65% of mapped EHR clinical terms. CONCLUSIONS In the EHR era, defining conditions preferentially should employ SNOMED CT concept hierarchy-based (intensional) value sets rather than extensional lists. By doing so, clinical guideline and eCQM authors can more readily engage specialists in vetting condition subtypes to include and exclude, and streamline broad EHR implementation of condition-specific decision support promoting guideline adherence for patient benefit.

For patients with a chronic disease such as prostate cancer, their possible journeys through treatment can be mapped as a state diagram, which now can be implemented as an electronic health record (EHR) Care Path, generating novel data for analysis and visualization. A prostate cancer Problem List form captured treatment path assignment, treatment response, and recurrence. Patients reported their symptom burden via the Expanded Prostate Cancer Index Composite (EPIC) questionnaire, completed by patients at defined intervals either at home via mobile device or computer, or in clinic on a tablet. Patients move through the Care Path via state transitions triggered automatically via rule. New types of EHR data on each patient's journey–pathway sequence and time-in-state–automatically ensue, enabling novel analyses. In the first 3 months after go-live, 408 patients were being actively managed on the Care Path. Data visualizations display not only each individual patient's journey through the system but also (using R) an aggregated view of the patterns of all patients' journeys. Combining a Care Path modeled as a state diagram with a Problem List form and online questionnaire(s) for patient-reported outcomes proves powerful for collecting chronic disease registry data as a byproduct of patient care–including novel state sequence and state dwell time data. Ready access to such data can accelerate the "Practice-to-Knowledge, Knowledge-to-Practice" cycles crucial to a Learning Health System.

The transformation of the American healthcare payment system from fee-for-service to value-based care increasingly makes it valuable to develop patient registries for specialized populations, to better assess healthcare quality and costs. Recent widespread adoption of Electronic Health Records (EHRs) in the U.S. now makes possible construction of EHR-based specialty registry data collection tools and reports, previously unfeasible using manual chart abstraction. But the complexities of specialty registry EHR tools and measures, along with the variety of stakeholders involved, can result in misunderstood requirements and frequent product change requests, as users first experience the tools in their actual clinical workflows. Such requirements churn could easily stall progress in specialty registry rollout. Modeling a system's requirements and solution design can be a powerful way to remove ambiguities, facilitate shared understanding, and help evolve a design to meet newly-discovered needs. "Agile Modeling" retains these values while avoiding excessive unused up-front modeling in favor of iterative incremental modeling. Using Agile Modeling principles and practices, in calendar year 2015 one institution developed 58 EHR-based specialty registries, with 111 new data collection tools, supporting 134 clinical process and outcome measures, and enrolling over 16,000 patients. The subset of UML and non-UML models found most consistently useful in designing, building, and iteratively evolving EHR-based specialty registries included User Stories, Domain Models, Use Case Diagrams, Decision Trees, Graphical User Interface Storyboards, Use Case text descriptions, and Solution Class Diagrams.

BACKGROUND Moving to electronic health records (EHRs) confers substantial benefits but risks unintended consequences. Modern EHRs consist of complex software code with extensive local configurability options, which can introduce defects. Defects in clinical decision support (CDS) tools are surprisingly common. Feasible approaches to prevent and detect defects in EHR configuration, including CDS tools, are needed. In complex software systems, use of test–driven development and automated regression testing promotes reliability. Test–driven development encourages modular, testable design and expanding regression test coverage. Automated regression test suites improve software quality, providing a “safety net” for future software modifications. Each automated acceptance test serves multiple purposes, as requirements (prior to build), acceptance testing (on completion of build), regression testing (once live), and “living” design documentation. Rapid-cycle development or “agile” methods are being successfully applied to CDS development. The agile practice of automated test–driven development is not widely adopted, perhaps because most EHR software code is vendor-developed. However, key CDS advisory configuration design decisions and rules stored in the EHR may prove amenable to automated testing as “executable requirements.” OBJECTIVE We aimed to establish feasibility of acceptance test–driven development of clinical decision support advisories in a commonly used EHR, using an open source automated acceptance testing framework (FitNesse). METHODS Acceptance tests were initially constructed as spreadsheet tables to facilitate clinical review. Each table specified one aspect of the CDS advisory’s expected behavior. Table contents were then imported into a test suite in FitNesse, which queried the EHR database to automate testing. Tests and corresponding CDS configuration were migrated together from the development environment to production, with tests becoming part of the production regression test suite. RESULTS We used test–driven development to construct a new CDS tool advising Emergency Department nurses to perform a swallowing assessment prior to administering oral medication to a patient with suspected stroke. Test tables specified desired behavior for (1) applicable clinical settings, (2) triggering action, (3) rule logic, (4) user interface, and (5) system actions in response to user input. Automated test suite results for the “executable requirements” are shown prior to building the CDS alert, during build, and after successful build. CONCLUSIONS Automated acceptance test–driven development and continuous regression testing of CDS configuration in a commercial EHR proves feasible with open source software. Automated test–driven development offers one potential contribution to achieving high-reliability EHR configuration. Vetting acceptance tests with clinicians elicits their input on crucial configuration details early during initial CDS design and iteratively during rapid-cycle optimization.

A tool that automatically identifies subject domains of examination questions is useful in at least three ways: (1) it can help learners hone their ability to perform this subject identification task, which is an important skill in several highstakes examinations, (2) in the context of educational content repositories, it can assist both maintainers in organizing and learners in querying content, and (3) it can be used to assess the appropriateness of questions on an examination. In this paper, we present such a tool for engineering examination questions using a novel technique that builds on existing efforts by India’s National Programme on Technology Enhanced Learning (NPTEL). In particular, we rely on textual transcripts as well as links between videos and questions from GATE (the annual Graduate Aptitude Test in Engineering taken by approximately one million candidates) that have been manually generated under this initiative. Our approach also addresses a key weakness in prior work based on queries by keywords, where terms used in multiple subject domains result in erroneous search results.

SummaryBackground: Creation of a new electronic health record (EHR)-based registry often can be a “one-off” complex endeavor: first developing new EHR data collection and clinical decision support tools, followed by developing registry-specific data extractions from the EHR for analysis. Each development phase typically has its own long development and testing time, leading to a prolonged overall cycle time for delivering one functioning registry with companion reporting into production. The next registry request then starts from scratch. Such an approach will not scale to meet the emerging demand for specialty registries to support population health and value-based care.Objective: To determine if the creation of EHR-based specialty registries could be markedly accelerated by employing (a) a finite core set of EHR data collection principles and methods, (b) concurrent engineering of data extraction and data warehouse design using a common dimensional data model for all registries, and (c) agile development methods commonly employed in new product development.Methods: We adopted as guiding principles to (a) capture data as a byproduct of care of the patient, (b) reinforce optimal EHR use by clinicians, (c) employ a finite but robust set of EHR data capture tool types, and (d) leverage our existing technology toolkit. Registries were defined by a shared condition (recorded on the Problem List) or a shared exposure to a procedure (recorded on the Surgical History) or to a medication (recorded on the Medication List). Any EHR fields needed - either to determine registry membership or to calculate a registry-associated clinical quality measure (CQM) - were included in the enterprise data warehouse (EDW) shared dimensional data model. Extract-transform-load (ETL) code was written to pull data at defined “grains” from the EHR into the EDW model. All calculated CQM values were stored in a single Fact table in the EDW crossing all registries. Registry-specific dashboards were created in the EHR to display both (a) real-time patient lists of registry patients and (b) EDW-gener-ated CQM data. Agile project management methods were employed, including co-development, lightweight requirements documentation with User Stories and acceptance criteria, and time-boxed iterative development of EHR features in 2-week “sprints” for rapid-cycle feedback and refinement.Results: Using this approach, in calendar year 2015 we developed a total of 43 specialty chronic disease registries, with 111 new EHR data collection and clinical decision support tools, 163 new clinical quality measures, and 30 clinic-specific dashboards reporting on both real-time patient care gaps and summarized and vetted CQM measure performance trends.Conclusions: This study suggests concurrent design of EHR data collection tools and reporting can quickly yield useful EHR structured data for chronic disease registries, and bodes well for efforts to migrate away from manual abstraction. This work also supports the view that in new EHR-based registry development, as in new product development, adopting agile principles and practices can help deliver valued, high-quality features early and often.

Background: Moving to electronic health records (EHRs) confers substantial benefits but risks unintended consequences. Modern EHRs consist of complex software code with extensive local configurability options, which can introduce defects. Defects in clinical decision support (CDS) tools are surprisingly common. Feasible approaches to prevent and detect defects in EHR configuration, including CDS tools, are needed. In complex software systems, use of test–driven development and automated regression testing promotes reliability. Test–driven development encourages modular, testable design and expanding regression test coverage. Automated regression test suites improve software quality, providing a “safety net” for future software modifications. Each automated acceptance test serves multiple purposes, as requirements (prior to build), acceptance testing (on completion of build), regression testing (once live), and “living” design documentation. Rapid-cycle development or “agile” methods are being successfully applied to CDS development. The agile practice of automated test–driven development is not widely adopted, perhaps because most EHR software code is vendor-developed. However, key CDS advisory configuration design decisions and rules stored in the EHR may prove amenable to automated testing as “executable requirements.” Objective: We aimed to establish feasibility of acceptance test–driven development of clinical decision support advisories in a commonly used EHR, using an open source automated acceptance testing framework (FitNesse). Methods: Acceptance tests were initially constructed as spreadsheet tables to facilitate clinical review. Each table specified one aspect of the CDS advisory’s expected behavior. Table contents were then imported into a test suite in FitNesse, which queried the EHR database to automate testing. Tests and corresponding CDS configuration were migrated together from the development environment to production, with tests becoming part of the production regression test suite. Results: We used test–driven development to construct a new CDS tool advising Emergency Department nurses to perform a swallowing assessment prior to administering oral medication to a patient with suspected stroke. Test tables specified desired behavior for (1) applicable clinical settings, (2) triggering action, (3) rule logic, (4) user interface, and (5) system actions in response to user input. Automated test suite results for the “executable requirements” are shown prior to building the CDS alert, during build, and after successful build. Conclusions: Automated acceptance test–driven development and continuous regression testing of CDS configuration in a commercial EHR proves feasible with open source software. Automated test–driven development offers one potential contribution to achieving high-reliability EHR configuration. Vetting acceptance tests with clinicians elicits their input on crucial configuration details early during initial CDS design and iteratively during rapid-cycle optimization. [JMIR Med Inform 2018;6(2):e23]