Search

Objective In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations. Materials and Methods We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements. Results Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback. Discussion We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate. Conclusion By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

ObjectiveCode sets play a central role in analytic work with clinical data warehouses, as components of phenotype, cohort, or analytic variable algorithms representing specific clinical phenomena. Code set quality has received critical attention and repositories for sharing and reusing code sets have been seen as a way to improve quality and reduce redundant effort. Nonetheless, concerns regarding code set quality persist. In order to better understand ongoing challenges in code set quality and reuse, and address them with software and infrastructure recommendations, we determined it was necessary to learn how code sets are constructed and validated in real-world settings.MethodsSurvey and field study using semi-structured interviews of a purposive sample of code set practitioners. Open coding and thematic analysis on interview transcripts, interview notes, and answers to open-ended survey questions.ResultsThirty-six respondents completed the survey, of whom 15 participated in follow-up interviews. We found great variability in the methods, degree of formality, tools, expertise, and data used in code set construction and validation. We found universal agreement that crafting high-quality code sets is difficult, but very different ideas about how this can be achieved and validated. A primary divide exists between those who rely on empirical techniques using patient-level data and those who only rely on expertise and semantic data. We formulated a method- and process-based model able to account for observed variability in formality, thoroughness, resources, and techniques.ConclusionOur model provides a structure for organizing a set of recommendations to facilitate reuse based on metadata capture during the code set development process. It classifies validation methods by the data they depend on — semantic, empirical, and derived — as they are applied over a sequence of phases: (1) code collection; (2) code evaluation; (3) code set evaluation; (4) code set acceptance; and, optionally, (5) reporting of methods used and validation results. This schematization of real-world practices informs our analysis of and response to persistent challenges in code set development. Potential re-users of existing code sets can find little evidence to support trust in their quality and fitness for use, particularly when reusing a code set in a new study or database context. Rather than allowing code set sharing and reuse to remain separate activities, occurring before and after the main action of code set development, sharing and reuse must permeate every step of the process in order to produce reliable evidence of quality and fitness for use.

Introduction Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine‐interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T. Methods We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject‐predicate‐object triples to more clearly differentiate metadata categories. Results We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK‐to‐CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories. Conclusion A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

Sharing of individual participant data is encouraged by the International Committee of Medical Journal Editors. We analyzed clinical trial registry data from ClinicalTrials.gov (CTG) and determined the proportion of trials sharing de-identified Individual Participant Data (IPD). We looked at 3,138 medical conditions (as Medical Subject Heading terms). Overall, 10.8% of trials with first registration date after December 1, 2015 answered ‘Yes’ to plan to share de-identified IPD data. This sharing rate ranges between 0% (biliary tract neoplasms) to 72.2% (meningitis, meningococcal) when analyzed by disease that is focus of a study. Via a predictive model, we found that studies that deposited basic summary results data to CTG results registry, large studies and phase 3 interventional studies are most likely to declare intent to share IPD data. As part of an HIV common data element analysis project, we further compared a body of HIV trials (24% sharing rate) to other diseases.

This paper focuses on value sets as an essential component in the health analytics ecosystem. We discuss shared repositories of reusable value sets and offer recommendations for their further development and adoption. In order to motivate these contributions, we explain how value sets fit into specific analytic tasks and the health analytics landscape more broadly; their growing importance and ubiquity with the advent of Common Data Models, Distributed Research Networks, and the availability of higher order, reusable analytic resources like electronic phenotypes and electronic clinical quality measures; the formidable barriers to value set reuse; and our introduction of a concept-agnostic orientation to vocabulary collections. The costs of ad hoc value set management and the benefits of value set reuse are described or implied throughout. Our standards, infrastructure, and design recommendations are not systematic or comprehensive but invite further work to support value set reuse for health analytics. The views represented in the paper do not necessarily represent the views of the institutions or of all the co-authors.

The Food and Drug Administration and Department of Defense were interested in detecting sub-optimal use of long-acting beta-agonists (LABAs) in asthmatics within the Military Health System (MHS). Visualizing the patterns of asthma medication use surrounding a LABA prescription is a quick way to detect possible sub-optimal use for further evaluation. Analysis of 100 asthma patients was conducted in EventFlow to display and summarize timepoint and interval data. Epidemiologists reported that EventFlow was a powerful tool for rapidly visualizing possible patterns of sub-optimal LABA use that can be targeted for intervention.

Objective: To explore approaches for integrating and visualizing time-oriented medication data in narrative and structured formats and to address related issues on handling temporal abstraction, granularity, and uncertainty. The ultimate goal is to improve medication reconciliation by providing clinicians with more accurate medication information in patient care. Methods: An event taxonomy was generated to capture different combinations of clinical and temporal uncertainties. A prototype of a temporal visualization system was implemented using an open source software package called Timeline. Medications were parsed and mapped to the event taxonomy, and then represented in Timelines. Seventy-five medications from narrative discharge summary reports and seventy-nine medications from structured orders were used as data input for temporal visualization. Five physicians served as domain experts and answered ten proof-of-concept survey questions. Results: Overall positive feedback from experts suggested the potential value of the proposed timeline visualization method. Challenges were also identified, and future work will include reconciliation of medications from various sources based on temporal attributes and medication classification.