Search

Integrated, up-to-date data about SARS-CoV-2 and coronavirus disease 2019 (COVID-19) is crucial for the ongoing response to the COVID-19 pandemic by the biomedical research community. While rich biological knowledge exists for SARS-CoV-2 and related viruses (SARS-CoV, MERS-CoV), integrating this knowledge is difficult and time consuming, since much of it is in siloed databases or in textual format. Furthermore, the data required by the research community varies drastically for different tasks - the optimal data for a machine learning task, for example, is much different from the data used to populate a browsable user interface for clinicians. To address these challenges, we created KG-COVID-19, a flexible framework that ingests and integrates biomedical data to produce knowledge graphs (KGs) for COVID-19 response. This KG framework can also be applied to other problems in which siloed biomedical data must be quickly integrated for different research applications, including future pandemics. Funding: This work was supported by grants from the Director, Office of Science, Office of Basic Energy Sciences of the U.S. Department of Energy [to J.R., D.U., S.C., N.L.H., M.J., C.J.M], the Laboratory Directed Research and Development (LDRD) Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02-05CH11231, the NIH (Monarch R24 OD011883, Illuminating the Druggable Genome U01 CA239108-01), a Training Grant from the NLM, NIH to the University of Colorado Anschutz Medical Campus Computational Bioscience Training Program [T15LM009451 to T.J.C.], the National Virtual Biotechnology Laboratory (NVBL), and the Google Cloud COVID-19 Research Grants program. Conflict of Interest: The authors declare no competing interests.

The development of ontology engineering tools has traditionally lacked a user-centred perspective, instead being guided by the need to address particular gaps indicated by anecdotal evidence. This has typically resulted in prototypes that do not obtain traction beyond a narrow scope. Understanding the authoring patterns of ontology engineers is crucial to informing the development of ontology engineering tools that cater for the activity workflows of the users and, consequently, boosting the adoption of these tools. We report evidence about how Protégé is used across three different authoring settings, addressing the threats to validity of relying on a single user study. These settings address the continuum of expertise (from intermediate to expert users), the type of tasks (whether they are free-form or prescriptive) and the effect of the location (laboratory, tutorial or on their own) and how the studies are administered (whether or not there is a close supervision). While there are activity workflows that are particular to settings, the results indicate a number of core workflows that are common to all of them. We discuss actionable recommendations for ontology engineering tools in light of these results.

Background. Classifying entities in domain ontologies under upper ontology classes is a recommended task in ontology engineering to facilitate semantic interoperability and modelling consistency. Integrating upper ontologies this way is difficult and, despite emerging automated methods, remains a largely manual task. Problem. Little is known about how well experts perform at upper ontology integration. To develop methodological and tool support, we first need to understand how well experts do this task. We designed a study to measure the performance of human experts at manually classifying classes in a general knowledge domain ontology with entities in the Basic Formal Ontology (BFO), an upper ontology used widely in the biomedical domain.Method. We recruited 8 BFO experts and asked them to classify 46 commonly known entities from the domain of travel with BFO entities. The tasks were delivered as part of a web survey.Results. We find that, even for a well understood general knowledge domain such as travel, the results of the manual classification tasks are highly inconsistent: the mean agreement of the participants with the classification decisions of an expert panel was only 51%, and the inter-rater agreement using Fleiss’ Kappa was merely moderate (0.52). We further follow up on the conjecture that the degree of classification consistency is correlated with the frequency the respective BFO classes are used in practice and find that this is only true to a moderate degree (0.52, Pearson).Conclusions. We conclude that manually classifying domain entities under upper ontology classes is indeed very difficult to do correctly. Given the importance of the task and the high degree of inconsistent classifications we encountered, we further conclude that it is necessary to improve the methodological framework surrounding the manual integration of domain and upper ontologies.

Ontologies are complex systems of axioms in which unanticipated consequences of changes are both frequent, and diffcult for ontology authors to apprehend. The effects of modelling actions range from unintended inferences to outright defects such as incoherency or even inconsistency. One of the central ontology authoring activities is verifying that a particular modelling step has had the intended consequences, often with the help of reasoners. For users ofProtégé, this involves, for example, exploring the inferred class hierarchy.This paper provides evidence that making entailment set changes explicit to authors significantly improves the understanding of authoring actions regarding both correctness and speed. This is tested by means of the InferenceInspector, a Protégé plugin we created that provides authors with specific details about the effects of an authoring action. We empirically validate the effectiveness of the Inference Inspector in two studies. In a first, exploratory study we determine the feasibility of the Inference Inspector for supporting verification and isolating authoring actions. In a second, controlled study we formally evaluate the Inference Inspector and determine that making changes to key entailment sets explicit significantly improves author verification compared to the standard static hierarchy/framebased approach. We discuss the advantages of the Inference Inspector for different types of verification questions and find that our approach is best suited for verifying added restrictions where no new signature, such as class names, is introduced, with a 42% improvement in verification correctness.