Search

In this work, we formulated the named entity recognition (NER) task as a multi-answer knowledge guided question-answer task (KGQA) and showed that the knowledge guidance helps to achieve state-of-the-art results for 11 of 18 biomedical NER datasets. We prepended five different knowledge contexts—entity types, questions, definitions, and examples—to the input text and trained and tested BERT-based neural models on such input sequences from a combined dataset of the 18 different datasets. This novel formulation of the task (a) improved named entity recognition and illustrated the impact of different knowledge contexts, (b) reduced system confusion by limiting prediction to a single entity-class for each input token (i.e., B , I , O only) compared to multiple entity-classes in traditional NER (i.e., B entity 1, B entity 2, I entity 1, I , O ), (c) made detection of nested entities easier, and (d) enabled the models to jointly learn NER-specific features from a large number of datasets. We performed extensive experiments of this KGQA formulation on the biomedical datasets, and through the experiments, we showed when knowledge improved named entity recognition. We analyzed the effect of the task formulation, the impact of the different knowledge contexts, the multi-task aspect of the generic format, and the generalization ability of KGQA. We also probed the model to better understand the key contributors for these improvements.

Natural language processing (NLP) of clinical trial documents can be useful in new trial design. Here we identify entity types relevant to clinical trial design and propose a framework called CT-BERT for information extraction from clinical trial text. We trained named entity recognition (NER) models to extract eligibility criteria entities by fine-tuning a set of pre-trained BERT models. We then compared the performance of CT-BERT with recent baseline methods including attention-based BiLSTM and Criteria2Query. The results demonstrate the superiority of CT-BERT in clinical trial NLP., HealthNLP 2021, IEEE International Conference on Healthcare Informatics (ICHI 2021)

Special purpose lexicons are invaluable in biomedical natural language processing. They are especially crucial for a task such as the 13-criteria based cohort identification from clinical notes, process in N2C2 2018 Track 1 Challenge. While manually developed lexicons helped us achieve high performance, the process was ad hoc and nonreproducible. This paper presents a semi-automated lexicon induction method, using Logistic Regression (LR) and word embeddings, which brings rigor to the process. The key idea was to use n-grams in the training corpus as features of LR and identify those features (n-grams) with the most impact on the outcome as the lexicon. The semi-automatically generated lexicons achieved overall F measure of 0.9166 versus 0.9003 with manually generated lexicons. Therefore, this study shows that lexicons generated using a rigorous, semi-automated approach can retain performance while bringing rigor to the process.

Adverse drug events (ADEs) occur in approximately 2–5% of hospitalized patients, often resulting in poor outcomes or even death. Extraction of ADEs from clinical narratives can accelerate and automate pharmacovigilance. Using state-of-the-art deep-learning neural networks to jointly model concept and relation extraction, we achieved the highest integrated task score in the 2018 Medication and Adverse Drug Event (MADE) 1.0 challenge. We used a combined bidirectional long short-term memory (BiLSTM) and conditional random fields (CRF) neural network to detect medical entities relevant to ADEs and a combined BiLSTM and attention network to determine relations, including the adverse drug reaction relation between medication and sign or symptom entities. Using these models, we conducted three experiments: (1) separate and sequential modeling of entities and relations; (2) joint modeling where relations between medications and sign or symptoms determined ADE and indication entities; (3) use of information from external resources such as the US FDA’s adverse event database as additional input to the second method. Joint modeling improved the overall task accuracy from 0.62 to 0.65 F measure, and the additional use of external resources improved the accuracy to 0.66 F measure. Given the gold-standard medical entity labels, the joint model plus external resources method achieved F measures of 0.83 for ADE-relevant medical entity detection and 0.87 for relation detection. It is important to use joint modeling techniques and external resources for effectively detecting ADEs from clinical narratives in electronic health record (EHR) systems. While the extraction of entities and relations individually achieved high accuracy, the integrated task still has room for further improvement.

Background Finding specific scientific articles in a large collection is an important natural language processing challenge in the biomedical domain. Systematic reviews and interactive article search are the type of downstream applications that benefit from addressing this problem. The task often involves screening articles for a combination of selection criteria. While machine learning was previously used for this purpose, it is not known if different criteria should be modeled together or separately in an ensemble model. The performance impact of the modern contextual language models on the task is also not known. Methods We framed the problem as text classification and conducted experiments to compare ensemble architectures, where the selection criteria were mapped to the components of the ensemble. We proposed a novel cascade ensemble analogous to the step-wise screening process employed in developing the gold standard. We compared performance of the ensembles with a single integrated model, which we refer to as the individual task learner (ITL). We used SciBERT, a variant of BERT pre-trained on scientific articles, and conducted experiments using a manually annotated dataset of ~49 K MEDLINE abstracts, known as Clinical Hedges. Results The cascade ensemble had significantly higher precision (0.663 vs. 0.388 vs. 0.478 vs. 0.320) and F measure (0.753 vs. 0.553 vs. 0.628 vs. 0.477) than ITL and ensembles using Boolean logic and a feed-forward network. However, ITL had significantly higher recall than the other classifiers (0.965 vs. 0.872 vs. 0.917 vs. 0.944). In fixed high recall studies, ITL achieved 0.509 precision @ 0.970 recall and 0.381 precision @ 0.985 recall on a subset that was studied earlier, and 0.295 precision @ 0.985 recall on the full dataset, all of which were improvements over the previous studies. Conclusion Pre-trained neural contextual language models (e.g. SciBERT) performed well for screening scientific articles. Performance at high fixed recall makes the single integrated model (ITL) more suitable among the architectures considered here, for systematic reviews. However, high F measure of the cascade ensemble makes it a better approach for interactive search applications. The effectiveness of the cascade ensemble architecture suggests broader applicability beyond this task and the dataset, and the approach is analogous to query optimization in Information Retrieval and query optimization in databases.

Abbreviations sense disambiguation is a special case of word sense disambiguation. Machine learning methods based on neural networks showed promising results for word sense disambiguation (Festag and Spreckelsen, 2017) [1] and, here we assess their effectiveness for abbreviation sense disambiguation.Convolutional Neural Network (CNN) models were trained, one for each abbreviation, to disambiguate abbreviation senses. A reverse substitution (of long forms with short forms) method from a previous study was used on clinical narratives from Cleveland Clinic, USA, to auto-generate training data. Accuracy of the CNN and traditional Support Vector Machine (SVM) models were studied using: (a) 5-fold cross validation on the auto-generated training data; (b) a manually created, set-aside gold standard; and (c) 10-fold cross validation on a publicly available dataset from a previous study.CNN improved accuracy by 1-4 percentage points on all the three datasets compared to SVM, and the improvement was the most for the set-aside dataset. The improvement was statistically significant at p 0.05 on the auto-generated dataset. We found that for some common abbreviations, sense distributions mismatch between the test and auto generated training data, and mitigating the mismatch significantly improved the model accuracy.The neural network models work well in disambiguating abbreviations in clinical narratives, and they are robust across datasets. This avoids feature-engineering for each dataset. Coupled with an enhanced auto-training data generation, neural networks can simplify development of a practical abbreviation disambiguation system.

This paper presents a natural language processing (NLP) based cognitive decision support system that automatically identifies the status of a disease from the clinical notes of a patient record. The system relies on IBM Watson Patient Record NLP analytics and supervised or semi-supervised learning techniques. It uses unstructured text in clinical notes, data from the structured part of a patient record, and disease control targets from the clinical guidelines. We evaluated the system using de-identified patient records of 414 hypertensive patients from a multi-specialty hospital system in the U.S. The experimental results show that, using supervised learning methods, our system can achieve an average 0.86 F1-score in identifying disease status passages and average accuracy of 0.77 in classifying the status as controlled or not. To the best of our knowledge, this is the first system to automatically identify disease control status from clinical notes.

Modern business information systems are typically multitiered distributed systems comprising Web services, application services, databases, enterprise information systems, file systems, storage controllers, and other storage systems. In such environments, data is stored in different forms at multiple tiers, with each tier associated with some level of data abstraction. An information entity owned by an application generally maps to several data entities, logically associated across tiers and related to the application. Discovery of such relationships in a distributed system is a challenging problem, complicated by the widespread adoption of virtualization technologies and by the traditional tendency to manage each tier as an independent domain. In this paper, we present a system and methodology for model-driven discovery of end-to-end application-data relationships spanning multiple tiers, from the applications to the lowest levels of the storage hierarchy. The key to our methodology involves modeling how data is used and transformed by distributed software components. An important benefit of our system, which we call Galapagos, is the ability to reflect business decisions expressed at the application level to the level of storage.

The JavaTM platform has the potential to revolutionize computing, with its promise of "write once, run anywhere"TM development. However, in order to realize this potential, Java applications must demonstrate satisfactory performance. Rapid progress has been made in addressing Java performance, although most of the initial efforts have targeted Java client applications. To make a significant impact in network computing, server applications written in the Java language, or those using Java extensions, frameworks, or components, must exhibit a competitive level of performance. One obstacle to obtaining this goal has been the lack of well-defined, server-specific, Java benchmarks. This paper helps address this shortcoming by defining representative Java server benchmarks. These benchmarks represent server application areas, including Web-based dynamic content delivery (servlets), business object frameworks, and multitier transactional data acquisition. Where applicable, we present benchmarks written using both the Java programming model (i.e., servlets) and the legacy model (i.e., the Common Gateway Interface) for direct comparisons of delivered performance. We also present performance measurements and analysis from multiple IBM server platforms, including both uniprocessor and multiprocessor systems.

This article presents the deign and implementation of the recovery scheme in Calypso. Calypso is a cluster-optimized, distributed file system for UNIX clusters. As in Sprite and AFS, Calypso servers are stateful and scale well to a large number of clients. The recovery scheme in Calypso is nondisruptive, meaning that open files remain open, client modified data are saved, and in-flight operations are properly handled across server recover. The scheme uses distributed state amount the clients to reconstruct the server state on a backup node if disks are multiported or on the rebooted server node. It guarantees data consistency during recovery and provides congestion control. Measurements show that the state reconstruction can be quite fast: for example, in a 32-node cluster, when an average node contains state for about 420 files, the reconstruction time is about 3.3 seconds. However, the time to update a file system after a failure can be a major factor in the overall recovery time, even when using journaling techniques.

Presents dynamic load-sharing heuristics that use predicted resource requirements of processes to manage workloads in a distributed system. A previously developed statistical pattern-recognition method is employed for resource prediction. While nonprediction-based heuristics depend on a rapidly changing system status, the new heuristics depend on slowly changing program resource usage patterns. Furthermore, prediction-based heuristics can be more effective since they use future requirements rather than just the current system state. Four prediction-based heuristics, two centralized and two distributed, are presented. Using trace driven simulations, they are compared against random scheduling and two effective nonprediction based heuristics. Results show that the prediction-based centralized heuristics achieve up to 30% better response times than the nonprediction centralized heuristic, and that the prediction-based distributed heuristics achieve up to 50% improvements relative to their nonpredictive counterpart. >

Discovering application dependency on data and storage is a key prerequisite for many storage optimization tasks such as data assignment to storage tiers, storage consolidation, virtualization, and handling unused data. However, in the real world these dependencies are rarely known, and discovering them is a challenge because of virtualization at various levels and the need for discovery methods to be non-intrusive. As a result, many optimization tasks are performed, if at all, without the full knowledge of application-to-storage dependencies.This paper presents a non-intrusive application-to-storage discovery method, and while it is built on our prior work, the storage discovery described here is entirely new. We used this discovery method in two production enterprise environments, consisting of about 323 servers, and we show how the discovered data enables three optimization tasks. First, we relate application criticality with storage tiers. Second, we find unused storage devices and we show how this information together with storage consolidation can be used to achieve power savings of up to two orders of magnitude. Third, we identify opportunities for database storage optimization.

IT out sourcing service providers are increasingly being challenged to reduce costs while improving the quality of service implementation and delivery. A key contributor to those goals can be an improved methodology for services solution engineering - which involves analysis of the client's requirements and environment, mapping the information to the capabilities of the service provider, documenting the design and implementation of each service, and defining transformations of the customer's environment to the target IT infrastructure defined by the overall design that facilitates more efficient service delivery. One such improved methodology leverages reuse, in the form of a taxonomy of standardized service offerings and a repository of standardized service designs they can be mapped to. In this paper, we describe a prototype system that provides automation for such a methodology by encoding design policies that are used to assist the services solution architect gather relevant information from the customer and cross check design decisions. The approach demonstrates how the notion of computer-aided design can be applied to the world of IT services and introduced in an incremental manner. This paper presents the system architecture, discusses the sources and models of knowledge, and illustrates how this knowledge can be used with specific examples from the services field.

Enterprise IT environments are complex: business applications rely on distributed middleware running on diverse hardware with components depending on each other in many unexpected ways. Discovery of applications' dependency on IT is a critical step in managing application and IT infrastructure together. Many tools and practices have emerged to discover and report IT assets and applications' dependency on the IT assets. However, our experience in the field shows that there are significant challenges in effectively deploying the tools. There is a critical need to research and develop flexible processes, methods, and practices, and architecture-level support for them in the tools to enable successful discovery using a "built-to-order" approach. In this paper we discuss our experiences with an advanced application-data relationship discovery tools in large scale enterprise environments and based on these experiences we identify three main challenges of effective discovery. They are: deployment process and related security issues; unavailability of software and administration-related information; and tool integration. To address these challenges, here we demonstrate a holistic approach that includes flexible processes, methods, and practices in the tools for achieving the necessary built-to order capability.

This paper addresses the important problem of finding relevant information in the context of a business process. It presents an information access solution called EIL (enterprise information leverage) which combines information extraction and semantic search to support information needs of professionals selling IT services. EIL leverages structured and unstructured data using novel architecture and special purpose algorithms. Our approach is to organize information around business activities (e.g. a sale), and the system supports semantic concept based information retrieval by utilizing both database query and document search where the relevant business activities act as a contextual constraint. We experimentally show that this approach is promising for reducing noise in search results. EIL is currently under pilot deployment in one of the IBM services sales units.

In outsourcing, service obligations broadly refer to the duties and responsibilities of the service provider related to the provision of services. It is critical that the provider clearly understands and properly implements its complete service obligations to the client. The BlueMoon project in IBM develops a novel approach to service obligations management in the context of IT outsourcing. BlueMoon conceptualizes different kinds of service obligations through a formal obligations model. It provides an overarching obligations configurator tool for gathering and validating obligation requirements throughout the outsourcing lifecycle. It further facilitates the implementation of service obligations by automatically binding them to the IT architecture planning and provisioning process and to their uses in various service documents. Work is under way to monitor service delivery operations and verify obligations compliance. Addressing a key challenge in IT outsourcing, bluemoon promises to significantly improve the client's satisfaction, the health of the outsourcing account, and the provider's bottom line.

To dynamically monitor services and diagnosing causes of potential problems, we need to acquire essential information about atomic services. A challenge is how we tap the service middleware and acquire low-level runtime data about atomic services. In this paper, we present a design of dynamic monitor which gathers quality-related data of running services deployed on Enterprise Service Bus. We show the architecture of dynamic monitor, and present an implementation of the monitor on open source platform.

In large networked systems, relationships between applications and the data that they use through multiple tiers of middleware systems are often invisible. While the benefits of knowing such relationships are clear from a systems management perspective, discovery of such relationships is complicated by the widespread adoption of virtualization technologies and the tendency to view each middleware tier as an independent "domain" from a systems management perspective. In this paper we present a methodology and a system for automatic discovery of end-to-end application-data relationships. The key to the methodology is the modeling of data locations from which applications use data and of how middleware systems make data available to software layers above them.

Large enterprises operate two or more data centers for scaling and fault tolerance. Each data center maintains a certain degree of autonomy while sharing resources with other data centers. In this paper we describe how policy-based resource management can be used in existing commercial products such that each data center can control when, what, and how to share resources with other data centers.

Policy-based file lifecycle management is important for balancing storage utilization and for regulatory conformance. It poses two important challenges, the need for simple yet effective policy design and an implementation that scales to billions of files. This paper describes the design and an innovative implementation technique of policy-based lifecycle management in a prototype built as a part of IBM's new SAN file system. The policy specification leverages a key abstraction in the file system called storage pools and its ability to support location independence for files. The policy implementation uses an innovative new technique that combines concurrent policy execution and a policy decisions cache, to enable scaling to billions of files under normal usage patterns.

We present a policy-based architecture STEPS for lifecycle management (LCM) in a mass scale distributed file system. The STEPS architecture is designed in the context of IBM's SAN file system (SFS) and leverages the parallelism and scalability offered by SFS, while providing a centralized point of control for policy-based management. The architecture uses novel concepts like policy cache and rate-controlled migration for efficient and non-intrusive execution of the LCM functions, while ensuring that the architecture scales with very large number of files. The architecture has been implemented and used for lifecycle management in a distributed deployment of SFS with heterogeneous data. We conduct experiments on the implementation to study the performance of the architecture. We observed that STEPS is highly scalable with increase in the number as well as the size of the file objects hosted by SFS. The performance study also demonstrated that most of the efficiency of policy execution is derived from policy cache. Further, a rate-control mechanism is necessary to ensure that users are isolated from LCM operations.

Grid computing, in the commercial space, builds upon a set of management disciplines, which aims at mapping available resource capabilities to application workloads, according to requirements these workloads depend upon and to business goals they must fulfill. This paper illustrates innovative technologies, developed at IBM Research, that address key issues found in commercial grid environments. These technologies fall into four main areas, workload virtualization, information virtualization, provisioning and orchestration, and application development.

The goal of policy-based storage management is to allow storage resources in an IT complex to be managed by setting comparatively high-level policies, rather than by doing low-level manual configuration. We describe the policy management and rule execution architecture in a prototype autonomic storage manager being developed in IBM Research. The prototype uses generic communication, a policy repository, and policy translation and execution services provided by an autonomic manager toolkit. The prototype supports a set of policy templates developed from a policy-based storage management framework.

Predicted task resource usage provides a basis for developing two centralized load-sharing policies: MinQ and MinResp. Trace-driven simulations are used to compare MinQ and MinResp against Centex, an efficient centralized scheme. Experimental results show that the use of prediction makes MinQ and MinResp significantly less sensitive to the status update rate than Centex. Consequently, the proposed algorithms perform better than Centex at slower update rates and are capable of handling larger workloads. The prediction-based policies are also highly effective for load-sharing in environments with widely varying CPU requirements. Using a real trace file,with an equal number of large and small tasks, MinResp consistently produced mean response times that were 9% to 35% lower than those of Centex. >

This paper presents the design and implementation of a distributed token manager for a cluster-optimized, distributed Unix file system. In this file system, tokens provide cache consistency and support for single-system Unix semantics. The paper describes the token types used, token arbitration protocol, deadlock-free implementation, fault-tolerance, and recovery. The key contributions of the work reported here are: (1) An efficient protocol for token arbitration, which minimizes bottlenecks and hence enhances scalability; (2) A practical approach to handling deadlocks, race conditions, and recovery issues, which complicate token manager design and implementation. >

In the single system UNIX, successful completion of a write system call implies a guarantee of adequate disk space for any new pages created by the system call. To support such a guarantee in a distributed file system designers need to solve the problems of accurately estimating the space needed, communication overhead, and fault tolerance. In the Calypso file system, which is a cluster-optimized, distributed UNIX file system, we solve these problems using an advance-reservation scheme. Measurements show that the overhead of this scheme for typical UNIX usage patterns is 1% to 3%.

This paper describes design and preliminary measurements of a file server recovery scheme that uses naturally replicated state among clients. This scheme, implemented in the Calypso file system, is truly transparent to the user and avoids the overhead of explicit replication. A three-phase protocol reconstructs the server state either on a backup node (if disks are multi-ported) or on the rebooted server node. Measurements show that the recovery time is about 21 seconds for a busy 10-node cluster. However, the time to rebuild the distributed state is only about 1.5 seconds, and most of the recovery time is spent in replaying the write-ahead log of the underlying file system. Fortunately, the log redo time is bounded by the log size.

The World-Wide Web is the 'killer application' of the mid-1990's, and as a result it serves as the market place for various services. This position document discusses the problem of creating complex Web services from simpler ones, under client control. For example, one might want an annotated forecast of a company's stock price from three simple services, viz., stock price data, forecasting and annotation. We propose meta-scripts as a solution. A client accesses a meta-script, which contains procedures for accessing and composing simpler services. Meta-identifiers are used in the meta-script instead of static links. Developments in type-consistency mechanisms and various enabling services of the Web help in realization of the meta-script based approach. This model of service composition separates control from data manipulation, and gives this control to a client. As clients are empowered with the selection and coordination of service providers, this can lead to a free market for services.

This position paper examines file system requirements of a few, large-scale, emerging applications. While these applications require some new functions that have never been implemented before, mostly they need customization of existing functions. Building a new file system for each application is not economically viable, but this is what is happening today either directly or indirectly.

We propose a new frequency-based replacement algorithm for managing caches used for disk blocks by a file system, database management system, or disk control unit, which we refer to here as data caches. Previously, LRU replacement has usually been used for such caches. We describe a replacement algorithm based on the concept of maintaining reference counts in which locality has been “factored out”. In this algorithm replacement choices are made using a combination of reference frequency and block age. Simulation results based on traces of file system and I/O activity from actual systems show that this algorithm can offer up to 34% performance improvement over LRU replacement, where the improvement is expressed as the fraction of the performance gain achieved between LRU replacement and the theoretically optimal policy in which the reference string must be known in advance. Furthermore, the implementation complexity and efficiency of this algorithm is comparable to one using LRU replacement.

A statistical approach is developed for predicting the CPU time, the file I/O, and the memory requirements of a program at the beginning of its life, given the identity of the program. Initially, statistical clustering is used to identify high-density regions of process resource usage. The identified regions form the states for building a state-transition model to characterize the resource usage of each program in its past executions. The prediction scheme uses the knowledge of the program's resource usage in its last execution together with its state-transition model to predict the resource usage in its next execution. The prediction scheme is shown to work using process resource-usage data collected from a VAX 11/780 running 4.3 BSD Unix. The results show that the predicted values correlate strongly with the actual; the coefficient of correlation between the predicted and actual values for CPU time is 0.84. The errors in prediction are mostly small and are heavily skewed toward small values. >