Search

A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.

In the world of public misinformation, there are many cases where the information is not false or fabricated, but rather has been manipulated using more subtle techniques such as word replacements, selection of details, omissions and argument distortion. These techniques can have the effect of influencing the reader’s frame of mind towards the events reported. We currently lack the necessary tools to uncover such manipulations automatically. In this position paper, we propose an integrated analysis framework and pipeline to identify various narrative signals in news articles; such as structural roles, framing, and subjectivity. By comparing these at the document level and sentence level, it will be possible to highlight differences of narrative techniques used to report the same news events.

Since the early years of the past century, many scholars have focused their efforts towards designing models to better understand the way listeners perceive musical tension. From the existing models, Lerdahl's has shown strong correlations against tension judgements provided by human listeners and has been used to make accurate predictions of musical tension. However, a full automation of Lerdahl's model of tension has not yet been made available. This paper presents a computational approach to automatically calculate musical tension according to Lerdahl's model, with a publicly available implementation.

The increasing use of social media around global news events, such as the London Olympics in 2012, raises questions for international broadcasters about how to engage with users via social media in order to best achieve their individual missions. Twitter is a highly diverse social network whose conversations are multi-directional involving individual users, political and cultural actors, athletes and a range of media professionals. In so doing, users form networks of influence via their interactions affecting the ways that information is shared about specific global events.\ud \ud This article attempts to understand how networks of influence are formed among Twitter users, and the relative influence of global news media organisations and information providers in the Twittersphere during such global news events. We build an analysis around a set of tweets collected during the 2012 London Olympics. To understand how different users influence the conversations across Twitter, we compare three types of accounts: those belonging to a number of well-known athletes, those belonging to some well-known commentators employed by the BBC, and a number of corporate accounts belonging to the BBC World Service and the official London Twitter account. We look at the data from two perspectives. First, to understand the structure of the social groupings formed among Twitter users, we use a network analysis to model social groupings in the Twittersphere across time and space. Second, to assess the influence of individual tweets, we investigate the ageing factor of tweets, which measures how long users continue to interact with a particular tweet after it is originally posted.\ud \ud We consider what the profile of particular tweets from corporate and athletes’ accounts can tell us about how networks of influence are forged and maintained. We use these analyses to answer the questions: How do different types of accounts help shape the social networks? and, What determines the level and type of influence of a particular account?

Current game music systems typically involve the playback of prerecorded audio tracks which are crossfaded in response to game events such as level changes. However, crossfading can limit the expressive power of musical transitions, and can make fine grained structural variations difficult to achieve. We therefore describe an alternative approach in which music is algorithmically generated based on a set of high-level musical features that can be controlled in real-time according to a player’s progression through a game narrative. We outline an implementation of the approach in an actual game, focusing primarily on how the music system traces the game’s emotional narrative by periodically querying certain narrative parameters and adjusting the musical features of its output accordingly.

This poster presents the work of the ComTax project to develop a community-driven curation process among practicing scientists and citizen scientists. The project provides tools to help scientists identify and validate appropriate taxonomic names from the scanned historical literature. The system operates on scanned documents, typically taken from the Biodiversity Heritage Library, although documents sourced from other repositories could be used.\ud \ud The system is intended to be used on uncorrected text after optical character recognition (OCR) on the scanned images. The key stages are:\ud \ud 1. Identify possible taxonomic names in the scanned text using machine learning techniques.\ud \ud 2. Verify the extracted names against existing databases. If present, the source scanned text can be automatically marked-up with the name.\ud \ud 3. Unverified names might mean they are not currently recorded in the verification databases, typically because the old name in the literature has been reclassified, or because erroneous OCR means that the name is incorrectly transcribed in the scanned text. In either case:\ud \ud 3.1. Present the proposed name to domain experts or citizen scientists for validation or correction, potentially through a voting mechanism to collect expert judgments on the putative taxonomic name.\ud \ud 3.2. Mark-up the scanned text with the corrected spelling of the name and offer validated taxonomic names for further use by the community.\ud \ud This poster will describe the technical challenges facing the ComTax project, and highlight potential extensions of the work to the curation of other entities of interest in the legacy literature or of different disciplines.

Digitized biodiversity literature provides a wealth of content for using biodiversity knowledge by machines. However, identifying taxonomic names and the associated semantic metadata is a difficult and labour intensive process. We present a system to support human assisted creation of semantic metadata. Information extraction techniques auto-matically identify taxonomic names from scanned documents. They are then presented to users for manual correction or verification. The tools that support the curation process include taxonomic name identification and mapping, and community-driven taxonomic name verification. Our research shows the potential for these information extrac-tion techniques to support research and curation in disciplines dependent upon scanned documents

Biological taxonomy is the classification of living and fossil organisms. Taxonomists have identified and named some 1.8 million species of animals, plants, and microorganisms, a fraction of Earth's estimated 5‐30 million species. Part of the effort of taxonomy lies in developing and curating taxonomic databases, which support access to the taxonomy literature, and provide basic knowledge needed for management and conservation of biodiversity.\ud \ud A major difficulty facing this task is incorporating knowledge that is currently contained only in the historical literature. Extracting this knowledge is a difficult and labour‐intensive process, as scanning errors and other variations in nomenclature mean that particular names must be manually verified as part of the process. For example, Actinobacillus actionomy, Actinobacillus actionomyce, and Actinobacillus actionomycetam could all be variants of the same name. ComTax is an ongoing project to develop a community‐driven curation process among taxonomists, by providing tools to help them identify and validate taxonomic names from the scanned historical literature. The system operates on scanned documents following optical character recognition (OCR). The key stages are:\ud \ud 1. Identify possible taxonomic names from scanned text. Names might be new either because they do not appear in existing databases, or because they have been incorrectly identified by OCR.\ud \ud 2. Present the proposed name to a domain expert for validation or correction.\ud \ud 3. Present validated taxonomic names for curation. Organizations like Global Biodiversity Information Facility (GBIF) manage the curation of taxonomic databases. This poster describes the technical challenges facing the ComTax project, and will discuss our work with the Natural History Museum to integrate the curation process into taxonomists' workflow. We also demonstrate the relevance of this work within the wider context of biological taxonomy curation.

peer-reviewed Stakeholders frequently use speculative language when they need to convey their requirements with some degree of uncertainty. Due to the intrinsic vagueness of speculative language, speculative requirements risk being misunderstood, and related uncertainty overlooked, and may benefit from careful treatment in the requirements engineering process. In this paper, we present a linguistically-oriented approach to automatic detection of uncertainty in natural language (NL) requirements. Our approach comprises two stages. First we identify speculative sentences by applying a machine learning algorithm called Conditional Random Fields (CRFs) to identify uncertainty cues. The algorithm exploits a rich set of lexical and syntactic features extracted from requirements sentences. Second, we try to determine the scope of uncertainty. We use a rule-based approach that draws on a set of hand-crafted linguistic heuristics to determine the uncertainty scope with the help of dependency structures present in the sentence parse tree. We report on a series of experiments we conducted to evaluate the performance and usefulness of our system.

We present the ABLE document collection, which consists of a set of annotated volumes of the Bulletin of the British Museum (Natural History). These follow our work on automating the markup of scanned copies of the biodiversity literature, for the purpose of supporting working taxonomists. We consider an enhanced TEI XML markup language, which is used as an intermediate stage in translating from the initial XML obtained from Optical Character Recognition to the target taXMLit. The intermediate representation allows additional information from external sources such as a taxonomic thesaurus to be incorporated before the final translation into taXMLit.

Accurately capturing the quantifier scope behaviour of coordinated NPs can be problematic for underspecification systems that define constraints over semantic constructors. We present an extension to a hole-semantics like language that allows a natural representation of coordinated NPs, and a translation from partial scope requirements into constraints on the constructors. We conclude that the efficient decision procedures developed for constraints on semantic constructors enable the possible meanings of sentences containing coordinated NPs to be fully underspecified.

In this paper we present heuristics for resolving coordination ambiguities. We test the hypothesis that the most likely reading of a coordination can be predicted using word distribution information from a generic corpus. Our heuristics are based upon the relative frequency of the coordination in the corpus, the distributional similarity of the coordinated words, and the collocation frequency between the coordinated words and their modifiers. These heuristics have varying but useful predictive power. They also take into account our view that many ambiguities cannot be effectively disambiguated, since human perceptions vary widely.

A metric for evaluating the creativity of a music-generating system is presented, the objective being to generate mazurka-style music that inherits salient patterns from an original excerpt by Frédéric Chopin. The metric acts as a filter within our overall system, causing rejection of generated passages that do not inherit salient patterns, until a generated passage survives. Over fifty iterations, the mean number of generations required until survival was 12.7, with standard deviation 13.2. In the interests of clarity and replicability, the system is described with reference to specific excerpts of music. Four concepts–Markov modelling for generation, pattern discovery, pattern quantification, and statistical testing–are presented quite distinctly, so that the reader might adopt (or ignore) each concept as they wish.

peer-reviewed Nocuous ambiguity occurs when a linguistic expression is interpreted differently by different readers in a given context. We present an approach to automatically identify nocuous ambiguity that is likely to lead to misunderstandings among readers. Our model is built on a machine learning architecture. It learns from a set of heuristics each of which predicts a factor that may lead a reader to favor a particular interpretation. An ambiguity threshold indicates the extent to which ambiguity can be tolerated in the application domain. Collections of human judgments are used to train heuristics and set ambiguity thresholds, and for evaluation. We report results from applying the methodology to coordination and anaphora ambiguity. Results show that the method can identify nocuous ambiguity in text, and may be widened to cover further types of ambiguity. We discuss approaches to evaluation.

We consider the problem of intra-opus pattern discovery, that is, the task of discovering patterns of a specified type within a piece of music. A music analyst undertook this task for works by Domenico Scarlattti and Johann Sebastian Bach, forming a benchmark of 'target' patterns. The performance of two existing algorithms and one of our own creation, called SIACT, is evaluated by comparison with this benchmark. SIACT out-performs the existing algorithms with regard to recall and, more often than not, precision. It is demonstrated that in all but the most carefully selected excerpts of music, the two existing algorithms can be affected by what is termed the 'problem of isolated membership'. Central to the relative success of SIACT is our intention that it should address this particular problem. The paper contrasts string-based and geometric approaches to pattern discovery, with an introduction to the latter. Suggestions for future work are given.

Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of disagreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks.

We study the problem of detecting sentences describing adverse drug reactions (ADRs) and frame the problem as binary classification. We investigate different neural network (NN) architectures for ADR classification. In particular, we propose two new neural network models, Convolutional Recurrent Neural Network (CRNN) by concatenating convolutional neural networks with recurrent neural networks, and Convolutional Neural Network with Attention (CNNA) by adding attention weights into convolutional neural networks. We evaluate various NN architectures on a Twitter dataset containing informal language and an Adverse Drug Effects (ADE) dataset constructed by sampling from MEDLINE case reports. Experimental results show that all the NN architectures outperform the traditional maximum entropy classifiers trained from n-grams with different weighting strategies considerably on both datasets. On the Twitter dataset, all the NN architectures perform similarly. But on the ADE dataset, CNN performs better than other more complex CNN variants. Nevertheless, CNNA allows the visualisation of attention weights of words when making classification decisions and hence is more appropriate for the extraction of word subsequences describing ADRs.

This literature review forms a deliverable in the European Network of Excellence on Semantic Interoperability and Data Mining in Biomedicine. More specifically, it is part of a work package (WP27) which aims to develop and evaluate generic methods and tools for assisting patients to understand their health and healthcare by generating patient-friendly readable texts that paraphrase the content of their electronic health records. We have reviewed the literature in topics that we consider to be relevant to this work package. When appropriate, we cover variations in conditions in the four countries of the collaborating research groups (France, Germany, Sweden and the UK) and we cover corpora, tools and language technologies for the European languages of interest to these groups.