Search

Due to the Covid-19 pandemic, universities were required to shift teaching online in 2020, including exams. This rapid move required many changes and compromises to be made in order for students to take their exams remotely. We report on the experiences of running online exams for a large CS1 course at an Australian university, describing the changes made to the online learning platform to support this task, as well as the authoring of the exam and the support provided to students. Several clear instances of academic misconduct are presented to illustrate some of the issues endemic to online exams. Broadly, this approach to examination was a success that we plan to continue in future years, and we reflect on how the approach can be improved.

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English., Findings of ACL 2021

The COVID-19 pandemic has driven ever-greater demand for tools which enable efficient exploration of biomedical literature. Although semi-structured information resulting from concept recognition and detection of the defining elements of clinical trials (e.g. PICO criteria) has been commonly used to support literature search, the contributions of this abstraction remain poorly understood, especially in relation to text-based retrieval. In this study, we compare the results retrieved by a standard search engine with those filtered using clinically-relevant concepts and their relations. With analysis based on the annotations from the TREC-COVID shared task, we obtain quantitative as well as qualitative insights into characteristics of relational and concept-based literature exploration. Most importantly, we find that the relational concept selection filters the original retrieved collection in a way that decreases the proportion of unjudged documents and increases the precision, which means that the user is likely to be exposed to a larger number of relevant documents., Accepted at HealthNLP'21

Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse -- but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models., Accepted at NAACL 2021

Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM., Accepted at COLING 2020 - The 28th International Conference on Computational Linguistics

Existing metonymy resolution approaches rely on features extracted from external resources like dictionaries and hand-crafted lexical resources. In this paper, we propose an end-to-end word-level classification approach based only on BERT, without dependencies on taggers, parsers, curated dictionaries of place names, or other external resources. We show that our approach achieves the state-of-the-art on 5 datasets, surpassing conventional BERT models and benchmarks by a large margin. We also show that our approach generalises well to unseen data., 12 pages

As part of growing NLP capabilities, coupled with an awareness of the ethical dimensions of research, questions have been raised about whether particular datasets and tasks should be deemed off-limits for NLP research. We examine this question with respect to a paper on automatic legal sentencing from EMNLP 2019 which was a source of some debate, in asking whether the paper should have been allowed to be published, who should have been charged with making such a decision, and on what basis. We focus in particular on the role of data statements in ethically assessing research, but also discuss the topic of dual use, and examine the outcomes of similar debates in other scientific disciplines., 6 pages; accepted for ACL2020

Summary: The medical burden of heart failure (HF) has spurred interest in clinicians and scientists to develop therapies to restore the function of a failing heart. To advance this agenda, the National Heart, Lung, and Blood Institute (NHLBI) convened a Working Group of experts from June 2 to 3, 2016, in Bethesda, Maryland, to develop NHLBI recommendations aimed at advancing the science of cardiac recovery in the setting of mechanical circulatory support (MCS). MCS devices effectively reduce volume and pressure overload that drives the cycle of progressive myocardial dysfunction, thereby triggering structural and functional reverse remodeling. Research in this field could be innovative in many ways, and the Working Group specifically discussed opportunities associated with genome-phenome systems biology approaches; genetic epidemiology; bioinformatics and precision medicine at the population level; advanced imaging modalities, including molecular and metabolic imaging; and the development of minimally invasive surgical and percutaneous bioengineering approaches. These new avenues of investigations could lead to new treatments that target phylogenetically conserved pathways involved in cardiac reparative mechanisms. A central point that emerged from the NHLBI Working Group meeting was that the lessons learned from the MCS investigational setting can be extrapolated to the broader HF population. With the precedents set by the significant effect of studies of other well-controlled and tractable subsets on larger populations, such as the genetic work in both cancer and cardiovascular disease, the work to improve our understanding of cardiac recovery and resilience in MCS patients could be transformational for the greater HF population. Key Words: cardiac remodeling, mechanical circulatory support, myocardial recovery, ventricular assist devices

INTRODUCTION Improved understanding of the clinical course of ambulatory advanced chronic systolic heart failure may improve the provision of appropriate care and is central to the design of clinical trials in this population. METHODS Twenty-one implanting ventricular assist device (VAD) centers enrolled 400 subjects in the Registry Evaluation of Vital Information for VADs in Ambulatory Life (REVIVAL), a prospective, observational study in ambulatory, chronic, advanced systolic heart failure, designed to identify a cohort with an approximately 25% 1-year risk of the primary composite outcome of death, urgent transplant, or durable mechanical circulatory support. Inclusion criteria utilized only information collected during routine clinical care. Exclusion criteria identified patients with contraindications to VAD. Study inclusion required at least 1 of 10 high-risk criteria derived from established hospitalization and non-hospitalization markers of increased mortality risk. We evaluated the test performance characteristics of the high-risk criteria. RESULTS Data on 373 subjects evaluable for the primary composite outcome at the 1-year visit are presented. Baseline data were consistent with a less advanced cohort than Medical Arm for Mechanically Assisted Circulatory Support or Risk Assessment (MedaMACS) and Comparative Effectiveness of Left Ventricular Assist Device and Medical Management in Ambulatory Heart Failure Patients (ROADMAP). Freedom from the primary composite outcome was 75.9%. Non-hospitalization inclusion criteria identified 89% of patients with events. CONCLUSIONS Using routinely obtained clinical information for enrollment, REVIVAL successfully recruited an ambulatory chronic systolic heart failure cohort with an approximately 25% annual risk of the primary composite outcome. Information from this registry will be relevant to the planning of future trials of earlier VAD use and of other interventions in this population.

Geotagged Twitter data allows us to investigate correlations of geographic language variation, both at an interlingual and intralingual level. Based on data-driven studies of such relationships, this paper investigates regional variation of language usage on Twitter across Europe and compares it to traditional research of regional variation. This paper presents a novel method to process large amounts of data and to capture gradual differences in language variation. Visualizing the results by deterministically translating linguistic features into color hues presents a novel view of language variation across Europe, as it is reflected on Twitter. The technique is easy to apply to large amounts of data and provides a fast visual reference that can serve as input for further qualitative studies. The general applicability is demonstrated on a number of studies both across and within national languages. This paper also discusses the unique challenges of large-scale analysis and visualization, and the complementary nature of traditional qualitative and data-driven quantitative methods, and argues for their possible synthesis.

This chapter presents a methodology for identifying and resolving various kinds of inconsistency in the context of merging dependency and multiword expression (MWE) annotations, to generate a dependency treebank with comprehensive MWE annotations. Candidates for correction are identified using a variety of heuristics, including an entirely novel one which identifies violations of MWE constituency in the dependency tree, and resolved by arbitration with minimal human intervention. Using this technique, we identified and corrected several hundred inconsistencies across both parse and MWE annotations, representing changes to a significant percentage (well over 10\%) of the MWE instances in the joint corpus and a large difference in MWE tagging performance relative to earlier versions.

In this paper, we propose a joint architecture that captures language, rhyme and meter for sonnet modelling. We assess the quality of generated poems using crowd and expert judgements. The stress and rhyme models perform very well, as generated poems are largely indistinguishable from human-written poems. Expert evaluation, however, reveals that a vanilla language model captures meter implicitly, and that machine-generated poems still underperform in terms of readability and emotion. Our research shows the importance expert evaluation for poetry generation, and that future research should look beyond rhyme/meter and focus on poetic language., 11 pages; ACL2018

While neural networks have been shown to achieve impressive results for sentence-level sentiment analysis, targeted aspect-based sentiment analysis (TABSA) --- extraction of fine-grained opinion polarity w.r.t. a pre-defined set of aspects --- remains a difficult task. Motivated by recent advances in memory-augmented models for machine reading, we propose a novel architecture, utilising external "memory chains" with a delayed memory update mechanism to track entities. On a TABSA task, the proposed model demonstrates substantial improvements over state-of-the-art approaches, including those using external knowledge bases., Accepted to NAACL 2018 (camera-ready)

We propose a method for embedding two-dimensional locations in a continuous vector space using a neural network-based model incorporating mixtures of Gaussian distributions, presenting two model variants for text-based geolocation and lexical dialectology. Evaluated over Twitter data, the proposed model outperforms conventional regression-based geolocation and provides a better estimate of uncertainty. We also show the effectiveness of the representation for predicting words from location in lexical dialectology, and evaluate it using the DARE dataset., Conference on Empirical Methods in Natural Language Processing (EMNLP 2017) September 2017, Copenhagen, Denmark

Simple Summary The VetCompass Australia program collects real-time clinical records from veterinary practices and aggregates them for researchers to interrogate. It delivers Australian researchers sustainable and cost-effective access to authoritative data from hundreds of veterinary practitioners, across Australia and opens up major international collaborative opportunities with related projects in the United Kingdom and elsewhere. Abstract VetCompass Australia is veterinary medical records-based research coordinated with the global VetCompass endeavor to maximize its quality and effectiveness for Australian companion animals (cats, dogs, and horses). Bringing together all seven Australian veterinary schools, it is the first nationwide surveillance system collating clinical records on companion-animal diseases and treatments. VetCompass data service collects and aggregates real-time, clinical records for researchers to interrogate, delivering sustainable and cost-effective access to data from hundreds of veterinary practitioners nationwide. Analysis of these clinical records will reveal geographical and temporal trends in the prevalence of inherited and acquired diseases, identify frequently prescribed treatments, revolutionize clinical auditing, help the veterinary profession to rank research priorities, and assure evidence-based companion-animal curricula in veterinary schools. VetCompass Australia will progress in three phases: (1) roll-out of the VetCompass platform to harvest Australian veterinary clinical record data; (2) development and enrichment of the coding (data-presentation) platform; and (3) creation of a world-first, real-time surveillance interface with natural language processing (NLP) technology. The first of these three phases is described in the current article. Advances in the collection and sharing of records from numerous practices will enable veterinary professionals to deliver a vastly improved level of care for companion animals that will improve their quality of life.

The Infant Jarvik ventricular assist device (VAD; Jarvik Heart, Inc., New York, NY) has been developed to support the circulation of infants and children with advanced heart failure. The first version of the device was determined to have elevated hemolysis under certain conditions. The objective of this work was to determine appropriate modifications to the Infant Jarvik VAD that would result in acceptably low hemolysis levels. In vitro hemolysis testing revealed that hemolysis was related to the shape of the pump blade tips and a critical speed over which hemolysis would occur. Various design modifications were tested and a final design was selected that met the hemolysis performance goal. The new version was named the Jarvik 2015 VAD. Chronic in vivo tests, virtual fit studies, and a series of other performance tests were carried out to assess the device's performance characteristics. In vivo test results revealed acceptable hemolysis levels in a series of animals and virtual fit studies showed that the device would fit into children 8 kg and above, but could fit in smaller children as well. Additional FDA-required testing has been completed and all of the data are being submitted to the FDA so that a clinical trial of the Jarvik 2015 VAD can begin. Development of a Jarvik VAD for use in young children has been challenging for various reasons. However, with the hemolysis issue addressed in the Jarvik 2015 VAD, the device is well-poised for the start of the PumpKIN clinical trial in the near future.

This paper examines the effect of online social network interactions on future attitudes. Specifically, we focus on how a person's online content and network dynamics can be used to predict future attitudes and stances in the aftermath of a major event. In this study, we focus on the attitudes of US Twitter users towards Islam and Muslims subsequent to the tragic Paris terrorist attacks that occurred on November 13, 2015. We quantitatively analyze 44K users' network interactions and historical tweets to predict their attitudes. We provide a description of the quantitative results based on the content (hashtags) and network interaction (retweets, replies, and mentions). We analyze two types of data: (1) we use post-event tweets to learn users' stated stances towards Muslims based on sampling methods and crowd-sourced annotations; and (2) we employ pre-event interactions on Twitter to build a classifier to predict post-event stances. We found that pre-event network interactions can predict someone' s attitudes towards Muslims with 82% macro F-measure, even in the absence of prior mentions of Islam, Muslims, or related terms.

Recent work on word embeddings has shown that simple vector subtraction over pre-trained embeddings is surprisingly effective at capturing different lexical relations, despite lacking explicit supervision. Prior work has evaluated this intriguing result using a word analogy prediction formulation and hand-selected relations, but the generality of the finding over a broader range of lexical relation types and different learning settings has not been evaluated. In this paper, we carry out such an evaluation in two learning settings: (1) spectral clustering to induce word relations, and (2) supervised learning to classify vector differences into relation types. We find that word embeddings capture a surprising amount of information, and that, under suitable supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items.

Topic models have been shown to be a useful way of representing the content of large document collections, for example, via visualization interfaces (topic browsers). These systems enable users to explore collections by way of latent topics. A standard way to represent a topic is using a term list; that is the top-n words with highest conditional probability within the topic. Other topic representations such as textual and image labels also have been proposed. However, there has been no comparison of these alternative representations. In this article, we compare 3 different topic representations in a document retrieval task. Participants were asked to retrieve relevant documents based on predefined queries within a fixed time limit, presenting topics in one of the following modalities: (a) lists of terms, (b) textual phrase labels, and (c) image labels. Results show that textual labels are easier for users to interpret than are term lists and image labels. Moreover, the precision of retrieved documents for textual and image labels is comparable to the precision achieved by representing topics using term lists, demonstrating that labeling methods are an effective alternative topic representation.

Spatial language, despite decades of research, still poses substantial challenges for automated systems, for instance in geographic information retrieval or human-robot interaction. We describe an approach to building a corpus of natural language expressions extracted from web documents for analyzing and modeling spatial relational expressions (SRE). The unique characteristic of this corpus is that it is built around georeferenced triplets, with each triplet containing two entities (including their latitude/longitude coordinates) related by a spatial expression such as near. While the approach is still experimental, our first results are promising, in that we believe they will form the foundation for a comprehensive contextualized model for interpreting spatial natural language expressions. For the time being, we are focusing on a single domain, hotel reviews. This domain restriction allowed us to implement a proof-of-concept that this approach, with advances in natural language technologies, will indeed deliver a comprehensive corpus. The potential to collect larger corpora, and associated challenges, is discussed.

This paper presents an approach to query construction to detect multilingual dictionaries for predetermined language combinations on the web, based on the identification of terms which are likely to occur in bilingual dictionaries but not in general web documents. We use eight target languages for our case study, and train our method on pre-identified multilingual dictionaries and theWikipedia dump for each of our languages. © 2014 Association for Computational Linguistics.

The severity of heart failure has traditionally been characterized using the New York Heart Association (NYHA) functional classification system and gauges the severity of symptoms for patients who have American Heart Association Stage C or D heart failure.1 Patients with advanced heart failure with NYHA Class III or IV symptoms have limitations in daily activities despite optimal medical management. However, not all patients with advanced heart failure have a similar prognosis. Investigators of the Interagency Registry for Mechanically Assisted Circulatory Support (INTERMACS, a prospective registry that collects data on all patients who receive a durable, FDA approved mechanical circulatory support [MCS] device in the United States) developed a classification system of patient profiles to further characterize the severity of heart failure in this high risk population of patients undergoing treatment with MCS.2 Seven patient profiles (1=critical cardiogenic shock to 7=advanced NYHA Class III symptoms) were defined based on clinical descriptors at the time of MCS implant.2, 3 INTERMACS profiles have been used to characterize risk for poor outcomes after MCS implantation. Stepanenko et al.,4 in their study on outcomes of elective versus emergent permanent (i.e., destination therapy) MCS in elderly patients with advanced heart failure, reported that patients not requiring inotrope support (INTERMACS profiles 4–7) had lower operative mortality than patients with cardiogenic shock on high dose inotropes (INTERMACS profiles 1–3), cautiously concluding that permanent MCS may benefit highly selected elderly patients. In another study, patients with INTERMACS profiles 4–7 had better actuarial survival and a shorter hospital length of stay than patients with INTERMACS profile 1.5 Kirklin et al.6 reported that INTERMACS profile 1 is a risk factor for death early after MCS implantation as destination therapy. Regarding patient-reported health related quality of life (HRQOL) outcomes, researchers have demonstrated improved HRQOL through as long as 2 years after MCS implantation for both bridge to transplant and destination therapy.7–10 However, improvement in HRQOL from before to after MCS implantation across the spectrum of heart failure severity is unknown. Given that pre-implant heart failure severity affects post implant clinical outcomes, it may affect post implant HRQOL outcomes as well. Determining MCS HRQOL benefits and risks, stratified by heart failure severity, may provide important information with which to inform patients who are considering MCS implantation, and thereby help patients make more informed health care decisions, given that severely ill patients make decisions on treatment options (e.g., including surgical therapies or continued medical therapy / palliative care / hospice), based on outcomes, including HRQOL, as well as survival.11–14 Such information may also guide development of more targeted strategies to enhance HRQOL after implant. Understanding HRQOL benefits of MCS is especially important for patients who receive MCS as destination therapy. In the future, indications for destination therapy MCS may expand to include patients with moderately advanced heart failure, thus providing further impetus for understanding variation in post MCS HRQOL outcomes by pre-implant advanced heart failure severity.15 The purpose of our study was to examine differences in HRQOL outcomes from before MCS implantation through 1 year after surgery among patients receiving continuous flow pumps, stratified by INTERMACS patient profiles. We defined HRQOL as “the functional effect of an illness and its consequent therapy upon a patient, as perceived by the patient.”16

This paper describes a method for detecting event trigger words in biomedical text based on a word sense disambiguation (WSD) approach. We first investigate the applicability of existing WSD techniques to trigger word disambiguation in the BioNLP 2009 shared task data, and find that we are able to outperform a traditional CRF-based approach for certain word types. On the basis of this finding, we combine the WSD approach with the CRF, and obtain significant improvements over the standalone CRF, gaining particularly in recall.

We propose a range of deep lexical acquisition methods which make use of morphological, syntactic and ontological language resources to model word similarity and bootstrap from a seed lexicon. The different methods are deployed in learning lexical items for a precision grammar, and shown to each have strengths and weaknesses over different word classes. A particular focus of this paper is the relative accessibility of different language resource types, and predicted ``bang for the buck'' associated with each in deep lexical acquisition applications.

This paper describes a method for learning the countability preferences of English nouns from raw text corpora. The method maps the corpus-attested lexico-syntactic properties of each noun onto a feature vector, and uses a suite of memory-based classifiers to predict membership in 4 countability classes. We were able to assign countability to English nouns with a precision of 94.6%. Accepted version

Multiword expressions are a key problem for the development of large-scale, linguistically sound natural language processing technology. This paper surveys the problem and some currently available analytic techniques. The various kinds of multiword expressions should be analyzed in distinct ways, including listing “words with spaces”, hierarchically organized lexicons, restricted combinatoric rules, lexical selection, “idiomatic constructions” and simple statistical affinity. An adequate comprehensive analysis of multiword expressions must employ both symbolic and statistical techniques. Accepted version

Background This work describes a system for identifying event mentions in bio-molecular research abstracts that are either speculative (e.g. analysis of IkappaBalpha phosphorylation, where it is not specified whether phosphorylation did or did not occur) or negated (e.g. inhibition of IkappaBalpha phosphorylation, where phosphorylation did not occur). The data comes from a standard dataset created for the BioNLP 2009 Shared Task. The system uses a machine-learning approach, where the features used for classification are a combination of shallow features derived from the words of the sentences and more complex features based on the semantic outputs produced by a deep parser. Method To detect event modification, we use a Maximum Entropy learner with features extracted from the data relative to the trigger words of the events. The shallow features are bag-of-words features based on a small sliding context window of 3-4 tokens on either side of the trigger word. The deep parser features are derived from parses produced by the English Resource Grammar and the RASP parser. The outputs of these parsers are converted into the Minimal Recursion Semantics formalism, and from this, we extract features motivated by linguistics and the data itself. All of these features are combined to create training or test data for the machine learning algorithm. Results Over the test data, our methods produce approximately a 4% absolute increase in F-score for detection of event modification compared to a baseline based only on the shallow bag-of-words features. Conclusions Our results indicate that grammar-based techniques can enhance the accuracy of methods for detecting event modification.

International audience; Computational linguistics builds models that can usefully process and produce language and that can increase our understanding of linguistic phenomena. From the computational perspective, language data are particularly challenging notably due to their variable degree of idiosyncrasy (unexpected properties shared by few peer objects), and the pervasiveness of non-compositional phenomena such as multiword expressions (whose meaning cannot be straightforwardly deduced from the meanings of their components, e.g. red tape, by and large, to pay a visit and to pull one’s leg) and constructions (conventional associations of forms and meanings). Additionally, if models and methods are to be consistent and valid across languages, they have to face specificities inherent either to particular languages, or to various linguistic traditions.These challenges were addressed by the Dagstuhl Seminar 21351 entitled "Universals of Linguistic Idiosyncrasy in Multilingual Computational Linguistics", which took place on 30-31 August 2021. Its main goal was to create synergies between three distinct though partly overlapping communities: experts in typology, in cross-lingual morphosyntactic annotation and in multiword expressions. This report documents the program and the outcomes of the seminar. We present the executive summary of the event, reports from the 3 Working Groups and abstracts of individual talks and open problems presented by the participants.