Search

Task-oriented dialogues often require agents to enact complex, multi-step procedures in order to meet user requests. While large language models have found success automating these dialogues in constrained environments, their widespread deployment is limited by the substantial quantities of task-specific data required for training. The following paper presents a data-efficient solution to constructing dialogue systems, leveraging explicit instructions derived from agent guidelines, such as company policies or customer service manuals. Our proposed Knowledge-Augmented Dialogue System (KADS) combines a large language model with a knowledge retrieval module that pulls documents outlining relevant procedures from a predefined set of policies, given a user-agent interaction. To train this system, we introduce a semi-supervised pre-training scheme that employs dialogue-document matching and action-oriented masked language modeling with partial parameter freezing. We evaluate the effectiveness of our approach on prominent task-oriented dialogue datasets, Action-Based Conversations Dataset and Schema-Guided Dialogue, for two dialogue tasks: action state tracking and workflow discovery. Our results demonstrate that procedural knowledge augmentation improves accuracy predicting in- and out-of-distribution actions while preserving high performance in settings with low or sparse data.

This project contains the data and code used to analyse and report on an experiment testing the effect of increased surface complexity of wood on in-stream macroinvertebrate assemblages across streams with varying degrees of catchment-scale urban impact (White and Walsh, 2020 Ecological Applications https://doi.org/10.1002/eap.2199). We used a hierarchical multi-taxon model to assess differences in abundance of taxa between two types of wood surface in streams with catchments ranging from forested to fully urban. We were particularly interested in the potential interaction between surface complexity and urban impact. Increased surface complexity increased abundance of most taxa, but this effect was less pronounced in urban streams, partly because of the reduced species pool tolerant of urban stormwater impacts, and partly because of a lesser response of some species to increased complexity in more urban streams. Collectively these taxon-specific effects resulted in small, uncertain increases in taxon richness with increased complexity in rural streams, and no change in richness of the less diverse assemblages of urban streams. The data are in 4 related tables (sites, samples, biota, taxonomy) in the worksheet WhiteWalsh_data.xlsx, with fields and relationships defined in the metadata table. Code for the multi-taxon model is contained in the RMarkdown document WhiteWalsh_S1.Rmd, which was used to produce Supplementary material S1 for the paper. Similarly, WhiteWalsh_S2.Rmd was used to produce Supplementary material S2. Code producing the figures of the paper are contained in WhiteWalshMethodsFigures.Rmd. Code for the map in Fig. 1 is also provided (WhiteWalshFig1Map.R), but the spatial data that it uses are not.

Language use differs dramatically from context to context. To some degree, modern language models like GPT-3 are able to account for such variance by conditioning on a string of previous input text, or prompt. Yet prompting is ineffective when contexts are sparse, out-of-sample, or extra-textual; for instance, accounting for when and where the text was produced or who produced it. In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes -- lightweight modules prepended to the input -- to account for structured variation. Specifically, we show how the popular class of mixed-effects models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout. We evaluate this approach on several domain-adaptation benchmarks, finding that it efficiently adapts to novel contexts with minimal data while still effectively generalizing to unseen contexts.

Psychometric testing is a valuable educational tool for the assessment and monitoring of students’ abilities in core subjects. However, the manual development of these tests is a tedious process requiring test specialists to produce and curate large volumes of high-quality items. In this paper, we consider whether automating test item generation with modern machine learning methods is a feasible solution for obtaining strong psychometric test items at low cost in the domain of sentence reading fluency. We assess the ability of the large neural language model GPT-3 to produce items ``few-shot’’--- from a short prompt with only a handful of examples. Our results show that generated items closely resemble standardized test items in terms of their factual ambiguity, content appropriateness, and complexity. Furthermore, after filtering for correct answer-labeling these generated items possess similar latent psychometric properties to standardized test items, even capturing subtle grade-level variation.

An overarching goal of natural language processing is to enable machines to communicate seamlessly with humans. However, natural language can be ambiguous or unclear. In cases of uncertainty, humans engage in an interactive process known as repair: asking questions and seeking clarification until their uncertainty is resolved. We propose a framework for building a visually grounded question-asking model capable of producing polar (yes-no) clarification questions to resolve misunderstandings in dialogue. Our model uses an expected information gain objective to derive informative questions from an off-the-shelf image captioner without requiring any supervised question-answer data. We demonstrate our model's ability to pose questions that improve communicative success in a goal-oriented 20 questions game with synthetic and human answerers., Comment: EMNLP 2021

A hallmark of human language is the ability to effectively and efficiently convey contextually relevant information. One theory for how humans reason about language is presented in the Rational Speech Acts (RSA) framework, which captures pragmatic phenomena via a process of recursive social reasoning (Goodman & Frank, 2016). However, RSA represents ideal reasoning in an unconstrained setting. We explore the idea that speakers might learn to amortize the cost of RSA computation over time by directly optimizing for successful communication with an internal listener model. In simulations with grounded neural speakers and listeners across two communication game datasets representing synthetic and human-generated data, we find that our amortized model is able to quickly generate language that is effective and concise across a range of contexts, without the need for explicit pragmatic reasoning., Accepted to CogSci 2020