Search

We introduce the Situated Corpus Of Understanding Transactions (SCOUT), a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multiple Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. SCOUT contains 89,056 utterances and 310,095 words from 278 dialogues averaging 320 utterances per dialogue. The dialogues are aligned with the multi-modal data streams available during the experiments: 5,785 images and 30 maps. The corpus has been annotated with Abstract Meaning Representation and Dialogue-AMR to identify the speaker's intent and meaning within an utterance, and with Transactional Units and Relations to track relationships between utterances to reveal patterns of the Dialogue Structure. We describe how the corpus and its annotations have been used to develop autonomous human-robot systems and enable research in open questions of how humans speak to robots. We release this corpus to accelerate progress in autonomous, situated, human-robot dialogue, especially in the context of navigation tasks where details about the environment need to be discovered., Comment: 14 pages, 7 figures

In this paper, we describe the development of symbolic representations annotated on human-robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue (occurring in disaster relief or search-and-rescue tasks), where a human and robot are engaged in a joint navigation and exploration task of an unfamiliar environment, but where the robot cannot immediately share high quality visual information due to limited communication constraints. Engaging in a dialogue provides an effective way to communicate, while on-demand or lower-quality visual information can be supplemented for establishing common ground. Within this paradigm, we capture propositional semantics and the illocutionary force of a single utterance within the dialogue through our Dialogue-AMR annotation, an augmentation of Abstract Meaning Representation. We then capture patterns in how different utterances within and across speaker floors relate to one another in our development of a multi-floor Dialogue Structure annotation schema. Finally, we begin to annotate and analyze the ways in which the visual modalities provide contextual information to the dialogue for overcoming disparities in the collaborators' understanding of the environment. We conclude by discussing the use-cases, architectures, and systems we have implemented from our annotations that enable physical robots to autonomously engage with humans in bi-directional dialogue and navigation., Comment: 52 pages, 14 figures

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

Complex news events, such as natural disasters and socio-political conflicts, require swift responses from the government and society. Relying on historical events to project the future is insufficient as such events are sparse and do not cover all possible conditions and nuanced situations. Simulation of these complex events can help better prepare and reduce the negative impact. We develop a controllable complex news event simulator guided by both the event schema representing domain knowledge about the scenario and user-provided assumptions representing case-specific conditions. As event dynamics depend on the fine-grained social and cultural context, we further introduce a geo-diverse commonsense and cultural norm-aware knowledge enhancement component. To enhance the coherence of the simulation, apart from the global timeline of events, we take an agent-based approach to simulate the individual character states, plans, and actions. By incorporating the schema and cultural norms, our generated simulations achieve much higher coherence and appropriateness and are received favorably by participants from a humanitarian assistance organization., Comment: Accepted as EMNLP 2024 Demo

Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research., Comment: 15 pages, 2 figures, project page: https://genea-workshop.github.io/leaderboard/

Hexagonal boron nitride encapsulation is the method of choice for protecting graphene from environmental doping and impurity scattering. It was previously demonstrated that metal-organic vapor phase epitaxy (MOVPE) grows epitaxially ordered, uniform BN layers on epigraphene (graphene grown on SiC). Due to graphene non-wetting properties, h-BN growth starts preferentially from the graphene ledges. We use this fact here to selectively promote growth of high-quality flat h-BN on epigraphene by patterning epigraphene microstructures prior to BN growth. Thin h-BN films (down to 6 nm) grown by MOVPE show smooth and pleated surface morphology on epigraphene, while crumpled BN is observed on the SiC. Cross-sectional high-resolution transmission electron microscopy images and fluorescence imaging confirm the higher BN quality grown on the epigraphene. Transport measurements reveal p-doping as expected from hydrogen intercalation of epigraphene and regions of high and low mobility. This method can be used to produce structurally uniform high-quality h-BN/epigraphene micro/nano scale heterostructure.

Alkaline-earth-like elements play pivotal roles in advanced quantum sensing technologies, notably optical clocks, with unprecedented precision achieved in recent years. Despite remarkable progress, current optical lattice clocks still face challenges in meeting the demanding size, weight, and power consumption constraints essential for space applications. Conventional atom sources, such as ovens or dispensers, require substantial heating power, making up a significant fraction of the system's overall power consumption. Addressing this challenge, we present a novel microstructured atomic oven based on fused silica, designed for miniaturization and low-power operation. We characterize the oven by loading a magneto-optical trap with Yb evaporated from the oven and demonstrate operation with a loading rate above $10^8$ $\mathrm{atoms}/\mathrm{s}$ for heating powers below $250$ $\mathrm{mW}$., Comment: 7 pages, 10 figures

For plane strain linear elasticity, given any anisotropic elasticity tensor $\mathbb{C}_{\rm aniso}$, we determine a best approximating isotropic counterpart $\mathbb{C}_{\rm iso}$. This is not done by using a distance measure on the space of positive definite elasticity tensors (Euclidean or logarithmic distance) but by considering two simple isotropic analytic solutions (center of dilatation and concentrated couple) and best fitting these radial solutions to the numerical anisotropic solution based on $\mathbb{C}_{\rm aniso}$. The numerical solution is done via a finite element calculation, and the fitting via a subsequent quadratic error minimization. Thus, we obtain the two Lam\'e-moduli $\mu$, $\lambda$ (or $\mu$ and the bulk-modulus $\kappa$) of $\mathbb{C}_{\rm aniso}$. We observe that our so-determined isotropic tensor $\mathbb{C}_{\rm iso}$ coincides with neither the best logarithmic fit of Norris nor the best Euclidean fit. Our result calls into question the very notion of a best-fit isotropic elasticity tensor to a given anisotropic material.

Coherence in writing, an aspect that second-language (L2) English learners often struggle with, is crucial in assessing L2 English writing. Existing automated writing evaluation systems primarily use basic surface linguistic features to detect coherence in writing. However, little effort has been made to correct the detected incoherence, which could significantly benefit L2 language learners seeking to improve their writing. To bridge this gap, we introduce DECOR, a novel benchmark that includes expert annotations for detecting incoherence in L2 English writing, identifying the underlying reasons, and rewriting the incoherent sentences. To our knowledge, DECOR is the first coherence assessment dataset specifically designed for improving L2 English writing, featuring pairs of original incoherent sentences alongside their expert-rewritten counterparts. Additionally, we fine-tuned models to automatically detect and rewrite incoherence in student essays. We find that incorporating specific reasons for incoherence during fine-tuning consistently improves the quality of the rewrites, achieving a result that is favored in both automatic and human evaluations., Comment: EMNLP 2024 Main, 23 pages, 5 figures, 20 tables

Quantum error correction suppresses noise in quantum systems to allow for high-precision computations. In this work, we introduce Multivariate Bicycle (MB) Quantum Low-Density Parity-Check (QLDPC) codes, via an extension of the framework developed by Bravyi et al. [Nature, 627, 778-782 (2024)] and particularly focus on Trivariate Bicycle (TB) codes. Unlike the weight-6 codes proposed in their study, we offer concrete examples of weight-4 and weight-5 TB-QLDPC codes which promise to be more amenable to near-term experimental setups. We show that our TB-QLDPC codes up to weight-6 have a bi-planar structure. Further, most of our new codes can also be arranged in a two-dimensional toric layout, and have substantially better encoding rates than comparable surface codes while offering similar error suppression capabilities. For example, we can encode 4 logical qubits with distance 5 into 30 physical qubits with weight-5 check measurements, while a surface code with these parameters requires 100 physical qubits. The high encoding rate and compact layout make our codes highly suitable candidates for near-term hardware implementations, paving the way for a realizable quantum error correction protocol., Comment: 4 + 14 pages, 17 figures

In human interaction, gestures serve various functions such as marking speech rhythm, highlighting key elements, and supplementing information. These gestures are also observed in explanatory contexts. However, the impact of gestures on explanations provided by virtual agents remains underexplored. A user study was carried out to investigate how different types of gestures influence perceived interaction quality and listener understanding. This study addresses the effect of gestures in explanation by developing an embodied virtual explainer integrating both beat gestures and iconic gestures to enhance its automatically generated verbal explanations. Our model combines beat gestures generated by a learned speech-driven synthesis module with manually captured iconic gestures, supporting the agent's verbal expressions about the board game Quarto! as an explanation scenario. Findings indicate that neither the use of iconic gestures alone nor their combination with beat gestures outperforms the baseline or beat-only conditions in terms of understanding. Nonetheless, compared to prior research, the embodied agent significantly enhances understanding.

An algorithm is provided to tile the plane with the aperiodic monotile Tile(1,1) recently discovered by Smith et al. (2023). Their geometric construction guidelines are expanded into a numerical MATLAB algorithm. The intention is to remove a possible obstacle for researchers interested in applications of this fundamental tiling of the plane., Comment: 7 pages, 2 figures

We propose that a high-quality flat interface or boundary may act as a long-range skew scatterer for charged quasiparticles in a metal. In the presence of electric current parallel to the interface the number of clockwise and anti-clockwise reflections from the interface are no longer compensated. This results in a net orbital magnetization that can be observed by magneto-electric Kerr effect in a vicinity of the mean free path from the edge. The value of effective magnetization is given by the product of charge current density and the electron mean free path. Unlike orbital Hall or orbital Edelstein effects the proposed phenomenon does not require inversion symmetry breaking in the bulk of the sample., Comment: 6 pages, 4 figures

The trustworthiness of AI applications has been the subject of recent research and is also addressed in the EU's recently adopted AI Regulation. The currently emerging foundation models in the field of text, speech and image processing offer completely new possibilities for developing AI applications. This whitepaper shows how the trustworthiness of an AI application developed with foundation models can be evaluated and ensured. For this purpose, the application-specific, risk-based approach for testing and ensuring the trustworthiness of AI applications, as developed in the 'AI Assessment Catalog - Guideline for Trustworthy Artificial Intelligence' by Fraunhofer IAIS, is transferred to the context of foundation models. Special consideration is given to the fact that specific risks of foundation models can have an impact on the AI application and must also be taken into account when checking trustworthiness. Chapter 1 of the white paper explains the fundamental relationship between foundation models and AI applications based on them in terms of trustworthiness. Chapter 2 provides an introduction to the technical construction of foundation models and Chapter 3 shows how AI applications can be developed based on them. Chapter 4 provides an overview of the resulting risks regarding trustworthiness. Chapter 5 shows which requirements for AI applications and foundation models are to be expected according to the draft of the European Union's AI Regulation and Chapter 6 finally shows the system and procedure for meeting trustworthiness requirements., Comment: 24 pages, 11 figures

Multicellular tumor spheroids (MCTS) are advanced cell culture systems for assessing the impact of combinatorial radio(chemo)therapy. They exhibit therapeutically relevant in-vivo-like characteristics from 3D cell-cell and cell-matrix interactions to radial pathophysiological gradients related to proliferative activity and nutrient/oxygen supply, altering cellular radioresponse. State-of-the-art assays quantify long-term curative endpoints based on collected brightfield image time series from large treated spheroid populations per irradiation dose and treatment arm. Here, spheroid control probabilities are documented analogous to in-vivo tumor control probabilities based on Kaplan-Meier curves. This analyses require laborious spheroid segmentation of up to 100.000 images per treatment arm to extract relevant structural information from the images, e.g., diameter, area, volume and circularity. While several image analysis algorithms are available for spheroid segmentation, they all focus on compact MCTS with clearly distinguishable outer rim throughout growth. However, treated MCTS may partly be detached and destroyed and are usually obscured by dead cell debris. We successfully train two Fully Convolutional Networks, UNet and HRNet, and optimize their hyperparameters to develop an automatic segmentation for both untreated and treated MCTS. We systematically validate the automatic segmentation on larger, independent data sets of spheroids derived from two human head-and-neck cancer cell lines. We find an excellent overlap between manual and automatic segmentation for most images, quantified by Jaccard indices at around 90%. For images with smaller overlap of the segmentations, we demonstrate that this error is comparable to the variations across segmentations from different biological experts, suggesting that these images represent biologically unclear or ambiguous cases., Comment: 30 pages, 23 figures

Incremental learning is the ability of systems to acquire knowledge over time, enabling their adaptation and generalization to novel tasks. It is a critical ability for intelligent, real-world systems, especially when data changes frequently or is limited. This review provides a comprehensive analysis of incremental learning in Large Language Models. It synthesizes the state-of-the-art incremental learning paradigms, including continual learning, meta-learning, parameter-efficient learning, and mixture-of-experts learning. We demonstrate their utility for incremental learning by describing specific achievements from these related topics and their critical factors. An important finding is that many of these approaches do not update the core model, and none of them update incrementally in real-time. The paper highlights current problems and challenges for future research in the field. By consolidating the latest relevant research developments, this review offers a comprehensive understanding of incremental learning and its implications for designing and developing LLM-based learning systems.

It is shown on the examples of Moore and Gosper curves that two spatially shifted or twisted, pre-asymptotic space-filling curves can produce large-scale superstructures akin to moir\'e patterns. To study physical phenomena emerging from these patterns, a geometrical coupling coefficient based on the Neumann integral is introduced. It is found that moir\'e patterns appear most defined at the peaks of those coefficients. A physical interpretation of these coefficients as a measure for inductive coupling between radiofrequency resonators leads to a design principle for strongly overlapping resonators with vanishing mutual inductance, which might be interesting for applications in MRI. These findings are demonstrated in graphical, numerical, and physical experiments., Comment: 10 pages, 6 figures

Wetting and dewetting dynamics of simple and complex liquids is described by kinetic equations in gradient dynamics form that incorporates the various coupled dissipative processes in a fully thermodynamically consistent manner. After briefly reviewing this, we also review how chemical reactions can be captured by a related gradient dynamics description, assuming detailed balanced mass action type kinetics. Then, we bring both aspects together and discuss mesoscopic reactive thin-film hydrodynamics illustrated by two examples, namely, models for reactive wetting and reactive surfactants. These models can describe the approach to equilibrium but may also be employed to study out-of-equilibrium chemo-mechanical dynamics. In the latter case, one breaks the gradient dynamics form by chemostatting to obtain active systems. In this way, for reactive wetting we recover running drops that are driven by chemically sustained wettability gradients and for drops covered by autocatalytic reactive surfactants we find complex forms of self-propulsion and self-excited oscillations.

Regulation of cell proliferation is a crucial aspect of tissue development and homeostasis and plays a major role in morphogenesis, wound healing, and tumor invasion. A phenomenon of such regulation is contact inhibition, which describes the dramatic slowing of proliferation, cell migration and individual cell growth when multiple cells are in contact with each other. While many physiological, molecular and genetic factors are known, the mechanism of contact inhibition is still not fully understood. In particular, the relevance of cellular signaling due to interfacial contact for contact inhibition is still debated. Cellular automata (CA) have been employed in the past as numerically efficient mathematical models to study the dynamics of cell ensembles, but they are not suitable to explore the origins of contact inhibition as such agent-based models assume fixed cell sizes. We develop a minimal, data-driven model to simulate the dynamics of planar cell cultures by extending a probabilistic CA to incorporate size changes of individual cells during growth and cell division. We successfully apply this model to previous in-vitro experiments on contact inhibition in epithelial tissue: After a systematic calibration of the model parameters to measurements of single-cell dynamics, our CA model quantitatively reproduces independent measurements of emergent, culture-wide features, like colony size, cell density and collective cell migration. In particular, the dynamics of the CA model also exhibit the transition from a low-density confluent regime to a stationary postconfluent regime with a rapid decrease in cell size and motion. This implies that the volume exclusion principle, a mechanical constraint which is the only inter-cellular interaction incorporated in the model, paired with a size-dependent proliferation rate is sufficient to generate the observed contact inhibition., Comment: 15 pages, 17 figures

Point defects have a strong influence on the physical properties of materials, often dominating the electronic and optical behavior in semiconductors and insulators. The simulation and analysis of point defects is therefore crucial for understanding the growth and operation of materials especially for optoelectronics applications. In this work, we present a general-purpose Python framework for the analysis of point defects in crystalline materials, as well as a generalized workflow for their treatment with high-throughput simulations. The distinguishing feature of our approach is an emphasis on a unique, unitcell, structure-only, definition of point defects which decouples the defect definition and the specific supercell representation used to simulate the defect. This allows the results of first-principles calculations to be aggregated into a database without extensive provenance information and is a crucial step in building a persistent database of point defects that can grow over time, a key component towards realizing the idea of a ``defect genome' that can yield more complex relationships governing the behavior of defects in materials. We demonstrate several examples of the approach for three technologically relevant materials and highlight current pitfalls that must be considered when employing these methodologies, as well as their potential solutions.

The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at https://wmdp.ai, Comment: See the project page at https://wmdp.ai

Cardiac pulsations in the human brain have received recent interest due to their possible role in the pathogenesis of neurodegenerative diseases. Further interest stems from their possible application as an endogenous signal source that can be utilized for brain imaging in general. The (pulse-)wave describing the blood flow velocity along an intracranial artery consists of a forward (anterograde) and a backward (retrograde, reflected) part, but measurements of this wave usually consist of a superposition of these components. In this paper, we provide a mathematical framework for the inverse problem of estimating the pulse wave velocity, as well as the forward and backward component of the pulse wave separately from MRI measurements on intracranial arteries. After a mathematical analysis of this problem, we consider possible reconstruction approaches, and derive an alternate direction approach for its solution. The resulting methods provide estimates for anterograde/retrograde wave forms and the pulse wave velocity under specified assumptions on a cerebrovascular model system. Numerical experiments on simulated and experimental data demonstrate the applicability and preliminary in vivo feasibility of the proposed methods., Comment: 31 pages, 11 figures