Search

We present MITO, the first dataset of multi-spectral millimeter-wave (mmWave) images of everyday objects. Unlike visible light, mmWave signals can image through everyday occlusions (e.g., cardboard boxes, fabric, plastic). However, due to the dearth of publicly-available mmWave images and the interdisciplinary challenges in collecting and processing mmWave signals, it remains difficult today for computer vision researchers to develop mmWave-based non-line-of-sight perception algorithms and models. To overcome these challenges, we introduce a real-world dataset and open-source simulation tool for mmWave imaging. The dataset is acquired using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Through a signal processing pipeline, we capture and create over 580 real-world 3D mmWave images from over 76 different objects in the YCB dataset, a standard dataset for robotics manipulation. We provide real-world mmWave images in line-of-sight and non-line-of-sight, as well as RGB-D images and ground truth segmentation masks. We also develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh, which achieves a median F-Score of 94% when compared to real-world mmWave images. We show the usefulness of this dataset and simulation tool in multiple CV tasks in non-line-of-sight. First, we perform object segmentation for mmWave images using the segment anything model (SAM), and achieve a median precision and recall of 92.6% and 64%. Second, we train a classifier that can recognize objects in non-line-of-sight. It is trained on synthetic images and can classify real-world images with 85% accuracy. We believe MITO will be a valuable resource for computer vision researchers in developing non-line-of-sight perception, similar to how early camera-based datasets shaped the field.

To optimize interventions for improving wellness, it is essential to understand habits, which wearable devices can measure with greater precision. Using high temporal resolution biometric data taken from the Oura Gen3 ring, we examine daily and weekly sleep and activity patterns of a cohort of young adults (N=582) in their first semester of college. A high compliance rate is observed for both daily and nightly wear, with slight dips in wear compliance observed shortly after waking up and also in the evening. Most students have a late-night chronotype with a median midpoint of sleep at 5AM, with males and those with mental health impairment having more delayed sleep periods. Social jetlag, or the difference in sleep times between free days and school days, is prevalent in our sample. While sleep periods generally shift earlier on weekdays and later on weekends, sleep duration on both weekdays and weekends is shorter than during prolonged school breaks, suggesting chronic sleep debt when school is in session. Synchronized spikes in activity consistent with class schedules are also observed, suggesting that walking in between classes is a widespread behavior in our sample that substantially contributes to physical activity. Lower active calorie expenditure is associated with weekends and a delayed but longer sleep period the night before, suggesting that for our cohort, active calorie expenditure is affected less by deviations from natural circadian rhythms and more by the timing associated with activities. Our study shows that regular sleep and activity routines may be inferred from consumer wearable devices if high temporal resolution and long data collection periods are available.

Foundation models have vast potential to enable diverse AI applications. The powerful yet incomplete nature of these models has spurred a wide range of mechanisms to augment them with capabilities such as in-context learning, information retrieval, and code interpreting. We propose Vieira, a declarative framework that unifies these mechanisms in a general solution for programming with foundation models. Vieira follows a probabilistic relational paradigm and treats foundation models as stateless functions with relational inputs and outputs. It supports neuro-symbolic applications by enabling the seamless combination of such models with logic programs, as well as complex, multi-modal applications by streamlining the composition of diverse sub-models. We implement Vieira by extending the Scallop compiler with a foreign interface that supports foundation models as plugins. We implement plugins for 12 foundation models including GPT, CLIP, and SAM. We evaluate Vieira on 9 challenging tasks that span language, vision, and structured and vector databases. Our evaluation shows that programs in Vieira are concise, can incorporate modern foundation models, and have comparable or better accuracy than competitive baselines.

Tokenization is a necessary component within the current architecture of many language models, including the transformer-based large language models (LLMs) of Generative AI, yet its impact on the model's cognition is often overlooked. We argue that LLMs demonstrate that the Distributional Hypothesis (DH) is sufficient for reasonably human-like language performance, and that the emergence of human-meaningful linguistic units among tokens motivates linguistically-informed interventions in existing, linguistically-agnostic tokenization techniques, particularly with respect to their roles as (1) semantic primitives and as (2) vehicles for conveying salient distributional patterns from human language to the model. We explore tokenizations from a BPE tokenizer; extant model vocabularies obtained from Hugging Face and tiktoken; and the information in exemplar token vectors as they move through the layers of a RoBERTa (large) model. Besides creating sub-optimal semantic building blocks and obscuring the model's access to the necessary distributional patterns, we describe how tokenization pretraining can be a backdoor for bias and other unwanted content, which current alignment practices may not remediate. Additionally, we relay evidence that the tokenization algorithm's objective function impacts the LLM's cognition, despite being meaningfully insulated from the main system intelligence.

The common phrase 'representation matters' asserts that media has a measurable and important impact on civic society's perception of self and others. The representation of health in media, in particular, may reflect and perpetuate a society's disease burden. Here, for the top 10 major causes of death in the United States, we examine how cinematic representation of overall and by-gender mortality diverges from reality. Using crowd-sourced data on film deaths from Cinemorgue Wiki, we employ natural language processing (NLP) techniques to analyze shifts in representation of deaths in movies versus the 2021 National Vital Statistic Survey (NVSS) top ten mortality causes. Overall, movies strongly overrepresent suicide and, to a lesser degree, accidents. In terms of gender, movies overrepresent men and underrepresent women for nearly every major mortality cause, including heart disease and cerebrovascular disease. The two exceptions for which women are overrepresented are suicide and accidents. We discuss the implications of under- and over-representing causes of death overall and by gender, as well as areas of future research., Comment: 10 pages, 2 figures, 1 appendix page

We present Crux, a cross-language verification tool for Rust and C/LLVM. Crux targets bounded, intricate pieces of code that are difficult for humans to get right: for example, cryptographic modules and serializer / deserializer pairs. Crux builds on the same framework as the mature SAW-Cryptol toolchain, but Crux provides an interface where proofs are phrased as symbolic unit tests. Crux is designed for use in production environments, and has already seen use in industry. In this paper, we focus on Crux-MIR, our verification tool for Rust. Crux-MIR provides a bit-precise model of safe and unsafe Rust which can be used to check both inline properties about Rust code, and extensional equality to executable specifications written in Cryptol or in the hacspec dialect of Rust. Notably, Crux-MIR supports compositional reasoning, which is necessary to scale to even moderately complex proofs. We demonstrate Crux-MIR by verifying the Ring library implementations of SHA1 and SHA2 against pre-existing functional specifications. Crux is available at https://crux.galois.com.

The effective adoption of responsible AI practices in journalism requires a concerted effort to bridge different perspectives, including technological, editorial, journalistic, and managerial. Among the many challenges that could impact information sharing around responsible AI inside news organizations are knowledge silos, where information is isolated within one part of the organization and not easily shared with others. This study aims to explore if, and if so, how, knowledge silos affect the adoption of responsible AI practices in journalism through a cross-case study of four major Dutch media outlets. We examine the individual and organizational barriers to AI knowledge sharing and the extent to which knowledge silos could impede the operationalization of responsible AI initiatives inside newsrooms. To address this question, we conducted 14 semi-structured interviews with editors, managers, and journalists at de Telegraaf, de Volkskrant, the Nederlandse Omroep Stichting (NOS), and RTL Nederland. The interviews aimed to uncover insights into the existence of knowledge silos, their effects on responsible AI practice adoption, and the organizational practices influencing these dynamics. Our results emphasize the importance of creating better structures for sharing information on AI across all layers of news organizations.

Developed as a response to the increasing popularity of data-driven journalism, automated journalism refers to the process of automating the collection, production, and distribution of news content and other data with the assistance of computer programs. Although the algorithmic technologies associated with automated journalism remain in the initial stage of development, early adopters have already praised the usefulness of automated journalism for generating routine news based on clean, structured data. Most noticeably, the Associated Press and The New York Times have been automating news content to cover financial and sports issues for over a decade. Nevertheless, research on automated journalism is also alerting to the dangers of using algorithms for news creation and distribution, including the possible bias behind AI systems or the human bias of those who develop computer programs. The popularization of automated news content also has important implications for the infrastructure of the newsroom, the role performance of journalists and other non-journalistic professionals, and the distribution of news content to a datafied audience., Comment: This is a preprint of an entry in Nai, A., Gr\"omping, M., & Wirz, D. (Eds). Elgar Encyclopedia of Political Communication. Edward Elgar Publishing

The National Science Foundation's Daniel K. Inouye Solar Telescope (DKIST) will provide high-resolution, multi-line spectropolarimetric observations that are poised to revolutionize our understanding of the Sun. Given the massive data volume, novel inference techniques are required to unlock its full potential. Here, we provide an overview of our "SPIn4D" project, which aims to develop deep convolutional neural networks (CNNs) for estimating the physical properties of the solar photosphere from DKIST spectropolarimetric observations. We describe the magnetohydrodynamic (MHD) modeling and the Stokes profile synthesis pipeline that produce the simulated output and input data, respectively. These data will be used to train a set of CNNs that can rapidly infer the four-dimensional MHD state vectors by exploiting the spatiotemporally coherent patterns in the Stokes profile time series. Specifically, our radiative MHD model simulates the small-scale dynamo actions that are prevalent in quiet-Sun and plage regions. Six cases with different mean magnetic fields have been conducted; each case covers six solar-hours, totaling 109 TB in data volume. The simulation domain covers at least $25\times25\times8$ Mm with $16\times16\times12$ km spatial resolution, extending from the upper convection zone up to the temperature minimum region. The outputs are stored at a 40 s cadence. We forward model the Stokes profile of two sets of Fe I lines at 630 and 1565 nm, which will be simultaneously observed by DKIST and can better constrain the parameter variations along the line of sight. The MHD model output and the synthetic Stokes profiles are publicly available, with 13.7 TB in the initial release., Comment: Submitted to AAS Journals after revision. Comments welcome

When attempting to understand the behavior of an executable, a binary analyst can make use of many different techniques. These include program slicing, dynamic instrumentation, binary-level rewriting, symbolic execution, and formal verification, all of which can uncover insights into how a piece of machine code behaves. As a result, there is no one-size-fits-all binary analysis tool, so a binary analysis researcher will often combine several different tools. Sometimes, a researcher will even need to design new tools to study problems that existing frameworks are not well equipped to handle. Designing such tools from complete scratch is rarely time- or cost-effective, however, given the scale and complexity of modern ISAs. We present Macaw, a modular framework that makes it possible to rapidly build reliable binary analysis tools across a range of use cases. Statically typed functional programming techniques are used pervasively throughout Macaw -- these range from using functional optimization passes to encoding tricky architectural invariants at the type level to statically check correctness properties. The level of assurance that functional programming ideas afford us allow us to iterate rapidly on Macaw while still having confidence that the underlying semantics are correct. Over a decade of development, we have used Macaw to support an industrial research team in building tools for machine code-related tasks. As such, the name 'Macaw' refers not just to the framework, but also a suite of tools that are built on top of it. We describe Macaw in depth and describe the different static and dynamic analyses that it performs, many powered by an SMT-based symbolic execution engine. We put a particular focus on interoperability between machine code and higher-level languages, including binary lifting from x86 to LLVM, as well verifying the correctness of mixed C and assembly code.

Large language models (LLMs) show promise in code translation - the task of translating code written in one programming language to another language - due to their ability to write code in most programming languages. However, LLM's effectiveness on translating real-world code remains largely unstudied. In this work, we perform the first substantial study on LLM-based translation to Rust by assessing the ability of five state-of-the-art LLMs, GPT4, Claude 3, Claude 2.1, Gemini Pro, and Mixtral. We conduct our study on code extracted from real-world open source projects. To enable our study, we develop FLOURINE, an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program, eliminating the need for pre-existing test cases. As part of our investigation, we assess both the LLM's ability to produce an initially successful translation, as well as their capacity to fix a previously generated buggy one. If the original and the translated programs are not I/O equivalent, we apply a set of automated feedback strategies, including feedback to the LLM with counterexamples. Our results show that the most successful LLM can translate 47% of our benchmarks, and also provides insights into next steps for improvements., Comment: 11 pages, 12 figures

With the increasing adoption of artificial intelligence (AI) technologies in the news industry, media organizations have begun publishing guidelines that aim to promote the responsible, ethical, and unbiased implementation of AI-based technologies. These guidelines are expected to serve journalists and media workers by establishing best practices and a framework that helps them navigate ever-evolving AI tools. Drawing on institutional theory and digital inequality concepts, this study analyzes 37 AI guidelines for media purposes in 17 countries. Our analysis reveals key thematic areas, such as transparency, accountability, fairness, privacy, and the preservation of journalistic values. Results highlight shared principles and best practices that emerge from these guidelines, including the importance of human oversight, explainability of AI systems, disclosure of automated content, and protection of user data. However, the geographical distribution of these guidelines, highlighting the dominance of Western nations, particularly North America and Europe, can further ongoing concerns about power asymmetries in AI adoption and consequently isomorphism outside these regions. Our results may serve as a resource for news organizations, policymakers, and stakeholders looking to navigate the complex AI development toward creating a more inclusive and equitable digital future for the media industry worldwide., Comment: 32 pages, 2 tables, 3 figures

Rust is a programming language that combines memory safety and low-level control, providing C-like performance while guaranteeing the absence of undefined behaviors by default. Rust's growing popularity has prompted research on safe and correct transpiling of existing code-bases to Rust. Existing work falls into two categories: rule-based and large language model (LLM)-based. While rule-based approaches can theoretically produce correct transpilations that maintain input-output equivalence to the original, they often yield unreadable Rust code that uses unsafe subsets of the Rust language. On the other hand, while LLM-based approaches typically produce more readable, maintainable, and safe code, they do not provide any guarantees about correctness. In this work, we present VERT, a tool that can produce readable Rust transpilations with formal guarantees of correctness. VERT's only requirement is that there is Web Assembly compiler for the source language, which is true for most major languages. VERT first uses the Web Assembly compiler to obtain an oracle Rust program. In parallel, VERT uses an LLM to generate a readable candidate Rust program. This candidate is verified against the oracle, and if verification fails, we regenerate a new candidate transpilation until verification succeeds. We evaluate VERT by transpiling a suite of 1,394 programs taken from competitive programming style benchmarks. Combining Anthropic's Claude-2 and VERT increases Rust transpilations passing property-based testing from 31% to 54% and bounded model-checking from 1% to 42% compared to using Claude alone. In addition, we evaluate VERT's ability to generate non-trivial safe Rust on programs taken from real-world C projects that make significant use of pointers. Our results provide insights into the limitations of LLMs to write safe Rust.

Generative AI tools are used to create art-like outputs and sometimes aid in the creative process. These tools have potential benefits for artists, but they also have the potential to harm the art workforce and infringe upon artistic and intellectual property rights. Without explicit consent from artists, Generative AI creators scrape artists' digital work to train Generative AI models and produce art-like outputs at scale. These outputs are now being used to compete with human artists in the marketplace as well as being used by some artists in their generative processes to create art. We surveyed 459 artists to investigate the tension between artists' opinions on Generative AI art's potential utility and harm. This study surveys artists' opinions on the utility and threat of Generative AI art models, fair practices in the disclosure of artistic works in AI art training models, ownership and rights of AI art derivatives, and fair compensation. Results show that a majority of artists believe creators should disclose what art is being used in AI training, that AI outputs should not belong to model creators, and express concerns about AI's impact on the art workforce and who profits from their art. We hope the results of this work will further meaningful collaboration and alignment between the art community and Generative AI researchers and developers.

Many computational chemistry and molecular simulation workflows can be expressed as graphs. This abstraction is useful to modularize and potentially reuse existing components, as well as provide parallelization and ease reproducibility. Existing tools represent the computation as a directed acyclic graph (DAG), thus allowing efficient execution by parallelization of concurrent branches. These systems can, however, generally not express cyclic and conditional workflows. We therefore developed Maize, a workflow manager for cyclic and conditional graphs based on the principles of flow-based programming. By running each node of the graph concurrently in separate processes and allowing communication at any time through dedicated inter-node channels, arbitrary graph structures can be executed. We demonstrate the effectiveness of the tool on a dynamic active learning task in computational drug design, involving the use of a small molecule generative model and an associated scoring system, and on a reactivity prediction pipeline using quantum-chemistry and semiempirical approaches.

Human feedback can prevent overtly harmful utterances in conversational models, but may not automatically mitigate subtle problematic behaviors such as a stated desire for self-preservation or power. Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles. We find this approach effectively prevents the expression of such behaviors. The success of simple principles motivates us to ask: can models learn general ethical behaviors from only a single written principle? To test this, we run experiments using a principle roughly stated as "do what's best for humanity". We find that the largest dialogue models can generalize from this short constitution, resulting in harmless assistants with no stated interest in specific motivations like power. A general principle may thus partially avoid the need for a long list of constitutions targeting potentially harmful behaviors. However, more detailed constitutions still improve fine-grained control over specific types of harms. This suggests both general and specific principles have value for steering AI safely.

Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses., Comment: 32 pages, 20 figures

We present the design, implementation, and evaluation of MiFly, a self-localization system for autonomous drones that works across indoor and outdoor environments, including low-visibility, dark, and GPS-denied settings. MiFly performs 6DoF self-localization by leveraging a single millimeter-wave (mmWave) anchor in its vicinity - even if that anchor is visually occluded. MmWave signals are used in radar and 5G systems and can operate in the dark and through occlusions. MiFly introduces a new mmWave anchor design and mounts light-weight high-resolution mmWave radars on a drone. By jointly designing the localization algorithms and the novel low-power mmWave anchor hardware (including its polarization and modulation), the drone is capable of high-speed 3D localization. Furthermore, by intelligently fusing the location estimates from its mmWave radars and its IMUs, it can accurately and robustly track its 6DoF trajectory. We implemented and evaluated MiFly on a DJI drone. We demonstrate a median localization error of 7cm and a 90th percentile less than 15cm, even when the anchor is fully occluded (visually) from the drone.

Despite a growing sample of precisely measured stellar rotation periods and ages, the strength of magnetic braking and the degree of departure from standard (Skumanich-like) spindown have remained persistent questions, particularly for stars more evolved than the Sun. Rotation periods can be measured for stars older than the Sun by leveraging asteroseismology, enabling models to be tested against a larger sample of old field stars. Because asteroseismic measurements of rotation do not depend on starspot modulation, they avoid potential biases introduced by the need for a stellar dynamo to drive starspot production. Using a neural network trained on a grid of stellar evolution models and a hierarchical model-fitting approach, we constrain the onset of weakened magnetic braking. We find that a sample of stars with asteroseismically-measured rotation periods and ages is consistent with models that depart from standard spindown prior to reaching the evolutionary stage of the Sun. We test our approach using neural networks trained on model grids produced by separate stellar evolution codes with differing physical assumptions and find that the choices of grid physics can influence the inferred properties of the braking law. We identify the normalized critical Rossby number ${\rm Ro}_{\rm crit}/{\rm Ro}_\odot = 0.91\pm0.03$ as the threshold for the departure from standard rotational evolution. This suggests that weakened magnetic braking poses challenges to gyrochronology for roughly half of the main sequence lifetime of sun-like stars., Comment: 26 pages, 10 figures

Over the past decade, the gender pay gap has remained steady with women earning 84 cents for every dollar earned by men on average. Many studies explain this gap through demand-side bias in the labor market represented through employers' job postings. However, few studies analyze potential bias from the worker supply-side. Here, we analyze the language in millions of US workers' resumes to investigate how differences in workers' self-representation by gender compare to differences in earnings. Across US occupations, language differences between male and female resumes correspond to 11% of the variation in gender pay gap. This suggests that females' resumes that are semantically similar to males' resumes may have greater wage parity. However, surprisingly, occupations with greater language differences between male and female resumes have lower gender pay gaps. A doubling of the language difference between female and male resumes results in an annual wage increase of $2,797 for the average female worker. This result holds with controls for gender-biases of resume text and we find that per-word bias poorly describes the variance in wage gap. The results demonstrate that textual data and self-representation are valuable factors for improving worker representations and understanding employment inequities., Comment: 24 pages, 15 figures

As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics. By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior., Comment: For few-shot examples and prompts, see https://github.com/anthropics/DecompositionFaithfulnessPaper

Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.