Search

The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-art performance on certain benchmarks for risks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks. Training models to incorporate a chain of thought before answering has the potential to unlock substantial benefits, while also increasing potential risks that stem from heightened intelligence. Our results underscore the need for building robust alignment methods, extensively stress-testing their efficacy, and maintaining meticulous risk management protocols. This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.

Colloid-polymer mixtures are an archetype for modeling phase transition processes, as they a exhibit low-density gas phase, high-density crystalline phase and an intervening liquid phase. While their equilibrium behavior has been studied extensively, the role of hydrodynamics in driving their phase separation is not yet understood. We present a theoretical model that describes hydrodynamic interactions in colloid-polymer mixtures in a microgravity environment. Our phase-field model consists of the Cahn-Hilliard equation, which describes phase separation processes in multicomponent mixtures, coupled with the Stokes equation for viscous fluid flow. We account for the dependence of the suspension viscosity on the colloid concentration, and the so-called Korteweg stresses that arise at the interfaces of colloidal phases. We process video microscopy images from NASA's Binary Colloid Alloy Test (BCAT) experiments, which were performed on the International Space Station. While terrestrial experiments would be dominated by gravitational forces and buoyancy-driven flows, the microgravity environment of the BCAT experiments allows for the visualization of phase separation by low interfacial tension, and thus enables a quantitative comparison between experiment and our model predictions.

We introduce an abstract notion of a 3D-rotation module for a group $G$ that does not require the module to carry a vector space structure, a priori nor a posteriori. Our first result shows that, under an expected irreducibility-like assumption, the only finite $G$ with such a module are those already known from the classical setting: $\operatorname{Alt}(4)$, $\operatorname{Sym}(4)$, and $\operatorname{Alt}(5)$. Our second result then studies the module structure when $G = \operatorname{Alt}(5)$ and shows that, under certain natural restrictions, it is fully determined and generalizes that of the classical icosahedral module. We include an application to the recently introduced setting of modules with an additive dimension, a general setting allowing for simultaneous treatment of classical representation theory of finite groups as well as representations within various well-behaved model-theoretic settings such as the $o$-minimal and finite Morley rank ones. Leveraging our recognition result for icosahedral modules, we classify the faithful $\operatorname{Alt}(5)$-modules with additive dimension that are dim-connected of dimension $3$ and without $2$-torsion.

As large language models (LLMs) are shaping the way information is shared and accessed online, their opinions have the potential to influence a wide audience. This study examines who the LLMs view as the most prominent figures across various fields, using prompts in ten different languages to explore the influence of linguistic diversity. Our findings reveal low diversity in responses, with a small number of figures dominating recognition across languages (also known as the "superstar effect"). These results highlight the risk of narrowing global knowledge representation when LLMs retrieve subjective information.

Intent inferral, the process by which a robotic device predicts a user's intent from biosignals, offers an effective and intuitive way to control wearable robots. Classical intent inferral methods treat biosignal inputs as unidirectional ground truths for training machine learning models, where the internal state of the model is not directly observable by the user. In this work, we propose reciprocal learning, a bidirectional paradigm that facilitates human adaptation to an intent inferral classifier. Our paradigm consists of iterative, interwoven stages that alternate between updating machine learning models and guiding human adaptation with the use of augmented visual feedback. We demonstrate this paradigm in the context of controlling a robotic hand orthosis for stroke, where the device predicts open, close, and relax intents from electromyographic (EMG) signals and provides appropriate assistance. We use LED progress-bar displays to communicate to the user the predicted probabilities for open and close intents by the classifier. Our experiments with stroke subjects show reciprocal learning improving performance in a subset of subjects (two out of five) without negatively impacting performance on the others. We hypothesize that, during reciprocal learning, subjects can learn to reproduce more distinguishable muscle activation patterns and generate more separable biosignals.

We developed a new sodium magnetic resonance fingerprinting ($^\text{23}\text{Na}$ MRF) method for the simultaneous mapping of $\text{T}_\text{1}$, $\text{T}_\text{2,long}^{*}$, $\text{T}_\text{2,short}^{*}$ and sodium density with built-in $\Delta\text{B}_{1}^{+}$ (radiofrequency transmission inhomogeneities) and $\Delta\text{f}_\text{0}$ corrections (frequency offsets). We based our $^\text{23}\text{Na}$ MRF implementation on a 3D FLORET sequence with 23 radiofrequency pulses. To capture the complex spin ${\frac{\text{3}}{\text{2}}}$ dynamics of the $^\text{23}\text{Na}$ nucleus, the fingerprint dictionary was simulated using the irreducible spherical tensor operators formalism. The dictionary contained 831,512 entries covering a wide range of $\text{T}_\text{1}$, $\text{T}_\text{2,long}^{*}$, $\text{T}_\text{2,short}^{*}$, $\Delta\text{B}_\text{1}^{+}$ factor and $\Delta\text{f}_\text{0}$ parameters. Fingerprint matching was performed using the Pearson correlation and the resulting relaxation maps were weighted with a subset of the highest correlation coefficients corresponding to signal matches for each voxel. Our $^\text{23}\text{Na}$ MRF method was compared against reference methods in a 7-compartment phantom, and applied in brain in five healthy volunteers at 7 T. In phantoms, $^\text{23}\text{Na}$ MRF produced values comparable to those obtained with reference methods. Average sodium relaxation time values in cerebrospinal fluid, gray matter and white matter across five healthy volunteers were in good agreement with values previously reported in the literature.

While there is widespread interest in supporting young people to critically evaluate machine learning-powered systems, there is little research on how we can support them in inquiring about how these systems work and what their limitations and implications may be. Outside of K-12 education, an effective strategy in evaluating black-boxed systems is algorithm auditing-a method for understanding algorithmic systems' opaque inner workings and external impacts from the outside in. In this paper, we review how expert researchers conduct algorithm audits and how end users engage in auditing practices to propose five steps that, when incorporated into learning activities, can support young people in auditing algorithms. We present a case study of a team of teenagers engaging with each step during an out-of-school workshop in which they audited peer-designed generative AI TikTok filters. We discuss the kind of scaffolds we provided to support youth in algorithm auditing and directions and challenges for integrating algorithm auditing into classroom activities. This paper contributes: (a) a conceptualization of five steps to scaffold algorithm auditing learning activities, and (b) examples of how youth engaged with each step during our pilot study.

Advances in artificial intelligence (AI) present significant risks and opportunities, requiring improved governance to mitigate societal harms and promote equitable benefits. Current incentive structures and regulatory delays may hinder responsible AI development and deployment, particularly in light of the transformative potential of large language models (LLMs). To address these challenges, we propose developing the following three contributions: (1) a large multimodal text and economic-timeseries foundation model that integrates economic and natural language policy data for enhanced forecasting and decision-making, (2) algorithmic mechanisms for eliciting diverse and representative perspectives, enabling the creation of data-driven public policy recommendations, and (3) an AI-driven web platform for supporting transparent, inclusive, and data-driven policymaking.

Wearable robotics have the capacity to assist stroke survivors in assisting and rehabilitating hand function. Many devices that use surface electromyographic (sEMG) for control rely on extrinsic muscle signals, since sEMG sensors are relatively easy to place on the forearm without interfering with hand activity. In this work, we target the intrinsic muscles of the thumb, which are superficial to the skin and thus potentially more accessible via sEMG sensing. However, traditional, rigid electrodes can not be placed on the hand without adding bulk and affecting hand functionality. We thus present a novel sensing sleeve that uses textile electrodes to measure sEMG activity of intrinsic thumb muscles. We evaluate the sleeve's performance on detecting thumb movements and muscle activity during both isolated and isometric muscle contractions of the thumb and fingers. This work highlights the potential of textile-based sensors as a low-cost, lightweight, and non-obtrusive alternative to conventional sEMG sensors for wearable robotics., Comment: 6 pages, 4 figures, ICORR 2025 submission

We adapt the social force model of crowd dynamics to capture the evacuation during a zombie outbreak from an academic building. Individuals navigate the building, opening doors, and evacuate to the nearest exit. Zombies chase the uninfected individuals, and once caught there is a probability of a susceptible individual being infected or killed, or for the zombie to be killed by the person being attacked. We find that the speed of the zombies plays a crucial role in the dynamics of the evacuation, the rate of infection, and the number of casualties during the outbreak. The model leads to insights that may be relevant to other, less fictitious, emergency situations.

Much effort has been made to explain and improve the success of transfer-based attacks (TBA) on black-box computer vision models. This work provides the first attempt at a priori prediction of attack success by identifying the presence of vulnerable features within target models. Recent work by Chen and Liu (2024) proposed the manifold attack model, a unifying framework proposing that successful TBA exist in a common manifold space. Our work experimentally tests the common manifold space hypothesis by a new methodology: first, projecting feature vectors from surrogate and target feature extractors trained on ImageNet onto the same low-dimensional manifold; second, quantifying any observed structure similarities on the manifold; and finally, by relating these observed similarities to the success of the TBA. We find that shared feature representation moderately correlates with increased success of TBA (\r{ho}= 0.56). This method may be used to predict whether an attack will transfer without information of the model weights, training, architecture or details of the attack. The results confirm the presence of shared feature representations between two feature extractors of different sizes and complexities, and demonstrate the utility of datasets from different target domains as test signals for interpreting black-box feature representations.

Social interactions are fundamental in animal groups, including humans, and can take various forms, such as competition, cooperation, or kinship. Understanding these interactions in marine environments has been historically challenging due to data collection difficulties. However, advancements in acoustic telemetry now enable remote analysis of such behaviors. This study proposes a method to derive leader-follower networks from presence data collected by a single acoustic receiver at a specific location. Using the Kolmogorov-Smirnov distance, the method analyzes lag times between consecutive presences of individuals to infer directed relationships. Tested on simulated data, it was then applied to detection data from acoustically tagged reef manta rays (\textit{Mobula~alfredi}) frequenting a known site. Results revealed temporal patterns, including circadian rhythms and burst-like behavior with power-law distributed time gaps between presences. The inferred leader-follower network highlighted key behavioral patterns: females followed males more often than expected, males showed stronger but fewer associations with specific females, and smaller individuals followed others less consistently than larger ones. These findings align with ecological insights, revealing structured social interactions and providing a novel framework for studying marine animal behavior through network theory., Comment: 24 pages, 6 figures

In this project, we describe a method of modeling semantic leadership across a set of communities associated with the #BlackLivesMatter movement, which has been informed by qualitative research on the structure of social media and Black Twitter in particular. We describe our bespoke approaches to time-binning, community clustering, and connecting communities over time, as well as our adaptation of state-of-the-art approaches to semantic change detection and semantic leadership induction. We find substantial evidence of the leadership role of BLM activists and progressives, as well as Black celebrities. We also find evidence of the sustained engagement of the conservative community with this discourse, suggesting an alternative explanation for how we arrived at the present moment, in which "anti-woke" and "anti-CRT" bills are being enacted nationwide., Comment: Accepted at ICWSM 2025; minor revisions forthcoming

Monitoring wastewater concentrations of SARS-CoV-2 yields a low-cost, noninvasive method for tracking disease prevalence and provides early warning signs of upcoming outbreaks in the serviced communities. There is tremendous clinical and public health interest in understanding the exact dynamics between wastewater viral loads and infection rates in the population. As both data sources may contain substantial noise and missingness, in addition to spatial and temporal dependencies, properly modeling this relationship must address these numerous complexities simultaneously while providing interpretable and clear insights. We propose a novel Bayesian functional concurrent regression model that accounts for both spatial and temporal correlations while estimating the dynamic effects between wastewater concentrations and positivity rates over time. We explicitly model the time lag between the two series and provide full posterior inference on the possible delay between spikes in wastewater concentrations and subsequent outbreaks. We estimate a time lag likely between 5 to 11 days between spikes in wastewater levels and reported clinical positivity rates. Additionally, we find a dynamic relationship between wastewater concentration levels and the strength of its association with positivity rates that fluctuates between outbreaks and non-outbreaks.

We report the first instance of an M dwarf/brown dwarf obliquity measurement for the TOI-2119 system using the Rossiter-McLaughlin effect. TOI-2119 b is a transiting brown dwarf orbiting a young, active early M dwarf ($T_{\rm{eff}}$ = 3553 K). It has a mass of 64.4 M$_{\rm{J}}$ and radius of 1.08 R$_{\rm{J}}$, with an eccentric orbit ($e$ = 0.3) at a period of 7.2 days. For this analysis, we utilise NEID spectroscopic transit observations and ground based simultaneous transit photometry from the Astrophysical Research Consortium (ARC) and the Las Campanas Remote Observatory (LCRO). We fit all available data of TOI-2119 b to refine the brown dwarf parameters and update the ephemeris. The classical Rossiter-McLaughlin technique yields a projected star-planet obliquity of $\lambda=-0.8\pm1.1^\circ$ and a three-dimensional obliquity of $\psi=15.7\pm5.5^\circ$. Additionally, we spatially resolve the stellar surface of TOI-2119 utilising the Reloaded Rossiter-McLaughlin technique to determine the projected star-planet obliquity as $\lambda=1.26 \pm 1.2^{\circ}$. Both of these results agree within $2\sigma$ and confirm the system is aligned, where TOI-2119 b joins an emerging group of aligned brown dwarf obliquities. We also probe stellar surface activity on the surface of TOI-2119 in the form of centre-to-limb variations as well as the potential for differential rotation. Overall, we find tentative evidence for centre-to-limb variations on the star but do not detect evidence of differential rotation., Comment: Accepted to MNRAS pending minor corrections which have been implemented in this version. 12 pages, 9 figures, 4 tables

Various watermarking methods (``watermarkers'') have been proposed to identify LLM-generated texts; yet, due to the lack of unified evaluation platforms, many critical questions remain under-explored: i) What are the strengths/limitations of various watermarkers, especially their attack robustness? ii) How do various design choices impact their robustness? iii) How to optimally operate watermarkers in adversarial environments? To fill this gap, we systematize existing LLM watermarkers and watermark removal attacks, mapping out their design spaces. We then develop WaterPark, a unified platform that integrates 10 state-of-the-art watermarkers and 12 representative attacks. More importantly, by leveraging WaterPark, we conduct a comprehensive assessment of existing watermarkers, unveiling the impact of various design choices on their attack robustness. We further explore the best practices to operate watermarkers in adversarial environments. We believe our study sheds light on current LLM watermarking techniques while WaterPark serves as a valuable testbed to facilitate future research., Comment: 22 pages

This paper explores and assesses in what ways generative AI can assist in translating survey instruments. Writing effective survey questions is a challenging and complex task, made even more difficult for surveys that will be translated and deployed in multiple linguistic and cultural settings. Translation errors can be detrimental, with known errors rendering data unusable for its intended purpose and undetected errors leading to incorrect conclusions. A growing number of institutions face this problem as surveys deployed by private and academic organizations globalize, and the success of their current efforts depends heavily on researchers' and translators' expertise and the amount of time each party has to contribute to the task. Thus, multilinguistic and multicultural surveys produced by teams with limited expertise, budgets, or time are at significant risk for translation-based errors in their data. We implement a zero-shot prompt experiment using ChatGPT to explore generative AI's ability to identify features of questions that might be difficult to translate to a linguistic audience other than the source language. We find that ChatGPT can provide meaningful feedback on translation issues, including common source survey language, inconsistent conceptualization, sensitivity and formality issues, and nonexistent concepts. In addition, we provide detailed information on the practicality of the approach, including accessing the necessary software, associated costs, and computational run times. Lastly, based on our findings, we propose avenues for future research that integrate AI into survey translation practices.

We propose a framework for two-player infinite-dimensional games with cooperative or competitive structure. These games take the form of coupled partial differential equations in which players optimize over a space of measures, driven by either a gradient descent or gradient descent-ascent in Wasserstein-2 space. We characterize the properties of the Nash equilibrium of the system, and relate it to the steady state of the dynamics. In the min-max setting, we show, under sufficient convexity conditions, that solutions converge exponentially fast and with explicit rate to the unique Nash equilibrium. Similar results are obtained for the cooperative setting. We apply this framework to distribution shift induced by interactions among a strategic population of agents and an algorithm, proving additional convergence results in the timescale-separated setting. We illustrate the performance of our model on (i) real data from an economics study on Colombia census data, (ii) feature modification in loan applications, and (iii) performative prediction. The numerical experiments demonstrate the importance of distribution-level, rather than moment-level, modeling., Comment: arXiv admin note: text overlap with arXiv:2307.01166

Vision Transformers (ViTs) have shown exceptional performance in vehicle re-identification (ReID) tasks. However, non-square aspect ratios of image or video inputs can negatively impact re-identification accuracy. To address this challenge, we propose a novel, human perception driven, and general ViT-based ReID framework that fuses models trained on various aspect ratios. Our key contributions are threefold: (i) We analyze the impact of aspect ratios on performance using the VeRi-776 and VehicleID datasets, providing guidance for input settings based on the distribution of original image aspect ratios. (ii) We introduce patch-wise mixup strategy during ViT patchification (guided by spatial attention scores) and implement uneven stride for better alignment with object aspect ratios. (iii) We propose a dynamic feature fusion ReID network to enhance model robustness. Our method outperforms state-of-the-art transformer-based approaches on both datasets, with only a minimal increase in inference time per image.

We investigate deformations of skew group algebras that arise from a finite cyclic group acting on a polynomial ring in positive characteristic, where characteristic divides the order of the group. We allow deformations which deform both the group action and the vector space multiplication. We fully characterize the Poincare-Birkhoff-Witt deformations which arise in this setting from multiple perspectives: a necessary and sufficient condition list, a practical road map from which one can generate examples corresponding to any choice of group algebra element, an explicit formula, and a combinatorial analysis of the class of algebras.

This document summarizes the discussions which took place during the PITT-PACC Workshop entitled "Non-Standard Cosmological Epochs and Expansion Histories," held in Pittsburgh, Pennsylvania, Sept. 5-7, 2024. Much like the non-standard cosmological epochs that were the subject of these discussions, the format of this workshop was also non-standard. Rather than consisting of a series of talks from participants, with each person presenting their own work, this workshop was instead organized around free-form discussion blocks, with each centered on a different overall theme and guided by a different set of Discussion Leaders. This document is not intended to serve as a comprehensive review of these topics, but rather as an informal record of the discussions that took place during the workshop, in the hope that the content and free-flowing spirit of these discussions may inspire new ideas and research directions., Comment: 33 pages, LaTeX, 2 figures

Many cultural institutions have made large digitized visual collections available online, often under permissible re-use licences. Creating interfaces for exploring and searching these collections is difficult, particularly in the absence of granular metadata. In this paper, we introduce a method for using state-of-the-art multimodal large language models (LLMs) to enable an open-ended, explainable search and discovery interface for visual collections. We show how our approach can create novel clustering and recommendation systems that avoid common pitfalls of methods based directly on visual embeddings. Of particular interest is the ability to offer concrete textual explanations of each recommendation without the need to preselect the features of interest. Together, these features can create a digital interface that is more open-ended and flexible while also being better suited to addressing privacy and ethical concerns. Through a case study using a collection of documentary photographs, we provide several metrics showing the efficacy and possibilities of our approach., Comment: 16 pages, CHR 2024: Computational Humanities Research Conference, December 4 - 6, 2024, Aarhus University, Denmark

In the 1970s, the United States Environmental Protection Agency sponsored Documerica, a large-scale photography initiative to document environmental subjects nation-wide. While over 15,000 digitized public-domain photographs from the collection are available online, most of the images were scanned from damaged copies of the original prints. We present and evaluate a modified histogram matching technique based on the underlying chemistry of the prints for correcting the damaged images by using training data collected from a small set of undamaged prints. The entire set of color-adjusted Documerica images is made available in an open repository., Comment: 11 pages, CHR 2024: Computational Humanities Research Conference, December 4 - 6, 2024, Aarhus University, Denmark

This paper considers cooperative control of robots involving two different testbed systems in remote locations with communication on the internet. This provides us the capability to exchange robots status like positions, velocities and directions needed for the swarming algorithm. The results show that all robots properly follow some leader defined one of the testbeds. Measurement of data exchange rates show no loss of packets, and average transfer delays stay within tolerance limits for practical applications. In our knowledge, the novelty of this paper concerns this kind of control over a large network like internet.

The neutron star low-mass X-ray binary, EXO 0748--676, recently returned to outburst after a $\sim$ 16 year-long quiescence. Since its return, there has been a global effort to capture the previously unseen rise of the source and to understand its somewhat early return to outburst, as it is typical for a source to spend longer in quiescence than in outburst. Here, we report on the simultaneous optical and X-ray detection of a type I X-ray burst, captured by XMM-Newton during a DDT observation on 30th June 2024. The data show 3 X-ray eclipses consistent with the known ephemeris and one type I X-ray burst at 60492.309 MJD. The X-ray burst is reprocessed into the optical band and captured by XMM-Newton's Optical Monitor during a 4399 s exposure with the B filter in image + fast mode. We determine that the optical peak lags the X-ray peak by 4.46 $\pm$ 1.71s. The optical and X-ray rise times are similar, but the optical decay timescale is shorter than the X-ray decay timescale. The reprocessing site is likely within a few light seconds of the X-ray emitting region, so the companion star, accretion disc and ablated material are all plausible., Comment: 6 Pages, 3 Figures, Accepted for Publication in MNRAS Letters

To track tumors during surgery, information from preoperative CT scans is used to determine their position. However, as the surgeon operates, the tumor may be deformed which presents a major hurdle for accurately resecting the tumor, and can lead to surgical inaccuracy, increased operation time, and excessive margins. This issue is particularly pronounced in robot-assisted partial nephrectomy (RAPN), where the kidney undergoes significant deformations during operation. Toward addressing this, we introduce a occupancy network-based method for the localization of tumors within kidney phantoms undergoing deformations at interactive speeds. We validate our method by introducing a 3D hydrogel kidney phantom embedded with exophytic and endophytic renal tumors. It closely mimics real tissue mechanics to simulate kidney deformation during in vivo surgery, providing excellent contrast and clear delineation of tumor margins to enable automatic threshold-based segmentation. Our findings indicate that the proposed method can localize tumors in moderately deforming kidneys with a margin of 6mm to 10mm, while providing essential volumetric 3D information at over 60Hz. This capability directly enables downstream tasks such as robotic resection., Comment: Accepted at IROS 2024

Tractography fiber clustering using diffusion MRI (dMRI) is a crucial strategy for white matter (WM) parcellation. Current methods primarily use the geometric information of fibers (i.e., the spatial trajectories) to group similar fibers into clusters, overlooking the important functional signals present along the fiber tracts. There is increasing evidence that neural activity in the WM can be measured using functional MRI (fMRI), offering potentially valuable multimodal information for fiber clustering. In this paper, we develop a novel deep learning fiber clustering framework, namely Deep Multi-view Fiber Clustering (DMVFC), that uses joint dMRI and fMRI data to enable functionally consistent WM parcellation. DMVFC can effectively integrate the geometric characteristics of the WM fibers with the fMRI BOLD signals along the fiber tracts. It includes two major components: 1) a multi-view pretraining module to compute embedding features from fiber geometric information and functional signals separately, and 2) a collaborative fine-tuning module to simultaneously refine the two kinds of embeddings. In the experiments, we compare DMVFC with two state-of-the-art fiber clustering methods and demonstrate superior performance in achieving functionally meaningful and consistent WM parcellation results., Comment: 5 pages, 3 figures

Work on shallow discourse parsing in English has focused on the Wall Street Journal corpus, the only large-scale dataset for the language in the PDTB framework. However, the data is not openly available, is restricted to the news domain, and is by now 35 years old. In this paper, we present and evaluate a new open-access, multi-genre benchmark for PDTB-style shallow discourse parsing, based on the existing UD English GUM corpus, for which discourse relation annotations in other frameworks already exist. In a series of experiments on cross-domain relation classification, we show that while our dataset is compatible with PDTB, substantial out-of-domain degradation is observed, which can be alleviated by joint training on both datasets., Comment: Accepted to EMNLP 2024 (main, long); camera-ready version

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

Infant crying can serve as a crucial indicator of various physiological and emotional states. This paper introduces a comprehensive approach for detecting infant cries within audio data. We integrate Meta's Wav2Vec with traditional audio features, such as Mel-frequency cepstral coefficients (MFCCs), chroma, and spectral contrast, employing Gradient Boosting Machines (GBM) for cry classification. We validate our approach on a real-world dataset, demonstrating significant performance improvements over existing methods.

There is a tremendous amount of interest in AI/ML technologies due to the proliferation of generative AI applications such as ChatGPT. This trend has significantly increased demand on GPUs, which are the workhorses for training AI models. Due to the high costs of GPUs and lacking supply, it has become of interest to optimize GPU usage in HPC centers. MIT Lincoln Laboratory Supercomputing Center (LLSC) has developed an easy-to-use GPU sharing feature supported by LLSC-developed tools including LLsub and LLMapReduce. This approach overcomes some of the limitations with the existing methods for GPU sharing. This allows users to apply GPU sharing whenever possible while they are developing their AI/ML models and/or doing parametric study on their AI models or executing other GPU applications. Based on our initial experimental results with GPU sharing, GPU sharing with triples mode is easy to use and achieved significant improvement in GPU usage and throughput performance for certain types of AI applications.

Brain imaging studies have demonstrated that diffusion MRI tractography geometric shape descriptors can inform the study of the brain's white matter pathways and their relationship to brain function. In this work, we investigate the possibility of utilizing a deep learning model to compute shape measures of the brain's white matter connections. We introduce a novel framework, TractShapeNet, that leverages a point cloud representation of tractography to compute five shape measures: length, span, volume, total surface area, and irregularity. We assess the performance of the method on a large dataset including 1065 healthy young adults. Experiments for shape measure computation demonstrate that our proposed TractShapeNet outperforms other point cloud-based neural network models in both the Pearson correlation coefficient and normalized error metrics. We compare the inference runtime results with the conventional shape computation tool DSI-Studio. Our results demonstrate that a deep learning approach enables faster and more efficient shape measure computation. We also conduct experiments on two downstream language cognition prediction tasks, showing that shape measures from TractShapeNet perform similarly to those computed by DSI-Studio. Our code will be available at: https://github.com/SlicerDMRI/TractShapeNet., Comment: 10 pages, 2 figures, 4 tables. This work has been submitted to the IEEE for possible publication

Reconstructing neuron morphology from 3D light microscope imaging data is critical to aid neuroscientists in analyzing brain networks and neuroanatomy. With the boost from deep learning techniques, a variety of learning-based segmentation models have been developed to enhance the signal-to-noise ratio of raw neuron images as a pre-processing step in the reconstruction workflow. However, most existing models directly encode the latent representative features of volumetric neuron data but neglect their intrinsic morphological knowledge. To address this limitation, we design a novel framework that distills the prior knowledge from a 2D Vision Transformer pre-trained on extensive 2D natural images to facilitate neuronal morphological learning of our 3D Vision Transformer. To bridge the knowledge gap between the 2D natural image and 3D microscopic morphologic domains, we propose a deformable tubular transferring strategy that adapts the pre-trained 2D natural knowledge to the inherent tubular characteristics of neuronal structure in the latent embedding space. The experimental results on the Janelia dataset of the BigNeuron project demonstrate that our method achieves a segmentation performance improvement of 4.53% in mean Dice and 3.56% in mean 95% Hausdorff distance., Comment: 9 pages, 3 figures, and 2 tables. This work has been submitted to the IEEE for possible publication

The increasing use and cost of high performance computing (HPC) requires new easy-to-use tools to enable HPC users and HPC systems engineers to transparently understand the utilization of resources. The MIT Lincoln Laboratory Supercomputing Center (LLSC) has developed a simple command, LLload, to monitor and characterize HPC workloads. LLload plays an important role in identifying opportunities for better utilization of compute resources. LLload can be used to monitor jobs both programmatically and interactively. LLload can characterize users' jobs using various LLload options to achieve better efficiency. This information can be used to inform the user to optimize HPC workloads and improve both CPU and GPU utilization. This includes improvements using judicious oversubscription of the computing resources. Preliminary results suggest significant improvement in GPU utilization and overall throughput performance with GPU overloading in some cases. By enabling users to observe and fix incorrect job submission and/or inappropriate execution setups, LLload can increase the resource usage and improve the overall throughput performance. LLload is a light-weight, easy-to-use tool for both HPC users and HPC systems engineers to monitor HPC workloads to improve system utilization and efficiency.

Metal additive manufacturing (AM) opens the possibility for spatial control of as-fabricated microstructure and properties. However, since the solid state diffusional transformations that drive microstructure outcomes are governed by nonlinear ODEs in terms of temperature, which is itself governed by PDEs over the entire part domain, solving for the system inputs needed to achieve desired microstructure distributions has proven difficult. In this work, we present a trajectory optimization approach for spatial control of microstructure in metal AM, which we demonstrate by controlling the hardness of a low-alloy steel in electron beam powder bed fusion (EB-PBF). To this end, we present models for thermal and microstructural dynamics. Next, we use experimental data to identify the parameters of the microstructure transformation dynamics. We then pose spatial microstructure control as a finite-horizon optimal control problem. The optimal power field trajectory is computed using an augmented Lagrangian differential dynamic programming (AL-DDP) method with GPU acceleration. The resulting time-varying power fields are then realized on an EB-PBF machine through an approximation scheme. Measurements of the resultant hardness shows that the optimized power field trajectory is able to closely produce the desired hardness distribution., Comment: 6 pages, 6 figures

We introduce a novel pipeline for segmenting and structuring color and depth (RGB-D) data. Existing processing pipelines for RGB-D data have focused on extracting geometric information alone. This approach precludes the development of more advanced robotic navigation and manipulation algorithms, which benefit from a semantic understanding of their environment. Our pipeline can segment RGB-D data into accurate semantic masks. These masks are then used to fuse raw captured point clouds into semantically separated point clouds. We store this information using the Universal Scene Description (USD) file format, a format suitable for easy querying by downstream robotics algorithms, human-friendly visualization, and robotics simulation.

metasnf is an R package that enables users to apply meta clustering, a method for efficiently searching a broad space of cluster solutions by clustering the solutions themselves, to clustering workflows based on similarity network fusion (SNF). SNF is a multi-modal data integration algorithm commonly used for biomedical subtype discovery. The package also contains functions to assist with cluster visualization, characterization, and validation. This package can help researchers identify SNF-derived cluster solutions that are guided by context-specific utility over context-agnostic measures of quality., Comment: 72 pages, 22 figures, submitted to Journal of Statistical Software

The shape of the brain's white matter connections is relatively unexplored in diffusion MRI tractography analysis. While it is known that tract shape varies in populations and across the human lifespan, it is unknown if the variability in dMRI tractography-derived shape may relate to the brain's functional variability across individuals. This work explores the potential of leveraging tractography fiber cluster shape measures to predict subject-specific cognitive performance. We implement machine learning models to predict individual cognitive performance scores. We study a large-scale database from the HCP-YA study. We apply an atlas-based fiber cluster parcellation to the dMRI tractography of each individual. We compute 15 shape, microstructure, and connectivity features for each fiber cluster. Using these features as input, we train a total of 210 models to predict 7 different NIH Toolbox cognitive performance assessments. We apply an explainable AI technique, SHAP, to assess the importance of each fiber cluster for prediction. Our results demonstrate that shape measures are predictive of individual cognitive performance. The studied shape measures, such as irregularity, diameter, total surface area, volume, and branch volume, are as effective for prediction as microstructure and connectivity measures. The overall best-performing feature is a shape feature, irregularity, which describes how different a cluster's shape is from an idealized cylinder. Further interpretation using SHAP values suggest that fiber clusters with features highly predictive of cognitive ability are widespread throughout the brain, including fiber clusters from the superficial association, deep association, cerebellar, striatal, and projection pathways. This study demonstrates the strong potential of shape descriptors to enhance the study of the brain's white matter and its relationship to cognitive function.

The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.

In recent years, advances in artificial intelligence (AI) have transformed structural biology, particularly protein structure prediction. Though AI-based methods, such as AlphaFold (AF), often predict single conformations of proteins with high accuracy and confidence, predictions of alternative folds are often inaccurate, low-confidence, or simply not predicted at all. Here, we review three blind spots that alternative conformations reveal about AF-based protein structure prediction. First, proteins that assume conformations distinct from their training-set homologs can be mispredicted. Second, AF overrelies on its training set to predict alternative conformations. Third, degeneracies in pairwise representations can lead to high-confidence predictions inconsistent with experiment. These weaknesses suggest approaches to predict alternative folds more reliably.

We introduce Controller-Embedded Language Model Interactions (CELI), a framework that integrates control logic directly within language model (LM) prompts, facilitating complex, multi-stage task execution. CELI addresses limitations of existing prompt engineering and workflow optimization techniques by embedding control logic directly within the operational context of language models, enabling dynamic adaptation to evolving task requirements. Our framework transfers control from the traditional programming execution environment to the LMs, allowing them to autonomously manage computational workflows while maintaining seamless interaction with external systems and functions. CELI supports arbitrary function calls with variable arguments, bridging the gap between LMs' adaptive reasoning capabilities and conventional software paradigms' structured control mechanisms. To evaluate CELI's versatility and effectiveness, we conducted case studies in two distinct domains: code generation (HumanEval benchmark) and multi-stage content generation (Wikipedia-style articles). The results demonstrate notable performance improvements across a range of domains. CELI achieved a 4.9 percentage point improvement over the best reported score of the baseline GPT-4 model on the HumanEval code generation benchmark. In multi-stage content generation, 94.4% of CELI-produced Wikipedia-style articles met or exceeded first draft quality when optimally configured, with 44.4% achieving high quality. These outcomes underscore CELI's potential for optimizing AI-driven workflows across diverse computational domains., Comment: 26 pages, 2 figures

Climate change is negatively impacting the world's biodiversity. To build automated systems to monitor these negative biodiversity impacts, large-scale, volunteer-collected datasets like iNaturalist are built from community-identified, natural imagery. However, such volunteer-based data are opportunistic and lack a structured sampling strategy, resulting in geographic, temporal, observation quality, and socioeconomic, biases that stymie uptake of these models for downstream biodiversity monitoring tasks. Here we introduce DivShift North American West Coast (DivShift-NAWC), a curated dataset of almost 8 million iNaturalist plant images across the western coast of North America, for exploring the effects of these biases on deep learning model performance. We compare model performance across four known biases and observe that they indeed confound model performance. We suggest practical strategies for curating datasets to train deep learning models for monitoring climate change's impacts on the world's biodiversity., Comment: Accepted to NeurIPS 2024 Workshop on Tackling Climate Change with Machine Learning

Many modern software applications present numerous ethical concerns due to conflicts between users' values and companies' priorities. Intersectional communities, those with multiple marginalized identities, are disproportionately affected by these ethical issues, leading to legal, financial, and reputational issues for software companies, as well as real-world harm for intersectional users. Historically, the voices of intersectional communities have been systematically marginalized and excluded from contributing their unique perspectives to software design, perpetuating software-related ethical concerns. This work aims to fill the gap in research on intersectional users' software-related perspectives and provide software practitioners with a starting point to address their ethical concerns. We aggregated and analyzed the intersectional users' ethical concerns over time and developed a prioritization method to identify critical concerns. To achieve this, we collected posts from over 700 intersectional subreddits discussing software applications, utilized deep learning to identify ethical concerns in these posts, and employed state-of-the-art techniques to analyze their content in relation to time and priority. Our findings revealed that intersectional communities report \textit{critical} complaints related to cyberbullying, inappropriate content, and discrimination, highlighting significant flaws in modern software, particularly for intersectional users. Based on these findings, we discuss how to better address the ethical concerns of intersectional users in software development.

Investigation of students' perceptions and opinions on the use of generative artificial intelligence (GenAI) in education is a topic gaining much interest. Studies addressing this are typically conducted with large heterogeneous groups, at one moment in time. However, how students perceive and use GenAI tools can potentially depend on many factors, including their background knowledge, familiarity with the tools, and the learning goals and policies of the courses they are taking. In this study we explore how students following computing courses use GenAI for programming-related tasks across different programs and courses: Bachelor and Master, in courses in which learning programming is the learning goal, courses that require programming as a means to achieve another goal, and in courses in which programming is optional, but can be useful. We are also interested in changes over time, since GenAI capabilities are changing at a fast pace, and users are adopting GenAI increasingly. We conducted three consecutive surveys (fall `23, winter `23, and spring `24) among students of all computing programs of a large European research university. We asked questions on the use in education, ethics, and job prospects, and we included specific questions on the (dis)allowed use of GenAI tools in the courses they were taking at the time. We received 264 responses, which we quantitatively and qualitatively analyzed, to find out how students have employed GenAI tools across 59 different computing courses, and whether the opinion of an average student about these tools evolves over time. Our study contributes to the emerging discussion of how to differentiate GenAI use across different courses, and how to align its use with the learning goals of a computing course., Comment: Accepted to Koli Calling 24. Numbers in Table 1, row 1 updated

Late-stage Ca-sulfate-filled fractures are common on Mars. Notably, the Shenandoah formation in the western edge of Jezero crater preserves a variety of Ca-sulfate minerals in the fine-grained siliciclastic rocks explored by the Perseverance rover. However, the depositional environment and timing of the formation of these sulfates is unknown. To address this outstanding problem, we developed a new technique to map the crystal textures of these sulfates in situ at two stratigraphically similar locations in the Shenandoah formation, allowing us to constrain the burial depth and paleoenvironment at the time of their deposition. Our results suggest that some Ca-sulfate analyzed was formed at a burial depth greater than 80m, whereas Ca-sulfates present at another outcrop likely precipitated in a shallow-subsurface environment. These results indicate that samples collected for potential return to Earth at the two studied locations capture two different times and distinct chemical conditions in the depositional history of the Shenandoah formation providing multiple opportunities to evaluate surface and subsurface habitability.

The isotopic compositions of samples returned from Cb-type asteroid Ryugu and Ivuna-type (CI) chondrites are distinct from other carbonaceous chondrites, which has led to the suggestion that Ryugu and CI chondrites formed in a different region of the accretion disk, possibly around the orbits of Uranus and Neptune. We show that, like for Fe, Ryugu and CI chondrites also have indistinguishable Ni isotope anomalies, which differ from those of other carbonaceous chondrites. We propose that this unique Fe and Ni isotopic composition reflects different accretion efficiencies of small FeNi metal grains among the carbonaceous chondrite parent bodies. The CI chondrites incorporated these grains more efficiently, possibly because they formed at the end of the disk's lifetime, when planetesimal formation was also triggered by photoevaporation of the disk. Isotopic variations among carbonaceous chondrites may thus reflect fractionation of distinct dust components from a common reservoir, implying CI chondrites and Ryugu may have formed in the same region of the accretion disk as other carbonaceous chondrites., Comment: Published open access in Science Advances

We propose a convex controller synthesis framework for a large class of constrained linear systems, including those described by (deterministic and stochastic) partial differential equations and integral equations, commonly used in fluid dynamics, thermo-mechanical systems, quantum control, or transportation networks. Most existing control techniques rely on a (finite-dimensional) discrete description of the system, via ordinary differential equations. Here, we work instead with more general (infinite-dimensional) Hilbert spaces. This enables the discretization to be applied after the optimization (optimize-then-discretize). Using output-feedback SLS, we formulate the controller synthesis as a convex optimization problem. Structural constraints like sensor and communication delays, and locality constraints, are incorporated while preserving convexity, allowing parallel implementation and extending key SLS properties to infinite dimensions. The proposed approach and its benefits are demonstrated on a linear Boltzmann equation.

Neurosymbolic (NeSy) artificial intelligence describes the combination of logic or rule-based techniques with neural networks. Compared to neural approaches, NeSy methods often possess enhanced interpretability, which is particularly promising for biomedical applications like drug discovery. However, since interpretability is broadly defined, there are no clear guidelines for assessing the biological plausibility of model interpretations. To assess interpretability in the context of drug discovery, we devise a novel prediction task, called drug mechanism-of-action (MoA) deconvolution, with an associated, tailored knowledge graph (KG), MoA-net. We then develop the MoA Retrieval System (MARS), a NeSy approach for drug discovery which leverages logical rules with learned rule weights. Using this interpretable feature alongside domain knowledge, we find that MARS and other NeSy approaches on KGs are susceptible to reasoning shortcuts, in which the prediction of true labels is driven by "degree-bias" rather than the domain-based rules. Subsequently, we demonstrate ways to identify and mitigate this. Thereafter, MARS achieves performance on par with current state-of-the-art models while producing model interpretations aligned with known MoAs., Comment: Under review. 10 pages, 5 supplementary pages. Corresponding code is here: https://github.com/laurendelong21/MARS and here: https://github.com/laurendelong21/MoA-Net