Search

Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult. We argue that these measurement tasks are highly reminiscent of measurement tasks found throughout the social sciences. With this in mind, we present a framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems. The framework distinguishes between four levels: the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves. This four-level approach differs from the way measurement is typically done in ML, where researchers and practitioners appear to jump straight from background concepts to measurement instruments, with little to no explicit systematization in between. As well as surfacing assumptions, thereby making it easier to understand exactly what the resulting measurements do and do not mean, this framework has two important implications for evaluating evaluations: First, it can enable stakeholders from different worlds to participate in conceptual debates, broadening the expertise involved in evaluating GenAI systems. Second, it brings rigor to operational debates by offering a set of lenses for interrogating the validity of measurement instruments and their resulting measurements., Comment: NeurIPS 2024 Workshop on Evaluating Evaluations (EvalEval)

We reanalyze the spectral lag data for of GRB 160625B using frequentist inference to constrain the energy scale ($E_{QG}$) of Lorentz Invariance Violation (LIV). For this purpose, we use profile likelihood to deal with the astrophysical nuisance parameters. This is in contrast to Bayesian inference implemented in previous works, where marginalization was carried out over the nuisance parameters. We show that with profile likelihood, we do not find a global minimum for $\chi^2$ as a function of $E_{QG}$ below the Planck scale for both the linear and quadratic models of LIV, whereas bounded credible intervals were obtained using Bayesian inference. Therefore, we can set lower limits in a straightforward manner. We find that $E_{QG} \geq 3.7 \times 10^{16}$ GeV and $E_{QG} \geq 2.6 \times 10^7$ GeV at 68\% c.l., for linear and quadratic LIV, respectively. Therefore, this is the first proof of principles application of profile likelihood method to the analysis of GRB spectral lag data to constrain LIV., Comment: 4 pages, 2 figures

Machine Learning and data mining techniques (i.e. supervised and unsupervised techniques) are used across domains to detect user safety violations. Examples include classifiers used to detect whether an email is spam or a web-page is requesting bank login information. However, existing ML/DM classifiers are limited in their ability to understand natural languages w.r.t the context and nuances. The aforementioned challenges are overcome with the arrival of Gen-AI techniques, along with their inherent ability w.r.t translation between languages, fine-tuning between various tasks and domains. In this manuscript, we provide a comprehensive overview of the various work done while using Gen-AI techniques w.r.t user safety. In particular, we first provide the various domains (e.g. phishing, malware, content moderation, counterfeit, physical safety) across which Gen-AI techniques have been applied. Next, we provide how Gen-AI techniques can be used in conjunction with various data modalities i.e. text, images, videos, audio, executable binaries to detect violations of user-safety. Further, also provide an overview of how Gen-AI techniques can be used in an adversarial setting. We believe that this work represents the first summarization of Gen-AI techniques for user-safety.

For image generation with diffusion models (DMs), a negative prompt n can be used to complement the text prompt p, helping define properties not desired in the synthesized image. While this improves prompt adherence and image quality, finding good negative prompts is challenging. We argue that this is due to a semantic gap between humans and DMs, which makes good negative prompts for DMs appear unintuitive to humans. To bridge this gap, we propose a new diffusion-negative prompting (DNP) strategy. DNP is based on a new procedure to sample images that are least compliant with p under the distribution of the DM, denoted as diffusion-negative sampling (DNS). Given p, one such image is sampled, which is then translated into natural language by the user or a captioning model, to produce the negative prompt n*. The pair (p, n*) is finally used to prompt the DM. DNS is straightforward to implement and requires no training. Experiments and human evaluations show that DNP performs well both quantitatively and qualitatively and can be easily combined with several DM variants.

The Affordable Care Act was signed into law in 2010, expanding Medicaid and improving access to care for millions of low-income Americans. Fewer uninsured individuals reduced the cost of uncompensated care, consequently improving the financial health of hospitals. We hypothesize that this amelioration in hospital finances resulted in a marked improvement of quality measures in states that chose to expand Medicaid. To our knowledge, the impact of Medicaid expansion on the Medicare population has not been investigated. Using a difference-in-difference analysis, we compare readmission rates for four measures from the Hospital Readmission Reduction Program: acute myocardial infarction, pneumonia, heart failure, and coronary artery bypass graft surgery. Our analysis provides evidence that between 2013 and 2021 expansion states improved hospital quality relative to non-expansion states as it relates to acute myocardial infarction readmissions (p = 0.015) and coronary artery bypass graft surgery readmissions (p = 0.039). Our analysis provides some evidence that expanding Medicaid improved hospital quality, as measured by a reduction in readmission rates. Using visualizations, we provide some evidence that hospital quality improved for the other two measures as well. We believe that a refinement of our estimation method and an improved dataset will increase our chances of finding significant results for these two other measures.

Recent theoretical work and targeted observational studies suggest that filaments are sites of galaxy preprocessing. The aim of the WISESize project is to directly probe galaxies over the full range of environments to quantify and characterize extrinsic galaxy quenching in the local Universe. In this paper, we use GALFIT to measure the infrared 12$\mu$m ($R_{12}$) and 3.4$\mu$m ($R_{3.4}$) effective radii of 603 late-type galaxies in and surrounding the Virgo cluster. We find that Virgo cluster galaxies show smaller star-forming disks relative to their field counterparts at the $2.5\sigma$ level, while filament galaxies show smaller star-forming disks to almost $1.5\sigma$. Our data, therefore, show that cluster galaxies experience significant effects on their star-forming disks prior to their final quenching period. There is also tentative support for the hypothesis that galaxies are preprocessed in filamentary regions surrounding clusters. On the other hand, galaxies belonging to rich groups and poor groups do not differ significantly from those in the field. We additionally find hints of a positive correlation between stellar mass and size ratio for both rich group and filament galaxies, though the uncertainties on these data are consistent with no correlation. We compare our size measurements with the predictions from two variants of a state-of-the-art semi-analytic model (SAM), one which includes starvation and the other incorporating both starvation and ram-pressure stripping (RPS). Our data appear to disfavor the SAM, which includes RPS for the rich group, filament, and cluster samples, which contributes to improved constraints for general models of galaxy quenching., Comment: 21 pages, 11 figures. Resubmitted second time to ApJ with only minor revision requested

In this paper, we propose a novel method for monocular depth estimation in dynamic scenes. We first explore the arbitrariness of object's movement trajectory in dynamic scenes theoretically. To overcome the arbitrariness, we use assume that points move along a straight line over short distances and then summarize it as a triangular constraint loss in two dimensional Euclidean space. To overcome the depth inconsistency problem around the edges, we propose a deformable support window module that learns features from different shapes of objects, making depth value more accurate around edge area. The proposed model is trained and tested on two outdoor datasets - KITTI and Make3D, as well as an indoor dataset - NYU Depth V2. The quantitative and qualitative results reported on these datasets demonstrate the success of our proposed model when compared against other approaches. Ablation study results on the KITTI dataset also validate the effectiveness of the proposed pixel movement prediction module as well as the deformable support window module.

We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction., Comment: Alphabetical author order

Agent learning from human interaction often relies on explicit signals, but implicit social cues, such as prosody in speech, could provide valuable information for more effective learning. This paper advocates for the integration of prosody as a teaching signal to enhance agent learning from human teachers. Through two exploratory studies--one examining voice feedback in an interactive reinforcement learning setup and the other analyzing restricted audio from human demonstrations in three Atari games--we demonstrate that prosody carries significant information about task dynamics. Our findings suggest that prosodic features, when coupled with explicit feedback, can enhance reinforcement learning outcomes. Moreover, we propose guidelines for prosody-sensitive algorithm design and discuss insights into teaching behavior. Our work underscores the potential of leveraging prosody as an implicit signal for more efficient agent learning, thus advancing human-agent interaction paradigms., Comment: Published at the 26th ACM International Conference on Multimodal Interaction (ICMI) 2024

In the rapidly evolving field of artificial intelligence (AI) agents, designing the agent's characteristics is crucial for shaping user experience. This workshop aims to establish a research community focused on AI agent persona design for various contexts, such as in-car assistants, educational tools, and smart home environments. We will explore critical aspects of persona design, such as voice, embodiment, and demographics, and their impact on user satisfaction and engagement. Through discussions and hands-on activities, we aim to propose practices and standards that enhance the ecological validity of agent personas. Topics include the design of conversational interfaces, the influence of agent personas on user experience, and approaches for creating contextually appropriate AI agents. This workshop will provide a platform for building a community dedicated to developing AI agent personas that better fit diverse, everyday interactions., Comment: 8 pages, the workshop accepted at the 23rd International Conference on Mobile and Ubiquitous Multimedia (MUM 2024)

Modeling the intrinsic alignment (IA) of galaxies poses a challenge to weak lensing analyses. The Dark Energy Survey is expected to be less impacted by IA when limited to blue, star-forming galaxies. The cosmological parameter constraints from this blue cosmic shear sample are stable to IA model choice, unlike passive galaxies in the full DES Y3 sample, the goodness-of-fit is improved and the $\Omega_{m}$ and $S_8$ better agree with the cosmic microwave background. Mitigating IA with sample selection, instead of flexible model choices, can reduce uncertainty in $S_8$ by a factor of 1.5., Comment: Data access available at https://jamiemccullough.github.io/data/blueshear/

We present results from data acquired by the SENSEI experiment at SNOLAB after a major upgrade in May 2023, which includes deploying 16 new sensors and replacing the copper trays that house the CCDs with a new light-tight design. We observe a single-electron event rate of $(1.39 \pm 0.11) \times 10^{-5}$ e$^-$/pix/day, corresponding to $(39.8 \pm 3.1)$ e$^-$/gram/day. This is an order-of-magnitude improvement compared to the previous lowest single-electron rate in a silicon detector and the lowest for any photon detector in the near-infrared-ultraviolet range. We use these data to obtain a 90% confidence level upper bound of $1.53 \times 10^{-5}$ e$^-$/pix/day and to set constraints on sub-GeV dark matter candidates that produce single-electron events. We hypothesize that the data taken at SNOLAB in the previous run, with an older tray design for the sensors, contained a larger rate of single-electron events due to light leaks. We test this hypothesis using data from the SENSEI detector located in the MINOS cavern at Fermilab.

Large Language Models (LLMs) have shown remarkable capabilities in various domains, yet their economic impact has been limited by challenges in tool use and function calling. This paper introduces ThorV2, a novel architecture that significantly enhances LLMs' function calling abilities. We develop a comprehensive benchmark focused on HubSpot CRM operations to evaluate ThorV2 against leading models from OpenAI and Anthropic. Our results demonstrate that ThorV2 outperforms existing models in accuracy, reliability, latency, and cost efficiency for both single and multi-API calling tasks. We also show that ThorV2 is far more reliable and scales better to multistep tasks compared to traditional models. Our work offers the tantalizing possibility of more accurate function-calling compared to today's best-performing models using significantly smaller LLMs. These advancements have significant implications for the development of more capable AI assistants and the broader application of LLMs in real-world scenarios., Comment: 15 pages for main paper, 21 pages in total including references and appendix, 10 figures

We report observations of direct evidence of energetic protons being accelerated above ~400 keV within the reconnection exhaust of a heliospheric current sheet (HCS) crossing by NASA's Parker Solar Probe (PSP) at a distance of ~16.25 solar radii (Rs) from the Sun. Inside the extended exhaust, both the reconnection-generated plasma jets and the accelerated protons propagated toward the Sun, unambiguously establishing their origin from HCS reconnection sites located beyond PSP. Within the core of the exhaust, PSP detected stably trapped energetic protons up to ~400 keV, which is approximately 1000 times greater than the available magnetic energy per particle. The differential energy spectrum of the accelerated protons behaved as a pure power-law with spectral index of about -5. Supporting simulations using the kglobal model suggest that the trapping and acceleration of protons up to ~400 keV in the reconnection exhaust is likely facilitated by merging magnetic islands with a guide field between ~0.2-0.3 of the reconnecting magnetic field, consistent with the observations. These new results, enabled by PSP's proximity to the Sun, demonstrate that magnetic reconnection in the HCS is a significant new source of energetic particles in the near-Sun solar wind. The discovery of in-situ particle acceleration via magnetic reconnection at the HCS provides valuable insights into this fundamental process which frequently converts the large magnetic field energy density in the near-Sun plasma environment and may be responsible for heating the sun's atmosphere, accelerating the solar wind, and energizing charged particles to extremely high energies in solar flares., Comment: 24 pages, 4 figures

For a finite simple graph $G=(V,\ E)$, the \emph{sum index} of $G$ is defined to be \begin{align*} S(G)=\min\{|\{f(u)+f(v):\ uv\in E\}|:\ f:V\lhook\joinrel\longrightarrow \mathbb{Z}\}. \end{align*} In this paper, from several different aspects, we show some new developments on graph sum index. Firstly, we determine the sum indices of the complete multipartite graphs, hypercubes, and some cluster graphs. Then, we study the maximum number of edges in a graph with a fixed sum index, which is related to the forbidden subgraph problem. Also, we show some relations between graph sum indices and results in additive combinatorics.

We search for a spatial coincidence between high energy neutrinos detected by the IceCube neutrino detector and ten red dwarfs which have been observed in gamma-rays. For our analysis, we use the unbinned maximum likelihood method to look for a statistically significant excess. We do not find any such spatial association between any of the red dwarfs and Icecube-detected neutrinos. Therefore, we conclude that none of the gamma-ray bright red dwarfs contribute to the diffuse neutrino flux measured by IceCube., Comment: 8 pages

Gen-AI techniques are able to improve understanding of context and nuances in language modeling, translation between languages, handle large volumes of data, provide fast, low-latency responses and can be fine-tuned for various tasks and domains. In this manuscript, we present a comprehensive overview of the applications of Gen-AI techniques in the finance domain. In particular, we present the opportunities and challenges associated with the usage of Gen-AI techniques. We also illustrate the various methodologies which can be used to train Gen-AI techniques and present the various application areas of Gen-AI technologies in the finance ecosystem. To the best of our knowledge, this work represents the most comprehensive summarization of Gen-AI techniques within the financial domain. The analysis is designed for a deep overview of areas marked for substantial advancement while simultaneously pin-point those warranting future prioritization. We also hope that this work would serve as a conduit between finance and other domains, thus fostering the cross-pollination of innovative concepts and practices.

We examined the origin of 3He abundance enhancement in 23 high-energy (25-50 MeV) solar proton events that coincide with 3He-rich periods detected by ACE ULEIS in 1997-2021. In seven events, 3He enhancement was due to 3He leftover from preceding events or independent 3He events occurring during proton events. One event is the most likely impulsive (3He-rich), and another is unclear. Reaccelerated remnant flare material was the most probable cause of 3He enhancements in the remaining 14 proton events. Imaging observations showed coronal jets in the parent active regions in six of these 14 events. Remarkably, the highest 3He/4He occurred in events with jets, implying their contribution to 3He enhancement., Comment: 6 pages, 4 figures, 2 tables, accepted for publication in IAU Symposium 388 Proceedings

Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions. The literature on behavior prediction spans various tasks, including action anticipation, activity forecasting, intent prediction, goal prediction, and so on. Our survey aims to tie together this fragmented literature, covering recent technical innovations as well as the development of new large-scale datasets for model training and evaluation. We also summarize the widely-used metrics for different tasks and provide a comprehensive performance comparison of existing approaches on eleven action anticipation datasets. This survey serves as not only a reference for contemporary methodologies in action anticipation, but also a guideline for future research direction of this evolving landscape., Comment: 30 pages, 9 figures, 12 tables

Our thesis is that operating in cyberspace is challenging because cyberspace exhibits extreme variety, high malleability, and extreme velocity. These properties make cyberspace largely inscrutable and limits one's agency in cyberspace, where agency is the ability to exert influence to transform the state or behaviour of the environment. With this thesis, we explore the nature of cyberspace, command and control (C2), and diagnose the challenges for cyber C2, with treatment to follow in future work., Comment: 16 pages. Published in proceedings of the 29th International Command and Control Research Symposium (ICCRTS), London UK, 2024

Recent LLM (Large Language Models) advancements benefit many fields such as education and finance, but HR has hundreds of repetitive processes, such as access requests, medical claim filing and time-off submissions, which are unaddressed. We relate these tasks to the LLM agent, which has addressed tasks such as writing assisting and customer support. We present HR-Agent, an efficient, confidential, and HR-specific LLM-based task-oriented dialogue system tailored for automating repetitive HR processes such as medical claims and access requests. Since conversation data is not sent to an LLM during inference, it preserves confidentiality required in HR-related tasks.

The first FASER search for a light, long-lived particle decaying into a pair of photons is reported. The search uses LHC proton-proton collision data at $\sqrt{s}=13.6~\text{TeV}$ collected in 2022 and 2023, corresponding to an integrated luminosity of $57.7\text{fb}^{-1}$. A model with axion-like particles (ALPs) dominantly coupled to weak gauge bosons is the primary target. Signal events are characterised by high-energy deposits in the electromagnetic calorimeter and no signal in the veto scintillators. One event is observed, compared to a background expectation of $0.44 \pm 0.39$ events, which is entirely dominated by neutrino interactions. World-leading constraints on ALPs are obtained for masses up to $300~\text{MeV}$ and couplings to the Standard Model W gauge boson, $g_{aWW}$, around $10^{-4}$ GeV$^{-1}$, testing a previously unexplored region of parameter space. Other new particle models that lead to the same experimental signature, including ALPs coupled to gluons or photons, U(1)$_B$ gauge bosons, up-philic scalars, and a Type-I two-Higgs doublet model, are also considered for interpretation, and new constraints on previously viable parameter space are presented in this paper., Comment: 37 pages, 22 figures

Background: Conventional prediction methods such as logistic regression and gradient boosting have been widely utilized for disease onset prediction for their reliability and interpretability. Deep learning methods promise enhanced prediction performance by extracting complex patterns from clinical data, but face challenges like data sparsity and high dimensionality. Methods: This study compares conventional and deep learning approaches to predict lung cancer, dementia, and bipolar disorder using observational data from eleven databases from North America, Europe, and Asia. Models were developed using logistic regression, gradient boosting, ResNet, and Transformer, and validated both internally and externally across the data sources. Discrimination performance was assessed using AUROC, and calibration was evaluated using Eavg. Findings: Across 11 datasets, conventional methods generally outperformed deep learning methods in terms of discrimination performance, particularly during external validation, highlighting their better transportability. Learning curves suggest that deep learning models require substantially larger datasets to reach the same performance levels as conventional methods. Calibration performance was also better for conventional methods, with ResNet showing the poorest calibration. Interpretation: Despite the potential of deep learning models to capture complex patterns in structured observational healthcare data, conventional models remain highly competitive for disease onset prediction, especially in scenarios involving smaller datasets and if lengthy training times need to be avoided. The study underscores the need for future research focused on optimizing deep learning models to handle the sparsity, high dimensionality, and heterogeneity inherent in healthcare datasets, and find new strategies to exploit the full capabilities of deep learning methods.

Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos. However, they can only generate short video clips, normally around 10 seconds or 240 frames, due to computation limitations during training. In this work, we show that existing models can be naturally extended to autoregressive video diffusion models without changing the architectures. Our key idea is to assign the latent frames with progressively increasing noise levels rather than a single noise level, which allows for fine-grained condition among the latents and large overlaps between the attention windows. Such progressive video denoising allows our models to autoregressively generate video frames without quality degradation or abrupt scene changes. We present state-of-the-art results on long video generation at 1 minute (1440 frames at 24 FPS). Videos from this paper are available at https://desaixie.github.io/pa-vdm/., Comment: 15 pages, 5 figures. Our video results and code are available at https://desaixie.github.io/pa-vdm/

Gravitational lensing magnification of Type Ia supernovae (SNe Ia) allows information to be obtained about the distribution of matter on small scales. In this paper, we derive limits on the fraction $\alpha$ of the total matter density in compact objects (which comprise stars, stellar remnants, small stellar groupings and primordial black holes) of mass $M > 0.03 M_{\odot}$ over cosmological distances. Using 1,532 SNe Ia from the Dark Energy Survey Year 5 sample (DES-SN5YR) combined with a Bayesian prior for the absolute magnitude $M$, we obtain $\alpha < 0.12$ at the 95\% confidence level after marginalisation over cosmological parameters, lensing due to large-scale structure, and intrinsic non-Gaussianity. Similar results are obtained using priors from the cosmic microwave background, baryon acoustic oscillations and galaxy weak lensing, indicating our results do not depend on the background cosmology. We argue our constraints are likely to be conservative (in the sense of the values we quote being higher than the truth), but discuss scenarios in which they could be weakened by systematics of the order of $\Delta \alpha \sim 0.04$, Comment: Accepted by MNRAS

We calculate the 2024 impact factors of 36 most widely used journals in Astrophysics, using the citations collated by NASA/ADS (Astrophysics Data System) and compare them to the official impact factors. This includes journals which publish papers outside of astrophysics such as PRD, EPJC, Nature etc. We also propose a new metric to gauge the impact factor based on the median number of citations in a journal and calculate the same for all the journals. We find that the ADS-based impact factors are mostly in agreement, albeit higher than the official impact factors for most journals. The journals with the maximum fractional difference in median-based and old impact factors are JHEAP and PTEP. We find the maximum difference between the ADS and official impact factor for Nature., Comment: 5 pages. Comments welcome

The scarcity of subspecialist medical expertise, particularly in rare, complex and life-threatening diseases, poses a significant challenge for healthcare delivery. This issue is particularly acute in cardiology where timely, accurate management determines outcomes. We explored the potential of AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based experimental AI system optimized for diagnostic dialogue, to potentially augment and support clinical decision-making in this challenging context. We curated a real-world dataset of 204 complex cases from a subspecialist cardiology practice, including results for electrocardiograms, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests. We developed a ten-domain evaluation rubric used by subspecialists to evaluate the quality of diagnosis and clinical management plans produced by general cardiologists or AMIE, the latter enhanced with web-search and self-critique capabilities. AMIE was rated superior to general cardiologists for 5 of the 10 domains (with preference ranging from 9% to 20%), and equivalent for the rest. Access to AMIE's response improved cardiologists' overall response quality in 63.7% of cases while lowering quality in just 3.4%. Cardiologists' responses with access to AMIE were superior to cardiologist responses without access to AMIE for all 10 domains. Qualitative examinations suggest AMIE and general cardiologist could complement each other, with AMIE thorough and sensitive, while general cardiologist concise and specific. Overall, our results suggest that specialized medical LLMs have the potential to augment general cardiologists' capabilities by bridging gaps in subspecialty expertise, though further research and validation are essential for wide clinical utility.

Self-driving research often underrepresents cyclist collisions and safety. To address this, we present CycleCrash, a novel dataset consisting of 3,000 dashcam videos with 436,347 frames that capture cyclists in a range of critical situations, from collisions to safe interactions. This dataset enables 9 different cyclist collision prediction and classification tasks focusing on potentially hazardous conditions for cyclists and is annotated with collision-related, cyclist-related, and scene-related labels. Next, we propose VidNeXt, a novel method that leverages a ConvNeXt spatial encoder and a non-stationary transformer to capture the temporal dynamics of videos for the tasks defined in our dataset. To demonstrate the effectiveness of our method and create additional baselines on CycleCrash, we apply and compare 7 models along with a detailed ablation. We release the dataset and code at https://github.com/DeSinister/CycleCrash/ ., Comment: Accepted to WACV 2025

Class imbalance in binary classification tasks remains a significant challenge in machine learning, often resulting in poor performance on minority classes. This study comprehensively evaluates three widely-used strategies for handling class imbalance: Synthetic Minority Over-sampling Technique (SMOTE), Class Weights tuning, and Decision Threshold Calibration. We compare these methods against a baseline scenario of no-intervention across 15 diverse machine learning models and 30 datasets from various domains, conducting a total of 9,000 experiments. Performance was primarily assessed using the F1-score, although our study also tracked results on additional 9 metrics including F2-score, precision, recall, Brier-score, PR-AUC, and AUC. Our results indicate that all three strategies generally outperform the baseline, with Decision Threshold Calibration emerging as the most consistently effective technique. However, we observed substantial variability in the best-performing method across datasets, highlighting the importance of testing multiple approaches for specific problems. This study provides valuable insights for practitioners dealing with imbalanced datasets and emphasizes the need for dataset-specific analysis in evaluating class imbalance handling techniques., Comment: 13 pages including appendix, 4 tables

Data preparation is the first and a very important step towards any Large Language Model (LLM) development. This paper introduces an easy-to-use, extensible, and scale-flexible open-source data preparation toolkit called Data Prep Kit (DPK). DPK is architected and designed to enable users to scale their data preparation to their needs. With DPK they can prepare data on a local machine or effortlessly scale to run on a cluster with thousands of CPU Cores. DPK comes with a highly scalable, yet extensible set of modules that transform natural language and code data. If the user needs additional transforms, they can be easily developed using extensive DPK support for transform creation. These modules can be used independently or pipelined to perform a series of operations. In this paper, we describe DPK architecture and show its performance from a small scale to a very large number of CPUs. The modules from DPK have been used for the preparation of Granite Models [1] [2]. We believe DPK is a valuable contribution to the AI community to easily prepare data to enhance the performance of their LLM models or to fine-tune models with Retrieval-Augmented Generation (RAG)., Comment: 10 pages, 7 figures

We consider dynamic pricing algorithms as applied to the online set cover problem. In the dynamic pricing framework, we assume the standard client server model with the additional constraint that the server can only place prices over the resources they maintain, rather than authoritatively assign them. In response, incoming clients choose the resource which minimizes their disutility when taking into account these additional prices. Our main contributions are the categorization of online algorithms which can be mimicked via dynamic pricing algorithms and the identification of a strongly competitive deterministic algorithm with respect to the frequency parameter of the online set cover input.

The use of creative writing as training data for large language models (LLMS) is highly contentious. While some argue that such use constitutes "fair use" and therefore does not require consent or compensation, others argue that consent and compensation is the morally correct approach. In this paper, we seek to understand how creative writers reason about the real or hypothetical use of their writing as training data and under what conditions, if any, they would consent to their writing being used. We interviewed 33 writers with variation across genre, method of publishing, degree of professionalization, and attitudes toward and engagement with LLMs. Through a grounded theory analysis, we report on core principles that writers express and how these principles can be at odds with their realistic expectations for how institutions engage with their work.

In many experimental contexts, it is necessary to statistically remove the impact of instrumental effects in order to physically interpret measurements. This task has been extensively studied in particle physics, where the deconvolution task is called unfolding. A number of recent methods have shown how to perform high-dimensional, unbinned unfolding using machine learning. However, one of the assumptions in all of these methods is that the detector response is accurately modeled in the Monte Carlo simulation. In practice, the detector response depends on a number of nuisance parameters that can be constrained with data. We propose a new algorithm called Profile OmniFold (POF), which works in a similar iterative manner as the OmniFold (OF) algorithm while being able to simultaneously profile the nuisance parameters. We illustrate the method with a Gaussian example as a proof of concept highlighting its promising capabilities.

The global population is rapidly aging, necessitating technologies that promote healthy aging. Voice User Interfaces (VUIs), leveraging natural language interaction, offer a promising solution for older adults due to their ease of use. However, current design practices often overemphasize functionality, neglecting older adults' complex aspirations, psychological well-being, and social connectedness. To address this gap, we conducted co-design sessions with 20 older adults employing an empathic design approach. Half of the participants interacted with a probe involving health information learning, while the others focused on a probe related to exercise. This method engaged participants in collaborative activities to uncover non-functional requirements early in the design process. Results indicate that when encouraged to share their needs within a social context, older adults revealed a range of sensory, aesthetic, hedonic, and social preferences and, more importantly, the specific personas of VUIs. These insights inform the relative importance of these factors in VUI design.

Virtual Reality (VR) is quickly establishing itself in various industries, including training, education, medicine, and entertainment, in which users are frequently required to carry out multiple complex cognitive and physical activities. However, the relationship between cognitive activities, physical activities, and familiar feelings of cybersickness is not well understood and thus can be unpredictable for developers. Researchers have previously provided labeled datasets for predicting cybersickness while users are stationary, but there have been few labeled datasets on cybersickness while users are physically walking. Thus, from 39 participants, we collected head orientation, head position, eye tracking, images, physiological readings from external sensors, and the self-reported cybersickness severity, physical load, and mental load in VR. Throughout the data collection, participants navigated mazes via real walking and performed tasks challenging their attention and working memory. To demonstrate the dataset's utility, we conducted a case study of training classifiers in which we achieved 95% accuracy for cybersickness severity classification. The noteworthy performance of the straightforward classifiers makes this dataset ideal for future researchers to develop cybersickness detection and reduction models. To better understand the features that helped with classification, we performed SHAP(SHapley Additive exPlanations) analysis, highlighting the importance of eye tracking and physiological measures for cybersickness prediction while walking. This open dataset can allow future researchers to study the connection between cybersickness and cognitive loads and develop prediction models. This dataset will empower future VR developers to design efficient and effective Virtual Environments by improving cognitive load management and minimizing cybersickness.

Statistics that capture the directional dependence of the baryon distribution in the cosmic web enable unique tests of cosmology and astrophysical feedback. We use constrained oriented stacking of thermal Sunyaev-Zel'dovich (tSZ) maps to measure the anisotropic distribution of hot gas $2.5-40$ Mpc away from galaxy clusters embedded in massive filaments and superclusters. The cluster selection and orientation (at a scale of $\sim15$ Mpc) use Dark Energy Survey (DES) Year 3 data, while expanded tSZ maps from the Atacama Cosmology Telescope Data Release 6 enable a $\sim3\times$ more significant measurement of the extended gas compared to the technique's proof-of-concept. Decomposing stacks into cosine multipoles of order $m$, we detect a dipole ($m=1$) and quadrupole ($m=2$) at $8-10\sigma$, as well as evidence for $m=4$ signal at up to $6\sigma$, indicating sensitivity to late-time non-Gaussianity. We compare to the Cardinal simulations with spherical gas models pasted onto dark matter halos. The fiducial tSZ data can discriminate between two models that deplete pressure differently in low-mass halos (mimicking astrophysical feedback), preferring higher average pressure in extended structures. However, uncertainty in the amount of cosmic infrared background contamination reduces the constraining power. Additionally, we apply the technique to DES galaxy density and weak lensing to study for the first time their oriented relationships with tSZ. In the tSZ-to-lensing relation, averaged on 7.5 Mpc (transverse) scales, we observe dependence on redshift but not shape or radial distance. Thus, on large scales, the superclustering of gas pressure, galaxies, and total matter is coherent in shape and extent., Comment: 45 pages, 18 figures, submitted to ApJ

Using a sample of 2800 galaxy clusters identified in the Dark Energy Survey across the redshift range $0.20 < z < 0.60$, we characterize the hierarchical assembly of Bright Central Galaxies (BCGs) and the surrounding intracluster light (ICL). To quantify hierarchical formation we use the stellar mass - halo mass (SMHM) relation for the BCG+ICL system and incorporate the magnitude gap (M14), the difference in brightness between the BCG (measured within 30kpc) and 4th brightest cluster member galaxy within 0.5 $R_{200,c}$. The inclusion of M14, which traces BCG hierarchical growth, increases the slope and decreases the intrinsic scatter in the SMHM relation, highlighting that it is a latent variable within the BCG+ICL SMHM relation. Moreover, the correlation with M14 decreases at large radii from the BCG's centre. However, the stellar light within the BCG+ICL transition region (30kpc - 80kpc) most strongly correlates with the dark matter halo mass and has a statistically significant correlation with M14. As the light in the transition region and M14 are independent measurements, the transition region may grow as a result of the BCG's hierarchical two-phase formation. Additionally, as M14 and ICL result from hierarchical growth, we use a stacked sample and find that clusters with large M14 values are characterized by larger ICL and BCG+ICL fractions, which illustrates that the merger processes that build the BCG stellar mass also grow the ICL. Furthermore, this may suggest that M14 combined with the ICL fraction can be used as a method to identify dynamically relaxed clusters., Comment: 16 pages, 5 Figures, submitted to MNRAS on 8/30/2024

Recent progress in large language model (LLM) technology has significantly enhanced the interaction experience between humans and voice assistants (VAs). This project aims to explore a user's continuous interaction with LLM-based VA (LLM-VA) during a complex task. We recruited 12 participants to interact with an LLM-VA during a cooking task, selected for its complexity and the requirement for continuous interaction. We observed that users show both verbal and nonverbal behaviors, though they know that the LLM-VA can not capture those nonverbal signals. Despite the prevalence of nonverbal behavior in human-human communication, there is no established analytical methodology or framework for exploring it in human-VA interactions. After analyzing 3 hours and 39 minutes of video recordings, we developed an analytical framework with three dimensions: 1) behavior characteristics, including both verbal and nonverbal behaviors, 2) interaction stages--exploration, conflict, and integration--that illustrate the progression of user interactions, and 3) stage transition throughout the task. This analytical framework identifies key verbal and nonverbal behaviors that provide a foundation for future research and practical applications in optimizing human and LLM-VA interactions.

Accurate classification of celestial objects is essential for advancing our understanding of the universe. MargNet is a recently developed deep learning-based classifier applied to SDSS DR16 dataset to segregate stars, quasars, and compact galaxies using photometric data. MargNet utilizes a stacked architecture, combining a Convolutional Neural Network (CNN) for image modelling and an Artificial Neural Network (ANN) for modelling photometric parameters. In this study, we propose enhancing MargNet's performance by incorporating attention mechanisms and Vision Transformer (ViT)-based models for processing image data. The attention mechanism allows the model to focus on relevant features and capture intricate patterns within images, effectively distinguishing between different classes of celestial objects. Additionally, we leverage ViTs, a transformer-based deep learning architecture renowned for exceptional performance in image classification tasks. We enhance the model's understanding of complex astronomical images by utilizing ViT's ability to capture global dependencies and contextual information. Our approach uses a curated dataset comprising 240,000 compact and 150,000 faint objects. The models learn classification directly from the data, minimizing human intervention. Furthermore, we explore ViT as a hybrid architecture that uses photometric features and images together as input to predict astronomical objects. Our results demonstrate that the proposed attention mechanism augmented CNN in MargNet marginally outperforms the traditional MargNet and the proposed ViT-based MargNet models. Additionally, the ViT-based hybrid model emerges as the most lightweight and easy-to-train model with classification accuracy similar to that of the best-performing attention-enhanced MargNet., Comment: 33 pages, 11 figures. Accepted for publication in APSS