Search

Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications. Existing approaches require prompt tuning or architectural adaptations, limiting zero-shot applicability. Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth. Using large language model insights on object co-occurrence, we introduce compound prompts grounded in realistic object combinations. Analysis of these prompt scores reveals VLM biases and ``AND''/``OR'' signal ambiguities, notably that maximum compound scores are surprisingly suboptimal compared to second-highest scores. We address these through a debiasing and score-fusion algorithm that corrects image bias and clarifies VLM response behaviors. Our method enhances other zero-shot approaches, consistently improving their results. Experiments show superior mean Average Precision (mAP) compared to methods requiring training data, achieved through refined object ranking for robust zero-shot MLR.

This paper introduces a new mathematical framework for analysis and optimization of tensor expressions within an enclosing loop. Tensors are multi-dimensional arrays of values. They are common in high performance computing (HPC) and machine learning domains. Our framework extends Scalar Evolution - an important optimization pass implemented in both LLVM and GCC - to tensors. Scalar Evolution (SCEV) relies on the theory of `Chain of Recurrences' for its mathematical underpinnings. We use the same theory for Tensor Evolution (TeV). While some concepts from SCEV map easily to TeV -- e.g. element-wise operations; tensors introduce new operations such as concatenation, slicing, broadcast, reduction, and reshape which have no equivalent in scalars and SCEV. Not all computations are amenable to TeV analysis but it can play a part in the optimization and analysis parts of ML and HPC compilers. Also, for many mathematical/compiler ideas, applications may go beyond what was initially envisioned, once others build on it and take it further. We hope for a similar trajectory for the tensor-evolution concept.

There is significant concern about the impact of generative AI on society. Modern AI tools are capable of generating ever more realistic text, images, and videos, and functional code, from minimal prompts. Accompanying this rise in ability and usability, there is increasing alarm about the misuses to which these tools can be put, and the intentional and unintentional harms to individuals and society that may result. In this paper, we argue that \emph{agency} is the appropriate lens to study these harms and benefits, but that doing so will require advancement in the theory of agency, and advancement in how this theory is applied in (agent-based) models.

The growing interest in employing large language models (LLMs) for decision-making in social and economic contexts has raised questions about their potential to function as agents in these domains. A significant number of societal problems involve the distribution of resources, where fairness, along with economic efficiency, play a critical role in the desirability of outcomes. In this paper, we examine whether LLM responses adhere to fundamental fairness concepts such as equitability, envy-freeness, and Rawlsian maximin, and investigate their alignment with human preferences. We evaluate the performance of several LLMs, providing a comparative benchmark of their ability to reflect these measures. Our results demonstrate a lack of alignment between current LLM responses and human distributional preferences. Moreover, LLMs are unable to utilize money as a transferable resource to mitigate inequality. Nonetheless, we demonstrate a stark contrast when (some) LLMs are tasked with selecting from a predefined menu of options rather than generating one. In addition, we analyze the robustness of LLM responses to variations in semantic factors (e.g. intentions or personas) or non-semantic prompting changes (e.g. templates or orderings). Finally, we highlight potential strategies aimed at enhancing the alignment of LLM behavior with well-established fairness concepts.

Magnetic topological insulators (MTIs) host topologically protected edge states, but the role that these edge states play in electronic transport remains unclear. Using scanning superconducting quantum interference device (SQUID) microscopy, we performed local measurements of the current distribution in a quantum anomalous Hall (QAH) insulator at large bias currents, where the quantization of the conductivity tensor breaks down. We find that bulk currents in the channel interior coexist with edge currents at the sample boundary. While the position of the edge current changes with the reversal of the magnetic field, it does not depend on the current direction. To understand our observations, we introduce a model which includes contributions from both the sample magnetization and currents driven by chemical potential gradients. To parameterize our model, we use local measurements of the chemical potential induced changes in the sample magnetization. Our model reveals that the observed edge currents can be understood as changes in the magnetization generated by the electrochemical potential distribution in the sample under bias. Our work underscores the complexity of electronic transport in MTIs and highlights both the value and challenges of using magnetic imaging to disentangle various contributions to the electronic transport signatures.

With rapid advances in qubit technologies, techniques for localizing, modulating, and measuring RF fields and their impact on qubit performance are of the utmost importance. Here, we demonstrate that flux-channeling from a permalloy nanowire can be used to achieve localized spatial modulation of an RF field and that the modulated field can be mapped with high resolution using the Rabi oscillations of an NV center. Rabi maps reveal ~100 mm wavelength microwaves concentrated in sub-300 nm-scale regions with up to ~16$\times$ power enhancement. This modulation is robust over a 20 dBm power range and has no adverse impact on NV $T_2$ coherence time. Micromagnetic simulations confirm that the modulated field results from the nanowire's stray field through its constructive/destructive interference with the incident RF field. Our findings provide a new pathway for controlling qubits, amplifying RF signals, and mapping local fields in various on-chip RF technologies.

We report charge to spin conversion in a hybrid heterostructure comprised of atomically thin bismuth (Bi) confined between a silicon carbide (SiC) substrate and epitaxial graphene (EG). We confirm composition, dimensionality, and a 96.5 \% intercalation coverage using X-ray photolectron spectroscopy, scanning transmission microscopy, low energy electron diffraction, and Raman spectroscopy. Electrical transport measurements show signs of weak antilocalization in the heterostructure, consistent with spin-orbit coupling in this hybrid heterostructure. Spin torque ferromagnetic resonance measurements in permalloy/EG/2D-Bi heterostructures probe charge-to-spin conversion and revealing that an in plane polarization of the spin current, perpendicular to the charge current. The ratio of the in-plane to out-of-plane torque is 3.75 times higher than in hydrogenated graphene control samples.

Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize "precision" and "recall," borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation), respectively. With the increase in metric proposals, there is a need for a unifying perspective, allowing for easier comparison and clearer explanation of their benefits and drawbacks. To this end, we unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. Additionally, we propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), which separately measure fidelity and two distinct aspects of diversity, inter- and intra-class. Our domain-agnostic metric, derived from the information-theoretic concepts of entropy and cross-entropy, can be dissected for both sample- and mode-level analysis. Our detailed experimental results demonstrate the sensitivity of our metric components to their respective qualities and reveal undesirable behaviors of other metrics.

Embedding models are crucial for tasks in Information Retrieval (IR) and semantic similarity measurement, yet their handling of longer texts and associated positional biases remains underexplored. In this study, we investigate the impact of content position and input size on text embeddings. Our experiments reveal that embedding models, irrespective of their positional encoding mechanisms, disproportionately prioritize the beginning of an input. Ablation studies demonstrate that insertion of irrelevant text or removal at the start of a document reduces cosine similarity between altered and original embeddings by up to 12.3% more than ablations at the end. Regression analysis further confirms this bias, with sentence importance declining as position moves further from the start, even with with content-agnosticity. We hypothesize that this effect arises from pre-processing strategies and chosen positional encoding techniques. These findings quantify the sensitivity of retrieval systems and suggest a new lens towards embedding model robustness., Comment: 13 pages, 11 figures, NeurIPS

Channel and spatial attention mechanisms introduced by earlier works enhance the representation abilities of deep convolutional neural networks (CNNs) but often lead to increased parameter and computation costs. While recent approaches focus solely on efficient feature context modeling for channel attention, we aim to model both channel and spatial attention comprehensively with minimal parameters and reduced computation. Leveraging the principles of relational modeling in graphs, we introduce a constant-parameter module, STEAM: Squeeze and Transform Enhanced Attention Module, which integrates channel and spatial attention to enhance the representation power of CNNs. To our knowledge, we are the first to propose a graph-based approach for modeling both channel and spatial attention, utilizing concepts from multi-head graph transformers. Additionally, we introduce Output Guided Pooling (OGP), which efficiently captures spatial context to further enhance spatial attention. We extensively evaluate STEAM for large-scale image classification, object detection and instance segmentation on standard benchmark datasets. STEAM achieves a 2% increase in accuracy over the standard ResNet-50 model with only a meager increase in GFLOPs. Furthermore, STEAM outperforms leading modules ECA and GCT in terms of accuracy while achieving a three-fold reduction in GFLOPs.

Source separation is the process of isolating individual sounds in an auditory mixture of multiple sounds [1], and has a variety of applications ranging from speech enhancement and lyric transcription [2] to digital audio production for music. Furthermore, Automatic Music Transcription (AMT) is the process of converting raw music audio into sheet music that musicians can read [3]. Historically, these tasks have faced challenges such as significant audio noise, long training times, and lack of free-use data due to copyright restrictions. However, recent developments in deep learning have brought new promising approaches to building low-distortion stems and generating sheet music from audio signals [4]. Using spectrogram masking, deep neural networks, and the MuseScore API, we attempt to create an end-to-end pipeline that allows for an initial music audio mixture (e.g...wav file) to be separated into instrument stems, converted into MIDI files, and transcribed into sheet music for each component instrument.

As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems

Solar meridional circulation, which manifests as poleward flow near the surface, is a relatively weak flow. While meridional circulation has been measured through various local helioseismic techniques, there is a lack of consensus about the nature of the depth profile and location of return flow, owing to its small amplitude and poor signal-to-noise ratio in observations. The measurements are strongly hampered by systematic effects, whose amplitudes are comparable to the signal induced by the flow and modelling them is therefore crucial. The removal of the center-to-limb systematic, which is the largest known feature hampering the inference of meridional flow, has been heuristically performed in helioseismic analyses, but it's effect on global modes is not fully understood or modelled. Here, we propose both a way to model the center-to-limb systematic and a method for estimation of meridional flow using global helioseismic cross-spectral analysis. We demonstrate that the systematic cannot be ignored while modelling the mode-coupling cross-spectral measurement and thus is critical for the inference of meridional circulation. We also show that inclusion of a model for the center-to-limb systematic improves shallow meridional circulation estimates from cross-spectral analysis., Comment: 26 pages, 23 figures, accepted for publication in ApJ

The advancement of 5G and NextG networks through Open Radio Access Network (O-RAN) architecture enables a shift toward virtualized, modular, and disaggregated configurations. A core component of O-RAN is the RAN Intelligent Controller (RIC), which manages RAN using machine learning-driven xApps that access sensitive data from RAN and User Equipment (UE), stored in the near Real-Time RIC (Near-RT RIC) database. This shared, open environment increases the risk of unauthorized data exposure. To address these concerns, this paper proposes a zero-trust RIC (ZT-RIC) framework that preserves data privacy across the RIC platform, including the RIC database, xApps, and E2 interface. ZT-RIC employs Inner Product Functional Encryption (IPFE) to encrypt RAN/UE data at the base station, preventing leaks through the E2 interface and shared database. Additionally, ZT-RIC enables xApps to perform inference on encrypted data without exposing sensitive information. For evaluation, a state-of-the-art InterClass xApp, which detects jamming signals using RAN key performance metrics (KPMs), is implemented. Testing on an LTE/5G O-RAN testbed shows that ZT-RIC preserves data confidentiality while achieving 97.9% accuracy in jamming detection and meeting sub-second latency requirements, with a round-trip time (RTT) of 0.527 seconds., Comment: This paper has been accepted to CCNC 2025

Topological effects manifest in a variety of lattice geometries. While square lattices, due to their simplicity, have been used for models supporting nontrivial topology, several exotic topological phenomena such as Dirac points, Weyl points and Haldane phases are most commonly supported by non-square lattices. Examples of prototypical non-square lattices include the honeycomb lattice of graphene and the Kagome lattice, both of which break fundamental symmetries and can exhibit quantized transport, especially when long-range hoppings and gauge fields are incorporated. The challenge of controllably realizing long-range hoppings and gauge fields has motivated a large body of research focused on harnessing lattices encoded in "synthetic" dimensions. Photons in particular have many internal degrees of freedom and hence show promise for implementing these synthetic dimensions; however, most photonic synthetic dimensions has hitherto created 1D or 2D square lattices. Here we show that non-square lattice Hamiltonians can be implemented using Floquet synthetic dimensions. Our construction uses dynamically modulated ring resonators and provides the capacity for direct $k$-space engineering of lattice Hamiltonians. Such a construction lifts constraints on the orthogonality of lattice vectors that make square geometries simpler to implement, and instead transfers the complexity to the engineering of complex Floquet drive signals. We simulate topological signatures of the Haldane and the brick-wall Haldane model and observe them to be robust in the presence of external optical drive and photon loss, and discuss unique characteristics of their topological transport when implemented on these Floquet lattices. Our proposal demonstrates the potential of driven-dissipative Floquet synthetic dimensions as a new architecture for $k$-space Hamiltonian simulation of high-dimensional lattice geometries.

One Health issues, such as the spread of highly pathogenic avian influenza (HPAI), present significant challenges at the intersection of human, animal, and environmental health. Recent H5N1 outbreaks underscore the need for comprehensive modeling that capture the complex interactions between various entities in these interconnected ecosystems, encompassing livestock, wild birds, and human populations. To support such efforts, we present a synthetic spatiotemporal gridded dataset for the contiguous United States, referred to as a digital similar. The methodology for constructing this digital similar involves fusing diverse datasets using statistical and optimization techniques. The livestock component includes farm-level representations of multiple livestock types -- cattle, poultry, hogs, and sheep -- including further categorization into subtypes, such as milk and beef cows, chicken, turkeys, ducks, etc. It also includes location-level data for livestock-product processing centers. Weekly abundance data for key wild bird species involved in avian flu transmission are included along with temporal networks of movements. Gridded distributions of the human population, along with demographic and occupational features, capture the placement of agricultural workers and the general population. The digital similar is verified and validated in multiple ways.This dataset aims to provide a comprehensive basis for modeling complex phenomena at the wild-domestic-human interfaces.

Music source separation (MSS) is a task that involves isolating individual sound sources, or stems, from mixed audio signals. This paper presents an ensemble approach to MSS, combining several state-of-the-art architectures to achieve superior separation performance across traditional Vocal, Drum, and Bass (VDB) stems, as well as expanding into second-level hierarchical separation for sub-stems like kick, snare, lead vocals, and background vocals. Our method addresses the limitations of relying on a single model by utilising the complementary strengths of various models, leading to more balanced results across stems. For stem selection, we used the harmonic mean of Signal-to-Noise Ratio (SNR) and Signal-to-Distortion Ratio (SDR), ensuring that extreme values do not skew the results and that both metrics are weighted effectively. In addition to consistently high performance across the VDB stems, we also explored second-level hierarchical separation, revealing important insights into the complexities of MSS and how factors like genre and instrumentation can influence model performance. While the second-level separation results show room for improvement, the ability to isolate sub-stems marks a significant advancement. Our findings pave the way for further research in MSS, particularly in expanding model capabilities beyond VDB and improving niche stem separations such as guitar and piano.

Ballots are the core records of elections. Electronic records of actual ballots cast (cast vote records) are available to the public in some jurisdictions. However, they have been released in a variety of formats and have not been independently evaluated. Here we introduce a database of cast vote records from the 2020 U.S. general election. We downloaded publicly available unstandardized cast vote records, standardized them into a multi-state database, and extensively compared their totals to certified election results. Our release includes vote records for President, Governor, U.S. Senate and House, and state upper and lower chambers -- covering 42.7 million voters in 20 states who voted for more than 2,204 candidates. This database serves as a uniquely granular administrative dataset for studying voting behavior and election administration. Using this data, we show that in battleground states, 1.9 percent of solid Republicans (as defined by their congressional and state legislative voting) in our database split their ticket for Joe Biden, while 1.2 percent of solid Democrats split their ticket for Donald Trump., Comment: 26 pages and appendix

Understanding the internal mechanisms of GPT-style transformers, particularly their capacity to perform in-context learning (ICL), is critical for advancing AI alignment and interpretability. In-context learning allows transformers to generalize during inference without modifying their weights, yet the precise operations driving this capability remain largely opaque. This paper presents an investigation into the mechanistic interpretability of these transformers, focusing specifically on their ability to learn and predict affine recurrences as an ICL task. To address this, we trained a custom three-layer transformer to predict affine recurrences and analyzed the model's internal operations using both empirical and theoretical approaches. Our findings reveal that the model forms an initial estimate of the target sequence using a copying mechanism in the zeroth layer, which is subsequently refined through negative similarity heads in the second layer. These insights contribute to a deeper understanding of transformer behaviors in recursive tasks and offer potential avenues for improving AI alignment through mechanistic interpretability. Finally, we discuss the implications of our results for future work, including extensions to higher-dimensional recurrences and the exploration of polynomial sequences., Comment: 21 pages, 18 figures

In this paper, we consider teams of robots with heterogeneous skills (e.g., sensing and manipulation) tasked with collaborative missions described by Linear Temporal Logic (LTL) formulas. These LTL-encoded tasks require robots to apply their skills to specific regions and objects in a temporal and logical order. While existing temporal logic planning algorithms can synthesize correct-by-construction paths, they typically lack reactivity to unexpected failures of robot skills, which can compromise mission performance. This paper addresses this challenge by proposing a reactive LTL planning algorithm that adapts to unexpected failures during deployment. Specifically, the proposed algorithm reassigns sub-tasks to robots based on their functioning skills and locally revises team plans to accommodate these new assignments and ensure mission completion. The main novelty of the proposed algorithm is its ability to handle cases where mission completion becomes impossible due to limited functioning robots. Instead of reporting mission failure, the algorithm strategically prioritizes the most crucial sub-tasks and locally revises the team's plans, as per user-specified priorities, to minimize mission violations. We provide theoretical conditions under which the proposed framework computes the minimum violation task reassignments and team plans. We provide numerical and hardware experiments to demonstrate the efficiency of the proposed method.

This paper addresses the challenge of improving user experience on e-commerce platforms by enhancing product ranking relevant to users' search queries. Ambiguity and complexity of user queries often lead to a mismatch between the user's intent and retrieved product titles or documents. Recent approaches have proposed the use of Transformer-based models, which need millions of annotated query-title pairs during the pre-training stage, and this data often does not take user intent into account. To tackle this, we curate samples from existing datasets at eBay, manually annotated with buyer-centric relevance scores and centrality scores, which reflect how well the product title matches the users' intent. We introduce a User-intent Centrality Optimization (UCO) approach for existing models, which optimises for the user intent in semantic product search. To that end, we propose a dual-loss based optimisation to handle hard negatives, i.e., product titles that are semantically relevant but do not reflect the user's intent. Our contributions include curating challenging evaluation sets and implementing UCO, resulting in significant product ranking efficiency improvements observed for different evaluation metrics. Our work aims to ensure that the most buyer-centric titles for a query are ranked higher, thereby, enhancing the user experience on e-commerce platforms., Comment: EMNLP 2024: Industry track

The constant shifts in social and political contexts, driven by emerging social movements and political events, lead to new forms of hate content and previously unrecognized hate patterns that machine learning models may not have captured. Some recent literature proposes the data augmentation-based techniques to enrich existing hate datasets by incorporating samples that reveal new implicit hate patterns. This approach aims to improve the model's performance on out-of-domain implicit hate instances. It is observed, that further addition of more samples for augmentation results in the decrease of the performance of the model. In this work, we propose a Knowledge Transfer-driven Concept Refinement method that distills and refines the concepts related to implicit hate samples through novel prototype alignment and concept losses, alongside data augmentation based on concept activation vectors. Experiments with several publicly available datasets show that incorporating additional implicit samples reflecting new hate patterns through concept refinement enhances the model's performance, surpassing baseline results while maintaining cross-dataset generalization capabilities.\footnote{DISCLAIMER: This paper contains explicit statements that are potentially offensive.}, Comment: 11 pages, 4 figures, 2 algorithms, 5 tables

Visual servoing, the method of controlling robot motion through feedback from visual sensors, has seen significant advancements with the integration of optical flow-based methods. However, its application remains limited by inherent challenges, such as the necessity for a target image at test time, the requirement of substantial overlap between initial and target images, and the reliance on feedback from a single camera. This paper introduces Imagine2Servo, an innovative approach leveraging diffusion-based image editing techniques to enhance visual servoing algorithms by generating intermediate goal images. This methodology allows for the extension of visual servoing applications beyond traditional constraints, enabling tasks like long-range navigation and manipulation without predefined goal images. We propose a pipeline that synthesizes subgoal images grounded in the task at hand, facilitating servoing in scenarios with minimal initial and target image overlap and integrating multi-camera feedback for comprehensive task execution. Our contributions demonstrate a novel application of image generation to robotic control, significantly broadening the capabilities of visual servoing systems. Real-world experiments validate the effectiveness and versatility of the Imagine2Servo framework in accomplishing a variety of tasks, marking a notable advancement in the field of visual servoing., Comment: Published at 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Neural Radiance Fields (NeRFs) have shown significant promise in 3D scene reconstruction and novel view synthesis. In agricultural settings, NeRFs can serve as digital twins, providing critical information about fruit detection for yield estimation and other important metrics for farmers. However, traditional NeRFs are not robust to challenging lighting conditions, such as low-light, extreme bright light and varying lighting. To address these issues, this work leverages three different sensors: an RGB camera, an event camera and a thermal camera. Our RGB scene reconstruction shows an improvement in PSNR and SSIM by +2.06 dB and +8.3% respectively. Our cross-spectral scene reconstruction enhances downstream fruit detection by +43.0% in mAP50 and +61.1% increase in mAP50-95. The integration of additional sensors leads to a more robust and informative NeRF. We demonstrate that our multi-modal system yields high quality photo-realistic reconstructions under various tree canopy covers and at different times of the day. This work results in the development of a resilient NeRF, capable of performing well in visibly degraded scenarios, as well as a learnt cross-spectral representation, that is used for automated fruit detection., Comment: 7 pages, 5 figures

In this article, we propose a new method of generating single microwave photons in superconducting circuits. We theoretically show that pure single microwave photons can be generated on demand and tuned over a large frequency band by making use of Landau-Zener transitions under a rapid sweep of a control parameter. We devise a protocol that enables pulse-level control of the frequency of the emitted photon over two octaves, without requiring extensive calibration. Additionally, we make theoretical estimates of the generation efficiency, tunability, purity, and linewidth of the photons emitted using this method for both charge and flux qubit-based architectures. We also provide estimates of optimal device parameters for these architectures in order to realize the device., Comment: 7+7 pages, 12 figures, Comments welcome

This paper introduces an intelligent lecturing assistant (ILA) system that utilizes a knowledge graph to represent course content and optimal pedagogical strategies. The system is designed to support instructors in enhancing student learning through real-time analysis of voice, content, and teaching methods. As an initial investigation, we present a case study on lecture voice sentiment analysis, in which we developed a training set comprising over 3,000 one-minute lecture voice clips. Each clip was manually labeled as either engaging or non-engaging. Utilizing this dataset, we constructed and evaluated several classification models based on a variety of features extracted from the voice clips. The results demonstrate promising performance, achieving an F1-score of 90% for boring lectures on an independent set of over 800 test voice clips. This case study lays the groundwork for the development of a more sophisticated model that will integrate content analysis and pedagogical practices. Our ultimate goal is to aid instructors in teaching more engagingly and effectively by leveraging modern artificial intelligence techniques., Comment: Accepted in the 4th Workshop on Knowledge Graphs and Big Data @ IEEE Big Data Conference 2024

Complex dialog systems often use retrieved evidence to facilitate factual responses. Such RAG (Retrieval Augmented Generation) systems retrieve from massive heterogeneous data stores that are usually architected as multiple indexes or APIs instead of a single monolithic source. For a given query, relevant evidence needs to be retrieved from one or a small subset of possible retrieval sources. Complex queries can even require multi-step retrieval. For example, a conversational agent on a retail site answering customer questions about past orders will need to retrieve the appropriate customer order first and then the evidence relevant to the customer's question in the context of the ordered product. Most RAG Agents handle such Chain-of-Thought (CoT) tasks by interleaving reasoning and retrieval steps. However, each reasoning step directly adds to the latency of the system. For large models this latency cost is significant -- in the order of multiple seconds. Multi-agent systems may classify the query to a single Agent associated with a retrieval source, though this means that a (small) classification model dictates the performance of a large language model. In this work we present REAPER (REAsoning-based PlannER) - an LLM based planner to generate retrieval plans in conversational systems. We show significant gains in latency over Agent-based systems and are able to scale easily to new and unseen use cases as compared to classification-based planning. Though our method can be applied to any RAG system, we show our results in the context of a conversational shopping assistant.

Graph clustering is an important unsupervised learning technique for partitioning graphs with attributes and detecting communities. However, current methods struggle to accurately capture true community structures and intra-cluster relations, be computationally efficient, and identify smaller communities. We address these challenges by integrating coarsening and modularity maximization, effectively leveraging both adjacency and node features to enhance clustering accuracy. We propose a loss function incorporating log-determinant, smoothness, and modularity components using a block majorization-minimization technique, resulting in superior clustering outcomes. The method is theoretically consistent under the Degree-Corrected Stochastic Block Model (DC-SBM), ensuring asymptotic error-free performance and complete label recovery. Our provably convergent and time-efficient algorithm seamlessly integrates with graph neural networks (GNNs) and variational graph autoencoders (VGAEs) to learn enhanced node features and deliver exceptional clustering performance. Extensive experiments on benchmark datasets demonstrate its superiority over existing state-of-the-art methods for both attributed and non-attributed graphs., Comment: The first two authors contributed equally to this work

We embed the Penrose limit into the Weyl classical double copy. Thereby, we provide a lift of the double copy properties of plane wave spacetimes into black hole geometries and we open a novel avenue towards taking the classical double copy beyond statements about algebraically special backgrounds. In particular, the Penrose limit, viewed as the leading order Fermi coordinate expansion around a null geodesic, complements approaches leveraging asymptotic flatness such as the asymptotic Weyl double copy. Along the way, we show how our embedding of the Penrose limit within the Weyl double copy naturally fixes the functional ambiguity in the double copy for Petrov type N spacetimes. We also highlight the utility of a spinorial approach to the Penrose limit. In particular, we use this spinorial approach to derive a simple analytical expression for arbitrary Penrose limits of four-dimensional, vacuum type D spacetimes., Comment: 44+18 pages

We introduce OpenBot-Fleet, a comprehensive open-source cloud robotics system for navigation. OpenBot-Fleet uses smartphones for sensing, local compute and communication, Google Firebase for secure cloud storage and off-board compute, and a robust yet low-cost wheeled robot toact in real-world environments. The robots collect task data and upload it to the cloud where navigation policies can be learned either offline or online and can then be sent back to the robot fleet. In our experiments we distribute 72 robots to a crowd of workers who operate them in homes, and show that OpenBot-Fleet can learn robust navigation policies that generalize to unseen homes with >80% success rate. OpenBot-Fleet represents a significant step forward in cloud robotics, making it possible to deploy large continually learning robot fleets in a cost-effective and scalable manner. All materials can be found at https://www.openbot.org. A video is available at https://youtu.be/wiv2oaDgDi8, Comment: Accepted at ICRA'24

To accelerate the process of materials design, materials science has increasingly used data driven techniques to extract information from collected data. Specially, machine learning (ML) algorithms, which span the ML discipline, have demonstrated ability to predict various properties of materials with the level of accuracy similar to explicit calculation of quantum mechanical theories, but with significantly reduced run time and computational resources. Within ML, graph neural networks have emerged as an important algorithm within the field of machine learning, since they are capable of predicting accurately a wide range of important physical, chemical and electronic properties due to their higher learning ability based on the graph representation of material and molecular descriptors through the aggregation of information embedded within the graph. In parallel with the development of state of the art classical machine learning applications, the fusion of quantum computing and machine learning have created a new paradigm where classical machine learning model can be augmented with quantum layers which are able to encode high dimensional data more efficiently. Leveraging the structure of existing algorithms, we developed a unique and novel gradient free hybrid quantum classical convoluted graph neural network (HyQCGNN) to predict formation energies of perovskite materials. The performance of our hybrid statistical model is competitive with the results obtained purely from a classical convoluted graph neural network, and other classical machine learning algorithms, such as XGBoost. Consequently, our study suggests a new pathway to explore how quantum feature encoding and parametric quantum circuits can yield drastic improvements of complex ML algorithm like graph neural network.

We have used a combination of linear response time-domain THz spectroscopy (TDTS) and high-field non-linear THz spectroscopy to separately probe the electronic momentum and energy relaxation rates respectively of the Dirac semimetal Cd$_3$As$_2$. We find, consistent with prior measurements, that Cd$_3$As$_2$ has an enormous nonlinearities in the THz frequency range. We extract the momentum relaxation rate of Cd$_3$As$_2$ using Drude fits to the optical conductivity. We also conduct THz range 2D coherent spectroscopy. The dominant response is a pump-probe signal, which allow us to separately extract the energy relaxation rate. We find that the rate of energy relaxation decreases down to the lowest measured temperatures. We connect this to Cd$_3$As$_2$ anomalous lattice dynamics, evidence for which is found in its low thermal conductivity and soft phonons in Raman scattering. The lack of a peak in the energy relaxation rate as a function of T can be connected to the linear in T dependence of the current relaxation e.g. the phonon scattering is elastic down to the lowest measured temperatures approximately 120 K., Comment: 10 pages, 4 figures, revised version to appear in PRB

We study nonequilibrium Andreev resonances in a voltage-biased graphene three-terminal Josephson junction (JJ). We observe periodic oscillations of resistance with maxima at multiples of the magnetic flux quantum (noninversion regime). As we increase the bias voltage, we further observe a transition point beyond which oscillations exhibit a $\pi$ phase shift (inversion regime) with maxima of resistance occuring at multiples of half-flux quantum. At this transition point, the frequency of the oscillations is doubled. We develop a model based on the coupling of the static Andreev bound states (ABSs) to the nonequilibrium Fermi surface of graphene to explain the observed noninversion to inversion crossovers. Our model associates these crossovers to microscopic phase-sensitive Andreev reflections which couple the normal and superfluid components of the current. Our findings show that multiterminal JJs can be used to engineer unconventional energy-phase relations such as those expected in the $\pi$-shifted ABSs without relying on quartet and Floquet physics. These nonequilibrium ABSs could potentially find applications in superconducting $\pi$ qubits.

Irrigation mapping plays a crucial role in effective water management, essential for preserving both water quality and quantity, and is key to mitigating the global issue of water scarcity. The complexity of agricultural fields, adorned with diverse irrigation practices, especially when multiple systems coexist in close quarters, poses a unique challenge. This complexity is further compounded by the nature of Landsat's remote sensing data, where each pixel is rich with densely packed information, complicating the task of accurate irrigation mapping. In this study, we introduce an innovative approach that employs a progressive training method, which strategically increases patch sizes throughout the training process, utilizing datasets from Landsat 5 and 7, labeled with the WRLU dataset for precise labeling. This initial focus allows the model to capture detailed features, progressively shifting to broader, more general features as the patch size enlarges. Remarkably, our method enhances the performance of existing state-of-the-art models by approximately 20%. Furthermore, our analysis delves into the significance of incorporating various spectral bands into the model, assessing their impact on performance. The findings reveal that additional bands are instrumental in enabling the model to discern finer details more effectively. This work sets a new standard for leveraging remote sensing imagery in irrigation mapping., Comment: Full version of the paper will be appearing in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024

Memory is a limiting resource for many deep learning tasks. Beside the neural network weights, one main memory consumer is the computation graph built up by automatic differentiation (AD) for backpropagation. We observe that PyTorch's current AD implementation neglects information about parameter differentiability when storing the computation graph. This information is useful though to reduce memory whenever gradients are requested for a parameter subset, as is the case in many modern fine-tuning tasks. Specifically, inputs to layers that act linearly in their parameters (dense, convolution, or normalization layers) can be discarded whenever the parameters are marked as non-differentiable. We provide a drop-in, differentiability-agnostic implementation of such layers and demonstrate its ability to reduce memory without affecting run time., Comment: The code is available at https://github.com/plutonium-239/memsave_torch . This paper was accepted to WANT@ICML'24

We use a concerted theory-experiment effort to investigate the formation of chiral real space spin texture when the archetypal Dirac semimetal Cd$_3$As$_2$ is interfaced with In$_{1-x}$Mn$_x$As, a ferromagnetic semiconductor with perpendicular magnetic anisotropy. Our calculations reveal a nonzero off-diagonal spin susceptibility in the Cd$_3$As$_2$ layer due to the Rashba spin-orbit coupling from broken inversion symmetry. This implies the presence of a Dzyaloshinskii-Moriya interaction between local moments in the In$_{1-x}$Mn$_x$As layer, mediated by Dirac electrons in the vicinal Cd$_3$As$_2$ layer, potentially creating the conditions for a real space chiral spin texture. Using electrical magnetoresistance measurements at low temperature, we observe an emergent excess contribution to the transverse magneto-resistance whose behavior is consistent with a topological Hall effect arising from the formation of an interfacial chiral spin texture. This excess Hall voltage varies with gate voltage, indicating a promising electrostatically-tunable platform for understanding the interplay between the helical momentum space states of a Dirac semimetal and chiral real space spin textures in a ferromagnet.

This paper presents SNOW-SCA, the first power side-channel analysis (SCA) attack of a 5G mobile communication security standard candidate, SNOW-V, running on a 32-bit ARM Cortex-M4 microcontroller. First, we perform a generic known-key correlation (KKC) analysis to identify the leakage points. Next, a correlation power analysis (CPA) attack is performed, which reduces the attack complexity to two key guesses for each key byte. The correct secret key is then uniquely identified utilizing linear discriminant analysis (LDA). The profiled SCA attack with LDA achieves 100% accuracy after training with $<200$ traces, which means the attack succeeds with just a single trace. Overall, using the \textit{combined CPA and LDA attack} model, the correct secret key byte is recovered with <50 traces collected using the ChipWhisperer platform. The entire 256-bit secret key of SNOW-V can be recovered incrementally using the proposed SCA attack. Finally, we suggest low-overhead countermeasures that can be used to prevent these SCA attacks.

The exponential growth in scale and relevance of social networks enable them to provide expansive insights. Predicting missing links in social networks efficiently can help in various modern-day business applications ranging from generating recommendations to influence analysis. Several categories of solutions exist for the same. Here, we explore various feature extraction techniques to generate representations of nodes and edges in a social network that allow us to predict missing links. We compare the results of using ten feature extraction techniques categorized across Structural embeddings, Neighborhood-based embeddings, Graph Neural Networks, and Graph Heuristics, followed by modeling with ensemble classifiers and custom Neural Networks. Further, we propose combining heuristic-based features and learned representations that demonstrate improved performance for the link prediction task on social network datasets. Using this method to generate accurate recommendations for many applications is a matter of further study that appears very promising. The code for all the experiments has been made public., Comment: Accepted to the MAISoN Workshop at IJCAI 2023

Optimal allocation of agricultural water in the event of droughts is an important global problem. In addressing this problem, many aspects, including the welfare of farmers, the economy, and the environment, must be considered. Under this backdrop, our work focuses on several resource-matching problems accounting for agents with multi-crop portfolios, geographic constraints, and fairness. First, we address a matching problem where the goal is to maximize a welfare function in two-sided markets where buyers' requirements and sellers' supplies are represented by value functions that assign prices (or costs) to specified volumes of water. For the setting where the value functions satisfy certain monotonicity properties, we present an efficient algorithm that maximizes a social welfare function. When there are minimum water requirement constraints, we present a randomized algorithm which ensures that the constraints are satisfied in expectation. For a single seller--multiple buyers setting with fairness constraints, we design an efficient algorithm that maximizes the minimum level of satisfaction of any buyer. We also present computational complexity results that highlight the limits on the generalizability of our results. We evaluate the algorithms developed in our work with experiments on both real-world and synthetic data sets with respect to drought severity, value functions, and seniority of agents.

We study the problem of capacity modification in the many-to-one stable matching of workers and firms. Our goal is to systematically study how the set of stable matchings changes when some seats are added to or removed from the firms. We make three main contributions: First, we examine whether firms and workers can improve or worsen upon changing the capacities under worker-proposing and firm-proposing deferred acceptance algorithms. Second, we study the computational problem of adding or removing seats to either match a fixed worker-firm pair in some stable matching or make a fixed matching stable with respect to the modified problem. We develop polynomial-time algorithms for these problems when only the overall change in the firms' capacities is restricted, and show NP-hardness when there are additional constraints for individual firms. Lastly, we compare capacity modification with the classical model of preference manipulation by firms and identify scenarios under which one mode of manipulation outperforms the other. We find that a threshold on a given firm's capacity, which we call its peak, crucially determines the effectiveness of different manipulation actions., Comment: Appears in the Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems 2024 (AAMAS 2024) (https://dl.acm.org/doi/10.5555/3635637.3662922)

Evacuation planning is an essential part of disaster management where the goal is to relocate people under imminent danger to safety. However, finding jointly optimal evacuation routes and schedule that minimizes the average evacuation time or evacuation completion time, is a computationally hard problem. As a result, large-scale evacuation routing and scheduling continues to be a challenge. In this paper, we present a game-theoretic approach to tackle this problem. We start by formulating a strategic routing and scheduling game, named the Evacuation Game: Routing and Scheduling (EGRES), where players choose their route and time of departure. We show that: (i) every instance of EGRES has at least one pure strategy Nash equilibrium, and (ii) an optimal outcome in an instance will always be an equilibrium in that instance. We then provide bounds on how bad an equilibrium can be compared to an optimal outcome. Additionally, we present a polynomial-time algorithm, the Sequential Action Algorithm (SAA), for finding equilibria in a given instance under a special condition. We use Virginia Beach City in Virginia, and Harris County in Houston, Texas as study areas and construct two EGRES instances. Our results show that, by utilizing SAA, we can efficiently find equilibria in these instances that have social objective close to the optimal value.

In cross-domain retrieval, a model is required to identify images from the same semantic category across two visual domains. For instance, given a sketch of an object, a model needs to retrieve a real image of it from an online store's catalog. A standard approach for such a problem is learning a feature space of images where Euclidean distances reflect similarity. Even without human annotations, which may be expensive to acquire, prior methods function reasonably well using unlabeled images for training. Our problem constraint takes this further to scenarios where the two domains do not necessarily share any common categories in training data. This can occur when the two domains in question come from different versions of some biometric sensor recording identities of different people. We posit a simple solution, which is to generate synthetic data to fill in these missing category examples across domains. This, we do via category preserving translation of images from one visual domain to another. We compare approaches specifically trained for this translation for a pair of domains, as well as those that can use large-scale pre-trained text-to-image diffusion models via prompts, and find that the latter can generate better replacement synthetic data, leading to more accurate cross-domain retrieval models. Our best SynCDR model can outperform prior art by up to 15\%. Code for our work is available at https://github.com/samarth4149/SynCDR ., Comment: Pre-print

Trapped ion systems are a leading platform for quantum information processing, but they are currently limited to 1D and 2D arrays, which imposes restrictions on both their scalability and their range of applications. Here, we propose a path to overcome this limitation by demonstrating that Penning traps can be used to realize remarkably clean bilayer crystals, wherein hundreds of ions self-organize into two well-defined layers. These bilayer crystals are made possible by the inclusion of an anharmonic trapping potential, which is readily implementable with current technology. We study the normal modes of this system and discover salient differences compared to the modes of single-plane crystals. The bilayer geometry and the unique properties of the normal modes open new opportunities, in particular in quantum sensing and quantum simulation, that are not straightforward in single-plane crystals. Furthermore, we illustrate that it may be possible to extend the ideas presented here to realize multilayer crystals with more than two layers. Our work increases the dimensionality of trapped ion systems by efficiently utilizing all three spatial dimensions and lays the foundation for a new generation of quantum information processing experiments with multilayer 3D crystals of trapped ions., Comment: 21+10 pages, 12 figures, 3 supplementary videos; Updated to version submitted to journal

When two different electronic materials are brought together, the resultant interface often shows unexpected quantum phenomena, including interfacial superconductivity and Fu-Kane topological superconductivity (TSC). Here, we use molecular beam epitaxy (MBE) to synthesize heterostructures formed by stacking together two magnetic materials, a ferromagnetic topological insulator (TI) and an antiferromagnetic iron chalcogenide (FeTe). We discover emergent interface-induced superconductivity in these heterostructures and demonstrate the trifecta occurrence of superconductivity, ferromagnetism, and topological band structure in the magnetic TI layer, the three essential ingredients of chiral TSC. The unusual coexistence of ferromagnetism and superconductivity can be attributed to the high upper critical magnetic field that exceeds the Pauli paramagnetic limit for conventional superconductors at low temperatures. The magnetic TI/FeTe heterostructures with robust superconductivity and atomically sharp interfaces provide an ideal wafer-scale platform for the exploration of chiral TSC and Majorana physics, constituting an important step toward scalable topological quantum computation., Comment: 14 pages, 4 figures. Accepted by Science. Comments are welcome

Plastic self-adaptation, nonlinear recurrent dynamics and multi-scale memory are desired features in hardware implementations of neural networks, because they enable them to learn, adapt and process information similarly to the way biological brains do. In this work, we experimentally demonstrate these properties occurring in arrays of photonic neurons. Importantly, this is realised autonomously in an emergent fashion, without the need for an external controller setting weights and without explicit feedback of a global reward signal. Using a hierarchy of such arrays coupled to a backpropagation-free training algorithm based on simple logistic regression, we are able to achieve a performance of 98.2% on the MNIST task, a popular benchmark task looking at classification of written digits. The plastic nodes consist of silicon photonics microring resonators covered by a patch of phase-change material that implements nonvolatile memory. The system is compact, robust, and straightforward to scale up through the use of multiple wavelengths. Moreover, it constitutes a unique platform to test and efficiently implement biologically plausible learning schemes at a high processing speed.

Bulk-boundary correspondence is a foundational principle underlying the electronic band structure and physical behavior of topological quantum materials. Although it has been rigorously tested in topological systems where the physical properties involve charge currents, it remains unclear whether bulk-boundary correspondence should also hold for non-conserved spin currents. We study charge-to-spin conversion in a canonical topological insulator, Bi$_{1-x}$Sb$_x$, to address this fundamentally unresolved question. We use spin-torque ferromagnetic resonance measurements to accurately probe the charge-to-spin conversion efficiency in epitaxial Bi$_{1-x}$Sb$_x$~thin films of high structural quality spanning the entire range of composition, including both trivial and topological band structures, as verified using {\it in vacuo} angle-resolved photoemission spectroscopy. From these measurements, we deduce the effective spin Hall conductivity (SHC) and find excellent agreement with the values predicted by tight-binding calculations for the intrinsic SHC of the bulk bands. These results provide strong evidence that the strong spin-orbit entanglement of bulk states well below the Fermi energy connects directly to the SHC in epitaxial Bi$_{1-x}$Sb$_x$~films interfaced with a metallic ferromagnet. The excellent agreement between theory and experiment points to the generic value of analyses focused entirely on bulk properties, even for topological systems involving non-conserved spin currents.

We report the molecular beam epitaxy of Bi_1-xSb_x thin films ($0 \leq x \leq 1$) on (0001) sapphire substrates using a thin (Bi,Sb)$_2$Te$_3$ buffer layer. Characterization of the films using reflection high energy diffraction, x-ray diffraction, atomic force microscopy, and scanning transmission electron microscopy reveals epitaxial growth of films of reasonable structural quality. This is further confirmed via x-ray diffraction pole figures that determine the epitaxial registry between the thin film and substrate. We further investigate the microscopic structure of thin films via Raman spectroscopy, demonstrating how the vibrational modes vary as the composition changes and discussing the implications for the crystal structure. We also characterize the samples using electrical transport measurements.

Pre-training on massive video datasets has become essential to achieve high action recognition performance on smaller downstream datasets. However, most large-scale video datasets contain images of people and hence are accompanied with issues related to privacy, ethics, and data protection, often preventing them from being publicly shared for reproducible research. Existing work has attempted to alleviate these problems by blurring faces, downsampling videos, or training on synthetic data. On the other hand, analysis on the transferability of privacy-preserving pre-trained models to downstream tasks has been limited. In this work, we study this problem by first asking the question: can we pre-train models for human action recognition with data that does not include real humans? To this end, we present, for the first time, a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Furthermore, we propose a novel pre-training strategy, called Privacy-Preserving MAE-Align, to effectively combine synthetic data and human-removed real data. Our approach outperforms previous baselines by up to 5% and closes the performance gap between human and no-human action recognition representations on downstream tasks, for both linear probing and fine-tuning. Our benchmark, code, and models are available at https://github.com/howardzh01/PPMA ., Comment: 19 pages, 7 figures, 2023 NeurIPS Datasets and Benchmarks Track

We use magnetoresistance measurements at high magnetic field (B \leq 65 T) and low temperature (T \geq 500 mK) to gain fresh insights into the behavior of the upper critical field, Hc2, in superconducting ultrathin FeSe films of varying degrees of disorder, grown by molecular beam epitaxy on SrTiO3. Measurements of Hc2 across samples with a widely varying superconducting critical temperature (1.2 K \leq Tc \leq 21 K) generically show similar qualitative temperature dependence. We analyze the temperature dependence of Hc2 in the context of Werthamer-Helfand-Hohenberg (WHH) theory. The analysis yields parameters that indicate a strong Pauli paramagnetic pair-breaking mechanism which is also reflected by pseudo-isotropic superconductivity in the limit of zero temperature. In the lower Tc samples, we observe a spin-orbit scattering driven enhancement of Hc2 above the strongly-coupled Pauli paramagnetic limit. We also observe clear deviations from WHH theory at low temperature, regardless of Tc. We attribute this to the multi-band superconductivity of FeSe and possibly to the emergence of a low temperature, high field superconducting phase.

Over the last decade, the possibility of realizing topological superconductivity (TSC) has generated much excitement, mainly due to the potential use of its excitations (Majorana zero modes) in a fault-tolerant topological quantum computer 1,2. TSC can be created in electronic systems where the topological and superconducting orders coexist3, motivating the continued exploration of candidate material platforms to this end. Here, we use molecular beam epitaxy (MBE) to synthesize heterostructures that host emergent interfacial superconductivity when a non-superconducting antiferromagnet (FeTe) is interfaced with a topological insulator (TI) (Bi, Sb)2Te3 wherein the chemical potential can be tuned through varying the Bi/Sb ratio. By performing in-vacuo angle-resolved photoemission spectroscopy (ARPES) and ex-situ electrical transport measurements, we find that the superconducting transition temperature and the upper critical magnetic field are suppressed when the chemical potential approaches the Dirac point. This observation implies a direct correlation between the interfacial superconductivity and Dirac electrons of the TI layer. We provide evidence to show that the observed interfacial superconductivity and its chemical potential dependence is the result of the competition between the Ruderman-Kittel-Kasuya-Yosida-type ferromagnetic coupling mediated by Dirac surface states and antiferromagnetic exchange couplings that generate the bicollinear antiferromagnetic order in the FeTe layer. The Dirac-fermion-assisted interfacial superconductivity in (Bi,Sb)2Te3/FeTe heterostructures provides a new approach to probe TSC and Majorana physics in hybrid devices and potentially constitutes an alternative platform for topological quantum computation., Comment: 32 pages and 4 figures. Accepted by Nature Communications