Search

Autonomous edge computing in robotics, smart cities, and autonomous vehicles relies on the seamless integration of sensing, processing, and actuation for real-time decision-making in dynamic environments. At its core is the sensing-to-action loop, which iteratively aligns sensor inputs with computational models to drive adaptive control strategies. These loops can adapt to hyper-local conditions, enhancing resource efficiency and responsiveness, but also face challenges such as resource constraints, synchronization delays in multi-modal data fusion, and the risk of cascading errors in feedback loops. This article explores how proactive, context-aware sensing-to-action and action-to-sensing adaptations can enhance efficiency by dynamically adjusting sensing and computation based on task demands, such as sensing a very limited part of the environment and predicting the rest. By guiding sensing through control actions, action-to-sensing pathways can improve task relevance and resource use, but they also require robust monitoring to prevent cascading errors and maintain reliability. Multi-agent sensing-action loops further extend these capabilities through coordinated sensing and actions across distributed agents, optimizing resource use via collaboration. Additionally, neuromorphic computing, inspired by biological systems, provides an efficient framework for spike-based, event-driven processing that conserves energy, reduces latency, and supports hierarchical control--making it ideal for multi-agent optimization. This article highlights the importance of end-to-end co-design strategies that align algorithmic models with hardware and environmental dynamics and improve cross-layer interdependencies to improve throughput, precision, and adaptability for energy-efficient edge autonomy in complex environments.

Object detection and tracking is an essential perception task for enabling fully autonomous navigation in robotic systems. Edge robot systems such as small drones need to execute complex maneuvers at high-speeds with limited resources, which places strict constraints on the underlying algorithms and hardware. Traditionally, frame-based cameras are used for vision-based perception due to their rich spatial information and simplified synchronous sensing capabilities. However, obtaining detailed information across frames incurs high energy consumption and may not even be required. In addition, their low temporal resolution renders them ineffective in high-speed motion scenarios. Event-based cameras offer a biologically-inspired solution to this by capturing only changes in intensity levels at exceptionally high temporal resolution and low power consumption, making them ideal for high-speed motion scenarios. However, their asynchronous and sparse outputs are not natively suitable with conventional deep learning methods. In this work, we propose TOFFE, a lightweight hybrid framework for performing event-based object motion estimation (including pose, direction, and speed estimation), referred to as Object Flow. TOFFE integrates bio-inspired Spiking Neural Networks (SNNs) and conventional Analog Neural Networks (ANNs), to efficiently process events at high temporal resolutions while being simple to train. Additionally, we present a novel event-based synthetic dataset involving high-speed object motion to train TOFFE. Our experimental results show that TOFFE achieves 5.7x/8.3x reduction in energy consumption and 4.6x/5.8x reduction in latency on edge GPU(Jetson TX2)/hybrid hardware(Loihi-2 and Jetson TX2), compared to previous event-based object detection baselines., Comment: 8 pages, 6 figures, 4 tables

The log canonical threshold (lct) is a fundamental invariant in birational geometry, essential for understanding the complexity of singularities in algebraic varieties. Its real counterpart, the real log canonical threshold (rlct), also known as the learning coefficient, has become increasingly relevant in statistics and machine learning, where it plays a critical role in model selection and error estimation for singular statistical models. In this paper, we investigate the rlct and its multiplicity for real (not necessarily reduced) hyperplane arrangements. We derive explicit combinatorial formulas for these invariants, generalizing earlier results that were limited to specific examples. Moreover, we provide a general algebraic theory for real log canonical thresholds, and present a SageMath implementation for efficiently computing the rlct and its multiplicity in the case or real hyperplane arrangements. Applications to examples are given, illustrating how the formulas also can be used to analyze the asymptotic behavior of high-dimensional volume integrals.

In this study, we investigate and compare formulations for computing shape derivatives in bi-material level-set optimization with precise modeling of the interface. The level-set function is parameterized using B-splines, whose coordinates serve as design variables. A precise mechanical model is obtained every design cycle, replicating the exact geometry of the bi-material design, using untrimming techniques and IGA on unstructured meshes. Design sensitivities are formulated by either a "discretize-then-differentiate" or a "differentiate-then-discretize" approach. A detailed comparative study shows the limitations of the latter in terms of accuracy, specifically when stresses near the material interface dominate the stress field. The precise representation of the interface facilitates an accurate evaluation of interfacial stresses, and the consistent discretized sensitivities enable to minimize them directly -- highlighting the main advantage of our framework. Furthermore, reducing the discretized approach by considering only interface control points and a selective set of adjacent control points provides an ideal trade-off between accuracy and numerical efficiency. This lays the foundations for multi-material shape and topology optimization procedures, considering accurate responses on the interface., Comment: 22 pages

Multiview systems have become a key technology in modern computer vision, offering advanced capabilities in scene understanding and analysis. However, these systems face critical challenges in bandwidth limitations and computational constraints, particularly for resource-limited camera nodes like drones. This paper presents a novel approach for communication-efficient distributed multiview detection and tracking using masked autoencoders (MAEs). We introduce a semantic-guided masking strategy that leverages pre-trained segmentation models and a tunable power function to prioritize informative image regions. This approach, combined with an MAE, reduces communication overhead while preserving essential visual information. We evaluate our method on both virtual and real-world multiview datasets, demonstrating comparable performance in terms of detection and tracking performance metrics compared to state-of-the-art techniques, even at high masking ratios. Our selective masking algorithm outperforms random masking, maintaining higher accuracy and precision as the masking ratio increases. Furthermore, our approach achieves a significant reduction in transmission data volume compared to baseline methods, thereby balancing multiview tracking performance with communication efficiency., Comment: 10 pages, conference

The ability of neural networks to perform robotic perception and control tasks such as depth and optical flow estimation, simultaneous localization and mapping (SLAM), and automatic control has led to their widespread adoption in recent years. Deep Reinforcement Learning has been used extensively in these settings, as it does not have the unsustainable training costs associated with supervised learning. However, DeepRL suffers from poor sample efficiency, i.e., it requires a large number of environmental interactions to converge to an acceptable solution. Modern RL algorithms such as Deep Q Learning and Soft Actor-Critic attempt to remedy this shortcoming but can not provide the explainability required in applications such as autonomous robotics. Humans intuitively understand the long-time-horizon sequential tasks common in robotics. Properly using such intuition can make RL policies more explainable while enhancing their sample efficiency. In this work, we propose SHIRE, a novel framework for encoding human intuition using Probabilistic Graphical Models (PGMs) and using it in the Deep RL training pipeline to enhance sample efficiency. Our framework achieves 25-78% sample efficiency gains across the environments we evaluate at negligible overhead cost. Additionally, by teaching RL agents the encoded elementary behavior, SHIRE enhances policy explainability. A real-world demonstration further highlights the efficacy of policies trained using our framework.

The distance reducing property for Markov bases is an important property that provides a bound on the mixing time of the associated Markov chain. The goal of this project is to understand properties of distance-reducing Markov bases. We explore the distance reducing property for monomial curves and give a complete characterisation of distance reduction in the case of complete intersection monomial curves. Our characterisation carefully uses the notion of gluings for numerical semigroups. We also characterise the distance reducing property for non-complete intersection monomial curves in small dimensions. We also explore the distance irreducible elements: the moves that appear in all distance reducing Markov bases., Comment: 42 pages, 4 figures, 1 table

Knowledge graphs are useful tools to organize, recommend and sort data. Hierarchies in knowledge graphs provide significant benefit in improving understanding and compartmentalization of the data within a knowledge graph. This work leverages large language models to generate and augment hierarchies in an existing knowledge graph. For small (<100,000 node) domain-specific KGs, we find that a combination of few-shot prompting with one-shot generation works well, while larger KG may require cyclical generation. We present techniques for augmenting hierarchies, which led to coverage increase by 98% for intents and 99% for colors in our knowledge graph., Comment: European Conference on Information Retrieval 2024

End-to-end generation of musical audio using deep learning techniques has seen an explosion of activity recently. However, most models concentrate on generating fully mixed music in response to abstract conditioning information. In this work, we present an alternative paradigm for producing music generation models that can listen and respond to musical context. We describe how such a model can be constructed using a non-autoregressive, transformer-based model architecture and present a number of novel architectural and sampling improvements. We train the described architecture on both an open-source and a proprietary dataset. We evaluate the produced models using standard quality metrics and a new approach based on music information retrieval descriptors. The resulting model reaches the audio quality of state-of-the-art text-conditioned models, as well as exhibiting strong musical coherence with its context., Comment: Accepted for publication at ICASSP 2024

Utilizing Low Earth Orbit (LEO) satellite networks equipped with Inter-Satellite Links (ISL) is envisioned to provide lower delay compared to traditional optical networks. However, LEO satellites have constrained energy resources as they rely on solar energy in their operations. Thus requiring special consideration when designing network topologies that do not only have low-delay link paths but also low-power consumption. In this paper, we study different satellite constellation types and network typologies and propose a novel power-efficient topology. As such, we compare three common satellite architectures, namely; (i) the theoretical random constellation, the widely deployed (ii) Walker-Delta, and (iii) Walker-Star constellations. The comparison is performed based on both the power efficiency and end-to-end delay. The results show that the proposed algorithm outperforms long-haul ISL paths in terms of energy efficiency with only a slight hit to delay performance relative to the conventional ISL topology.

In the field of robotics, event-based cameras are emerging as a promising low-power alternative to traditional frame-based cameras for capturing high-speed motion and high dynamic range scenes. This is due to their sparse and asynchronous event outputs. Spiking Neural Networks (SNNs) with their asynchronous event-driven compute, show great potential for extracting the spatio-temporal features from these event streams. In contrast, the standard Analog Neural Networks (ANNs) fail to process event data effectively. However, training SNNs is difficult due to additional trainable parameters (thresholds and leaks), vanishing spikes at deeper layers, and a non-differentiable binary activation function. Furthermore, an additional data structure, membrane potential, responsible for keeping track of temporal information, must be fetched and updated at every timestep in SNNs. To overcome these challenges, we propose a novel SNN-ANN hybrid architecture that combines the strengths of both. Specifically, we leverage the asynchronous compute capabilities of SNN layers to effectively extract the input temporal information. Concurrently, the ANN layers facilitate training and efficient hardware deployment on traditional machine learning hardware such as GPUs. We provide extensive experimental analysis for assigning each layer to be spiking or analog, leading to a network configuration optimized for performance and ease of training. We evaluate our hybrid architecture for optical flow estimation on DSEC-flow and Multi-Vehicle Stereo Event-Camera (MVSEC) datasets. On the DSEC-flow dataset, the hybrid SNN-ANN architecture achieves a 40% reduction in average endpoint error (AEE) with 22% lower energy consumption compared to Full-SNN, and 48% lower AEE compared to Full-ANN, while maintaining comparable energy usage.

The ability of resource-constrained biological systems such as fruitflies to perform complex and high-speed maneuvers in cluttered environments has been one of the prime sources of inspiration for developing vision-based autonomous systems. To emulate this capability, the perception pipeline of such systems must integrate information cues from tasks including optical flow and depth estimation, object detection and tracking, and segmentation, among others. However, the conventional approach of employing slow, synchronous inputs from standard frame-based cameras constrains these perception capabilities, particularly during high-speed maneuvers. Recently, event-based sensors have emerged as low latency and low energy alternatives to standard frame-based cameras for capturing high-speed motion, effectively speeding up perception and hence navigation. For coherence, all the perception tasks must be trained on the same input data. However, present-day datasets are curated mainly for a single or a handful of tasks and are limited in the rate of the provided ground truths. To address these limitations, we present Flying Event Dataset fOr Reactive behAviour (FEDORA) - a fully synthetic dataset for perception tasks, with raw data from frame-based cameras, event-based cameras, and Inertial Measurement Units (IMU), along with ground truths for depth, pose, and optical flow at a rate much higher than existing datasets.

To every simple toric ideal $I_T$ one can associate the strongly robust simplicial complex $\Delta _T$, which determines the strongly robust property for all ideals that have $I_T$ as their bouquet ideal. We show that for the simple toric ideals of monomial curves in $\mathbb{A}^{s}$, the strongly robust simplicial complex $\Delta _T$ is either $\{\emptyset \}$ or contains exactly one 0-dimensional face. In the case of monomial curves in $\mathbb{A}^{3}$, the strongly robust simplicial complex $\Delta _T$ contains one 0-dimensional face if and only if the toric ideal $I_T$ is a complete intersection ideal with exactly two Betti degrees. Finally, we provide a construction to produce infinitely many strongly robust ideals with bouquet ideal the ideal of a monomial curve and show that they are all produced this way., Comment: 17 pages

With the rapid growth of IoT networks, ubiquitous coverage is becoming increasingly necessary. Low Earth Orbit (LEO) satellite constellations for IoT have been proposed to provide coverage to regions where terrestrial systems cannot. However, LEO constellations for uplink communications are severely limited by the high density of user devices, which causes a high level of co-channel interference. This research presents a novel framework that utilizes spiking neural networks (SNNs) to detect IoT signals in the presence of uplink interference. The key advantage of SNNs is the extremely low power consumption relative to traditional deep learning (DL) networks. The performance of the spiking-based neural network detectors is compared against state-of-the-art DL networks and the conventional matched filter detector. Results indicate that both DL and SNN-based receivers surpass the matched filter detector in interference-heavy scenarios, owing to their capacity to effectively distinguish target signals amidst co-channel interference. Moreover, our work highlights the ultra-low power consumption of SNNs compared to other DL methods for signal detection. The strong detection performance and low power consumption of SNNs make them particularly suitable for onboard signal detection in IoT LEO satellites, especially in high interference conditions.

Creating a pop song melody according to pre-written lyrics is a typical practice for composers. A computational model of how lyrics are set as melodies is important for automatic composition systems, but an end-to-end lyric-to-melody model would require enormous amounts of paired training data. To mitigate the data constraints, we adopt a two-stage approach, dividing the task into lyric-to-rhythm and rhythm-to-melody modules. However, the lyric-to-rhythm task is still challenging due to its multimodality. In this paper, we propose a novel lyric-to-rhythm framework that includes part-of-speech tags to achieve better text setting, and a Transformer architecture designed to model long-term syllable-to-note associations. For the rhythm-to-melody task, we adapt a proven chord-conditioned melody Transformer, which has achieved state-of-the-art results. Experiments for Chinese lyric-to-melody generation show that the proposed framework is able to model key characteristics of rhythm and pitch distributions in the dataset, and in a subjective evaluation, the melodies generated by our system were rated as similar to or better than those of a state-of-the-art alternative., Comment: Published in ISMIR 2022

Conventional virtual memory (VM) frameworks enable a virtual address to flexibly map to any physical address. This flexibility necessitates large data structures to store virtual-to-physical mappings, which leads to high address translation latency and large translation-induced interference in the memory hierarchy. On the other hand, restricting the address mapping so that a virtual address can only map to a specific set of physical addresses can significantly reduce address translation overheads by using compact and efficient translation structures. However, restricting the address mapping flexibility across the entire main memory severely limits data sharing across different processes and increases data accesses to the swap space of the storage device, even in the presence of free memory. We propose Utopia, a new hybrid virtual-to-physical address mapping scheme that allows both flexible and restrictive hash-based address mapping schemes to harmoniously co-exist in the system. The key idea of Utopia is to manage physical memory using two types of physical memory segments: restrictive and flexible segments. A restrictive segment uses a restrictive, hash-based address mapping scheme that maps virtual addresses to only a specific set of physical addresses and enables faster address translation using compact translation structures. A flexible segment employs the conventional fully-flexible address mapping scheme. By mapping data to a restrictive segment, Utopia enables faster address translation with lower translation-induced interference. Utopia improves performance by 24% in a single-core system over the baseline system, whereas the best prior state-of-the-art contiguity-aware translation scheme improves performance by 13%., Comment: To appear in 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2023

Event cameras detect changes in per-pixel intensity to generate asynchronous `event streams'. They offer great potential for accurate semantic map retrieval in real-time autonomous systems owing to their much higher temporal resolution and high dynamic range (HDR) compared to conventional cameras. However, existing implementations for event-based segmentation suffer from sub-optimal performance since these temporally dense events only measure the varying component of a visual signal, limiting their ability to encode dense spatial context compared to frames. To address this issue, we propose a hybrid end-to-end learning framework HALSIE, utilizing three key concepts to reduce inference cost by up to $20\times$ versus prior art while retaining similar performance: First, a simple and efficient cross-domain learning scheme to extract complementary spatio-temporal embeddings from both frames and events. Second, a specially designed dual-encoder scheme with Spiking Neural Network (SNN) and Artificial Neural Network (ANN) branches to minimize latency while retaining cross-domain feature aggregation. Third, a multi-scale cue mixer to model rich representations of the fused embeddings. These qualities of HALSIE allow for a very lightweight architecture achieving state-of-the-art segmentation performance on DDD-17, MVSEC, and DSEC-Semantic datasets with up to $33\times$ higher parameter efficiency and favorable inference cost (17.9mJ per cycle). Our ablation study also brings new insights into effective design choices that can prove beneficial for research across other vision tasks., Comment: Accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

Classification of cancer cellularity within tissue samples is currently a manual process performed by pathologists. This process of correctly determining cancer cellularity can be time intensive. Deep Learning (DL) techniques in particular have become increasingly more popular for this purpose, due to the accuracy and performance they exhibit, which can be comparable to the pathologists. This work investigates the capabilities of two DL approaches to assess cancer cellularity in whole slide images (WSI) in the SPIE-AAPM-NCI BreastPathQ challenge dataset. The effects of training on augmented data via rotations, and combinations of multiple architectures into a single network were analyzed using a modified Kendall Tau-b prediction probability metric known as the average prediction probability PK. A deep, transfer learned, Convolutional Neural Network (CNN) InceptionV3 was used as a baseline, achieving an average PK value of 0.884, showing improvement from the average PK value of 0.83 achieved by pathologists. The network was then trained on additional training datasets which were rotated between 1 and 360 degrees, which saw a peak increase of PK up to 4.2%. An additional architecture consisting of the InceptionV3 network and VGG16, a shallow, transfer learned CNN, was combined in a parallel architecture. This parallel architecture achieved a baseline average PK value of 0.907, a statistically significantly improvement over either of the architectures' performances separately (p<0.0001 by unpaired t-test)., Comment: 7 pages (includes a cover page), 5 figures, 1 table, work addresses the BreastPathQ challenge

Spiking Neural Networks (SNNs) are bio-plausible models that hold great potential for realizing energy-efficient implementations of sequential tasks on resource-constrained edge devices. However, commercial edge platforms based on standard GPUs are not optimized to deploy SNNs, resulting in high energy and latency. While analog In-Memory Computing (IMC) platforms can serve as energy-efficient inference engines, they are accursed by the immense energy, latency, and area requirements of high-precision ADCs (HP-ADC), overshadowing the benefits of in-memory computations. We propose a hardware/software co-design methodology to deploy SNNs into an ADC-Less IMC architecture using sense-amplifiers as 1-bit ADCs replacing conventional HP-ADCs and alleviating the above issues. Our proposed framework incurs minimal accuracy degradation by performing hardware-aware training and is able to scale beyond simple image classification tasks to more complex sequential regression tasks. Experiments on complex tasks of optical flow estimation and gesture recognition show that progressively increasing the hardware awareness during SNN training allows the model to adapt and learn the errors due to the non-idealities associated with ADC-Less IMC. Also, the proposed ADC-Less IMC offers significant energy and latency improvements, $2-7\times$ and $8.9-24.6\times$, respectively, depending on the SNN model and the workload, compared to HP-ADC IMC., Comment: 13 pages, 14 figures