Search

Unsupervised Domain Adaptation (UDA) is the task of bridging the domain gap between a labeled source domain, e.g., synthetic data, and an unlabeled target domain. We observe that current UDA methods show inferior results on fine structures and tend to oversegment objects with ambiguous appearance. To address these shortcomings, we propose to leverage geometric information, i.e., depth predictions, as depth discontinuities often coincide with segmentation boundaries. We show that naively incorporating depth into current UDA methods does not fully exploit the potential of this complementary information. To this end, we present MICDrop, which learns a joint feature representation by masking image encoder features while inversely masking depth encoder features. With this simple yet effective complementary masking strategy, we enforce the use of both modalities when learning the joint feature representation. To aid this process, we propose a feature fusion module to improve both global as well as local information sharing while being robust to errors in the depth predictions. We show that our method can be plugged into various recent UDA methods and consistently improve results across standard UDA benchmarks, obtaining new state-of-the-art performances.

Neuromorphic computing offers a transformative pathway to overcome the computational and energy challenges faced in deploying robotic localization and navigation systems at the edge. Visual place recognition, a critical component for navigation, is often hampered by the high resource demands of conventional systems, making them unsuitable for small-scale robotic platforms which still require to perform complex, long-range tasks. Although neuromorphic approaches offer potential for greater efficiency, real-time edge deployment remains constrained by the complexity and limited scalability of bio-realistic networks. Here, we demonstrate a neuromorphic localization system that performs accurate place recognition in up to 8km of traversal using models as small as 180 KB with 44k parameters, while consuming less than 1% of the energy required by conventional methods. Our Locational Encoding with Neuromorphic Systems (LENS) integrates spiking neural networks, an event-based dynamic vision sensor, and a neuromorphic processor within a single SPECK(TM) chip, enabling real-time, energy-efficient localization on a hexapod robot. LENS represents the first fully neuromorphic localization system capable of large-scale, on-device deployment, setting a new benchmark for energy efficient robotic place recognition., Comment: 28 pages, 4 main figures, 4 supplementary figures, 1 supplementary table, and 1 movie. Under review

We introduce RMMI, a novel reactive control framework for mobile manipulators operating in complex, static environments. Our approach leverages a neural Signed Distance Field (SDF) to model intricate environment details and incorporates this representation as inequality constraints within a Quadratic Program (QP) to coordinate robot joint and base motion. A key contribution is the introduction of an active collision avoidance cost term that maximises the total robot distance to obstacles during the motion. We first evaluate our approach in a simulated reaching task, outperforming previous methods that rely on representing both the robot and the scene as a set of primitive geometries. Compared with the baseline, we improved the task success rate by 25% in total, which includes increases of 10% by using the active collision cost. We also demonstrate our approach on a real-world platform, showing its effectiveness in reaching target poses in cluttered and confined spaces using environment models built directly from sensor data. For additional details and experiment videos, visit https://rmmi.github.io/., Comment: 8 pages, 6 figures, Paper under review

Hierarchical methods represent state-of-the-art visual localization, optimizing search efficiency by using global descriptors to focus on relevant map regions. However, this state-of-the-art performance comes at the cost of substantial memory requirements, as all database images must be stored for feature matching. In contrast, direct 2D-3D matching algorithms require significantly less memory but suffer from lower accuracy due to the larger and more ambiguous search space. We address this ambiguity by fusing local and global descriptors using a weighted average operator within a 2D-3D search framework. This fusion rearranges the local descriptor space such that geographically nearby local descriptors are closer in the feature space according to the global descriptors. Therefore, the number of irrelevant competing descriptors decreases, specifically if they are geographically distant, thereby increasing the likelihood of correctly matching a query descriptor. We consistently improve the accuracy over local-only systems and achieve performance close to hierarchical methods while halving memory requirements. Extensive experiments using various state-of-the-art local and global descriptors across four different datasets demonstrate the effectiveness of our approach. For the first time, our approach enables direct matching algorithms to benefit from global descriptors while maintaining memory efficiency. The code for this paper will be published at \href{https://github.com/sontung/descriptor-disambiguation}{github.com/sontung/descriptor-disambiguation}.

Complementary to high-energy experimental efforts, indirect astrophysical searches of particles beyond the standard model have long been pursued. The present article follows the latter approach and considers, for the first time, the self-consistent treatment of the energy losses from dark flavored particles produced in the decay of hyperons during a core-collapse supernova (CCSN). To this end, general relativistic supernova simulations in spherical symmetry are performed, featuring six-species Boltzmann neutrino transport, and covering the long-term evolution of the nascent remnant proto-neutron star (PNS) deleptonization for several tens of seconds. A well-calibrated hyperon equation of state (EOS) is therefore implemented into the supernova simulations and tested against the corresponding nucleonic model. It is found that supernova observables, such as the neutrino signal, are robustly insensitive to the appearance of hyperons for the simulation times considered in the present study. The presence of hyperons enables an additional channel for the appearance of dark sector particles, which is considered at the level of the $\Lambda$ hyperon decay. Assuming massless particles that escape the PNS after being produced, these channels expedite the deleptonizing PNS and the cooling behaviour. This, in turn, shortens the neutrino emission timescale. The present study confirms the previously estimated upper limits on the corresponding branching ratios for low and high mass PNS, by effectively reducing the neutrino emission timescale by a factor of two. This is consistent with the classical argument deduced from the neutrino detection associated with SN1987A., Comment: 24 pages, 7 figures, submitted to JCAP

We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200 times in rendering speed., Comment: Project page is available at https://tobiasfshr.github.io/pub/4dgf/

Identifying and controlling an unstable, underactuated robot to enable reference tracking is a challenging control problem. In this paper, a ballbot (robot balancing on a ball) is used as an experimental setup to demonstrate and test proposed strategies to tackle this control problem. A double-loop control system, including a state-feedback gain in the outer-loop and a Proportional-Integral-Derivative (PID) controller in the inner-loop, is presented to balance the system in its unstable equilibrium. Once stability is reached, the plant's response to a designed excitation signal is measured and interpreted to identify the system's dynamics. Hereby, the parameters of a linearized model of the ballbot are identified with prior knowledge about the structure of the nonlinear dynamics of the system. Based on an identified linear time-invariant (LTI) state-space model, a double-loop control strategy is considered to balance the real system and to allow reference tracking. A linear quadratic regulator (LQR) is designed offline and implemented in the inner-loop to ensure balance. In the outer-loop, the estimated dynamics forecast the system's behavior online using a model-predictive-control (MPC) design to find the optimal control input for reference tracking. The experimental results demonstrate the applicability of the proposed strategies., Comment: 6 pages, 10 figures

Broad-scale marine surveys performed by underwater vehicles significantly increase the availability of coral reef imagery, however it is costly and time-consuming for domain experts to label images. Point label propagation is an approach used to leverage existing image data labeled with sparse point labels. The resulting augmented ground truth generated is then used to train a semantic segmentation model. Here, we first demonstrate that recent advances in foundation models enable generation of multi-species coral augmented ground truth masks using denoised DINOv2 features and K-Nearest Neighbors (KNN), without the need for any pre-training or custom-designed algorithms. For extremely sparsely labeled images, we propose a labeling regime based on human-in-the-loop principles, resulting in significant improvement in annotation efficiency: If only 5 point labels per image are available, our proposed human-in-the-loop approach improves on the state-of-the-art by 17.3% for pixel accuracy and 22.6% for mIoU; and by 10.6% and 19.1% when 10 point labels per image are available. Even if the human-in-the-loop labeling regime is not used, the denoised DINOv2 features with a KNN outperforms the prior state-of-the-art by 3.5% for pixel accuracy and 5.7% for mIoU (5 grid points). We also provide a detailed analysis of how point labeling style and the quantity of points per image affects the point label propagation quality and provide general recommendations on maximizing point label efficiency., Comment: Accepted at the CVPR2024 3rd Workshop on Learning with Limited Labelled Data for Image and Video Understanding (L3D-IVU), 10 pages, 6 figures, an additional 4 pages of supplementary material

Recovering the 3D scene geometry from a single view is a fundamental yet ill-posed problem in computer vision. While classical depth estimation methods infer only a 2.5D scene representation limited to the image plane, recent approaches based on radiance fields reconstruct a full 3D representation. However, these methods still struggle with occluded regions since inferring geometry without visual observation requires (i) semantic knowledge of the surroundings, and (ii) reasoning about spatial context. We propose KYN, a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We introduce a vision-language modulation module to enrich point features with fine-grained semantic information. We aggregate point representations across the scene through a language-guided spatial attention mechanism to yield per-point density predictions aware of the 3D semantic context. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work. Project page: https://ruili3.github.io/kyn., Comment: CVPR 2024. Project page: https://ruili3.github.io/kyn

We estimate the radiance field of large-scale dynamic areas from multiple vehicle captures under varying environmental conditions. Previous works in this domain are either restricted to static environments, do not scale to more than a single short video, or struggle to separately represent dynamic object instances. To this end, we present a novel, decomposable radiance field approach for dynamic urban environments. We propose a multi-level neural scene graph representation that scales to thousands of images from dozens of sequences with hundreds of fast-moving objects. To enable efficient training and rendering of our representation, we develop a fast composite ray sampling and rendering scheme. To test our approach in urban driving scenarios, we introduce a new, novel view synthesis benchmark. We show that our approach outperforms prior art by a significant margin on both established and our proposed benchmark while being faster in training and rendering., Comment: CVPR 2024. Project page is available at https://tobiasfshr.github.io/pub/ml-nsg/

Event cameras are increasingly popular in robotics due to beneficial features such as low latency, energy efficiency, and high dynamic range. Nevertheless, their downstream task performance is greatly influenced by the optimization of bias parameters. These parameters, for instance, regulate the necessary change in light intensity to trigger an event, which in turn depends on factors such as the environment lighting and camera motion. This paper introduces feedback control algorithms that automatically tune the bias parameters through two interacting methods: 1) An immediate, on-the-fly \textit{fast} adaptation of the refractory period, which sets the minimum interval between consecutive events, and 2) if the event rate exceeds the specified bounds even after changing the refractory period repeatedly, the controller adapts the pixel bandwidth and event thresholds, which stabilizes after a short period of noise events across all pixels (\textit{slow} adaptation). Our evaluation focuses on the visual place recognition task, where incoming query images are compared to a given reference database. We conducted comprehensive evaluations of our algorithms' adaptive feedback control in real-time. To do so, we collected the QCR-Fast-and-Slow dataset that contains DAVIS346 event camera streams from 366 repeated traversals of a Scout Mini robot navigating through a 100 meter long indoor lab setting (totaling over 35km distance traveled) in varying brightness conditions with ground truth location information. Our proposed feedback controllers result in superior performance when compared to the standard bias settings and prior feedback control methods. Our findings also detail the impact of bias adjustments on task performance and feature ablation studies on the fast and slow adaptation mechanisms., Comment: 8 pages, 9 figures, paper accepted to the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)

To enable self-driving vehicles accurate detection and tracking of surrounding objects is essential. While Light Detection and Ranging (LiDAR) sensors have set the benchmark for high-performance systems, the appeal of camera-only solutions lies in their cost-effectiveness. Notably, despite the prevalent use of Radio Detection and Ranging (RADAR) sensors in automotive systems, their potential in 3D detection and tracking has been largely disregarded due to data sparsity and measurement noise. As a recent development, the combination of RADARs and cameras is emerging as a promising solution. This paper presents Camera-RADAR 3D Detection and Tracking (CR3DT), a camera-RADAR fusion model for 3D object detection, and Multi-Object Tracking (MOT). Building upon the foundations of the State-of-the-Art (SotA) camera-only BEVDet architecture, CR3DT demonstrates substantial improvements in both detection and tracking capabilities, by incorporating the spatial and velocity information of the RADAR sensor. Experimental results demonstrate an absolute improvement in detection performance of 5.3% in mean Average Precision (mAP) and a 14.9% increase in Average Multi-Object Tracking Accuracy (AMOTA) on the nuScenes dataset when leveraging both modalities. CR3DT bridges the gap between high-performance and cost-effective perception systems in autonomous driving, by capitalizing on the ubiquitous presence of RADAR in automotive applications. The code is available at: https://github.com/ETH-PBL/CR3DT.

Fast flavor conversions (FFCs) of neutrinos, which can occur in core-collapse supernovae (CCSNe), are multiangle effects. They depend on the angular distribution of the neutrino's electron lepton number (ELN). In this work, we present a comprehensive study of the FFCs by solving the multienergy and multiangle quantum kinetic equations with an extended set of collisional weak processes based on a static and spherically symmetric CCSN matter background profile. We investigate the emergence and evolution of FFCs in models featuring different ELN angular distributions, considering scenarios with two and three neutrino flavors. The spectrogram method is utilized to illustrate the small-scale spatial structure, and we show that this structure of neutrino flavor coherence and number densities in the nonlinear regime is qualitatively consistent with the dispersion relation analysis. On the coarse-grained level, we find that different asymptotic states can be achieved following the FFCs depending on the locations and shapes of the ELN distributions, despite sharing a common feature of the elimination of the ELN angular crossing. While equilibration among different neutrino flavors may be achieved immediately after the prompt FFCs, it is not a general outcome of the asymptotic state, as subsequent feedback effects from collisional neutrino-matter interactions come into play, particularly for cases where FFCs occur inside the neutrinosphere. The impacts of FFCs and the feedback effect on the net neutrino heating rates, the equilibrium electron fraction of CCSN matter, and the free-streaming neutrino energy spectra are quantitatively assessed. Other aspects including the impact of the vacuum term and the coexistence with other type of flavor instabilities are also discussed., Comment: 32 pages, 22 figures, published in PRD

Neutron stars (NS) that are born in binary systems with a main-sequence star companion can experience mass transfer, resulting in the accumulation of material at the surface of the NS. This, in turn, leads to the continuous growth of the NS mass and the associated steepening of the gravitational potential. Supposing the central density surpasses the onset for the phase transition from nuclear, generally hadronic matter to deconfined quark-gluon plasma, which is a quantity currently constrained solely from an upper limit by asymptotic freedom in quantum chromodynamics (QCD), the system may experience a dynamic response due to the appearance of additional degrees of freedom in the equation of state (EOS). This dynamical response might give rise to a rapid softening of the EOS during the transition in the hadron-quark matter co-existence region. While this phenomenon has long been studied in the context of hydrostatic configurations, the dynamical implications of this problem are still incompletely understood. It is the purpose of the present paper to simulate the dynamics of NSs with previously accreted envelopes caused by the presence of a first-order QCD phase transition. Therefore, we employed the neutrino radiation hydrodynamics treatment based on the fully general relativistic approach in spherical symmetry, implementing a three-flavor Boltzmann neutrino transport and a microscopic model EOS that contains a first-order hadron-quark phase transition. The associated neutrino signal shows a sudden rise in the neutrino fluxes and average energies, becoming observable for the present generation of neutrino detectors for a galactic event, and a gravitational wave mode analysis revealed the behaviors of the dominant $f$ mode and the first and the second gravity $g$ modes that are excited during the NS evolution across the QCD phase transition., Comment: 16 pages, 14 figures,to be published in A&A Journal

The advent of autonomous vehicle technologies has significantly impacted various sectors, including motorsport, where Formula Student and Formula: Society of Automotive Engineers introduced autonomous racing classes. These offer new challenges to aspiring engineers, including the team at QUT Motorsport, but also raise the entry barrier due to the complexity of high-speed navigation and control. This paper presents an open-source solution using the Robot Operating System 2, specifically its open-source navigation stack, to address these challenges in autonomous Formula Student race cars. We compare off-the-shelf navigation libraries that this stack comprises of against traditional custom-made programs developed by QUT Motorsport to evaluate their applicability in autonomous racing scenarios and integrate them onto an autonomous race car. Our contributions include quantitative and qualitative comparisons of these packages against traditional navigation solutions, aiming to lower the entry barrier for autonomous racing. This paper also serves as a comprehensive tutorial for teams participating in similar racing disciplines and other autonomous mobile robot applications., Comment: 10 pages, 6 figures

In robotics, Spiking Neural Networks (SNNs) are increasingly recognized for their largely-unrealized potential energy efficiency and low latency particularly when implemented on neuromorphic hardware. Our paper highlights three advancements for SNNs in Visual Place Recognition (VPR). Firstly, we propose Modular SNNs, where each SNN represents a set of non-overlapping geographically distinct places, enabling scalable networks for large environments. Secondly, we present Ensembles of Modular SNNs, where multiple networks represent the same place, significantly enhancing accuracy compared to single-network models. Each of our Modular SNN modules is compact, comprising only 1500 neurons and 474k synapses, making them ideally suited for ensembling due to their small size. Lastly, we investigate the role of sequence matching in SNN-based VPR, a technique where consecutive images are used to refine place recognition. We analyze the responsiveness of SNNs to ensembling and sequence matching compared to other VPR techniques. Our contributions highlight the viability of SNNs for VPR, offering scalable and robust solutions, and paving the way for their application in various energy-sensitive robotic tasks., Comment: 20 pages, 10 figures, under review

We propose FocusTune, a focus-guided sampling technique to improve the performance of visual localization algorithms. FocusTune directs a scene coordinate regression model towards regions critical for 3D point triangulation by exploiting key geometric constraints. Specifically, rather than uniformly sampling points across the image for training the scene coordinate regression model, we instead re-project 3D scene coordinates onto the 2D image plane and sample within a local neighborhood of the re-projected points. While our proposed sampling strategy is generally applicable, we showcase FocusTune by integrating it with the recently introduced Accelerated Coordinate Encoding (ACE) model. Our results demonstrate that FocusTune both improves or matches state-of-the-art performance whilst keeping ACE's appealing low storage and compute requirements, for example reducing translation error from 25 to 19 and 17 to 15 cm for single and ensemble models, respectively, on the Cambridge Landmarks dataset. This combination of high performance and low compute and storage requirements is particularly promising for applications in areas like mobile robotics and augmented reality. We made our code available at \url{https://github.com/sontung/focus-tune}.

Robot navigation requires an autonomy pipeline that is robust to environmental changes and effective in varying conditions. Teach and Repeat (T&R) navigation has shown high performance in autonomous repeated tasks under challenging circumstances, but research within T&R has predominantly focused on motion planning as opposed to motion control. In this paper, we propose a novel T&R system based on a robust motion control technique for a skid-steering mobile robot using sliding-mode control that effectively handles uncertainties that are particularly pronounced in the T&R task, where sensor noises, parametric uncertainties, and wheel-terrain interaction are common challenges. We first theoretically demonstrate that the proposed T&R system is globally stable and robust while considering the uncertainties of the closed-loop system. When deployed on a Clearpath Jackal robot, we then show the global stability of the proposed system in both indoor and outdoor environments covering different terrains, outperforming previous state-of-the-art methods in terms of mean average trajectory error and stability in these challenging environments. This paper makes an important step towards long-term autonomous T&R navigation with ensured safety guarantees., Comment: Accepted to IEEE International Conference on Robotics and Automation 2024 (ICRA2024)

Spiking Neural Networks (SNNs) are at the forefront of neuromorphic computing thanks to their potential energy-efficiency, low latencies, and capacity for continual learning. While these capabilities are well suited for robotics tasks, SNNs have seen limited adaptation in this field thus far. This work introduces a SNN for Visual Place Recognition (VPR) that is both trainable within minutes and queryable in milliseconds, making it well suited for deployment on compute-constrained robotic systems. Our proposed system, VPRTempo, overcomes slow training and inference times using an abstracted SNN that trades biological realism for efficiency. VPRTempo employs a temporal code that determines the timing of a single spike based on a pixel's intensity, as opposed to prior SNNs relying on rate coding that determined the number of spikes; improving spike efficiency by over 100%. VPRTempo is trained using Spike-Timing Dependent Plasticity and a supervised delta learning rule enforcing that each output spiking neuron responds to just a single place. We evaluate our system on the Nordland and Oxford RobotCar benchmark localization datasets, which include up to 27k places. We found that VPRTempo's accuracy is comparable to prior SNNs and the popular NetVLAD place recognition algorithm, while being several orders of magnitude faster and suitable for real-time deployment -- with inference speeds over 50 Hz on CPU. VPRTempo could be integrated as a loop closure component for online SLAM on resource-constrained systems such as space and underwater robots., Comment: 8 pages, 3 figures, accepted to the IEEE International Conference on Robotics and Automation (ICRA) 2024

We present results from fully general relativistic (GR), three-dimensional (3D), neutrino-radiation magneto-hydrodynamic (MHD) simulations of stellar core collapse of a 20 M$_\odot$ star with spectral neutrino transport. Our focus is to study the gravitational-wave (GW) signatures from the magnetorotationally (MR)-driven models. By parametrically changing the initial angular velocity and the strength of the magnetic fields in the core, we compute four models. Our results show that the MHD outflows are produced only for models (two out of four), to which magnetic field strengths of 10$^{12}$ G and rotation rates of 1 or 2 rad s$^{-1}$ are initially imposed in the core. Seen from the direction perpendicular to the rotational axis, a characteristic waveform is obtained exhibiting a monotonic time increase in the wave amplitude. As previously identified, this stems from the propagating MHD outflows along the axis. We show that the GW amplitude from anisotropic neutrino emission becomes more than one order-of-magnitude bigger than that from the matter contribution, whereas seen from the rotational axis, both of the two components are in the same order-of-magnitudes. Due to the memory effect, the frequency of the neutrino GW from our full-fledged 3D-MHD models is in the range less than $\sim$10 Hz. Toward the future GW detection for a Galactic core-collapse supernova, if driven by the MR mechanism, the planned next-generation detector as DECIGO is urgently needed to catch the low-frequency signals., Comment: 12 pages, 10 figures

Dense 3D reconstruction and ego-motion estimation are key challenges in autonomous driving and robotics. Compared to the complex, multi-modal systems deployed today, multi-camera systems provide a simpler, low-cost alternative. However, camera-based 3D reconstruction of complex dynamic scenes has proven extremely difficult, as existing solutions often produce incomplete or incoherent results. We propose R3D3, a multi-camera system for dense 3D reconstruction and ego-motion estimation. Our approach iterates between geometric estimation that exploits spatial-temporal information from multiple cameras, and monocular depth refinement. We integrate multi-camera feature correlation and dense bundle adjustment operators that yield robust geometric depth and pose estimates. To improve reconstruction where geometric depth is unreliable, e.g. for moving objects or low-textured regions, we introduce learnable scene priors via a depth refinement network. We show that this design enables a dense, consistent 3D reconstruction of challenging, dynamic outdoor environments. Consequently, we achieve state-of-the-art dense depth prediction on the DDAD and NuScenes benchmarks., Comment: Accepted to ICCV 2023. Project page is available at https://www.vis.xyz/pub/r3d3/

Neutrinos are known to play important roles in many astrophysical scenarios from the early period of the big bang to current stellar evolution being a unique messenger of the fusion reactions occurring in the center of our sun. In particular, neutrinos are crucial in determining the dynamics and the composition evolution in explosive events such as core-collapse supernovae and the merger of two neutron stars. In this paper, we review the current understanding of supernovae and binary neutron star mergers by focusing on the role of neutrinos therein. Several recent improvements on the theoretical modeling of neutrino interaction rates in nuclear matter as well as their impact on the heavy element nucleosynthesis in the supernova neutrino-driven wind are discussed, including the neutrino-nucleon opacity at the mean field level taking into account the relativistic kinematics of nucleons, the effect due to the nucleon-nucleon correlation, and the nucleon-nucleon bremsstrahlung. We also review the framework used to compute the neutrino-nucleus interactions and the up-to-date yield prediction for isotopes from neutrino nucleosynthesis occurring in the outer envelope of the supernova progenitor star during the explosion. Here improved predictions of energy spectra of supernova neutrinos of all flavors have had significant impact on the nucleosynthesis yields. Rapid progresses in modeling the flavor oscillations of neutrinos in these environments, including several novel mechanisms for collective neutrino oscillations and their potential impacts on various nucleosynthesis processes are summarized., Comment: Matches the published version on PPNP

Animals and insects showcase remarkably robust and adept navigational abilities, up to literally circumnavigating the globe. Primary progress in robotics inspired by these natural systems has occurred in two areas: highly theoretical computational neuroscience models, and handcrafted systems like RatSLAM and NeuroSLAM. In this research, we present work bridging the gap between the two, in the form of Multiscale Continuous Attractor Networks (MCAN), that combine the multiscale parallel spatial neural networks of the previous theoretical models with the real-world robustness of the robot-targeted systems, to enable trajectory tracking over large velocity ranges. To overcome the limitations of the reliance of previous systems on hand-tuned parameters, we present a genetic algorithm-based approach for automated tuning of these networks, substantially improving their usability. To provide challenging navigational scale ranges, we open source a flexible city-scale navigation simulator that adapts to any street network, enabling high throughput experimentation. In extensive experiments using the city-scale navigation environment and Kitti, we show that the system is capable of stable dead reckoning over a wide range of velocities and environmental scales, where a single-scale approach fails., Comment: 8 Pages, 8 Figures, accepted at IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)

The occurrence of a first-order hadron-quark matter phase transition at high baryon densities is investigated in astrophysical simulations of core-collapse supernovae, to decipher yet incompletely understood properties of the dense matter equation of state (EOS) using neutrinos from such cosmic events. It is found that the emission of a nonstandard second neutrino burst, dominated by electron antineutrinos, is not only a measurable signal for the appearance of deconfined quark matter but also reveals information about the state of matter at extreme conditions encountered at the supernova (SN) interior. To this end, a large set of spherically symmetric SN models is investigated, studying the dependence on the EOS and the stellar progenitor. General relativistic neutrino-radiation hydrodynamics is employed featuring three-flavor Boltzmann neutrino transport and a microscopic hadron-quark hybrid matter EOS class. Therefore, the DD2 relativistic mean-field hadronic model is employed, and several variations of it, and the string-flip model for the description of deconfined quark matter. The resulting hybrid model covers a representative range of onset densities for the phase transition and latent heats. This facilitates the direct connection between intrinsic signatures of the neutrino signal and properties of the EOS. In particular, a set of linear relations has been found empirically. These potentially provide a constraint for the onset density of a possible QCD phase transition from the future neutrino observation of the next galactic core-collapse SN, if a millisecond electron anti-neutrino burst is present around or less than 1s., Comment: 22 pages, 8 figures

The ability to recognize, localize and track dynamic objects in a scene is fundamental to many real-world applications, such as self-driving and robotic systems. Yet, traditional multiple object tracking (MOT) benchmarks rely only on a few object categories that hardly represent the multitude of possible objects that are encountered in the real world. This leaves contemporary MOT methods limited to a small set of pre-defined object categories. In this paper, we address this limitation by tackling a novel task, open-vocabulary MOT, that aims to evaluate tracking beyond pre-defined training categories. We further develop OVTrack, an open-vocabulary tracker that is capable of tracking arbitrary object classes. Its design is based on two key ingredients: First, leveraging vision-language models for both classification and association via knowledge distillation; second, a data hallucination strategy for robust appearance feature learning from denoising diffusion probabilistic models. The result is an extremely data-efficient open-vocabulary tracker that sets a new state-of-the-art on the large-scale, large-vocabulary TAO benchmark, while being trained solely on static images. Project page: https://www.vis.xyz/pub/ovtrack/, Comment: CVPR 2023

Localization is an essential capability for mobile robots. A rapidly growing field of research in this area is Visual Place Recognition (VPR), which is the ability to recognize previously seen places in the world based solely on images. This present work is the first tutorial paper on visual place recognition. It unifies the terminology of VPR and complements prior research in two important directions: 1) It provides a systematic introduction for newcomers to the field, covering topics such as the formulation of the VPR problem, a general-purpose algorithmic pipeline, an evaluation methodology for VPR approaches, and the major challenges for VPR and how they may be addressed. 2) As a contribution for researchers acquainted with the VPR problem, it examines the intricacies of different VPR problem types regarding input, data processing, and output. The tutorial also discusses the subtleties behind the evaluation of VPR algorithms, e.g., the evaluation of a VPR system that has to find all matching database images per query, as opposed to just a single match. Practical code examples in Python illustrate to prospective practitioners and researchers how VPR is implemented and evaluated., Comment: IEEE Robotics & Automation Magazine (RAM)

Seagrass meadows serve as critical carbon sinks, but estimating the amount of carbon they store requires knowledge of the seagrass species present. Underwater and surface vehicles equipped with machine learning algorithms can help to accurately estimate the composition and extent of seagrass meadows at scale. However, previous approaches for seagrass detection and classification have required supervision from patch-level labels. In this paper, we reframe seagrass classification as a weakly supervised coarse segmentation problem where image-level labels are used during training (25 times fewer labels compared to patch-level labeling) and patch-level outputs are obtained at inference time. To this end, we introduce SeaFeats, an architecture that uses unsupervised contrastive pre-training and feature similarity, and SeaCLIP, a model that showcases the effectiveness of large language models as a supervisory signal in domain-specific applications. We demonstrate that an ensemble of SeaFeats and SeaCLIP leads to highly robust performance. Our method outperforms previous approaches that require patch-level labels on the multi-species 'DeepSeagrass' dataset by 6.8% (absolute) for the class-weighted F1 score, and by 12.1% (absolute) for the seagrass presence/absence F1 score on the 'Global Wetlands' dataset. We also present two case studies for real-world deployment: outlier detection on the Global Wetlands dataset, and application of our method on imagery collected by the FloatyBoat autonomous surface vehicle., Comment: 10 pages, 4 figures, additional 3 pages of supplementary material

To track the 3D locations and trajectories of the other traffic participants at any given time, modern autonomous vehicles are equipped with multiple cameras that cover the vehicle's full surroundings. Yet, camera-based 3D object tracking methods prioritize optimizing the single-camera setup and resort to post-hoc fusion in a multi-camera setup. In this paper, we propose a method for panoramic 3D object tracking, called CC-3DT, that associates and models object trajectories both temporally and across views, and improves the overall tracking consistency. In particular, our method fuses 3D detections from multiple cameras before association, reducing identity switches significantly and improving motion modeling. Our experiments on large-scale driving datasets show that fusion before association leads to a large margin of improvement over post-hoc fusion. We set a new state-of-the-art with 12.6% improvement in average multi-object tracking accuracy (AMOTA) among all camera-based methods on the competitive NuScenes 3D tracking benchmark, outperforming previously published methods by 6.5% in AMOTA with the same 3D detector., Comment: Project page: https://www.vis.xyz/pub/cc-3dt/

We present a relativistic density functional approach to color superconducting quark matter that mimics quark confinement by a fast growth of the quasiparticle selfenergy in the confining region. The approach is shown to be equivalent to a chiral model of quark matter with medium dependent couplings. While the (pseudo)scalar sector of the model is fitted to the vacuum phenomenology of quantum chromodynamics, the strength of interaction in the vector and diquark channels is varied in order to provide the best agreement with the observational constraints on the mass-radius relation and tidal deformability of neutron stars modelled with our approach. In order to recover the conformal behavior of quark matter at asymptotically high densities we introduce a medium dependence of the vector and diquark couplings motivated by the nonperturbative gluon exchange. Our analysis signals that the onset of deconfinement to color superconducting quark matter is likely to occur in neutron stars with masses below 1.0 $M_\odot$., Comment: 8 pages, 4 figuures, Contribution to Proceedings of Quark Confinement and the Hadron Spectrum XV, August 1-6, 2022, Stavanger, Norway

We implement a multi-group and discrete-ordinate neutrino transport model in spherical symmetry which allows to simulate collective neutrino oscillations by including realistic collisional rates in a self-consistent way. We utilize this innovative model, based on strategic parameter rescaling, to study a recently proposed collisional flavor instability caused by the asymmetry of emission and absorption rates between $\nu_e$ and $\bar\nu_e$ for four different static backgrounds taken from different stages in a core-collapse supernova simulation. Our results confirm that collisional instabilities generally exist around the neutrinosphere during the SN accretion and post-accretion phase, as suggested by [arXiv:2104.11369]. However, the growth and transport of flavor instabilities can only be fully captured by models with global simulations as done in this work. With minimal ingredient to trigger collisional instabilities, we find that the flavor oscillations and transport mainly affect (anti)neutrinos of heavy lepton flavors around their decoupling sphere, which then leave imprints on their energy spectra in the free-streaming regime. For electron (anti)neutrinos, their properties remain nearly intact. We also explore various effects due to the decoherence from neutrino-nucleon scattering, artificially enhanced decoherence from emission and absorption, neutrino vacuum mixing, and inhomogeneous matter profile, and discuss the implication of our work., Comment: 20 pages, 13 figures

One recent promising approach to the Visual Place Recognition (VPR) problem has been to fuse the place recognition estimates of multiple complementary VPR techniques using methods such as SRAL and multi-process fusion. These approaches come with a substantial practical limitation: they require all potential VPR methods to be brute-force run before they are selectively fused. The obvious solution to this limitation is to predict the viable subset of methods ahead of time, but this is challenging because it requires a predictive signal within the imagery itself that is indicative of high performance methods. Here we propose an alternative approach that instead starts with a known single base VPR technique, and learns to predict the most complementary additional VPR technique to fuse with it, that results in the largest improvement in performance. The key innovation here is to use a dimensionally reduced difference vector between the query image and the top-retrieved reference image using this baseline technique as the predictive signal of the most complementary additional technique, both during training and inference. We demonstrate that our approach can train a single network to select performant, complementary technique pairs across datasets which span multiple modes of transportation (train, car, walking) as well as to generalise to unseen datasets, outperforming multiple baseline strategies for manually selecting the best technique pairs based on the same training data., Comment: 7 pages, 5 figures. arXiv admin note: text overlap with arXiv:2112.04701

Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions in images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of object regions on a pair of images for contrastive learning. We combine this similarity learning with multiple existing object detectors to build Quasi-Dense Tracking (QDTrack), which does not require displacement regression or motion priors. We find that the resulting distinctive feature space admits a simple nearest neighbor search at inference time for object association. In addition, we show that our similarity learning scheme is not limited to video data, but can learn effective instance similarity even from static input, enabling a competitive tracking performance without training on videos or using tracking supervision. We conduct extensive experiments on a wide variety of popular MOT benchmarks. We find that, despite its simplicity, QDTrack rivals the performance of state-of-the-art tracking methods on all benchmarks and sets a new state-of-the-art on the large-scale BDD100K MOT benchmark, while introducing negligible computational overhead to the detector.

Spiking neural networks have significant potential utility in robotics due to their high energy efficiency on specialized hardware, but proof-of-concept implementations have not yet typically achieved competitive performance or capability with conventional approaches. In this paper, we tackle one of the key practical challenges of scalability by introducing a novel modular ensemble network approach, where compact, localized spiking networks each learn and are solely responsible for recognizing places in a local region of the environment only. This modular approach creates a highly scalable system. However, it comes with a high-performance cost where a lack of global regularization at deployment time leads to hyperactive neurons that erroneously respond to places outside their learned region. Our second contribution introduces a regularization approach that detects and removes these problematic hyperactive neurons during the initial environmental learning phase. We evaluate this new scalable modular system on benchmark localization datasets Nordland and Oxford RobotCar, with comparisons to standard techniques NetVLAD, DenseVLAD, and SAD, and a previous spiking neural network system. Our system substantially outperforms the previous SNN system on its small dataset, but also maintains performance on 27 times larger benchmark datasets where the operation of the previous system is computationally infeasible, and performs competitively with the conventional localization systems., Comment: 8 pages, 6 figures, accepted to the IEEE International Conference on Robotics and Automation (ICRA) 2023

We address the problem of out-of-distribution (OOD) detection for the task of object detection. We show that residual convolutional layers with batch normalisation produce Sensitivity-Aware FEatures (SAFE) that are consistently powerful for distinguishing in-distribution from out-of-distribution detections. We extract SAFE vectors for every detected object, and train a multilayer perceptron on the surrogate task of distinguishing adversarially perturbed from clean in-distribution examples. This circumvents the need for realistic OOD training data, computationally expensive generative models, or retraining of the base object detector. SAFE outperforms the state-of-the-art OOD object detectors on multiple benchmarks by large margins, e.g. reducing the FPR95 by an absolute 30.6% from 48.3% to 17.7% on the OpenImages dataset.

Based on a recently developed relativistic density functional approach to color-superconducting quark matter and a novel quark-hadron transition construction which phenomenologically accounts for the effects of inhomogeneous pasta phases and quark-hadron continuity, we construct a class of hybrid equations of state applicable at the regimes typical for compact star astrophysics and heavy ion collisions. We outline that early quark deconfinement is a notable consequence of strong diquark pairing providing a good agreement with the observational data and driving the trajectories of the matter evolution during the supernovae explosions toward the regimes typical for the compact star mergers and heavy-ion collisions., Comment: Presented at the $29^{\rm th}$ Conference ''Quark Matter 2022'' on ultrarelativistic nucleus-nucleus collisions, April 4-10, 2022, Krak\'ow, Poland; minor typo corrections in v2

Event cameras continue to attract interest due to desirable characteristics such as high dynamic range, low latency, virtually no motion blur, and high energy efficiency. One of the potential applications that would benefit from these characteristics lies in visual place recognition for robot localization, i.e. matching a query observation to the corresponding reference place in the database. In this letter, we explore the distinctiveness of event streams from a small subset of pixels (in the tens or hundreds). We demonstrate that the absolute difference in the number of events at those pixel locations accumulated into event frames can be sufficient for the place recognition task, when pixels that display large variations in the reference set are used. Using such sparse (over image coordinates) but varying (variance over the number of events per pixel location) pixels enables frequent and computationally cheap updates of the location estimates. Furthermore, when event frames contain a constant number of events, our method takes full advantage of the event-driven nature of the sensory stream and displays promising robustness to changes in velocity. We evaluate our proposed approach on the Brisbane-Event-VPR dataset in an outdoor driving scenario, as well as the newly contributed indoor QCR-Event-VPR dataset that was captured with a DAVIS346 camera mounted on a mobile robotic platform. Our results show that our approach achieves competitive performance when compared to several baseline methods on those datasets, and is particularly well suited for compute- and energy-constrained platforms such as interplanetary rovers., Comment: 8 pages

We outline a theoretical approach supporting strong phase transitions from normal nuclear matter to the deconfined quark-gluon plasma, in the equation of state (EOS) for compact star matter. Implications of this hypothesis are discussed for astrophysical applications. Special emphasis is devoted to potentially detectable signatures, which can be directly related with the occurrence of a sufficiently strong phase transition. Therefore, simulations of core-collapse supernovae and binary compact star mergers are considered, including the subsequent emission of gravitational waves and, in the case of supernova, in addition the neutrinos play the role of messengers., Comment: 39 pages, 13 figures, contribution submitted on 20.02.2022 to the Book "Astrophysics in the XXI Century with Compact Stars", edited by Cesar Augusto Zen Vasconcellos and Fridolin Weber, World Scientific

Monitoring coral reefs using underwater vehicles increases the range of marine surveys and availability of historical ecological data by collecting significant quantities of images. Analysis of this imagery can be automated using a model trained to perform semantic segmentation, however it is too costly and time-consuming to densely label images for training supervised models. In this letter, we leverage photo-quadrat imagery labeled by ecologists with sparse point labels. We propose a point label aware method for propagating labels within superpixel regions to obtain augmented ground truth for training a semantic segmentation model. Our point label aware superpixel method utilizes the sparse point labels, and clusters pixels using learned features to accurately generate single-species segments in cluttered, complex coral images. Our method outperforms prior methods on the UCSD Mosaics dataset by 3.62% for pixel accuracy and 8.35% for mean IoU for the label propagation task, while reducing computation time reported by previous approaches by 76%. We train a DeepLabv3+ architecture and outperform state-of-the-art for semantic segmentation by 2.91% for pixel accuracy and 9.65% for mean IoU on the UCSD Mosaics dataset and by 4.19% for pixel accuracy and 14.32% mean IoU for the Eilat dataset.

Several global parameters of compact stars are related via empirical relations, which are (nearly) independent of the underlying equation of state (EoS) of dense matter and, therefore, are said to be universal. We investigate the universality of relations that express the maximum mass and the radius of non-rotating and maximally rapidly rotating configurations, as well as their moment of inertia, in terms of the compactness of the star. For this, we first utilize a collection of cold (zero-temperature) and hot (isentropic) nucleonic EoS and confirm that the universal relations are holding for our collection of EoS. We then go on, to add to our collection and test for the same universality models of EoS that admit a strong first-order phase transition from nucleonic to deconfined quark matter. Also in this case we find that the universal relations hold, in particular for hot, isentropic hybrid stars. By fitting the universal relations to our computed data, we determine the coefficients entering these relations and the accuracy to which they hold., Comment: 16 pages, 8 figures, published in MNRAS, Vol. 515, p. 3539 (2022)

We introduce powerful ideas from Hyperdimensional Computing into the challenging field of Out-of-Distribution (OOD) detection. In contrast to most existing work that performs OOD detection based on only a single layer of a neural network, we use similarity-preserving semi-orthogonal projection matrices to project the feature maps from multiple layers into a common vector space. By repeatedly applying the bundling operation $\oplus$, we create expressive class-specific descriptor vectors for all in-distribution classes. At test time, a simple and efficient cosine similarity calculation between descriptor vectors consistently identifies OOD samples with better performance than the current state-of-the-art. We show that the hyperdimensional fusion of multiple network layers is critical to achieve best general performance., Comment: Accepted to WACV2023

A recent approach to the Visual Place Recognition (VPR) problem has been to fuse the place recognition estimates of multiple complementary VPR techniques simultaneously. However, selecting the optimal set of techniques to use in a specific deployment environment a-priori is a difficult and unresolved challenge. Further, to the best of our knowledge, no method exists which can select a set of techniques on a frame-by-frame basis in response to image-to-image variations. In this work, we propose an unsupervised algorithm that finds the most robust set of VPR techniques to use in the current deployment environment, on a frame-by-frame basis. The selection of techniques is determined by an analysis of the similarity scores between the current query image and the collection of database images and does not require ground-truth information. We demonstrate our approach on a wide variety of datasets and VPR techniques and show that the proposed dynamic multi-process fusion (Dyn-MPF) has superior VPR performance compared to a variety of challenging competitive methods, some of which are given an unfair advantage through access to the ground-truth information.

In this paper, we present a novel approach to compute ambiguities in thinned uniform linear arrays, i.e., sparse non-uniform linear arrays, via a mixed-integer program. Ambiguities arise when there exists a set of distinct directions-of-arrival, for which the corresponding steering matrix is rank-deficient and are associated with nonunique parameter estimation. Our approach uses Young tableaux for which a submatrix of the steering matrix has a vanishing determinant, which can be expressed through vanishing sums of unit roots. Each of these vanishing sums then corresponds to an ambiguous set of directions-of-arrival. We derive a method to enumerate such ambiguous sets using a mixed-integer program and present results on several examples.

Axion-like particles (ALPs) may be abundantly produced in core-collapse (CC) supernovae (SNe), hence the cumulative signal from all past SN events can create a diffuse flux peaked at energies of about 25~MeV. We improve upon the modeling of the ALPs flux by including a set of CC SN models with different progenitor masses, as well as the effects of failed CC SNe -- which yield the formation of black holes instead of explosions. Relying on the coupling strength of ALPs to photons and the related Primakoff process, the diffuse SN ALP flux is converted into gamma rays while traversing the magnetic field of the Milky Way. The spatial morphology of this signal is expected to follow the shape of the Galactic magnetic field lines. We make use of this via a template-based analysis that utilizes 12 years of $Fermi$-LAT data in the energy range from 50 MeV to 500 GeV. In our benchmark case of the realization of astrophysical and cosmological parameters, we find an upper limit of $g_{a\gamma} \lesssim 3.76\times10^{-11}\;\mathrm{GeV}^{-1}$ at 95$\%$ confidence level for $m_a \ll 10^{-11}$ eV, while we find that systematic deviations from this benchmark scenario induce an uncertainty as large as about a factor of two. Our result slightly improves the CAST bound, while still being a factor of six (baseline scenario) weaker than the SN1987A gamma-ray burst limit., Comment: 26 pages, 17 figures; v2 changes: inclusion of gravitational redshift on CC SN ALP emission spectra, impact of the creation of alpha particles during a CC SN on ALP emission spectra

Spiking neural networks (SNNs) offer both compelling potential advantages, including energy efficiency and low latencies and challenges including the non-differentiable nature of event spikes. Much of the initial research in this area has converted deep neural networks to equivalent SNNs, but this conversion approach potentially negates some of the advantages of SNN-based approaches developed from scratch. One promising area for high-performance SNNs is template matching and image recognition. This research introduces the first high-performance SNN for the Visual Place Recognition (VPR) task: given a query image, the SNN has to find the closest match out of a list of reference images. At the core of this new system is a novel assignment scheme that implements a form of ambiguity-informed salience, by up-weighting single-place-encoding neurons and down-weighting "ambiguous" neurons that respond to multiple different reference places. In a range of experiments on the challenging Nordland, Oxford RobotCar, SPEDTest, Synthia, and St Lucia datasets, we show that our SNN achieves comparable VPR performance to state-of-the-art and classical techniques, and degrades gracefully in performance with an increasing number of reference places. Our results provide a significant milestone towards SNNs that can provide robust, energy-efficient, and low latency robot localization., Comment: 8 pages, 6 figures, IEEE Robotics and Automation Letters (RA-L), also accepted to IEEE International Conference on Robotics and Automation (ICRA 2022)

We investigate observable signatures of a first-order quantum chromodynamics (QCD) phase transition in the context of core collapse supernovae. To this end, we conduct axially symmetric numerical relativity simulations with multi-energy neutrino transport, using a hadron-quark hybrid equation of state (EOS). We consider four non-rotating progenitor models, whose masses range from $9.6$ to $70$ M$_\odot$. We find that the two less massive progenitor stars (9.6 and 11.2 M$_\odot$) show a successful explosion, which is driven by the neutrino heating. They do not undergo the QCD phase transition and leave behind a neutron star (NS). As for the more massive progenitor stars (50 and 70 M$_\odot$), the proto-neutron star (PNS) core enters the phase transition region and experiences the second collapse. Because of a sudden stiffening of the EOS entering to the pure quark matter regime, a strong shock wave is formed and blows off the PNS envelope in the 50 M$_\odot$ model. Consequently, the remnant becomes a quark core surrounded by hadronic matters, leading to the formation of the hybrid star. However for the 70 M$_\odot$ model, the shock wave cannot overcome the continuous mass accretion and it readily becomes a black hole. We find that the neutrino and gravitational wave (GW) signals from supernova explosions driven by the hadron-quark phase transition are detectable for the present generation of neutrino and GW detectors. Furthermore, the analysis of the GW detector response reveals unique kHz signatures, which will allow us to distinguish this class of supernova explosions from failed and neutrino-driven explosions., Comment: 24 pages, 16 figures, Accepted for publication in ApJ