Search

Using techniques from many-body quantum theory, we propose a framework for representing the evolution of observables of measure-preserving ergodic flows through infinite-dimensional rotation systems on tori. This approach is based on a class of weighted Fock spaces $F_w(\mathcal H_\tau)$ generated by a 1-parameter family of reproducing kernel Hilbert spaces $\mathcal H_\tau$, and endowed with commutative Banach algebra structure under the symmetric tensor product using a subconvolutive weight $w$. We describe the construction of the spaces $F_w(\mathcal H_\tau)$ and show that their Banach algebra spectra, $\sigma(F_w(\mathcal H_\tau))$, decompose into a family of tori of potentially infinite dimension. Spectrally consistent unitary approximations $U^t_\tau$ of the Koopman operator acting on $\mathcal H_\tau$ are then lifted to rotation systems on these tori akin to the topological models of ergodic systems with pure point spectra in the Halmos--von Neumann theorem. Our scheme also employs a procedure for representing observables of the original system by polynomial functions on finite-dimensional tori in $\sigma(F_w(\mathcal H_\tau))$ of arbitrarily large degree, with coefficients determined from pointwise products of eigenfunctions of $U^t_\tau$. This leads to models for the Koopman evolution of observables on $L^2$ built from tensor products of finite collections of approximate Koopman eigenfunctions. Numerically, the scheme is amenable to consistent data-driven implementation using kernel methods. We illustrate it with applications to Stepanoff flows on the 2-torus and the Lorenz 63 system. Connections with quantum computing are also discussed., Comment: 50 pages, 9 figures

Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks., Comment: Accepted by International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2025

Ground-penetrating radar (GPR) is a mature geophysical method that has gained increasing popularity in planetary science over the past decade. GPR has been utilised both for Lunar and Martian missions providing pivotal information regarding the near surface geology of Terrestrial planets. Within that context, numerous processing pipelines have been suggested to address the unique challenges present in planetary setups. These processing pipelines often require manual tuning resulting to ambiguous outputs open to non-unique interpretations. These pitfalls combined with the large number of planetary GPR data (kilometers in magnitude), highlight the necessity for automatic, objective and advanced processing and interpretation schemes. The current paper investigates the potential of deep learning for interpreting and processing GPR data. The one-shot multi-offset configuration is investigated via a coherent numerical case study, showcasing the potential of deep learning for A) reconstructing the dielectric distribution of the the near surface of Terrestrial planets, and B) filling missing or bad-quality traces. Special care was taken for the numerical data to be both realistic and challenging. Moreover, the generated synthetic data are properly labelled and made publicly available for training future data-driven pipelines and contributing towards developing pre-trained foundation models for GPR.

The Gaussian process (GP) is a Bayesian nonparametric paradigm that is widely adopted for uncertainty quantification (UQ) in a number of safety-critical applications, including robotics, healthcare, as well as surveillance. The consistency of the resulting uncertainty values however, hinges on the premise that the learning function conforms to the properties specified by the GP model, such as smoothness, periodicity and more, which may not be satisfied in practice, especially with data arriving on the fly. To combat against such model mis-specification, we propose to wed the GP with the prevailing conformal prediction (CP), a distribution-free post-processing framework that produces it prediction sets with a provably valid coverage under the sole assumption of data exchangeability. However, this assumption is usually violated in the online setting, where a prediction set is sought before revealing the true label. To ensure long-term coverage guarantee, we will adaptively set the key threshold parameter based on the feedback whether the true label falls inside the prediction set. Numerical results demonstrate the merits of the online GP-CP approach relative to existing alternatives in the long-term coverage performance.

Koopman operators and transfer operators represent nonlinear dynamics in state space through its induced action on linear spaces of observables and measures, respectively. This framework enables the use of linear operator theory for supervised and unsupervised learning of nonlinear dynamical systems, and has received considerable interest in recent years. Here, we propose a data-driven technique for spectral approximation of Koopman operators of continuous-time, measure-preserving ergodic systems that is asymptotically consistent and makes direct use of known equations of motion (physics). Our approach is based on a bounded transformation of the Koopman generator (an operator implementing directional derivatives of observables along the dynamical flow), followed by smoothing by a Markov semigroup of kernel integral operators. This results in a skew-adjoint, compact operator whose eigendecomposition is expressible as a variational generalized eigenvalue problem. We develop Galerkin methods to solve this eigenvalue problem and study their asymptotic consistency in the large-data limit. A key aspect of these methods is that they are physics-informed, in the sense of making direct use of dynamical vector field information through automatic differentiation of kernel functions. Solutions of the eigenvalue problem reconstruct evolution operators that preserve unitarity of the underlying Koopman group while spectrally converging to it in a suitable limit. In addition, the computed eigenfunctions have representatives in a reproducing kernel Hilbert space, enabling out-of-sample evaluation of learned dynamical features. Numerical experiments performed with this method on integrable and chaotic low-dimensional systems demonstrate its efficacy in extracting dynamically coherent observables under complex dynamics.

One of the primary catalysts fueling advances in artificial intelligence (AI) and machine learning (ML) is the availability of massive, curated datasets. A commonly used technique to curate such massive datasets is crowdsourcing, where data are dispatched to multiple annotators. The annotator-produced labels are then fused to serve downstream learning and inference tasks. This annotation process often creates noisy labels due to various reasons, such as the limited expertise, or unreliability of annotators, among others. Therefore, a core objective in crowdsourcing is to develop methods that effectively mitigate the negative impact of such label noise on learning tasks. This feature article introduces advances in learning from noisy crowdsourced labels. The focus is on key crowdsourcing models and their methodological treatments, from classical statistical models to recent deep learning-based approaches, emphasizing analytical insights and algorithmic developments. In particular, this article reviews the connections between signal processing (SP) theory and methods, such as identifiability of tensor and nonnegative matrix factorization, and novel, principled solutions of longstanding challenges in crowdsourcing -- showing how SP perspectives drive the advancements of this field. Furthermore, this article touches upon emerging topics that are critical for developing cutting-edge AI/ML systems, such as crowdsourcing in reinforcement learning with human feedback (RLHF) and direct preference optimization (DPO) that are key techniques for fine-tuning large language models (LLMs).

We propose a tensor network framework for approximating the evolution of observables of measure-preserving ergodic systems. Our approach is based on a spectrally-convergent approximation of the skew-adjoint Koopman generator by a diagonalizable, skew-adjoint operator $W_\tau$ that acts on a reproducing kernel Hilbert space $\mathcal H_\tau$ with coalgebra structure and Banach algebra structure under the pointwise product of functions. Leveraging this structure, we lift the unitary evolution operators $e^{t W_\tau}$ (which can be thought of as regularized Koopman operators) to a unitary evolution group on the Fock space $F(\mathcal H_\tau)$ generated by $\mathcal H_\tau$ that acts multiplicatively with respect to the tensor product. Our scheme also employs a representation of classical observables ($L^\infty$ functions of the state) by quantum observables (self-adjoint operators) acting on the Fock space, and a representation of probability densities in $L^1$ by quantum states. Combining these constructions leads to an approximation of the Koopman evolution of observables that is representable as evaluation of a tree tensor network built on a tensor product subspace $\mathcal H_\tau^{\otimes n} \subset F(\mathcal H_\tau)$ of arbitrarily high grading $n \in \mathbb N$. A key feature of this quantum-inspired approximation is that it captures information from a tensor product space of dimension $(2d+1)^n$, generated from a collection of $2d + 1$ eigenfunctions of $W_\tau$. Furthermore, the approximation is positivity preserving. The paper contains a theoretical convergence analysis of the method and numerical applications to two dynamical systems on the 2-torus: an ergodic torus rotation as an example with pure point Koopman spectrum and a Stepanoff flow as an example with topological weak mixing., Comment: 52 pages, 10 figures

We define the notion of a thick open set $\Omega$ in a Euclidean space and show that a local Hardy-Littlewood inequality holds in $L^p(\Omega)$, $p \in (1, \infty]$. We then establish pointwise and $L^p(\Omega)$ convergence for families of convolution operators with a Markov normalization on $\Omega$. We demonstrate application of such smoothing operators to piecewise-continuous density, velocity, and stress fields from discrete element models of sea ice dynamics., Comment: 28 pages, 5 figures

Reproducing kernel Hilbert spaces (RKHSs) are Hilbert spaces of functions where pointwise evaluation is continuous. There are known examples of RKHSs that are Banach algebras under pointwise multiplication. These examples are built from weights on the dual of a locally compact abelian group. In this paper we define an algebra structure on an RKHS that is equivalent to subconvolutivity of the weight for known examples (referred to as reproducing kernel Hilbert algebras, or RKHAs). We show that the class of RKHAs is closed under the Hilbert space tensor product and the pullback construction on the category of RKHSs. The subcategory of RKHAs becomes a monoidal category with the spectrum as a monoidal functor to the category of topological spaces. The image of this functor is shown to contain all compact subspaces of $\mathbb R^n$ for $n>0$., Comment: 21 pages, 6 figures

Utilizing task-invariant prior knowledge extracted from related tasks, meta-learning is a principled framework that empowers learning a new task especially when data records are limited. A fundamental challenge in meta-learning is how to quickly "adapt" the extracted prior in order to train a task-specific model within a few optimization steps. Existing approaches deal with this challenge using a preconditioner that enhances convergence of the per-task training process. Though effective in representing locally a quadratic training loss, these simple linear preconditioners can hardly capture complex loss geometries. The present contribution addresses this limitation by learning a nonlinear mirror map, which induces a versatile distance metric to enable capturing and optimizing a wide range of loss geometries, hence facilitating the per-task training. Numerical tests on few-shot learning datasets demonstrate the superior expressiveness and convergence of the advocated approach., Comment: Accepted by 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-24)

On-device memory concerns in distributed deep learning have become severe due to (i) the growth of model size in multi-GPU training, and (ii) the wide adoption of deep neural networks for federated learning on IoT devices which have limited storage. In such settings, communication efficient optimization methods are attractive alternatives, however they still struggle with memory issues. To tackle these challenges, we propose an communication efficient method called contractive error feedback (ConEF). As opposed to SGD with error-feedback (EFSGD) that inefficiently manages memory, ConEF obtains the sweet spot of convergence and memory usage, and achieves communication efficiency by leveraging biased and all-reducable gradient compression. We empirically validate ConEF on various learning tasks that include image classification, language modeling, and machine translation and observe that ConEF saves 80\% - 90\% of the extra memory in EFSGD with almost no loss on test performance, while also achieving 1.3x - 5x speedup of SGD. Through our work, we also demonstrate the feasibility and convergence of ConEF to clear up the theoretical barrier of integrating ConEF to popular memory efficient frameworks such as ZeRO-3.

3D reconstruction is a fundamental task in robotics that gained attention due to its major impact in a wide variety of practical settings, including agriculture, underwater, and urban environments. This task can be carried out via view planning (VP), which aims to optimally place a certain number of cameras in positions that maximize the visual information, improving the resulting 3D reconstruction. Nonetheless, in most real-world settings, existing environmental noise can significantly affect the performance of 3D reconstruction. To that end, this work advocates a novel geometric-based reconstruction quality function for VP, that accounts for the existing noise of the environment, without requiring its closed-form expression. With no analytic expression of the objective function, this work puts forth an adaptive Bayesian optimization algorithm for accurate 3D reconstruction in the presence of noise. Numerical tests on noisy agricultural environments showcase the merits of the proposed approach for 3D reconstruction with even a small number of available cameras.

Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation. Embracing the geometry of the loss function, where neighborhoods of 'flat minima' heighten generalization ability, SAM seeks 'flat valleys' by minimizing the maximum loss caused by an adversary perturbing parameters within the neighborhood. Although critical to account for sharpness of the loss function, such an 'over-friendly adversary' can curtail the outmost level of generalization. The novel approach of this contribution fosters stabilization of adversaries through variance suppression (VaSSO) to avoid such friendliness. VaSSO's provable stability safeguards its numerical improvement over SAM in model-agnostic tasks, including image classification and machine translation. In addition, experiments confirm that VaSSO endows SAM with robustness against high levels of label noise., Comment: Accepted to NeurIPS 2023

Koopman operators and transfer operators represent dynamical systems through their induced linear action on vector spaces of observables, enabling the use of operator-theoretic techniques to analyze nonlinear dynamics in state space. The extraction of approximate Koopman or transfer operator eigenfunctions (and the associated eigenvalues) from an unknown system is nontrivial, particularly if the system has mixed or continuous spectrum. In this paper, we describe a spectrally accurate approach to approximate the Koopman operator on $L^2$ for measure-preserving, continuous-time systems via a ``compactification'' of the resolvent of the generator. This approach employs kernel integral operators to approximate the skew-adjoint Koopman generator by a family of skew-adjoint operators with compact resolvent, whose spectral measures converge in a suitable asymptotic limit, and whose eigenfunctions are approximately periodic. Moreover, we develop a data-driven formulation of our approach, utilizing data sampled on dynamical trajectories and associated dictionaries of kernel eigenfunctions for operator approximation. The data-driven scheme is shown to converge in the limit of large training data under natural assumptions on the dynamical system and observation modality. We explore applications of this technique to dynamical systems on tori with pure point spectra and the Lorenz 63 system as an example with mixing dynamics., Comment: 60 pages, 7 figures. Revision includes removal of errors in notation and improved clarity

Conic programming has well-documented merits in a gamut of signal processing and machine learning tasks. This contribution revisits a recently developed first-order conic descent (CD) solver, and advances it in three aspects: intuition, theory, and algorithmic implementation. It is found that CD can afford an intuitive geometric derivation that originates from the dual problem. This opens the door to novel algorithmic designs, with a momentum variant of CD, momentum conic descent (MOCO) exemplified. Diving deeper into the dual behavior CD and MOCO reveals: i) an analytically justified stopping criterion; and, ii) the potential to design preconditioners to speed up dual convergence. Lastly, to scale semidefinite programming (SDP) especially for low-rank solutions, a memory efficient MOCO variant is developed and numerically validated.

An important problem in modern applied science is characterizing the behavior of systems with complex internal dynamics subjected to external forcings from their environment. While a great variety of techniques has been developed to analyze such non-autonomous systems, many approaches rely on the availability of ensembles of experiments or simulations in order to generate sufficient information to encapsulate the external forcings. This makes them unsuitable to study important classes of natural systems such as climate dynamics where only a single realization is observed. Here, we show that operator-theoretic techniques previously developed to identify slowly decaying observables of autonomous dynamical systems provide a powerful means for identifying trends and persistent cycles of non-autonomous systems using data from a \emph{single} trajectory of the system. Using systematic mathematical analysis and prototype examples, we demonstrate that eigenfunctions of Koopman and transfer operators provide coordinates that simultaneously capture nonlinear trends and coherent modes of internal variability. In addition, we apply our framework to two real-world examples from present and past climate dynamics: Variability of sea surface temperature (SST) over the industrial era and the mid-Pleistocene transition (MPT) of Quaternary glaciation cycles. Our results provide a nonparametric representation of SST and surface air temperature (SAT) trends over the industrial era, while also capturing the response of the seasonal precipitation cycle to these trends. In addition, our paleo-climate analysis reveals the dominant glaciation cycles over the past 3 million years and the MPT with a high level of granularity.

We introduce a linear operator on a Hilbert $C^*$-module for analyzing skew-product dynamical systems. The operator is defined by composition and multiplication. We show that it admits a decomposition in the Hilbert $C^*$-module, called eigenoperator decomposition, that generalizes the concept of the eigenvalue decomposition. This decomposition reconstructs the Koopman operator of the system in a manner that represents the continuous spectrum through eigenoperators. In addition, it is related to the notions of cocycle and Oseledets subspaces and it is useful for characterizing coherent structures under skew-product dynamics. We present numerical applications to simple systems on two-dimensional domains.

Craters are amongst the most important morphological features in planetary exploration. To that extent, detecting, mapping and counting craters is a mainstream process in planetary science, done primarily manually, which is a very laborious and time-consuming process. Recently, machine learning (ML) and computer vision have been successfully applied for both detecting craters and estimating their size. Existing ML approaches for automated crater detection have been trained in specific types of data e.g. digital elevation model (DEM), images and associated metadata for orbiters such as the Lunar Reconnaissance Orbiter Camera (LROC) etc.. Due to that, each of the resulting ML schemes is applicable and reliable only to the type of data used during the training process. Data from different sources, angles and setups can compromise the reliability of these ML schemes. In this paper we present a universal crater detection scheme that is based on the recently proposed Segment Anything Model (SAM) from META AI. SAM is a prompt-able segmentation system with zero-shot generalization to unfamiliar objects and images without the need for additional training. Using SAM we can successfully identify crater-looking objects in any type of data (e,g, raw satellite images Level-1 and 2 products, DEMs etc.) for different setups (e.g. Lunar, Mars) and different capturing angles. Moreover, using shape indexes, we only keep the segmentation masks of crater-like features. These masks are subsequently fitted with an ellipse, recovering both the location and the size/geometry of the detected craters., Comment: 11 pages, 7 Figures, preprint of a submitted paper in Icarus (under review)

Meta-learning owns unique effectiveness and swiftness in tackling emerging tasks with limited data. Its broad applicability is revealed by viewing it as a bi-level optimization problem. The resultant algorithmic viewpoint however, faces scalability issues when the inner-level optimization relies on gradient-based iterations. Implicit differentiation has been considered to alleviate this challenge, but it is restricted to an isotropic Gaussian prior, and only favors deterministic meta-learning approaches. This work markedly mitigates the scalability bottleneck by cross-fertilizing the benefits of implicit differentiation to probabilistic Bayesian meta-learning. The novel implicit Bayesian meta-learning (iBaML) method not only broadens the scope of learnable priors, but also quantifies the associated uncertainty. Furthermore, the ultimate complexity is well controlled regardless of the inner-level optimization trajectory. Analytical error bounds are established to demonstrate the precision and efficiency of the generalized implicit gradient over the explicit one. Extensive numerical tests are also carried out to empirically validate the performance of the proposed method., Comment: Accepted as a poster paper in the main track of Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23)

Recent years have witnessed the emergence of mobile edge computing (MEC), on the premise of a cost-effective enhancement in the computational ability of hardware-constrained wireless devices (WDs) comprising the Internet of Things (IoT). In a general multi-server multi-user MEC system, each WD has a computational task to execute and has to select binary (off)loading decisions, along with the analog-amplitude resource allocation variables in an online manner, with the goal of minimizing the overall energy-delay cost (EDC) with dynamic system states. While past works typically rely on the explicit expression of the EDC function, the present contribution considers a practical setting, where in lieu of system state information, the EDC function is not available in analytical form, and instead only the function values at queried points are revealed. Towards tackling such a challenging online combinatorial problem with only bandit information, novel Bayesian optimization (BO) based approaches are put forth by leveraging the multi-armed bandit (MAB) framework. Per time slot, the discrete offloading decisions are first obtained via the MAB method, and the analog resource allocation variables are subsequently optimized using the BO selection rule. By exploiting both temporal and contextual information, two novel BO approaches, termed time-varying BO and contextual time-varying BO, are developed. Numerical tests validate the merits of the proposed BO approaches compared with contemporary benchmarks under different MEC network sizes., Comment: This work has been submitted to the IEEE for possible publication

We propose a scheme for data-driven parameterization of unresolved dimensions of dynamical systems based on the mathematical framework of quantum mechanics and Koopman operator theory. Given a system in which some components of the state are unknown, this method involves defining a surrogate system in a time-dependent quantum state which determines the fluxes from the unresolved degrees of freedom at each timestep. The quantum state is a density operator on a finite-dimensional Hilbert space of classical observables and evolves over time under an action induced by the Koopman operator. The quantum state also updates with new values of the resolved variables according to a quantum Bayes' law, implemented via an operator-valued feature map. Kernel methods are utilized to learn data-driven basis functions and represent quantum states, observables, and evolution operators as matrices. The resulting computational schemes are automatically positivity-preserving, aiding in the physical consistency of the parameterized system. We analyze the results of two different modalities of this methodology applied to the Lorenz 63 and Lorenz 96 multiscale systems, and show how this approach preserves important statistical and qualitative properties of the underlying chaotic systems., Comment: 49 pages, 10 figures

We develop an algebraic framework for sequential data assimilation of partially observed dynamical systems. In this framework, Bayesian data assimilation is embedded in a non-abelian operator algebra, which provides a representation of observables by multiplication operators and probability densities by density operators (quantum states). In the algebraic approach, the forecast step of data assimilation is represented by a quantum operation induced by the Koopman operator of the dynamical system. Moreover, the analysis step is described by a quantum effect, which generalizes the Bayesian observational update rule. Projecting this formulation to finite-dimensional matrix algebras leads to new computational data assimilation schemes that are (i) automatically positivity-preserving; and (ii) amenable to consistent data-driven approximation using kernel methods for machine learning. Moreover, these methods are natural candidates for implementation on quantum computers. Applications to data assimilation of the Lorenz 96 multiscale system and the El Nino Southern Oscillation in a climate model show promising results in terms of forecast skill and uncertainty quantification., Comment: 46 pages, 4 figures

Labeled data can be expensive to acquire in several application domains, including medical imaging, robotics, and computer vision. To efficiently train machine learning models under such high labeling costs, active learning (AL) judiciously selects the most informative data instances to label on-the-fly. This active sampling process can benefit from a statistical function model, that is typically captured by a Gaussian process (GP). While most GP-based AL approaches rely on a single kernel function, the present contribution advocates an ensemble of GP models with weights adapted to the labeled data collected incrementally. Building on this novel EGP model, a suite of acquisition functions emerges based on the uncertainty and disagreement rules. An adaptively weighted ensemble of EGP-based acquisition functions is also introduced to further robustify performance. Extensive tests on synthetic and real datasets showcase the merits of the proposed EGP-based approaches with respect to the single GP-based AL alternatives.

Bayesian optimization (BO) has well-documented merits for optimizing black-box functions with an expensive evaluation cost. Such functions emerge in applications as diverse as hyperparameter tuning, drug discovery, and robotics. BO hinges on a Bayesian surrogate model to sequentially select query points so as to balance exploration with exploitation of the search space. Most existing works rely on a single Gaussian process (GP) based surrogate model, where the kernel function form is typically preselected using domain knowledge. To bypass such a design process, this paper leverages an ensemble (E) of GPs to adaptively select the surrogate model fit on-the-fly, yielding a GP mixture posterior with enhanced expressiveness for the sought function. Acquisition of the next evaluation input using this EGP-based function posterior is then enabled by Thompson sampling (TS) that requires no additional design parameters. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. The novel EGP-TS readily accommodates parallel operation. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret for both sequential and parallel settings. Tests on synthetic functions and real-world applications showcase the merits of the proposed method.

Kron reduction is a network-reduction method that eliminates nodes with zero current injections from electrical networks operating in sinusoidal steady state. In the time domain, the state-of-the-art application of Kron reduction has been in networks with transmission lines that have constant R/L ratios. This paper considers RL networks without such restriction and puts forth a provably exact time-domain generalization of Kron reduction. Exemplifying empirical tests on a wye-delta network are provided to validate the analytical results.