Search

For multiple scientific endeavors it is common to measure a phenomenon of interest in more than one ways. We make observations of objects from several different perspectives in space, at different points in time; we may also measure different properties of a mixture using different types of instruments. After collecting this heterogeneous information, it is necessary to be able to synthesize a complete picture of what is `common' across its sources: the subject we ultimately want to study. However, isolated (`clean') observations of a system are not always possible: observations often contain information about other systems in its environment, or about the measuring instruments themselves. In that sense, each observation may contain information that `does not matter' to the original object of study; this `uncommon' information between sensors observing the same object may still be important, and decoupling it from the main signal(s) useful. We introduce a neural network autoencoder framework capable of both tasks: it is structured to identify `common' variables, and, making use of orthogonality constraints to define geometric independence, to also identify disentangled `uncommon' information originating from the heterogeneous sensors. We demonstrate applications in several computational examples.

Before we attempt to learn a function between two (sets of) observables of a physical process, we must first decide what the inputs and what the outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of initially deciding ``the right quantities'' to relate through such a function, and then proceed to learn it. This is accomplished by processing multiple simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: multiple observation processes of the system. We thus determine (a) what subsets of observables are common between the observation processes (and therefore observable from each other, relatable through a function); and (b) what information is unrelated to these common observables, and therefore particular to each observation process, and not contributing to the desired function. Any data-driven function approximation technique can subsequently be used to learn the input-output relation, from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular ``twists'' of the approach are discussed. The first has to do with the identifiability of particular quantities of interest from the measurements. We now construct mappings from a single set of observations of one process to entire level sets of measurements of the process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures ``now'', while the second observation process measures ``in the future'', the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution.

Deep Operator Networks (DeepOnets) have revolutionized the domain of scientific machine learning for the solution of the inverse problem for dynamical systems. However, their implementation necessitates optimizing a high-dimensional space of parameters and hyperparameters. This fact, along with the requirement of substantial computational resources, poses a barrier to achieving high numerical accuracy. Here, inpsired by DeepONets and to address the above challenges, we present Random Projection-based Operator Networks (RandONets): shallow networks with random projections that learn linear and nonlinear operators. The implementation of RandONets involves: (a) incorporating random bases, thus enabling the use of shallow neural networks with a single hidden layer, where the only unknowns are the output weights of the network's weighted inner product; this reduces dramatically the dimensionality of the parameter space; and, based on this, (b) using established least-squares solvers (e.g., Tikhonov regularization and preconditioned QR decomposition) that offer superior numerical approximation properties compared to other optimization techniques used in deep-learning. In this work, we prove the universal approximation accuracy of RandONets for approximating nonlinear operators and demonstrate their efficiency in approximating linear nonlinear evolution operators (right-hand-sides (RHS)) with a focus on PDEs. We show, that for this particular task, RandONets outperform, both in terms of numerical approximation accuracy and computational cost, the ``vanilla" DeepOnets.

Singularly perturbed dynamical systems, commonly known as fast-slow systems, play a crucial role in various applications such as plasma physics. They are closely related to reduced order modeling, closures, and structure-preserving numerical algorithms for multiscale modeling. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, the Fenichel normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on several examples that exhibit multiple timescales, including the Grad moment system from hydrodynamics, two-scale Lorentz96 equations for modeling atmospheric dynamics, and Abraham-Lorentz dynamics modeling radiation reaction of electrons in a magnetic field.

We use Physics-Informed Neural Networks (PINNs) to solve the discrete-time nonlinear observer state estimation problem. Integrated within a single-step exact observer linearization framework, the proposed PINN approach aims at learning a nonlinear state transformation map by solving a system of inhomogeneous functional equations. The performance of the proposed PINN approach is assessed via two illustrative case studies for which the observer linearizing transformation map can be derived analytically. We also perform an uncertainty quantification analysis for the proposed PINN scheme and we compare it with conventional power-series numerical implementations, which rely on the computation of a power series solution.

In this work we introduce a manifold learning-based surrogate modeling framework for uncertainty quantification in high-dimensional stochastic systems. Our first goal is to perform data mining on the available simulation data to identify a set of low-dimensional (latent) descriptors that efficiently parameterize the response of the high-dimensional computational model. To this end, we employ Principal Geodesic Analysis on the Grassmann manifold of the response to identify a set of disjoint principal geodesic submanifolds, of possibly different dimension, that captures the variation in the data. Since operations on the Grassmann require the data to be concentrated, we propose an adaptive algorithm based on Riemanniann K-means and the minimization of the sample Frechet variance on the Grassmann manifold to identify "local" principal geodesic submanifolds that represent different system behavior across the parameter space. Polynomial chaos expansion is then used to construct a mapping between the random input parameters and the projection of the response on these local principal geodesic submanifolds. The method is demonstrated on four test cases, a toy-example that involves points on a hypersphere, a Lotka-Volterra dynamical system, a continuous-flow stirred-tank chemical reactor system, and a two-dimensional Rayleigh-Benard convection problem, Comment: 50 pages, 17 figures

A valuable step in the modeling of multiscale dynamical systems in fields such as computational chemistry, biology, materials science and more, is the representative sampling of the phase space over long timescales of interest; this task is not, however, without challenges. For example, the long term behavior of a system with many degrees of freedom often cannot be efficiently computationally explored by direct dynamical simulation; such systems can often become trapped in local free energy minima. In the study of physics-based multi-time-scale dynamical systems, techniques have been developed for enhancing sampling in order to accelerate exploration beyond free energy barriers. On the other hand, in the field of Machine Learning, a generic goal of generative models is to sample from a target density, after training on empirical samples from this density. Score based generative models (SGMs) have demonstrated state-of-the-art capabilities in generating plausible data from target training distributions. Conditional implementations of such generative models have been shown to exhibit significant parallels with long-established -- and physics based -- solutions to enhanced sampling. These physics-based methods can then be enhanced through coupling with the ML generative models, complementing the strengths and mitigating the weaknesses of each technique. In this work, we show that that SGMs can be used in such a coupling framework to improve sampling in multiscale dynamical systems., Comment: 20 pages, 9 figures

We study the tipping point collective dynamics of an adaptive susceptible-infected-susceptible (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a parameter-dependent effective stochastic differential equation (eSDE) in terms of physically meaningful coarse mean-field variables through a deep-learning ResNet architecture inspired by numerical stochastic integrators. We construct an approximate effective bifurcation diagram based on the identified drift term of the eSDE and contrast it with the mean-field SIS model bifurcation diagram. We observe a subcritical Hopf bifurcation in the evolving network's effective SIS dynamics, that causes the tipping point behavior; this takes the form of large amplitude collective oscillations that spontaneously -- yet rarely -- arise from the neighborhood of a (noisy) stationary state. We study the statistics of these rare events both through repeated brute force simulations and by using established mathematical/computational tools exploiting the right-hand-side of the identified SDE. We demonstrate that such a collective SDE can also be identified (and the rare events computations also performed) in terms of data-driven coarse observables, obtained here via manifold learning techniques, in particular Diffusion Maps. The workflow of our study is straightforwardly applicable to other complex dynamics problems exhibiting tipping point dynamics., Comment: 22 pages, 12 figures

Deriving closed-form, analytical expressions for reduced-order models, and judiciously choosing the closures leading to them, has long been the strategy of choice for studying phase- and noise-induced transitions for agent-based models (ABMs). In this paper, we propose a data-driven framework that pinpoints phase transitions for an ABM- the Desai-Zwanzig model in its mean-field limit, using a smaller number of variables than traditional closed-form models. To this end, we use the manifold learning algorithm Diffusion Maps to identify a parsimonious set of data-driven latent variables, and show that they are in one-to-one correspondence with the expected theoretical order parameter of the ABM. We then utilize a deep learning framework to obtain a conformal reparametrization of the data-driven coordinates that facilitates, in our example, the identification of a single parameter-dependent ODE in these coordinates. We identify this ODE through a residual neural network inspired by a numerical integration scheme (forward Euler). We then use the identified ODE - enabled through an odd symmetry transformation - to construct the bifurcation diagram exhibiting the phase transition., Comment: 13 pages, 10 Figures

This study presents a collection of purely data-driven workflows for constructing reduced-order models (ROMs) for distributed dynamical systems. The ROMs we focus on, are data-assisted models inspired by, and templated upon, the theory of Approximate Inertial Manifolds (AIMs); the particular motivation is the so-called post-processing Galerkin method of Garcia-Archilla, Novo and Titi. Its applicability can be extended: the need for accurate truncated Galerkin projections and for deriving closed-formed corrections can be circumvented using machine learning tools. When the right latent variables are not a priori known, we illustrate how autoencoders as well as Diffusion Maps (a manifold learning scheme) can be used to discover good sets of latent variables and test their explainability. The proposed methodology can express the ROMs in terms of (a) theoretical (Fourier coefficients), (b) linear data-driven (POD modes) and/or (c) nonlinear data-driven (Diffusion Maps) coordinates. Both Black-Box and (theoretically-informed and data-corrected) Gray-Box models are described; the necessity for the latter arises when truncated Galerkin projections are so inaccurate as to not be amenable to post-processing. We use the Chafee-Infante reaction-diffusion and the Kuramoto-Sivashinsky dissipative partial differential equations to illustrate and successfully test the overall framework., Comment: 27 pages, 22 figures

This chapter discusses the development and implementation of algorithms based on Equation-Free/Dynamic Data Driven Applications Systems (EF/DDDAS) protocols for the computer-assisted study of the bifurcation structure of complex dynamical systems, such as those that arise in biology (neuronal networks, cell populations), multiscale systems in physics, chemistry and engineering, and system modeling in the social sciences. An illustrative example demonstrates the experimental realization of a chain of granular particles (a so-called engineered granular chain). In particular, the focus is on the detection/stability analysis of time-periodic, spatially localized structures referred to as "dark breathers". Results in this chapter highlight, both experimentally and numerically, that the number of breathers can be controlled by varying the frequency as well as the amplitude of an "out of phase" actuation, and that a "snaking" structure in the bifurcation diagram (computed through standard, model-based numerical methods for dynamical systems) is also recovered through the EF/DDDAS methods operating on a black-box simulator. The EF/DDDAS protocols presented here are, therefore, a step towards general purpose protocols for performing detailed bifurcation analyses directly on laboratory experiments, not only on their mathematical models, but also on measured data., Comment: Accepted for publication as a chapter in the Handbook of Dynamic Data Driven Applications Systems

We present a machine learning (ML)-assisted framework bridging manifold learning, neural networks, Gaussian processes, and Equation-Free multiscale modeling, for (a) detecting tipping points in the emergent behavior of complex systems, and (b) characterizing probabilities of rare events (here, catastrophic shifts) near them. Our illustrative example is an event-driven, stochastic agent-based model (ABM) describing the mimetic behavior of traders in a simple financial market. Given high-dimensional spatiotemporal data -- generated by the stochastic ABM -- we construct reduced-order models for the emergent dynamics at different scales: (a) mesoscopic Integro-Partial Differential Equations (IPDEs); and (b) mean-field-type Stochastic Differential Equations (SDEs) embedded in a low-dimensional latent space, targeted to the neighborhood of the tipping point. We contrast the uses of the different models and the effort involved in learning them., Comment: 29 pages, 8 figures, 6 tables

Fed-batch culture is an established operation mode for the production of biologics using mammalian cell cultures. Quantitative modeling integrates both kinetics for some key reaction steps and optimization-driven metabolic flux allocation, using flux balance analysis; this is known to lead to certain mathematical inconsistencies. Here, we propose a physically-informed data-driven hybrid model (a "gray box") to learn models of the dynamical evolution of Chinese Hamster Ovary (CHO) cell bioreactors from process data. The approach incorporates physical laws (e.g. mass balances) as well as kinetic expressions for metabolic fluxes. Machine learning (ML) is then used to (a) directly learn evolution equations (black-box modelling); (b) recover unknown physical parameters ("white-box" parameter fitting) or -- importantly -- (c) learn partially unknown kinetic expressions (gray-box modelling). We encode the convex optimization step of the overdetermined metabolic biophysical system as a differentiable, feed-forward layer into our architectures, connecting partial physical knowledge with data-driven machine learning.

We present a physics-informed machine learning (PIML) scheme for the feedback linearization of nonlinear discrete-time dynamical systems. The PIML finds the nonlinear transformation law, thus ensuring stability via pole placement, in one step. In order to facilitate convergence in the presence of steep gradients in the nonlinear transformation law, we address a greedy-wise training procedure. We assess the performance of the proposed PIML approach via a benchmark nonlinear discrete map for which the feedback linearization transformation law can be derived analytically; the example is characterized by steep gradients, due to the presence of singularities, in the domain of interest. We show that the proposed PIML outperforms, in terms of numerical approximation accuracy, the traditional numerical implementation, which involves the construction--and the solution in terms of the coefficients of a power-series expansion--of a system of homological equations as well as the implementation of the PIML in the entire domain, thus highlighting the importance of continuation techniques in the training procedure of PIML.

Origami structures often serve as the building block of mechanical systems due to their rich static and dynamic behaviors. Experimental observation and theoretical modeling of origami dynamics have been reported extensively, whereas the data-driven modeling of origami dynamics is still challenging due to the intrinsic nonlinearity of the system. In this study, we show how the dynamic mode decomposition (DMD) method can be enhanced by integrating geometry information of the origami structure to model origami dynamics in an efficient and accurate manner. In particular, an improved version of DMD with control, that we term geometry-informed dynamic mode decomposition~(giDMD), is developed and evaluated on the origami chain and dual Kresling origami structure to reveal the efficacy and interpretability. We show that giDMD can accurately predict the dynamics of an origami chain across frequencies, where the topological boundary state can be identified by the characteristics of giDMD. Moreover, the periodic intrawell motion can be accurately predicted in the dual origami structure. The type of dynamics in the dual origami structure can also be identified. The model learned by the giDMD also reveals the influential geometrical parameters in the origami dynamics, indicating the interpretability of this method. The accurate prediction of chaotic dynamics remains a challenge for the method. Nevertheless, we expect that the proposed giDMD approach will be helpful towards the prediction and identification of dynamics in complex origami structures, while paving the way to the application to a wider variety of lightweight and deployable structures., Comment: 11 pages, 7 figures

Study of the nonlinear evolution deep neural network (DNN) parameters undergo during training has uncovered regimes of distinct dynamical behavior. While a detailed understanding of these phenomena has the potential to advance improvements in training efficiency and robustness, the lack of methods for identifying when DNN models have equivalent dynamics limits the insight that can be gained from prior work. Topological conjugacy, a notion from dynamical systems theory, provides a precise definition of dynamical equivalence, offering a possible route to address this need. However, topological conjugacies have historically been challenging to compute. By leveraging advances in Koopman operator theory, we develop a framework for identifying conjugate and non-conjugate training dynamics. To validate our approach, we demonstrate that it can correctly identify a known equivalence between online mirror descent and online gradient descent. We then utilize it to: identify non-conjugate training dynamics between shallow and wide fully connected neural networks; characterize the early phase of training dynamics in convolutional neural networks; uncover non-conjugate training dynamics in Transformers that do and do not undergo grokking. Our results, across a range of DNN architectures, illustrate the flexibility of our framework and highlight its potential for shedding new light on training dynamics., Comment: 18 pages, 6 figures, 3 supplemental figures

Finding saddle points of dynamical systems is an important problem in practical applications such as the study of rare events of molecular systems. Gentlest ascent dynamics (GAD) is one of a number of algorithms in existence that attempt to find saddle points in dynamical systems. It works by deriving a new dynamical system in which saddle points of the original system become stable equilibria. GAD has been recently generalized to the study of dynamical systems on manifolds (differential algebraic equations) described by equality constraints and given in an extrinsic formulation. In this paper, we present an extension of GAD to manifolds defined by point-clouds, formulated using the intrinsic viewpoint. These point-clouds are adaptively sampled during an iterative process that drives the system from the initial conformation (typically in the neighborhood of a stable equilibrium) to a saddle point. Our method requires the reactant (initial conformation), does not require the explicit constraint equations to be specified, and is purely data-driven.

In the present work, we study coherent structures in a one-dimensional discrete nonlinear Schr\"odinger lattice in which the coupling between waveguides is periodically modulated. Numerical experiments with single-site initial conditions show that, depending on the power, the system exhibits two fundamentally different behaviors. At low power, initial conditions with intensity concentrated in a single site give rise to transport, with the energy moving unidirectionally along the lattice, whereas high power initial conditions yield stationary solutions. We explain these two behaviors, as well as the nature of the transition between the two regimes, by analyzing a simpler model where the couplings between waveguides are given by step functions. In this case, we numerically construct both stationary and moving coherent structures, which are solutions reproducing themselves exactly after an integer multiple of the coupling period. For the stationary solutions, which are true periodic orbits, we use Floquet analysis to determine the parameter regime for which they are spectrally stable. Typically, the traveling solutions are characterized by having small-amplitude, oscillatory tails, although we identify a set of parameters for which these tails disappear. These parameters turn out to be independent of the lattice size, and our simulations suggest that for these parameters, numerically exact traveling solutions are stable., Comment: 14 pages, 21 figures

Many multiscale wave systems exhibit macroscale emergent behaviour, for example, the fluid dynamics of floods and tsunamis. Resolving a large range of spatial scales typically requires a prohibitively high computational cost. The small dissipation in wave systems poses a significant challenge to further developing multiscale modelling methods in multiple dimensions. This article develops and evaluates two families of equation-free multiscale methods on novel 2D staggered patch schemes, and demonstrates the power and utility of these multiscale schemes for weakly damped linear waves. A detailed study of sensitivity to numerical roundoff errors establishes the robustness of developed staggered patch schemes. Comprehensive eigenvalue analysis over a wide range of parameters establishes the stability, accuracy, and consistency of the multiscale schemes. Analysis of the computational complexity shows that the measured compute times of the multiscale schemes may be 10^5 times smaller than the compute time for the corresponding full-domain computation. This work provides the essential foundation for efficient large-scale simulation of challenging nonlinear multiscale waves., Comment: 35 pages, 12 figures, and 6 tables

Iterative algorithms are of utmost importance in decision and control. With an ever growing number of algorithms being developed, distributed, and proprietarized, there is a similarly growing need for methods that can provide classification and comparison. By viewing iterative algorithms as discrete-time dynamical systems, we leverage Koopman operator theory to identify (semi-)conjugacies between algorithms using their spectral properties. This provides a general framework with which to classify and compare algorithms., Comment: 6 pages, 5 figures, accepted to IEEE CDC 2022

E. coli chemotactic motion in the presence of a chemoattractant field has been extensively studied using wet laboratory experiments, stochastic computational models as well as partial differential equation-based models (PDEs). The most challenging step in bridging these approaches, is establishing a closed form of the so-called chemotactic term, which describes how bacteria bias their motion up chemonutrient concentration gradients, as a result of a cascade of biochemical processes. Data-driven models can be used to learn the entire evolution operator of the chemotactic PDEs (black box models), or, in a more targeted fashion, to learn just the chemotactic term (gray box models). In this work, data-driven Machine Learning approaches for learning the underlying model PDEs are (a) validated through the use of simulation data from established continuum models and (b) used to infer chemotactic PDEs from experimental data. Even when the data at hand are sparse (coarse in space and/or time), noisy (due to inherent stochasticity in measurements) or partial (e.g. lack of measurements of the associated chemoattractant field), we can attempt to learn the right-hand-side of a closed PDE for an evolving bacterial density. In fact we show that data-driven PDEs including a short history of the bacterial density field (e.g. in the form of higher-order in time PDEs in terms of the measurable bacterial density) can be successful in predicting further bacterial density evolution, and even possibly recovering estimates of the unmeasured chemonutrient field. The main tool in this effort is the effective low-dimensionality of the dynamics (in the spirit of the Whitney and Takens embedding theorems). The resulting data-driven PDE can then be simulated to reproduce/predict computational or experimental bacterial density profile data, and estimate the underlying (unmeasured) chemonutrient field evolution., Comment: 44 pages, 20 figures, 3 tables

Circadian rhythmicity lies at the center of various important physiological and behavioral processes in mammals, such as sleep, metabolism, homeostasis, mood changes and more. It has been shown that this rhythm arises from self-sustained biomolecular oscillations of a neuronal network located in the Suprachiasmatic Nucleus (SCN). Under normal circumstances, this network remains synchronized to the day-night cycle due to signaling from the retina. Misalignment of these neuronal oscillations with the external light signal can disrupt numerous physiological functions and take a long-lasting toll on health and well-being. In this work, we study a modern computational neuroscience model to determine the limits of circadian synchronization to external light signals of different frequency and duty cycle. We employ a matrix-free approach to locate periodic steady states of the high-dimensional model for various driving conditions. Our algorithmic pipeline enables numerical continuation and construction of bifurcation diagrams w.r.t. forcing parameters. We computationally explore the effect of heterogeneity in the circadian neuronal network, as well as the effect of corrective therapeutic interventions, such as that of the drug molecule Longdaysin. Lastly, we employ unsupervised learning to construct a data-driven embedding space for representing neuronal heterogeneity., Comment: 14 pages, 15 figures

Sampling the phase space of molecular systems -- and, more generally, of complex systems effectively modeled by stochastic differential equations -- is a crucial modeling step in many fields, from protein folding to materials discovery. These problems are often multiscale in nature: they can be described in terms of low-dimensional effective free energy surfaces parametrized by a small number of "slow" reaction coordinates; the remaining "fast" degrees of freedom populate an equilibrium measure on the reaction coordinate values. Sampling procedures for such problems are used to estimate effective free energy differences as well as ensemble averages with respect to the conditional equilibrium distributions; these latter averages lead to closures for effective reduced dynamic models. Over the years, enhanced sampling techniques coupled with molecular simulation have been developed. An intriguing analogy arises with the field of Machine Learning (ML), where Generative Adversarial Networks can produce high dimensional samples from low dimensional probability distributions. This sample generation returns plausible high dimensional space realizations of a model state, from information about its low-dimensional representation. In this work, we present an approach that couples physics-based simulations and biasing methods for sampling conditional distributions with ML-based conditional generative adversarial networks for the same task. The "coarse descriptors" on which we condition the fine scale realizations can either be known a priori, or learned through nonlinear dimensionality reduction. We suggest that this may bring out the best features of both approaches: we demonstrate that a framework that couples cGANs with physics-based enhanced sampling techniques can improve multiscale SDE dynamical systems sampling, and even shows promise for systems of increasing complexity., Comment: 26 pages, 14 figures, 3 tables

We report the computational discovery of complex, topologically charged, and spectrally stable states in three-dimensional multi-component nonlinear wave systems of nonlinear Schr{\"o}dinger type. While our computations relate to two-component atomic Bose-Einstein condensates in parabolic traps, our methods can be broadly applied to high-dimensional, nonlinear systems of partial differential equations. The combination of the so-called deflation technique with a careful selection of initial guesses enables the computation of an unprecedented breadth of patterns, including ones combining vortex lines, rings, stars, and ``vortex labyrinths''. Despite their complexity, they may be dynamically robust and amenable to experimental observation, as confirmed by Bogolyubov-de Gennes spectral analysis and numerical evolution simulations., Comment: 8 pages, 5 figures

Numerical schemes for wave-like systems with small dissipation are often inaccurate and unstable due to truncation errors and numerical roundoff errors. Hence, numerical simulations of wave-like systems lacking proper handling of these numerical issues often fail to represent the physical characteristics of wave phenomena. This challenge gets even more intricate for multiscale modelling, especially in multiple dimensions. When using the usual collocated grid, about two-thirds of the resolved wave modes are incorrect with significant dispersion. But, numerical schemes on staggered grids (with alternating variable arrangement) are significantly less dispersive and preserve much of the wave characteristics. Also, the group velocity of the energy propagation in the numerical waves on a staggered grid is in the correct direction, in contrast to the collocated grid. For high accuracy and to preserve much of the wave characteristics, this article extends the concept of staggered grids in full-domain modelling to multidimensional multiscale modelling. Specifically, this article develops 120 multiscale staggered grids and demonstrates their stability, accuracy, and wave-preserving characteristic for equation-free multiscale modelling of weakly damped linear waves. But most characteristics of the developed multiscale staggered grids must also hold in general for multiscale modelling of many complex spatio-temporal physical phenomena such as the general computational fluid dynamics., Comment: 42 pages, 13 figures, and 1 table

We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modelled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models.~The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning (in particular, non-linear manifold learning (Diffusion Maps (DMs)) helps identify a set of coarse-grained variables that parametrize the low-dimensional manifold on which the emergent/collective dynamics evolve. The out-of-sample extension and pre-image problems, i.e. the construction of non-linear mappings from the high-dimensional input space to the low-dimensional manifold and back, are solved by coupling DMs with the Nystrom extension and Geometric Harmonics, respectively; (B) having identified the manifold and its coordinates, we exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics; then (C) based on the previous steps, we design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states, thus demonstrating that the scheme is robust against numerical approximation errors and modelling uncertainty.~The efficiency of the framework is illustrated by controlling emergent unstable (i) traveling waves of a deterministic agent-based model of traffic dynamics, and (ii) equilibria of a stochastic financial market agent model with mimesis.

We present a data-driven approach to learning surrogate models for amplitude equations, and illustrate its application to interfacial dynamics of phase field systems. In particular, we demonstrate learning effective partial differential equations describing the evolution of phase field interfaces from full phase field data. We illustrate this on a model phase field system, where analytical approximate equations for the dynamics of the phase field interface (a higher order eikonal equation and its approximation, the Kardar-Parisi-Zhang (KPZ) equation) are known. For this system, we discuss data-driven approaches for the identification of equations that accurately describe the front interface dynamics. When the analytical approximate models mentioned above become inaccurate, as we move beyond the region of validity of the underlying assumptions, the data-driven equations outperform them. In these regimes, going beyond black-box identification, we explore different approaches to learn data-driven corrections to the analytically approximate models, leading to effective gray box partial differential equations.

We propose a machine learning framework for the data-driven discovery of macroscopic chemotactic Partial Differential Equations (PDEs) -- and the closures that lead to them -- from high-fidelity, individual-based stochastic simulations of E.coli bacterial motility. The fine scale, detailed, hybrid (continuum - Monte Carlo) simulation model embodies the underlying biophysics, and its parameters are informed from experimental observations of individual cells. We exploit Automatic Relevance Determination (ARD) within a Gaussian Process framework for the identification of a parsimonious set of collective observables that parametrize the law of the effective PDEs. Using these observables, in a second step we learn effective, coarse-grained "Keller-Segel class" chemotactic PDEs using machine learning regressors: (a) (shallow) feedforward neural networks and (b) Gaussian Processes. The learned laws can be black-box (when no prior knowledge about the PDE law structure is assumed) or gray-box when parts of the equation (e.g. the pure diffusion part) is known and "hardwired" in the regression process. We also discuss data-driven corrections (both additive and functional) of analytically known, approximate closures., Comment: 33 pages, 5 figures, 1 table

We construct a reduced, data-driven, parameter dependent effective Stochastic Differential Equation (eSDE) for electric-field mediated colloidal crystallization using data obtained from Brownian Dynamics Simulations. We use Diffusion Maps (a manifold learning algorithm) to identify a set of useful latent observables. In this latent space we identify an eSDE using a deep learning architecture inspired by numerical stochastic integrators and compare it with the traditional Kramers-Moyal expansion estimation. We show that the obtained variables and the learned dynamics accurately encode the physics of the Brownian Dynamic Simulations. We further illustrate that our reduced model captures the dynamics of corresponding experimental data. Our dimension reduction/reduced model identification approach can be easily ported to a broad class of particle systems dynamics experiments/models., Comment: 21 pages, 16 figures, 2 tables

We introduce a data-driven approach to building reduced dynamical models through manifold learning; the reduced latent space is discovered using Diffusion Maps (a manifold learning technique) on time series data. A second round of Diffusion Maps on those latent coordinates allows the approximation of the reduced dynamical models. This second round enables mapping the latent space coordinates back to the full ambient space (what is called lifting); it also enables the approximation of full state functions of interest in terms of the reduced coordinates. In our work, we develop and test three different reduced numerical simulation methodologies, either through pre-tabulation in the latent space and integration on the fly or by going back and forth between the ambient space and the latent space. The data-driven latent space simulation results, based on the three different approaches, are validated through (a) the latent space observation of the full simulation through the Nystr\"om Extension formula, or through (b) lifting the reduced trajectory back to the full ambient space, via Latent Harmonics. Latent space modeling often involves additional regularization to favor certain properties of the space over others, and the mapping back to the ambient space is then constructed mostly independently from these properties; here, we use the same data-driven approach to construct the latent space and then map back to the ambient space., Comment: 25 pages,21 figures, 4 tables

Starting with sets of disorganized observations of spatially varying and temporally evolving systems, obtained at different (also disorganized) sets of parameters, we demonstrate the data-driven derivation of parameter dependent, evolutionary partial differential equation (PDE) models capable of generating the data. This tensor type of data is reminiscent of shuffled (multi-dimensional) puzzle tiles. The independent variables for the evolution equations (their "space" and "time") as well as their effective parameters are all "emergent", i.e., determined in a data-driven way from our disorganized observations of behavior in them. We use a diffusion map based "questionnaire" approach to build a parametrization of our emergent space/time/parameter space for the data. This approach iteratively processes the data by successively observing them on the "space", the "time", and the "parameter" axes of a tensor. Once the data are organized, we use machine learning (here, neural networks) to approximate the operators governing the evolution equations in this emergent space. Our illustrative example is based on a previously developed vertex-plus-signaling model of Drosophila embryonic development. This allows us to discuss features of the process like symmetry breaking, translational invariance, and autonomousness of the emergent PDE model, as well as its interpretability., Comment: 9 main pages (8 figures) with 7 SI pages (5 figures) Submitted to PNAS Nexus

We introduce a method to successively locate equilibria (steady states) of dynamical systems on Riemannian manifolds. The manifolds need not be characterized by an a priori known atlas or by the zeros of a smooth map. Instead, they can be defined by point-clouds and sampled as needed through an iterative process. If the manifold is an Euclidean space, our method follows isoclines, curves along which the direction of the vector field $X$ is constant. For a generic vector field $X$, isoclines are smooth curves and every equilibrium lies on isoclines. We generalize the definition of isoclines to Riemannian manifolds through the use of parallel transport: generalized isoclines are curves along which the directions of $X$ are parallel transports of each other. As in the Euclidean case, generalized isoclines of generic vector fields $X$ are smooth curves that connect equilibria of $X$. Our algorithm can be regarded as an extension of the method of Newton trajectories to the manifold setting when the manifold is unknown. This work is motivated by computational statistical mechanics, specifically high dimensional (stochastic) differential equations that model the dynamics of molecular systems. Often, these dynamics concentrate near low-dimensional manifolds and have transitions (saddle points with a single unstable direction) between metastable equilibria. We employ iteratively sampled data and isoclines to locate these saddle points. Coupling a black-box sampling scheme (e.g., Markov chain Monte Carlo) with manifold learning techniques (diffusion maps in the case presented here), we show that our method reliably locates equilibria of $X$.

In the present work, we revisit a recently proposed and experimentally realized topological 2D lattice with periodically time-dependent interactions. We identify the fundamental solitons, previously observed in experiments and direct numerical simulations, as exact, exponentially localized, periodic in time solutions. This is done for a variety of phase-shift angles of the central nodes upon a period oscillation of the coupling strength. Subsequently, we perform a systematic Floquet stability analysis of the relevant structures. We analyze both their point and their continuous spectrum and find that the solutions are generically stable, aside from the possible emergence of complex quartets due to the collision of bands of continuous spectrum. The relevant instabilities become weaker as the lattice size gets larger. Finally, we also consider multi-soliton analogues of these Floquet states, inspired by the corresponding discrete nonlinear Schr\"odinger (DNLS) lattice. When exciting initially multiple sites in phase, we find that the solutions reflect the instability of their DNLS multi-soliton counterparts, while for configurations with multiple excited sites in alternating phases, the Floquet states are spectrally stable, again in analogy to their DNLS counterparts., Comment: 9 pages, 14 figures

The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the na\"ive approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and find that they can be used to shed light on magnitude pruning's performance during the early part of training., Comment: 14 pages, 5 figures

We present a data-driven approach to characterizing nonidentifiability of a model's parameters and illustrate it through dynamic as well as steady kinetic models. By employing Diffusion Maps and their extensions, we discover the minimal combinations of parameters required to characterize the output behavior of a chemical system: a set of effective parameters for the model. Furthermore, we introduce and use a Conformal Autoencoder Neural Network technique, as well as a kernel-based Jointly Smooth Function technique, to disentangle the redundant parameter combinations that do not affect the output behavior from the ones that do. We discuss the interpretability of our data-driven effective parameters, and demonstrate the utility of the approach both for behavior prediction and parameter estimation. In the latter task, it becomes important to describe level sets in parameter space that are consistent with a particular output behavior. We validate our approach on a model of multisite phosphorylation, where a reduced set of effective parameters (nonlinear combinations of the physical ones) has previously been established analytically., Comment: 47 pages, 23 figures, 4 tables, submitted to PNAS Nexus, revised and expanded in response to reviewers' comments

The Equation-Free approach to efficient multiscale numerical computation marries trusted micro-scale simulations to a framework for numerical macro-scale reduction -- the patch dynamics scheme. A recent novel patch scheme empowered the Equation-Free approach to simulate systems containing shocks on the macro-scale. However, the scheme did not predict the formation of shocks accurately, and it could not simulate moving shocks. This article resolves both issues, as a first step in one spatial dimension, by embedding the Equation-Free, shock-resolving patch scheme within a classic framework for adaptive moving meshes. Our canonical micro-scale problems exhibit heterogeneous nonlinear advection and heterogeneous diffusion. We demonstrate many remarkable benefits from the moving patch scheme, including efficient and accurate macro-scale prediction despite the unknown macro-scale closure. Equation-free methods are here extended to simulate moving, forming and merging shocks without a priori knowledge of the existence or closure of the shocks. Whereas adaptive moving mesh equations are typically stiff, typically requiring small time-steps on the macro-scale, the moving macro-scale mesh of patches is typically not stiff given the context of the micro-scale time-steps required for the sub-patch dynamics., Comment: 35 A5 pages, 12 figures, submitted to SISC

In this work, we propose a method to learn multivariate probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker-Planck equations). We analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker-Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a multivariate normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can approximate probability density function evolutions in time from observed sampled data for systems driven by both Brownian and L\'evy noise. We present examples with two- and three-dimensional, uni- and multimodal distributions to validate the method.

We consider the existence and spectral stability of static multi-kink structures in the discrete sine-Gordon equation, as a representative example of the family of discrete Klein-Gordon models. The multi-kinks are constructed using Lin's method from an alternating sequence of well-separated kink and antikink solutions. We then locate the point spectrum associated with these multi-kink solutions by reducing the spectral problem to a matrix equation. For an $m$-structure multi-kink, there will be $m$ eigenvalues in the point spectrum near each eigenvalue of the primary kink, and, as long as the spectrum of the primary kink is imaginary, the spectrum of the multi-kink will be as well. We obtain analytic expressions for the eigenvalues of a multi-kink in terms of the eigenvalues and corresponding eigenfunctions of the primary kink, and these are in very good agreement with numerical results. We also perform numerical time-stepping experiments on perturbations of multi-kinks, and the outcomes of these simulations are interpreted using the spectral results., Comment: 23 pages, 11 figures

We study the learning of numerical algorithms for scientific computing, which combines mathematically driven, handcrafted design of general algorithm structure with a data-driven adaptation to specific classes of tasks. This represents a departure from the classical approaches in numerical analysis, which typically do not feature such learning-based adaptations. As a case study, we develop a machine learning approach that automatically learns effective solvers for initial value problems in the form of ordinary differential equations (ODEs), based on the Runge-Kutta (RK) integrator architecture. We show that we can learn high-order integrators for targeted families of differential equations without the need for computing integrator coefficients by hand. Moreover, we demonstrate that in certain cases we can obtain superior performance to classical RK methods. This can be attributed to certain properties of the ODE families being identified and exploited by the approach. Overall, this work demonstrates an effective learning-based approach to the design of algorithms for the numerical solution of differential equations. This can be readily extended to other numerical tasks.

In the present work we revisit the $b$-family model of peakon equations, containing as special cases the $b=2$ (Camassa-Holm) and $b=3$ (Degasperis-Procesi) integrable examples. We establish information about the point spectrum of the peakon solutions and notably find that for suitably smooth perturbations there exists point spectrum in the right half plane rendering the peakons unstable for $b<1$. We explore numerically these ideas in the realm of fixed-point iterations, spectral stability analysis and time-stepping of the model for the different parameter regimes. In particular, we identify exact, stationary (spectrally stable) lefton solutions for $b<-1$, and for $-11$. While many of the above dynamical features had been explored in earlier studies, in the present work, we supplement them, wherever possible, with spectral stability computations., Comment: 33 pages, 10 figures

Equation-free macroscale modelling is a systematic and rigorous computational methodology for efficiently predicting the dynamics of a microscale system at a desired macroscale system level. In this scheme, the given microscale model is computed in small patches spread across the space-time domain, with patch coupling conditions bridging the unsimulated space. For accurate simulations, care must be taken in designing the patch coupling conditions. Here we construct novel coupling conditions which preserve translational invariance, rotational invariance, and self-adjoint symmetry, thus guaranteeing that conservation laws associated with these symmetries are preserved in the macroscale simulation. Spectral and algebraic analyses of the proposed scheme in both one and two dimensions reveal mechanisms for further improving the accuracy of the simulations. Consistency of the patch scheme's macroscale dynamics with the original microscale model is proved. This new self-adjoint patch scheme provides an efficient, flexible, and accurate computational homogenisation in a wide range of multiscale scenarios of interest to scientists and engineers.

We present previously unknown solutions to the 3D Gross--Pitaevskii equation describing atomic Bose-Einstein condensates. This model supports elaborate patterns, including excited states bearing vorticity. The discovered coherent structures exhibit striking topological features, involving combinations of vortex rings and multiple, possibly bent vortex lines. Although unstable, many of them persist for long times in dynamical simulations. These solutions were identified by a state-of-the-art numerical technique called deflation, which is expected to be applicable to many problems from other areas of physics., Comment: 9 pages, 11 figures

Scientists and engineers often create accurate, trustworthy, computational simulation schemes - but all too often these are too computationally expensive to execute over the time or spatial domain of interest. The equation-free approach is to marry such trusted simulations to a framework for numerical macroscale reduction - the patch dynamics scheme. This article extends the patch scheme to scenarios in which the trusted simulation resolves abrupt state changes on the microscale that appear as shocks on the macroscale. Accurate simulation for problems in these scenarios requires extending the patch scheme by capturing the shock within a novel patch, and also modifying the patch coupling rules in the vicinity in order to maintain accuracy. With these two extensions to the patch scheme, straightforward arguments derive consistency conditions that match the usual order of accuracy for patch schemes. The new scheme is successfully tested on four archetypal problems. This technique will empower scientists and engineers to accurately and efficiently simulate, over large spatial domains, multiscale multiphysics systems that have rapid transition layers on the microscale., Comment: 21 pages, 9 figures

The multiscale patch scheme is built from given small micro-scale simulations of complicated physical processes to empower large macro-scale simulations. By coupling small patches of simulations over unsimulated spatial gaps, large savings in computational time are possible. Here we discuss generalising the patch scheme to the case of wave systems on staggered grids in 2D space. Classic macro-scale interpolation provides a generic coupling between patches that achieves arbitrarily high order consistency between the emergent macro-scale simulation and the underlying micro-scale dynamics. Eigen-analysis indicates that the resultant scheme empowers feasible computation of large macro-scale simulations of wave systems even with complicated underlying physics. As examples we use the scheme to simulate some wave scenarios via a turbulent shallow water model.

Recently, a novel bifurcation technique known as the deflated continuation method (DCM) was applied to the single-component nonlinear Schr\"odinger (NLS) equation with a parabolic trap in two spatial dimensions. The bifurcation analysis carried out by a subset of the present authors shed light on the configuration space of solutions of this fundamental problem in the physics of ultracold atoms. In the present work, we take this a step further by applying the DCM to two coupled NLS equations in order to elucidate the considerably more complex landscape of solutions of this system. Upon identifying branches of solutions, we construct the relevant bifurcation diagrams and perform spectral stability analysis to identify parametric regimes of stability and instability and to understand the mechanisms by which these branches emerge. The method reveals a remarkable wealth of solutions: these do not only include some of the well-known ones including, e.g., from the Cartesian or polar small amplitude limits of the underlying linear problem but also a significant number of branches that arise through (typically pitchfork) bifurcations. In addition to presenting a ``cartography'' of the landscape of solutions, we comment on the challenging task of identifying {\it all} solutions of such a high-dimensional, nonlinear problem., Comment: 27 pages, 19 figures

In the present work, we consider the existence and spectral stability of multi-pulse solutions in Hamiltonian lattice systems. We provide a general framework for the study of such wave patterns based on a discrete analogue of Lin's method, previously used in the continuum realm. We develop explicit conditions for the existence of $m$-pulse structures and subsequently develop a reduced matrix allowing us to address their spectral stability. As a prototypical example for the manifestation of the details of the formulation, we consider the discrete nonlinear Schr\"{o}dinger equation. Different families of $2$- and $3$-pulse solitary waves are discussed, and analytical expressions for the corresponding stability eigenvalues are obtained which are in very good agreement with numerical results., Comment: 42 pages, 4 figures

A systematic mathematical framework for the study of numerical algorithms would allow comparisons, facilitate conjugacy arguments, as well as enable the discovery of improved, accelerated, data-driven algorithms. Over the course of the last century, the Koopman operator has provided a mathematical framework for the study of dynamical systems, which facilitates conjugacy arguments and can provide efficient reduced descriptions. More recently, numerical approximations of the operator have enabled the analysis of a large number of deterministic and stochastic dynamical systems in a completely data-driven, essentially equation-free pipeline. Discrete or continuous time numerical algorithms (integrators, nonlinear equation solvers, optimization algorithms) are themselves dynamical systems. In this paper, we use this insight to leverage the Koopman operator framework in the data-driven study of such algorithms and discuss benefits for analysis and acceleration of numerical computation. For algorithms acting on high-dimensional spaces by quickly contracting them towards low-dimensional manifolds, we demonstrate how basis functions adapted to the data help to construct efficient reduced representations of the operator. Our illustrative examples include the gradient descent and Nesterov optimization algorithms, as well as the Newton-Raphson algorithm., Comment: 27 pages, 11 figures

Large scale dynamical systems (e.g. many nonlinear coupled differential equations) can often be summarized in terms of only a few state variables (a few equations), a trait that reduces complexity and facilitates exploration of behavioral aspects of otherwise intractable models. High model dimensionality and complexity makes symbolic, pen--and--paper model reduction tedious and impractical, a difficulty addressed by recently developed frameworks that computerize reduction. Symbolic work has the benefit, however, of identifying both reduced state variables and parameter combinations that matter most (effective parameters, "inputs"); whereas current computational reduction schemes leave the parameter reduction aspect mostly unaddressed. As the interest in mapping out and optimizing complex input--output relations keeps growing, it becomes clear that combating the curse of dimensionality also requires efficient schemes for input space exploration and reduction. Here, we explore systematic, data-driven parameter reduction by means of effective parameter identification, starting from current nonlinear manifold-learning techniques enabling state space reduction. Our approach aspires to extend the data-driven determination of effective state variables with the data-driven discovery of effective model parameters, and thus to accelerate the exploration of high-dimensional parameter spaces associated with complex models.

We study quasiperiodicity-induced localization of waves in strongly precompressed granular chains. We propose three different setups, inspired by the Aubry--Andr\'e (AA) model, of quasiperiodic chains; and we use these models to compare the effects of on-site and off-site quasiperiodicity in nonlinear lattices. When there is purely on-site quasiperiodicity, which we implement in two different ways, we show for a chain of spherical particles that there is a localization transition (as in the original AA model). However, we observe no localization transition in a chain of cylindrical particles in which we incorporate quasiperiodicity in the distribution of contact angles between adjacent cylinders by making the angle periodicity incommensurate with that of the chain. For each of our three models, we compute the Hofstadter spectrum and the associated Minkowski--Bouligand fractal dimension, and we demonstrate that the fractal dimension decreases as one approaches the localization transition (when it exists). Finally, in a suite of numerical computations, we demonstrate both localization and that there exist regimes of ballistic, superdiffusive, diffusive, and subdiffusive transport. Our models provide a flexible set of systems to study quasiperiodicity-induced analogs of Anderson phenomena in granular chains that one can tune controllably from weakly to strongly nonlinear regimes., Comment: 16 pages

We study - experimentally, theoretically, and numerically - nonlinear excitations in lattices of magnets with long-range interactions. We examine breather solutions, which are spatially localized and periodic in time, in a chain with algebraically-decaying interactions. It was established two decades ago [S. Flach, Phys. Rev. E 58, R4116 (1998)] that lattices with long-range interactions can have breather solutions in which the spatial decay of the tails has a crossover from exponential to algebraic decay. In this Letter, we revisit this problem in the setting of a chain of repelling magnets with a mass defect and verify, both numerically and experimentally, the existence of breathers with such a crossover.