Search

Transformations are a key tool in the qualitative study of dynamical systems: transformations to a normal form, for example, underpin the study of instabilities and bifurcations. In this work, we test, and when possible establish, an equivalence between two different artificial neural networks by attempting to construct a data-driven transformation between them, using diffusion maps with a Mahalanobis-like metric. If the construction succeeds, the two networks can be thought of as belonging to the same equivalence class. We first discuss transformation functions between only the outputs of the two networks; we then also consider transformations that take into account outputs (activations) of a number of internal neurons from each network. Whitney's theorem dictates the number of (generic) measurements from one of the networks required to reconstruct each and every feature of the second network. The construction of the transformation function relies on a consistent, intrinsic representation of the network input space. We illustrate our algorithm by matching neural network pairs trained to learn (a) observations of scalar functions, (b) observations of two-dimensional vector fields, and (c) representations of images of a moving three-dimensional object (a rotating horse). We also demonstrate reconstruction of a network's input (and output) from minimal partial observations of intermediate neuron activations. The construction of equivalences across different network instantiations clearly relates to transfer learning and will also be valuable in establishing equivalence between different machine learning-based tools., (© 2024 Author(s). Published under an exclusive license by AIP Publishing.)

We study the tipping point collective dynamics of an adaptive susceptible-infected-susceptible (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a parameter-dependent effective stochastic differential equation (eSDE) in terms of physically meaningful coarse mean-field variables through a deep-learning ResNet architecture inspired by numerical stochastic integrators. We construct an approximate effective bifurcation diagram based on the identified drift term of the eSDE and contrast it with the mean-field SIS model bifurcation diagram. We observe a subcritical Hopf bifurcation in the evolving network's effective SIS dynamics that causes the tipping point behavior; this takes the form of large amplitude collective oscillations that spontaneously-yet rarely-arise from the neighborhood of a (noisy) stationary state. We study the statistics of these rare events both through repeated brute force simulations and by using established mathematical/computational tools exploiting the right-hand side of the identified SDE. We demonstrate that such a collective SDE can also be identified (and the rare event computations also performed) in terms of data-driven coarse observables, obtained here via manifold learning techniques, in particular, Diffusion Maps. The workflow of our study is straightforwardly applicable to other complex dynamic problems exhibiting tipping point dynamics., (© 2024 Author(s). Published under an exclusive license by AIP Publishing.)

A valuable step in the modeling of multiscale dynamical systems in fields such as computational chemistry, biology, and materials science is the representative sampling of the phase space over long time scales of interest; this task is not, however, without challenges. For example, the long term behavior of a system with many degrees of freedom often cannot be efficiently computationally explored by direct dynamical simulation; such systems can often become trapped in local free energy minima. In the study of physics-based multi-time-scale dynamical systems, techniques have been developed for enhancing sampling in order to accelerate exploration beyond free energy barriers. On the other hand, in the field of machine learning (ML), a generic goal of generative models is to sample from a target density, after training on empirical samples from this density. Score-based generative models (SGMs) have demonstrated state-of-the-art capabilities in generating plausible data from target training distributions. Conditional implementations of such generative models have been shown to exhibit significant parallels with long-established-and physics-based-solutions to enhanced sampling. These physics-based methods can then be enhanced through coupling with the ML generative models, complementing the strengths and mitigating the weaknesses of each technique. In this work, we show that SGMs can be used in such a coupling framework to improve sampling in multiscale dynamical systems., (© 2024 Author(s). Published under an exclusive license by AIP Publishing.)

We identify effective stochastic differential equations (SDEs) for coarse observables of fine-grained particle- or agent-based simulations; these SDEs then provide useful coarse surrogate models of the fine scale dynamics. We approximate the drift and diffusivity functions in these effective SDEs through neural networks, which can be thought of as effective stochastic ResNets. The loss function is inspired by, and embodies, the structure of established stochastic numerical integrators (here, Euler-Maruyama and Milstein); our approximations can thus benefit from backward error analysis of these underlying numerical schemes. They also lend themselves naturally to "physics-informed" gray-box identification when approximate coarse models, such as mean field equations, are available. Existing numerical integration schemes for Langevin-type equations and for stochastic partial differential equations can also be used for training; we demonstrate this on a stochastically forced oscillator and the stochastic wave equation. Our approach does not require long trajectories, works on scattered snapshot data, and is designed to naturally handle different time steps per snapshot. We consider both the case where the coarse collective observables are known in advance, as well as the case where they must be found in a data-driven manner.

Circadian rhythmicity lies at the center of various important physiological and behavioral processes in mammals, such as sleep, metabolism, homeostasis, mood changes, and more. Misalignment of intrinsic neuronal oscillations with the external day-night cycle can disrupt such processes and lead to numerous disorders. In this work, we computationally determine the limits of circadian synchronization to external light signals of different frequency, duty cycle, and simulated amplitude. Instead of modeling circadian dynamics with generic oscillator models (e.g., Kuramoto-type), we use a detailed computational neuroscience model, which integrates biomolecular dynamics, neuronal electrophysiology, and network effects. This allows us to investigate the effect of small drug molecules, such as Longdaysin, and connect our results with experimental findings. To combat the high dimensionality of such a detailed model, we employ a matrix-free approach, while our entire algorithmic pipeline enables numerical continuation and construction of bifurcation diagrams using only direct simulation. We, thus, computationally explore the effect of heterogeneity in the circadian neuronal network, as well as the effect of the corrective therapeutic intervention of Longdaysin. Last, we employ unsupervised learning to construct a data-driven embedding space for representing neuronal heterogeneity.

We address a three-tier numerical framework based on nonlinear manifold learning for the forecasting of high-dimensional time series, relaxing the "curse of dimensionality" related to the training phase of surrogate/machine learning models. At the first step, we embed the high-dimensional time series into a reduced low-dimensional space using nonlinear manifold learning (local linear embedding and parsimonious diffusion maps). Then, we construct reduced-order surrogate models on the manifold (here, for our illustrations, we used multivariate autoregressive and Gaussian process regression models) to forecast the embedded dynamics. Finally, we solve the pre-image problem, thus lifting the embedded time series back to the original high-dimensional space using radial basis function interpolation and geometric harmonics. The proposed numerical data-driven scheme can also be applied as a reduced-order model procedure for the numerical solution/propagation of the (transient) dynamics of partial differential equations (PDEs). We assess the performance of the proposed scheme via three different families of problems: (a) the forecasting of synthetic time series generated by three simplistic linear and weakly nonlinear stochastic models resembling electroencephalography signals, (b) the prediction/propagation of the solution profiles of a linear parabolic PDE and the Brusselator model (a set of two nonlinear parabolic PDEs), and (c) the forecasting of a real-world data set containing daily time series of ten key foreign exchange rates spanning the time period 3 September 2001-29 October 2020.

In this work, we propose a method to learn multivariate probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker-Planck equations). We analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker-Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a multivariate normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can approximate probability density function evolutions in time from observed sampled data for systems driven by both Brownian and Lévy noise. We present examples with two- and three-dimensional, uni- and multimodal distributions to validate the method.

We present an approach, based on learning an intrinsic data manifold, for the initialization of the internal state values of long short-term memory (LSTM) recurrent neural networks, ensuring consistency with the initial observed input data. Exploiting the generalized synchronization concept, we argue that the converged, "mature" internal states constitute a function on this learned manifold. The dimension of this manifold then dictates the length of observed input time series data required for consistent initialization. We illustrate our approach through a partially observed chemical model system, where initializing the internal LSTM states in this fashion yields visibly improved performance. Finally, we show that learning this data manifold enables the transformation of partially observed dynamics into fully observed ones, facilitating alternative identification paths for nonlinear dynamical systems.

Large collections of coupled, heterogeneous agents can manifest complex dynamical behavior presenting difficulties for simulation and analysis. However, if the collective dynamics lie on a low-dimensional manifold, then the original agent-based model may be approximated with a simplified surrogate model on and near the low-dimensional space where the dynamics live. Analytically identifying such simplified models can be challenging or impossible, but here we present a data-driven coarse-graining methodology for discovering such reduced models. We consider two types of reduced models: globally based models that use global information and predict dynamics using information from the whole ensemble and locally based models that use local information, that is, information from just a subset of agents close (close in heterogeneity space, not physical space) to an agent, to predict the dynamics of an agent. For both approaches, we are able to learn laws governing the behavior of the reduced system on the low-dimensional manifold directly from time series of states from the agent-based system. These laws take the form of either a system of ordinary differential equations (ODEs), for the globally based approach, or a partial differential equation (PDE) in the locally based case. For each technique, we employ a specialized artificial neural network integrator that has been templated on an Euler time stepper (i.e., a ResNet) to learn the laws of the reduced model. As part of our methodology, we utilize the proper orthogonal decomposition (POD) to identify the low-dimensional space of the dynamics. Our globally based technique uses the resulting POD basis to define a set of coordinates for the agent states in this space and then seeks to learn the time evolution of these coordinates as a system of ODEs. For the locally based technique, we propose a methodology for learning a partial differential equation representation of the agents; the PDE law depends on the state variables and partial derivatives of the state variables with respect to model heterogeneities. We require that the state variables are smooth with respect to model heterogeneities, which permit us to cast the discrete agent-based problem as a continuous one in heterogeneity space. The agents in such a representation bear similarity to the discretization points used in typical finite element/volume methods. As an illustration of the efficacy of our techniques, we consider a simplified coupled neuron model for rhythmic oscillations in the pre-Bötzinger complex and demonstrate how our data-driven surrogate models are able to produce dynamics comparable to the dynamics of the full system. A nontrivial conclusion is that the dynamics can be equally well reproduced by an all-to-all coupled and by a locally coupled model of the same agents.

Equations governing physico-chemical processes are usually known at microscopic spatial scales, yet one suspects that there exist equations, e.g., in the form of partial differential equations (PDEs), that can explain the system evolution at much coarser, meso-, or macroscopic length scales. Discovering those coarse-grained effective PDEs can lead to considerable savings in computation-intensive tasks like prediction or control. We propose a framework combining artificial neural networks with multiscale computation, in the form of equation-free numerics, for the efficient discovery of such macro-scale PDEs directly from microscopic simulations. Gathering sufficient microscopic data for training neural networks can be computationally prohibitive; equation-free numerics enable a more parsimonious collection of training data by only operating in a sparse subset of the space-time domain. We also propose using a data-driven approach, based on manifold learning (including one using the notion of unnormalized optimal transport of distributions and one based on moment-based description of the distributions), to identify macro-scale dependent variable(s) suitable for the data-driven discovery of said PDEs. This approach can corroborate physically motivated candidate variables or introduce new data-driven variables, in terms of which the coarse-grained effective PDE can be formulated. We illustrate our approach by extracting coarse-grained evolution equations from particle-based simulations with a priori unknown macro-scale variable(s) while significantly reducing the requisite data collection computational effort.

Data mining is routinely used to organize ensembles of short temporal observations so as to reconstruct useful, low-dimensional realizations of an underlying dynamical system. In this paper, we use manifold learning to organize unstructured ensembles of observations ("trials") of a system's response surface. We have no control over where every trial starts, and during each trial, operating conditions are varied by turning "agnostic" knobs, which change system parameters in a systematic, but unknown way. As one (or more) knobs "turn," we record (possibly partial) observations of the system response. We demonstrate how such partial and disorganized observation ensembles can be integrated into coherent response surfaces whose dimension and parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through different types of observations. We demonstrate our approach by organizing unstructured observations of response surfaces, including the reconstruction of a cusp bifurcation surface for hydrogen combustion in a continuous stirred tank reactor. Finally, we demonstrate how this observation-based reconstruction naturally leads to informative transport maps between the input parameter space and output/state variable spaces.

Complex spatiotemporal dynamics of physicochemical processes are often modeled at a microscopic level (through, e.g., atomistic, agent-based, or lattice models) based on first principles. Some of these processes can also be successfully modeled at the macroscopic level using, e.g., partial differential equations (PDEs) describing the evolution of the right few macroscopic observables (e.g., concentration and momentum fields). Deriving good macroscopic descriptions (the so-called "closure problem") is often a time-consuming process requiring deep understanding/intuition about the system of interest. Recent developments in data science provide alternative ways to effectively extract/learn accurate macroscopic descriptions approximating the underlying microscopic observations. In this paper, we introduce a data-driven framework for the identification of unavailable coarse-scale PDEs from microscopic observations via machine-learning algorithms. Specifically, using Gaussian processes, artificial neural networks, and/or diffusion maps, the proposed framework uncovers the relation between the relevant macroscopic space fields and their time evolution (the right-hand side of the explicitly unavailable macroscopic PDE). Interestingly, several choices equally representative of the data can be discovered. The framework will be illustrated through the data-driven discovery of macroscopic, concentration-level PDEs resulting from a fine-scale, lattice Boltzmann level model of a reaction/transport process. Once the coarse evolution law is identified, it can be simulated to produce long-term macroscopic predictions. Different features (pros as well as cons) of alternative machine-learning algorithms for performing this task (Gaussian processes and artificial neural networks) are presented and discussed.

Concise, accurate descriptions of physical systems through their conserved quantities abound in the natural sciences. In data science, however, current research often focuses on regression problems, without routinely incorporating additional assumptions about the system that generated the data. Here, we propose to explore a particular type of underlying structure in the data: Hamiltonian systems, where an "energy" is conserved. Given a collection of observations of such a Hamiltonian system over time, we extract phase space coordinates and a Hamiltonian function of them that acts as the generator of the system dynamics. The approach employs an autoencoder neural network component to estimate the transformation from observations to the phase space of a Hamiltonian system. An additional neural network component is used to approximate the Hamiltonian function on this constructed space, and the two components are trained jointly. As an alternative approach, we also demonstrate the use of Gaussian processes for the estimation of such a Hamiltonian. After two illustrative examples, we extract an underlying phase space as well as the generating Hamiltonian from a collection of movies of a pendulum. The approach is fully data-driven and does not assume a particular form of the Hamiltonian function.

Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD) 51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems.

We propose to compute approximations to invariant sets in dynamical systems by minimizing an appropriate distance between a suitably selected finite set of points and its image under the dynamics. We demonstrate, through computational experiments, that this approach can successfully converge to approximations of (maximal) invariant sets of arbitrary topology, dimension, and stability, such as, e.g., saddle type invariant sets with complicated dynamics. We further propose to extend this approach by adding a Lennard-Jones type potential term to the objective function, which yields more evenly distributed approximating finite point sets, and illustrate the procedure through corresponding numerical experiments.

We present a universal characterization scheme for chimera states applicable to both numerical and experimental data sets. The scheme is based on two correlation measures that enable a meaningful definition of chimera states as well as their classification into three categories: stationary, turbulent, and breathing. In addition, these categories can be further subdivided according to the time-stationarity of these two measures. We demonstrate that this approach is both consistent with previously recognized chimera states and enables us to classify states as chimeras which have not been categorized as such before. Furthermore, the scheme allows for a qualitative and quantitative comparison of experimental chimeras with chimeras obtained through numerical simulations.

Kinetic Monte Carlo simulations are used to study the stochastic two-species Lotka-Volterra model on a square lattice. For certain values of the model parameters, the system constitutes an excitable medium: travelling pulses and rotating spiral waves can be excited. Stable solitary pulses travel with constant (modulo stochastic fluctuations) shape and speed along a periodic lattice. The spiral waves observed persist sometimes for hundreds of rotations, but they are ultimately unstable and break-up (because of fluctuations and interactions between neighboring fronts) giving rise to complex dynamic behavior in which numerous small spiral waves rotate and interact with each other. It is interesting that travelling pulses and spiral waves can be exhibited by the model even for completely immobile species, due to the non-local reaction kinetics.

We consider the simplest network of coupled non-identical phase oscillators capable of displaying a "chimera" state (namely, two subnetworks with strong coupling within the subnetworks and weaker coupling between them) and systematically investigate the effects of gradually removing connections within the network, in a random but systematically specified way. We average over ensembles of networks with the same random connectivity but different intrinsic oscillator frequencies and derive ordinary differential equations (ODEs), whose fixed points describe a typical chimera state in a representative network of phase oscillators. Following these fixed points as parameters are varied we find that chimera states are quite sensitive to such random removals of connections, and that oscillations of chimera states can be either created or suppressed in apparent bifurcation points, depending on exactly how the connections are gradually removed.

This paper focuses on subsurface oxygen and its influence on pattern formation during CO-oxidation on platinum surfaces. For the observation of spatiotemporal pattern formation during catalytic reactions the photoelectron emission microscope (PEEM) has proven to be an excellent real-time imaging instrument, capable of tracking local work function changes. The existence of subsurface oxygen on platinumlike surfaces has been extensively discussed and for palladium its presence has been clearly established during rate oscillations. Subsurface oxygen is defined at this point as an atomic O species located directly underneath the uppermost metal crystal layer; its dipole moment therefore considerably lowers the work function of the surface. Here we review some of the investigations involving subsurface oxygen, focusing on the role subsurface oxygen might play in pattern formation during CO-oxidation on platinum. We will also present some new results, where this species clearly interacts with chemisorbed oxygen under restrictions by boundary conditions on the Pt(110) single crystal. These previously (through microlithography) constructed domain boundaries on the surface are made out of Rh or Pd, and they are acting as an additional source of CO molecules for the Pt surface. (c) 2002 American Institute of Physics.

We first briefly review the subject of spatiotemporal pattern formation on microdesigned composite catalysts. One of the most significant interaction mechanisms between different reacting domains (consisting of different metal catalysts such as Pt and Rh, coupled through surface diffusion) is the initiation of reaction fronts at the interface between them. We then explore in some detail the effect of two-dimensional composite geometry on this basic building block of composite catalyst dynamics. (c) 2002 American Institute of Physics.

A simple system of five nonlinear ordinary differential equations is shown to reproduce many dynamical features of spiral waves in two-dimensional excitable media.