Search

We introduce a new training algorithm for variety of deep neural networks that utilize random complex exponential activation functions. Our approach employs a Markov Chain Monte Carlo sampling procedure to iteratively train network layers, avoiding global and gradient-based optimization while maintaining error control. It consistently attains the theoretical approximation rate for residual networks with complex exponential activation functions, determined by network complexity. Additionally, it enables efficient learning of multiscale and high-frequency features, producing interpretable parameter distributions. Despite using sinusoidal basis functions, we do not observe Gibbs phenomena in approximating discontinuous target functions.

In this work, we consider the approximation of a large class of bounded functions, with minimal regularity assumptions, by ReLU neural networks. We show that the approximation error can be bounded from above by a quantity proportional to the uniform norm of the target function and inversely proportional to the product of network width and depth. We inherit this approximation error bound from Fourier features residual networks, a type of neural network that uses complex exponential activation functions. Our proof is constructive and proceeds by conducting a careful complexity analysis associated with the approximation of a Fourier features residual network by a ReLU network.

In this work, we consider the general problem of constructing a neural network surrogate model using multi-fidelity information. Motivated by rigorous error and complexity estimates for ReLU neural networks, given an inexpensive low-fidelity and an expensive high-fidelity computational model, we present a residual multi-fidelity computational framework that formulates the correlation between models as a residual function, a possibly non-linear mapping between 1) the shared input space of the models together with the low-fidelity model output and 2) the discrepancy between the two model outputs. To accomplish this, we train two neural networks to work in concert. The first network learns the residual function on a small set of high-fidelity and low-fidelity data. Once trained, this network is used to generate additional synthetic high-fidelity data, which is used in the training of a second network. This second network, once trained, acts as our surrogate for the high-fidelity quantity of interest. We present three numerical examples to demonstrate the power of the proposed framework. In particular, we show that dramatic savings in computational cost may be achieved when the output predictions are desired to be accurate within small tolerances.

We present a Bayesian quantum characterization framework that takes into account uncertainties from both experimental data and the dynamical modeling of the experiment. We demonstrate that ignoring dynamical model uncertainty may lead to predictions that are inconsistent with the uncertainty present in quantum data. A Bayesian framework is proposed and applied to characterize a tantalum-based transmon device at Lawrence Livermore National Laboratory. We demonstrate how modeling uncertainty can be represented by a Gaussian process within the Bayesian framework, and how this approach improves the predictive capability of Lindblad's master equations applied to the Ramsey measurement protocol., Comment: 23 pages, 4 figures

Motivated by the noisy and fluctuating behavior of current quantum computing devices, this paper presents a data-driven characterization approach for estimating transition frequencies and decay times in a Lindbladian dynamical model of a superconducting quantum device. The data includes parity events in the transition frequency between the first and second excited states. A simple but effective mathematical model, based upon averaging solutions of two Lindbladian models, is demonstrated to accurately capture the experimental observations. A deterministic point estimate of the device parameters is first performed to minimize the misfit between data and Lindbladian simulations. These estimates are used to make an informed choice of prior distributions for the subsequent Bayesian inference. An additive Gaussian noise model is developed for the likelihood function, which includes two hyper-parameters to capture the noise structure of the data. The outcome of the Bayesian inference are posterior probability distributions of the transition frequencies, which for example can be utilized to design risk neutral optimal control pulses. The applicability of our approach is demonstrated on experimental data from the Quantum Device and Integration Testbed (QuDIT) at Lawrence Livermore National Laboratory, using a tantalum-based superconducting transmon device., Comment: 20 pages, 13 figures

The goal of this survey is to present an explanatory review of the approximation properties of deep neural networks. Specifically, we aim at understanding how and why deep neural networks outperform other classical linear and nonlinear approximation methods. This survey consists of three chapters. In Chapter 1 we review the key ideas and concepts underlying deep networks and their compositional nonlinear structure. We formalize the neural network problem by formulating it as an optimization problem when solving regression and classification problems. We briefly discuss the stochastic gradient descent algorithm and the back-propagation formulas used in solving the optimization problem and address a few issues related to the performance of neural networks, including the choice of activation functions, cost functions, overfitting issues, and regularization. In Chapter 2 we shift our focus to the approximation theory of neural networks. We start with an introduction to the concept of density in polynomial approximation and in particular study the Stone-Weierstrass theorem for real-valued continuous functions. Then, within the framework of linear approximation, we review a few classical results on the density and convergence rate of feedforward networks, followed by more recent developments on the complexity of deep networks in approximating Sobolev functions. In Chapter 3, utilizing nonlinear approximation theory, we further elaborate on the power of depth and approximation superiority of deep ReLU networks over other classical methods of nonlinear approximation., Comment: 82 pages, 31 figures

Sinkhorn divergence is a measure of dissimilarity between two probability measures. It is obtained through adding an entropic regularization term to Kantorovich's optimal transport problem and can hence be viewed as an entropically regularized Wasserstein distance. Given two discrete probability vectors in the $n$-simplex and supported on two bounded spaces in ${\mathbb R}^d$, we present a fast method for computing Sinkhorn divergence when the cost matrix can be decomposed into a $d$-term sum of asymptotically smooth Kronecker product factors. The method combines Sinkhorn's matrix scaling iteration with a low-rank hierarchical representation of the scaling matrices to achieve a near-linear complexity ${\mathcal O}(n \log^3 n)$. This provides a fast and easy-to-implement algorithm for computing Sinkhorn divergence, enabling its applicability to large-scale optimization problems, where the computation of classical Wasserstein metric is not feasible. We present a numerical example related to signal processing to demonstrate the applicability of quadratic Sinkhorn divergence in comparison with quadratic Wasserstein distance and to verify the accuracy and efficiency of the proposed method., Comment: 23 pages

We propose a multi-fidelity neural network surrogate sampling method for the uncertainty quantification of physical/biological systems described by ordinary or partial differential equations. We first generate a set of low/high-fidelity data by low/high-fidelity computational models, e.g. using coarser/finer discretizations of the governing differential equations. We then construct a two-level neural network, where a large set of low-fidelity data are utilized in order to accelerate the construction of a high-fidelity surrogate model with a small set of high-fidelity data. We then embed the constructed high-fidelity surrogate model in the framework of Monte Carlo sampling. The proposed algorithm combines the approximation power of neural networks with the advantages of Monte Carlo sampling within a multi-fidelity framework. We present two numerical examples to demonstrate the accuracy and efficiency of the proposed method. We show that dramatic savings in computational cost may be achieved when the output predictions are desired to be accurate within small tolerances., Comment: 23 pages, 13 figures

We present a Bayesian framework based on a new exponential likelihood function driven by the quadratic Wasserstien metric. Compared to conventional Bayesian models based on Gaussian likelihood functions driven by the least-squares norm ($L_2$ norm), the new framework features several advantages. First, the new framework does not rely on the likelihood of the measurement noise and hence can treat complicated noise structures such as combined additive and multiplicative noise. Secondly, unlike the normal likelihood function, the Wasserstein-based exponential likelihood function does not usually generate multiple local extrema. As a result, the new framework features better convergence to correct posteriors when a Markov Chain Monte Carlo sampling algorithm is employed. Thirdly, in the particular case of signal processing problems, while a normal likelihood function measures only the amplitude differences between the observed and simulated signals, the new likelihood function can capture both the amplitude and the phase differences. We apply the new framework to a class of signal processing problems, that is, the inverse uncertainty quantification of waveforms, and demonstrate its advantages compared to Bayesian models with normal likelihood functions., Comment: 24 pages

We introduce and study a new class of partial differential equations (PDEs) with hybrid fuzzy-stochastic parameters, coined fuzzy-stochastic PDEs. Compared to purely stochastic PDEs or purely fuzzy PDEs, fuzzy-stochastic PDEs offer powerful models for accurate representation and propagation of hybrid aleatoric-epistemic uncertainties inevitable in many real-world problems. We will use the level-set representation of fuzzy functions and define the solution to fuzzy-stochastic PDE problems through a corresponding parametric problem, and further present theoretical results on the well-posedness and regularity of such problems. We also propose a numerical strategy for computing output fuzzy-stochastic quantities, such as fuzzy failure probabilities and fuzzy probability distributions. We present two numerical examples to compute various fuzzy-stochastic quantities and to demonstrate the applicability of fuzzy-stochastic PDEs to complex engineering problems., Comment: 31 pages

We study mathematical and computational models for computing the deformation of fiber-reinforced cross-plied laminates due to external forces. This requires an understanding of both micro-structural effects and different sources of uncertainty in the problem. We first show that the uncertainties in the problem are of both statistical (aleatoric) and systematic (epistemic) types and that current multiscale stochastic models, such as stationary random fields, which are based on precise probability theory, are not capable of correctly characterizing uncertainty in fiber composites. Next, we motivate the applicability of models based on imprecise uncertainty theory and present a novel fuzzy-stochastic model, which can more accurately describe uncertainties in fiber composites. The new model is constructed by combining stochastic fields and fuzzy variables through a simple calibration-validation approach. Finally, we construct a global-local multiscale algorithm for efficiently computing output quantities of interest. The method aims at approximating required quantities, such as displacements and stresses, in regions of relatively small size, e.g. hot spots or zones. The algorithm uses the concept of representative volume elements and computes a global solution to construct a local approximation that captures the microscale features of the solution. The results are based on and backed by real experimental data.

We consider the wave equation with highly oscillatory initial data, where there is uncertainty in the wave speed, initial phase and/or initial amplitude. To estimate quantities of interest related to the solution and their statistics, we combine a high-frequency method based on Gaussian beams with sparse stochastic collocation. Although the wave solution, $u^\varepsilon$, is highly oscillatory in both physical and stochastic spaces, we provide theoretical arguments and numerical evidence that quantities of interest based on local averages of $|u^\varepsilon|^2$ are smooth, with derivatives in the stochastic space uniformly bounded in $\varepsilon$, where $\varepsilon$ denotes the short wavelength. This observable related regularity makes the sparse stochastic collocation approach more efficient than Monte Carlo methods. We present numerical tests that demonstrate this advantage.

We develop a fast method for optimally designing experiments in the context of statistical seismic source inversion. In particular, we efficiently compute the optimal number and locations of the receivers or seismographs. The seismic source is modeled by a point moment tensor multiplied by a time-dependent function. The parameters include the source location, moment tensor components, and start time and frequency in the time function. The forward problem is modeled by elastodynamic wave equations. We show that the Hessian of the cost functional, which is usually defined as the square of the weighted L2 norm of the difference between the experimental data and the simulated data, is proportional to the measurement time and the number of receivers. Consequently, the posterior distribution of the parameters, in a Bayesian setting, concentrates around the "true" parameters, and we can employ Laplace approximation and speed up the estimation of the expected Kullback-Leibler divergence (expected information gain), the optimality criterion in the experimental design procedure. Since the source parameters span several magnitudes, we use a scaling matrix for efficient control of the condition number of the original Hessian matrix. We use a second-order accurate finite difference method to compute the Hessian matrix and either sparse quadrature or Monte Carlo sampling to carry out numerical integration. We demonstrate the efficiency, accuracy, and applicability of our method on a two-dimensional seismic source inversion problem.

The Gaussian beam superposition method is an asymptotic method for computing high frequency wave fields in smoothly varying inhomogeneous media. In this paper we study the accuracy of the Gaussian beam superposition method and derive error estimates related to the discretization of the superposition integral and the Taylor expansion of the phase and amplitude off the center of the beam. We show that in the case of odd order beams, the error is smaller than a simple analysis would indicate because of error cancellation effects between the beams. Since the cancellation happens only when odd order beams are used, there is no remarkable gain in using even order beams. Moreover, applying the error estimate to the problem with constant speed of propagation, we show that in this case the local beam width is not a good indicator of accuracy, and there is no direct relation between the error and the beam width. We present numerical examples to verify the error estimates.

In the harmonic description of general relativity, the principle part of Einstein's equations reduces to 10 curved space wave equations for the componenets of the space-time metric. We present theorems regarding the stability of several evolution-boundary algorithms for such equations when treated in second order differential form. The theorems apply to a model black hole space-time consisting of a spacelike inner boundary excising the singularity, a timelike outer boundary and a horizon in between. These algorithms are implemented as stable, convergent numerical codes and their performance is compared in a 2-dimensional excision problem., Comment: 19 pages, 9 figures