Search

We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective. In particular, we establish general sufficient conditions for universal approximation using continuoustime deep residual networks, which can also be understood as approximation theories in Lp using flow maps of dynamical systems. In specific cases, rates of approximation in terms of the time horizon are also established. Overall, these results reveal that composition function approximation through flow maps presents a new paradigm in approximation theory and contributes to building a useful mathematical framework to investigate deep learning. [ABSTRACT FROM AUTHOR]

Despite the wide applications of neural networks, there have been increasing concerns about their vulnerability issue. While numerous attack and defense techniques have been developed, this work investigates the robustness issue from a new angle: can we design a self-healing neural network that can automatically detect and fix the vulnerability issue by itself? A typical self-healing mechanism is the immune system of a human body. This biology-inspired idea has been used in many engineering designs but has rarely been investigated in deep learning. This paper considers the post-training self-healing of a neural network, and proposes a closed-loop control formulation to automatically detect and fix the errors caused by various attacks or perturbations. We provide a margin-based analysis to explain how this formulation can improve the robustness of a classifier. To speed up the inference, we convert the optimal control problem to Pontryagon's Maximum Principle and solve it via the method of successive approximation. Lastly, we present an error estimation of the proposed framework for neural networks with nonlinear activation functions. We validate the performance of several network architectures against various perturbations. Since the self-healing method does not need a-priori information about data perturbations or attacks, it can handle a broad class of unforeseen perturbations. [ABSTRACT FROM AUTHOR]

We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent (SGD), momentum SGD and stochastic Nesterov's accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover important analytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting. [ABSTRACT FROM AUTHOR]

The continuous dynamical system approach to deep learning is explored in order to devise alternative frameworks for training algorithms. Training is recast as a control problem and this allows us to formulate necessary optimality conditions in continuous time using the Pontryagin's maximum principle (PMP). A modification of the method of successive approximations is then used to solve the PMP, giving rise to an alternative training algorithm for deep learning. This approach has the advantage that rigorous error estimates and convergence results can be established. We also show that it may avoid some pitfalls of gradient-based methods, such as slow convergence on at landscapes near saddle points. Furthermore, we demonstrate that it obtains favorable initial convergence rate periteration, provided Hamiltonian maximization can be efficiently carried out - a step which is still in need of improvement. Overall, the approach opens up new avenues to attack problems associated with deep learning, such as trapping in slow manifolds and inapplicability of gradient-based methods for discrete trainable variables. [ABSTRACT FROM AUTHOR]

Mice that lacked manganese-superoxide dismutase (Mn-SOD) activity exhibited the typical pathology of dilated cardiomyopathy (DCM). The aim of the present study was to investigate the effect of supplementation with omega-3 polyunsaturated fatty acids (n-3 PUFA) on heart function and oxidative stress biomarkers in mice with DCM. In the present study, heart/muscle-specific Mn-SOD-deficient mice (H/M-Sod2-/-) were treated with n-3 PUFA (30 mg/kg/day) for 10 weeks, and the reactive oxygen species (ROS) production in their heart mitochondria and cardiac function was subsequently assessed. n-3 PUFA treatment diminished ROS production and suppressed the progression of cardiac dysfunction. Furthermore, n-3 PUFA treatment effectively reversed the cardiac dysfunction and dilatation observed in symptomatic H/M-Sod2-/- mice. Notably, n-3 PUFA treatment ameliorated a molecular defect in connexin 43. Hematoxylin-eosin staining indicated that the phenotype of DCM was also ameliorated following n-3 PUFA treatment. Furthermore, echocardiography demonstrated that cardiac function was significantly improved in the mice treated with n-3 PUFA (P<0.05). Meanwhile, pre-treatment with n-3 PUFA significantly decreased cardiomyocyte apoptosis (P<0.001). In conclusion, n-3 PUFA treatment is able to prevent murine DCM, primarily by reducing ROS production and improving myocardial apoptosis. Therefore, the impairment of ROS production is proposed as a potential therapy for DCM. [ABSTRACT FROM AUTHOR]

Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD)51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems. [ABSTRACT FROM AUTHOR]

The stability of the plane Poiseuille flow is analyzed using a thermodynamic formalism by considering the deterministic Navier-Stokes equation with Gaussian random initial data. A unique critical Reynolds number, Rec ≈2,332, at which the probability of observing puffs in the solution changes from 0 to 1, is numerically demonstrated to exist in the thermodynamic limit and is found to be independent of the noise amplitude. Using the puff density as the macrostate variable, the free energy of such a system is computed and analyzed. The puff density approaches zero as the critical Reynolds number is approached from above, signaling a continuous transition despite the fact that the bifurcation is subcritical for a finite-sized system. An action function is found for the probability of observing puffs in a small subregion of the flow, and this action function depends only on the Reynolds number. The strategy used here should be applicable to a wide range of other problems exhibiting subcritical instabilities. [ABSTRACT FROM AUTHOR]

Metalearning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the metalearning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta neural network to a recursively recurrent neural network superstructure for the design of customized iterative algorithms. In contrast to off-the-shelf deep learning approaches, it features a distinct division into modules for generation of information and for the subsequent assembly of this information towards a solution. Local information in the form of a subspace is generated by subordinate, inner, iterations of recurrent function evaluations starting at the current outer iterate. The update to the next outer iterate is computed as a linear combination of these evaluations, reducing the residual in this space, and constitutes the output of the network. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields iterations similar to Krylov solvers for linear equation systems, Newton--Krylov solvers for nonlinear equation systems, and Runge-Kutta integrators for ordinary differential equations. Due to its modularity, the superstructure can be readily extended with functionalities needed to represent more general classes of iterative algorithms traditionally based on Taylor series expansions. [ABSTRACT FROM AUTHOR]

We study the learning of numerical algorithms for scientific computing, which combines mathematically driven, handcrafted design of general algorithm structure with a data-driven adaptation to specific classes of tasks. This represents a departure from the classical approaches in numerical analysis, which typically do not feature such learning-based adaptations. As a case study, we develop a machine learning approach that automatically learns effective solvers for initial value problems in the form of ordinary differential equations (ODEs), based on the Runge--Kutta (RK) integrator architecture. We show that we can learn high-order integrators for targeted families of differential equations without the need for computing integrator coefficients by hand. Moreover, we demonstrate that in certain cases we can obtain superior performance to classical RK methods. This can be attributed to certain properties of the ODE families being identified and exploited by the approach. Overall, this work demonstrates an effective learning-based approach to the design of algorithms for the numerical solution of differential equations. This can be readily extended to other numerical tasks. [ABSTRACT FROM AUTHOR]

We perform a systematic study of the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider the simple but representative setting of using continuous-time linear RNNs to learn from data generated by linear relationships. On the approximation side, we prove a direct and an inverse approximation theorem of linear functionals using RNNs, which reveal the intricate connections between memory structures in the target and the corresponding approximation effciency. In particular, we show that temporal relationships can be effectively approximated by RNNs if and only if the former possesses suffcient memory decay. On the optimization front, we perform detailed analysis of the optimization dynamics, including a precise understanding of the diffculty that may arise in learning relationships with long-term memory. The term "curse of memory" is coined to describe the uncovered phenomena, akin to the "curse of dimension" that plagues highdimensional function approximation. These results form a relatively complete picture of the interaction of memory and recurrent structures in the linear dynamical setting. [ABSTRACT FROM AUTHOR]