Search

This paper presents the Standalone Neural ODE (sNODE), a continuous-depth neural ODE model capable of describing a full deep neural network. This uses a novel nonlinear conjugate gradient (NCG) descent optimization scheme for training, where the Sobolev gradient can be incorporated to improve smoothness of model weights. We also present a general formulation of the neural sensitivity problem and show how it is used in the NCG training. The sensitivity analysis provides a reliable measure of uncertainty propagation throughout a network, and can be used to study model robustness and to generate adversarial attacks. Our evaluations demonstrate that our novel formulations lead to increased robustness and performance as compared to ResNet models, and that it opens up for new opportunities for designing and developing machine learning with improved explainability., Comment: 25 pages, 15 figures; typos corrected

The inverse problem of supervised reconstruction of depth-variable (time-dependent) parameters in a neural ordinary differential equation (NODE) is considered, that means finding the weights of a residual network with time continuous layers. The NODE is treated as an isolated entity describing the full network as opposed to earlier research, which embedded it between pre- and post-appended layers trained by conventional methods. The proposed parameter reconstruction is done for a general first order differential equation by minimizing a cost functional covering a variety of loss functions and penalty terms. A nonlinear conjugate gradient method (NCG) is derived for the minimization. Mathematical properties are stated for the differential equation and the cost functional. The adjoint problem needed is derived together with a sensitivity problem. The sensitivity problem can estimate changes in the network output under perturbation of the trained parameters. To preserve smoothness during the iterations the Sobolev gradient is calculated and incorporated. As a proof-of-concept, numerical results are included for a NODE and two synthetic datasets, and compared with standard gradient approaches (not based on NODEs). The results show that the proposed method works well for deep learning with infinite numbers of layers, and has built-in stability and smoothness., Comment: 26 pages, 3 figures

We define the space of functions of bounded variation ($BV$) on the graph. Using the notion of divergence of flows on graphs, we show that the unit ball of the dual space to $BV$ in the graph setting can be described as the image of the unit ball of the space $\ell^{\infty}$ by the divergence operator. Based on this result, we propose a new iterative algorithm to find the exact minimizer for the total variation (TV) denoising problem on the graph. The proposed algorithm is provable convergent and its performance on image denoising examples is compared with the Split Bregman and Primal-Dual algorithms as benchmarks for iterative methods and with BM3D as a benchmark for other state-of-the-art denoising methods. The experimental results show highly competitive empirical convergence rate and visual quality for the proposed algorithm.

In this paper, we introduce a new image denoising model: the damped flow (DF), which is a second order nonlinear evolution equation associated with a class of energy functionals of image. The existence, uniqueness and regularization property of DF are proven. For the numerical implementation, based on the St\"{o}rmer-Verlet method, a discrete damped flow, SV-DDF, is developed. The convergence of SV-DDF is studied as well. Several numerical experiments, as well as a comparison with other methods, are provided to demonstrate the feasibility and effectiveness of the SV-DDF.