A new hybrid optimizer for stochastic optimization acceleration of deep neural networks: Dynamical system perspective.

Stochastic optimization acceleration is extremely significant and challenging for deep neural networks (DNNs). In recent years, several novel proportional-integral–differential-based (PID-based) optimizers have been proposed to speed up the optimization by alleviating the oscillation behavior of stochastic gradient descent with momentum (SGD-M), yet lacked theoretical analysis. Along this line of research, this paper adopts dynamical system theory to design a new hybrid optimizer and present theoretical analysis. Firstly, it is found that DNN optimization is equivalent to a discrete time dynamical system. Building upon the equivalence, high order augmented dynamical system viewpoint is utilized to design a PI-like optimizer for ensuring high accuracy, which is more stable than SGD-M. Then, hybrid dynamical system viewpoint is employed to improve the PI-like optimizer as a new hybrid form for suppressing oscillation and accelerating optimization. Lyapunov method, Taylor series, matrix theory and equilibrium are combined to theoretically investigate the convergence and the oscillation of loss function, showing that the proposed hybrid optimizer can alleviate oscillation, boost optimization speed, and maintain high accuracy. In theoretical analyses, explicit conditions of hyper-parameters that guarantee training stability are calculated and presented, practically guiding the adjustment of hyper-parameters and promoting the application of hybrid optimizer. Experiments are presented on three commonly used benchmark datasets, i.e., MNIST, CIFAR10 and CIFAR100, demonstrating that the hybrid optimizer obtains up to 42% acceleration with competitive accuracy relative to state-of-the-art optimizers. In short, this paper not only presents a new hybrid optimizer for accelerating optimization, but also provides a novel, theoretical and systematic perspective to find and analyze new optimizer for DNNs. [ABSTRACT FROM AUTHOR]