Back to Search
Start Over
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
- Publication Year :
- 2021
- Publisher :
- arXiv, 2021.
-
Abstract
- Deep neural networks are notorious for defying theoretical treatment. However, when the number of parameters in each layer tends to infinity the network function is a Gaussian process (GP) and quantitatively predictive description is possible. Gaussian approximation allows to formulate criteria for selecting hyperparameters, such as variances of weights and biases, as well as the learning rate. These criteria rely on the notion of criticality defined for deep neural networks. In this work we describe a new practical way to diagnose criticality. We introduce \emph{partial Jacobians} of a network, defined as derivatives of preactivations in layer $l$ with respect to preactivations in layer $l_0\leq l$. We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections. We derive and implement a simple and cheap numerical test that allows to select optimal initialization for a broad class of deep neural networks. Using these tools we show quantitatively that proper stacking of the LayerNorm (applied to preactivations) and residual connections leads to an architecture that is critical for any initialization. Finally, we apply our methods to analyze the MLP-Mixer architecture and show that it is everywhere critical.<br />Comment: 43 pages, 9 figures. Added residual connections and MLP-Mixer
- Subjects :
- High Energy Physics - Theory
FOS: Computer and information sciences
Computer Science - Machine Learning
High Energy Physics - Theory (hep-th)
Statistics - Machine Learning
FOS: Physical sciences
Machine Learning (stat.ML)
Disordered Systems and Neural Networks (cond-mat.dis-nn)
Condensed Matter - Disordered Systems and Neural Networks
Machine Learning (cs.LG)
Subjects
Details
- Database :
- OpenAIRE
- Accession number :
- edsair.doi.dedup.....2a63a94ef3e4626836ca9c84c16a415b
- Full Text :
- https://doi.org/10.48550/arxiv.2111.12143