Search

• NNGPs establish the equivalence between Gaussian processes (GPs) and infinitely wide deep neural networks (NNs). • We consider the impact of noise regularisation (e.g. dropout) on NNGPs using signal propagation theory. • We find that the best NNGPs have kernels matching that of an optimal initialisation for noise regularised ReLU networks. • We also show how noise influences the NNGP's covariance matrix, resulting in simpler posterior functions. • We verify our theoretical findings with experiments on MNIST and CIFAR-10 and synthetic data. Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation (e.g. dropout) on NNGPs, and relate their behaviour to signal propagation theory in noise regularised deep neural networks. For ReLU activations, we find that the best performing NNGPs have kernel parameters that correspond to a recently proposed initialisation scheme for noise regularised ReLU networks. In addition, we show how the noise influences the covariance matrix of the NNGP, producing a stronger prior towards simple functions away from the training points. We verify our theoretical findings with experiments on MNIST and CIFAR-10 as well as on synthetic data. [ABSTRACT FROM AUTHOR]

• Recent work has shown that dropout limits the depth to which information can propagate through a neural network. • We investigate the effect of initialisation on training speed and generalisation within this depth limit. • We ask specifically: if dropout limits depth, does initialising critically still matter? • We conduct a large-scale controlled experiment and perform a statistical analysis of over 12 000 trained networks. • We show that at moderate depths, critical initialisation gives no performance gains over off-critical initialisations. Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over 12 000 trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour. [ABSTRACT FROM AUTHOR]