Surrogate dropout: Learning optimal drop rate through proxy.

Dropout is commonly used in deep neural networks to alleviate the problem of overfitting. Conventionally, the neurons in a layer indiscriminately share a fixed drop probability, which results in difficulty in determining the appropriate value for different tasks. Moreover, this static strategy will also incur serious degradation on performance when the conventional dropout is extensively applied to both shallow and deep layers. A question is whether selectively dropping the neurons would realize a better regularization effect. This paper proposes a simple and effective surrogate dropout method whereby neurons are dropped according to their importance. The proposed method has two main stages. The first stage trains a surrogate module that can be jointly optimized along with the neural network to evaluate the importance of each neuron. In the second stage, the output of the surrogate module is regarded as a guidance signal for dropping certain neurons, approximating the optimal per-neuron drop rate when the network converges. Various convolutional neural network architectures and multiple datasets, including CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and two medical image datasets are used to evaluate the surrogate dropout method. The experimental results demonstrate that the proposed method achieves a better regularization effect than the baseline methods. • A simple and effective regularization method called surrogate dropout is proposed, which regards the surrogate module as a proxy for approximating the optimal drop rate of each neuron. • Compared with conventional dropout, the surrogate dropout method has fewer restrictions. Both the shallow and deep layers in CNNs can benefit from the usage of surrogate dropout. • The superior regularization effect of surrogate dropout has been empirically verified using multiple datasets and networks with various depths. [ABSTRACT FROM AUTHOR]