Search

It is an important problem to design resource-efficient neural architectures. One solution is adjusting the number of channels in each layer and the number of blocks in each network stage. This paper presents a novel framework named network adjustment which considers accuracy as a function of the computational resource (e.g., FLOPs or parameters), so that architecture design becomes an optimization problem and can be solved with the gradient-based optimization method. The gradient is defined as the resource utilization ratio (RUR) of each changeable module (layer or block) in a network and is accurate only in a small neighborhood of the current status. Therefore, we estimate it using Dropout, a probabilistic operation, and optimize the network architecture iteratively. The computational overhead of the entire process is comparable to that of re-training the final model from scratch. We investigate two versions of RUR where the resource usage is measured by FLOPs and latency. Experiments on standard image classification datasets and a few base networks including ResNet and EfficientNet demonstrate the effectiveness of our approach, which consistently outperforms the pruning-based counterparts. [ABSTRACT FROM AUTHOR]

With the rapid development of neural architecture search (NAS), researchers found powerful network architectures for a wide range of vision tasks. Like the manually designed counterparts, we desire the automatically searched architectures to have the ability of being freely transferred to different scenarios. This paper formally puts forward this problem, referred to as NAS in the wild, which explores the possibility of finding the optimal architecture in a proxy dataset and then deploying it to mostly unseen scenarios. We instantiate this setting using a currently popular algorithm named differentiable architecture search (DARTS), which often suffers unsatisfying performance while being transferred across different tasks. We argue that the accuracy drop originates from the formulation that uses a super-network for search but a sub-network for re-training. The different properties of these stages have resulted in a significant optimization gap, and consequently, the architectural parameters "over-fit" the super-network. To alleviate the gap, we present a progressive method that gradually increases the network depth during the search stage, which leads to the Progressive DARTS (P-DARTS) algorithm. With a reduced search cost (7 hours on a single GPU), P-DARTS achieves improved performance on both the proxy dataset (CIFAR10) and a few target problems (ImageNet classification, COCO detection and three ReID benchmarks). Our code is available at https://github.com/chenxin061/pdarts. [ABSTRACT FROM AUTHOR]