Multi-core parallel architecture design and experiment for deep learning model training.

The parallel architecture can improve the training speed of Deep learning models and it is beneficial to model optimization. This research designs Optimal Interleaved Distributed Architecture (OIDA). Its characteristics are (1) the data set that each child node participates in training is fixed and unique; (2) multiple approaches ensure that the model parameters of each child node participating in training each time are always optimal; (3) the computing units of each child node are interleaved for training without stopping. The architecture can be deployed as Single Machine Parallel (SMP) or Multi Machine Distributed (MMD), or a combination of both. The experimental results show that its efficiency is obvious in a multi-machine multi-card environment, and the model speed of training millions of parameters in a 5-machine 10-card lightweight (Graphics Processing Unit (GPU) of 2GB memory) cluster can reach megapixels/s. When training the same amount of training data, GPU can process data faster than Central Processing Unit (CPU) with the same number of clusters. 4-machine 4-core GPU parallel saves 65.61% of time compared with 4-machine 4-core CPU parallel, while 4-machine 8-core GPU saves 83.32% of time compared with 4-machine 4-core CPU parallel. For polarized Synthetic Aperture Radar (SAR) Deep learning data sets, increasing the number of computers in the cluster can effectively save training time. In the experimental environment of this research, there is no situation that when the number of computers in the cluster increases due to network data transmission, the training time does not decrease. [ABSTRACT FROM AUTHOR]