Search

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Network slicing has been considered as a promising candidate to provide customized services for vehicular applications that have extremely high requirements of latency and reliability. However, the high mobility of vehicles poses significant challenges to resource management in such a stochastic vehicular environment with time-varying service demands. In this paper, we develop an online network slicing scheduling strategy for joint resource block (RB) allocation and power control in vehicular networks. The long-term time-averaged total system capacity is maximized while guaranteeing strict ultra-reliable and low-latency requirements of vehicle communication links, subject to stability constraints of task queues. The formulated problem is a mixed integer nonlinear stochastic optimization problem, which is decoupled into three subproblems by leveraging Lyapunov optimization. In order to tackle this problem, we propose an online algorithm, namely JRPSV, to obtain the optimal RB allocation and power control at each time slot according to the current network state. Furthermore, rigorous theoretical analysis is conducted for the proposed JRPSV algorithm, indicating that the system capacity and the system average latency obey a $[ {{\mathcal O}({1/V}),{\mathcal O}(V)} ]$ trade-off with the control parameter $V$. Extensive simulation results are provided to validate the theoretical analysis and demonstrate the effectiveness of JRPSV as well as the impacts of various parameters. [ABSTRACT FROM AUTHOR]

The optimization of electrical machines and energy-harvesting devices has been one of the hot topics of research in recent years. In this context, advanced computational intelligence algorithms are suitable design tools, since they are able to optimize complex systems with a large number of constraints and free design parameters. This paper presents the social network optimization (SNO): a new population-based algorithm developed to guarantee an effective and faster exploration of the solution domain with respect to traditional optimization techniques. The proposed technique will be described and validated with a benchmark case study. Finally, the optimization problem of a tubular permanent magnet linear generator will be formulated in the context of vehicular energy-harvesting systems. [ABSTRACT FROM PUBLISHER]

Algorithms that jointly allocate resources across different layers are envisioned to boost the performance of wireless systems. Recent results have revealed that two of the most important parameters that critically affect the resulting cross-layer designs are channel- and queue-state information (QSI). Motivated by these results, this paper relies on stochastic convex optimization to develop optimal algorithms that use instantaneous fading and queue length information to allocate resources at the transport (flow-control), link, and physical layers. Focus is placed on a cellular system, where an access point exchanges information with different users over flat-fading orthogonal channels. Both uplink and downlink setups are considered. The allocation strategies are obtained as the solution of a constrained utility maximization problem that involves average performance metrics. It turns out that the optimal allocation at a given instant depends on the instantaneous channel-state information (CSI) and Lagrange multipliers, which are associated with the quality-of-service (QoS) requirements and the operating conditions of the system. The multipliers are estimated online using stochastic approximation tools and are linked with the window-averaged length of the queues. Capitalizing on those links, queue stability and average queuing delay of the developed algorithms are characterized, and a simple mechanism is devised to effect delay priorities among users. [ABSTRACT FROM PUBLISHER]

In this paper, we study Stochastic Composite Optimization (SCO) for sparse learning that aims to learn a sparse solution from a composite function. Most of the recent SCO algorithms have already reached the optimal expected convergence rate \mathcal O(1/\lambda T) , but they often fail to deliver sparse solutions at the end either due to the limited sparsity regularization during stochastic optimization (SO) or due to the limitation in online-to-batch conversion. Even when the objective function is strongly convex, their high probability bounds can only attain \mathcal O(\sqrt\log (1/\delta)/T) with $\delta$ is the failure probability, which is much worse than the expected convergence rate. To address these limitations, we propose a simple yet effective two-phase Stochastic Composite Optimization scheme by adding a novel powerful sparse online-to-batch conversion to the general Stochastic Optimization algorithms. We further develop three concrete algorithms, OptimalSL, LastSL and AverageSL, directly under our scheme to prove the effectiveness of the proposed scheme. Both the theoretical analysis and the experiment results show that our methods can really outperform the existing methods at the ability of sparse learning and at the meantime we can improve the high probability bound to approximately \mathcal {O}(\log (\log (T)/\delta)/\lambda T) . [ABSTRACT FROM AUTHOR]

We consider stochastic optimization of a smooth non-convex loss function with a convex non-smooth regularizer. In the online setting, where a single sample of the stochastic gradient of the loss is available at every iteration, the problem can be solved using the proximal stochastic gradient descent (SGD) algorithm and its variants. However in many problems, especially those arising in communications and signal processing, information beyond the stochastic gradient may be available thanks to the structure of the loss function. By constructing a strongly convex surrogate of the loss function at every iteration, the stochastic SCA algorithms can exploit the structural properties of the loss function. In this work, we take a closer look at the stochastic SCA algorithm and develop its asynchronous variant which can be used for resource allocation in wireless networks. While the stochastic SCA algorithm is known to converge asymptotically, its iteration complexity has not been well-studied, and is the focus of the current work. The insights obtained from the non-asymptotic analysis allow us to develop a more practical asynchronous variant of the stochastic SCA algorithm which allows the use of surrogate calculated in earlier iterations. We characterize precise bound on the maximum delay the algorithm can tolerate, while still achieving the same convergence rate. We apply the algorithm to the problem of linear precoding in wireless sensor networks, where it can be implemented at low complexity but is shown to perform well in practice. [ABSTRACT FROM AUTHOR]

Sparse Tucker Decomposition (STD) algorithms learn a core tensor and a group of factor matrices to obtain an optimal low-rank representation feature for the High-Order, High-Dimension, and Sparse Tensor (HOHDST). However, existing STD algorithms face the problem of intermediate variables explosion which results from the fact that the formation of those variables, i.e., matrices Khatri-Rao product, Kronecker product, and matrix-matrix multiplication, follows the whole elements in sparse tensor. The above problems prevent deep fusion of efficient computation and big data platforms. To overcome the bottleneck, a novel stochastic optimization strategy (SGD_Tucker) is proposed for STD which can automatically divide the high-dimension intermediate variables into small batches of intermediate matrices. Specifically, SGD_Tucker only follows the randomly selected small samples rather than the whole elements, while maintaining the overall accuracy and convergence rate. In practice, SGD_Tucker features the two distinct advancements over the state of the art. First, SGD_Tucker can prune the communication overhead for the core tensor in distributed settings. Second, the low data-dependence of SGD_Tucker enables fine-grained parallelization, which makes SGD_Tucker obtaining lower computational overheads with the same accuracy. Experimental results show that SGD_Tucker runs at least 2 X faster than the state of the art. [ABSTRACT FROM AUTHOR]