Search

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

It is common for communities in real-world networks to possess hierarchical and overlapping structures, which make community detection even more challenging. In this paper, by investigating consensus process of the classical DeGroot model in opinion dynamics, we propose a novel method based on the cumulative opinion distance (COD) to discover hierarchical and overlapping communities. It is shown that this method is different from those classical algorithms relying on static fitness metrics that depict the inhomogeneous connectivity across the network. The proposed method is validated from two aspects. First, by estimating the eigenvectors of adjacency matrices, we investigate the detectability limit of our algorithms on random networks, which together with the results concerning the convergence speed of consensus guarantees the performance of our method theoretically. Second, experiments on both large scale real-world networks and artificial benchmarks show that our method is very effective and competitive on hierarchical modular graphs. In particular, it outperforms the state-of-the-art algorithms on overlapping community detection. [ABSTRACT FROM AUTHOR]

Dual Coordinate Descent (DCD) is one of the most popular optimization methods. The parallelization of DCD is difficult, as DCD is sequential in nature. As such, simultaneously running multiple DCD threads on batches of data elements causes result inaccuracy and slow convergence, due to the concurrent updates of multiple coordinates. Some parallelization methods adopt separable approximate functions that depend on the degree of parallelism. Such dependencies result in both poor scalability and slow-convergence. To address these challenges, in this paper we present a new parallel primal-dual algorithm for DCD, called PPD. In PPD, the block data distribution is utilized to obtain a new approximate function that is independent from the parallelism. Moreover, PPD is designed with a novel primal-dual acceleration scheme to approach the optimal solution closely and quickly. We demonstrate the advantages of PPD in terms of scalability and efficiency through experiments. [ABSTRACT FROM AUTHOR]

We consider a class of nonconvex regularized optimization problems, which appear frequently in machine learning and data processing. Due to the structure of the problems, the iteratively reweighted algorithm was developed and applied to the consensus optimization. In this paper, we propose the acceleration of this scheme by adding an inertial term in each iteration. The proposed algorithms inherit the advantages of classical decentralized algorithms: they can be implemented over a connected network, in which the agents communicate with their neighbors and perform local computations. We also employ the diminishing stepsizes technique for the iteratively reweighted algorithm and consider its acceleration. In specific cases, our algorithms reduce to existing decentralized schemes and also indicate novel ones. Mathematically, we prove the convergence for both algorithms with several assumptions on the objective functions. With Kurdyka-Łojasiewicz property, convergence rates can be derived for constant stepsize case. Numerical results demonstrate the efficiency of the algorithms. [ABSTRACT FROM AUTHOR]

Coupled Matrix Tensor Factorization (CMTF) facilitates the integration and analysis of multiple data sources and helps discover meaningful information. Nonnegative CMTF (N-CMTF) has been employed in many applications for identifying latent patterns, prediction, and recommendation. However, due to the added complexity with coupling between tensor and matrix data, existing N-CMTF algorithms exhibit poor computation efficiency. In this paper, a computationally efficient N-CMTF factorization algorithm is presented based on the column-wise element selection, preventing frequent gradient updates. Theoretical and empirical analyses show that the proposed N-CMTF factorization algorithm is not only more accurate but also more computationally efficient than existing algorithms in approximating the tensor as well as in identifying the underlying nature of factors. [ABSTRACT FROM AUTHOR]

Recently, reconciling social networks, identifying the accounts belonging to the same individual across social networks, receives significant attention from both academic and industry. Most of the existing studies have limitations in the following three aspects: multiplicity, comprehensiveness and robustness. To address these limitations, we rethink this problem and, for the first time, robustly and comprehensively reconcile multiple social networks. In this paper, we propose two frameworks, MASTER and MASTER+, i.e., across Multiple social networks, integrate Attribute and STructure Embedding for Reconciliation. In MASTER, we first design a novel Constrained Dual Embedding model, simultaneously embedding and reconciling multiple social networks, to formulate this problem into a unified optimization. To address this optimization, we then design an effective NS-Alternating algorithm and prove it converges to KKT points. To further speed up MASTER, we propose a scalable framework, namely MASTER+. The core idea is to group accounts into clusters and then perform MASTER in each cluster in parallel. Specifically, we design an efficient Augmented Pre-Embedding model and Balance-aware Fuzzy Clustering algorithm for the high efficiency and the high accuracy. Extensive experiments demonstrate that both MASTER and MASTER+ outperform the state-of-the-art approaches. Moreover, MASTER+ inherits the effectiveness of MASTER and enjoys higher efficiency. [ABSTRACT FROM AUTHOR]

Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data. [ABSTRACT FROM AUTHOR]