Search

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

Finding top- $k$ k frequent items has been a hot issue in databases. Finding top- $k$ k persistent items is a new issue, and has attracted increasing attention in recent years. In practice, users often want to know which items are significant, i.e., not only frequent but also persistent. No prior art can address both of the above two issues at the same time. Also, for high-speed data streams, prior art cannot achieve high accuracy when the memory is tight. In this paper, we define a new issue, named finding significant items, and propose a novel algorithm namely LTC to address this issue. It includes two key techniques, Long-tail Restoring and CLOCK, as well as three optimizations. In addition, LTC is extended to support finding significant items with thresholds. We theoretically derive the correct rate and error bound, and conduct extensive experiments on three real datasets to test the performance of LTC. Our experimental results show that LTC can achieve $10^5$ 10 5 times higher accuracy in terms of average relative error than other related algorithms. Lastly, LTC is applied to a DDoS detection task and it shows that finding significant items is more powerful than finding frequent items. [ABSTRACT FROM AUTHOR]

Dual Coordinate Descent (DCD) is one of the most popular optimization methods. The parallelization of DCD is difficult, as DCD is sequential in nature. As such, simultaneously running multiple DCD threads on batches of data elements causes result inaccuracy and slow convergence, due to the concurrent updates of multiple coordinates. Some parallelization methods adopt separable approximate functions that depend on the degree of parallelism. Such dependencies result in both poor scalability and slow-convergence. To address these challenges, in this paper we present a new parallel primal-dual algorithm for DCD, called PPD. In PPD, the block data distribution is utilized to obtain a new approximate function that is independent from the parallelism. Moreover, PPD is designed with a novel primal-dual acceleration scheme to approach the optimal solution closely and quickly. We demonstrate the advantages of PPD in terms of scalability and efficiency through experiments. [ABSTRACT FROM AUTHOR]

We consider a class of nonconvex regularized optimization problems, which appear frequently in machine learning and data processing. Due to the structure of the problems, the iteratively reweighted algorithm was developed and applied to the consensus optimization. In this paper, we propose the acceleration of this scheme by adding an inertial term in each iteration. The proposed algorithms inherit the advantages of classical decentralized algorithms: they can be implemented over a connected network, in which the agents communicate with their neighbors and perform local computations. We also employ the diminishing stepsizes technique for the iteratively reweighted algorithm and consider its acceleration. In specific cases, our algorithms reduce to existing decentralized schemes and also indicate novel ones. Mathematically, we prove the convergence for both algorithms with several assumptions on the objective functions. With Kurdyka-Łojasiewicz property, convergence rates can be derived for constant stepsize case. Numerical results demonstrate the efficiency of the algorithms. [ABSTRACT FROM AUTHOR]

Community detection is one of the fundamental tasks in graph mining, which aims to identify group assignment of nodes in a complex network. Recently, network embedding techniques have demonstrated their strong power in advancing the community detection task and achieve better performance than various traditional methods. Despite their empirical success, most of the existing algorithms directly leverage the observed coarse network structure for community detection. Therefore, they often lead to suboptimal performance as the observed connections fail to capture the essential tie strength information among nodes precisely and account for the impact of noisy links. In this paper, an optimal network structure for community detection is introduced to characterize the fine-grained tie strength information between connected nodes and alleviate the adverse effects of noisy links. To obtain an expressive node representation for community detection, we learn the optimal network structure and network embeddings in a joint framework, instead of using a two-stage approach to derive the node embeddings from the coarse network topology. In particular, we formulate the joint framework as an optimization problem and an alternating optimization algorithm is exploited to solve the proposed optimization problem. Additionally, theoretical analyses regarding the computational complexity and the convergence of the optimization algorithm are also provided. Extensive experiments on both synthetic and real-world networks demonstrate the effectiveness and superiority of the proposed framework. [ABSTRACT FROM AUTHOR]

Coupled Matrix Tensor Factorization (CMTF) facilitates the integration and analysis of multiple data sources and helps discover meaningful information. Nonnegative CMTF (N-CMTF) has been employed in many applications for identifying latent patterns, prediction, and recommendation. However, due to the added complexity with coupling between tensor and matrix data, existing N-CMTF algorithms exhibit poor computation efficiency. In this paper, a computationally efficient N-CMTF factorization algorithm is presented based on the column-wise element selection, preventing frequent gradient updates. Theoretical and empirical analyses show that the proposed N-CMTF factorization algorithm is not only more accurate but also more computationally efficient than existing algorithms in approximating the tensor as well as in identifying the underlying nature of factors. [ABSTRACT FROM AUTHOR]

Recently, reconciling social networks, identifying the accounts belonging to the same individual across social networks, receives significant attention from both academic and industry. Most of the existing studies have limitations in the following three aspects: multiplicity, comprehensiveness and robustness. To address these limitations, we rethink this problem and, for the first time, robustly and comprehensively reconcile multiple social networks. In this paper, we propose two frameworks, MASTER and MASTER+, i.e., across Multiple social networks, integrate Attribute and STructure Embedding for Reconciliation. In MASTER, we first design a novel Constrained Dual Embedding model, simultaneously embedding and reconciling multiple social networks, to formulate this problem into a unified optimization. To address this optimization, we then design an effective NS-Alternating algorithm and prove it converges to KKT points. To further speed up MASTER, we propose a scalable framework, namely MASTER+. The core idea is to group accounts into clusters and then perform MASTER in each cluster in parallel. Specifically, we design an efficient Augmented Pre-Embedding model and Balance-aware Fuzzy Clustering algorithm for the high efficiency and the high accuracy. Extensive experiments demonstrate that both MASTER and MASTER+ outperform the state-of-the-art approaches. Moreover, MASTER+ inherits the effectiveness of MASTER and enjoys higher efficiency. [ABSTRACT FROM AUTHOR]

Subspace clustering plays an important role in the tasks such as data processing and pattern recognition. Since the high-dimensional data may contain complex noise, as well as non-linear structure, learning low-dimensional subspace structures is a challenging task. However, the existing methods to deal with both problems relax the original problem convexly. The results of solving by these methods deviate from the solution of the original problem. In this paper, to overcome this deficiency, we propose a robust low-rank kernel subspace clustering model, which coalesces the non-convex Schatten p-norm ($0 < p \leq 1$ 0 < p ≤ 1 ) regularizer with “kernel trick” and correntropy. Our “kernel trick” extends linear subspace clustering to non-linear counterparts, the Schatten p-norm regularizer can approximate the rank of the data in feature space effectively, and the correntropy is a robust measure to large corruptions. Furthermore, an efficient iterative algorithm (HQ-ADMM) is designed to solve the formulated problem, which coalesces the half-quadratic technique and Alternating Direction Method of Multipliers. This algorithm can ensure the closed form solutions at each iteration, which improves the computation speed of the algorithm. Extensive experiments on face/object clustering and motion segmentation clearly attest the ascendancy of the proposed method over several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Multi-label learning methods assign multiple labels to one object. In practice, in addition to differentiating relevant labels from irrelevant ones, it is often desired to rank relevant labels for an object, whereas the ranking of irrelevant labels is not important. Thus, we require an algorithm to do classification and ranking of relevant labels simultaneously. Such a requirement, however, cannot be met because most existing methods were designed to optimize existing criteria, yet there is no criterion which encodes the aforementioned requirement. In this paper, we present a new criterion, PRO Loss, concerning the prediction of all labels as well as the ranking of only relevant labels. We then propose ProSVM which optimizes PRO Loss efficiently using alternating direction method of multipliers. We further improve its efficiency with an upper approximation that reduces the number of constraints from $O(T^2)$ O (T 2) to $O(T)$ O (T) , where $T$ T is the number of labels. We then notice that in real applications, it is difficult to get full supervised information for multi-label data. To make the proposed algorithm more robust to supervised information, we adapt ProSVM to deal with the multi-label learning with partial labels problem. Experiments show that our proposal is not only superior on PRO Loss, but also highly competitive on existing evaluation criteria. [ABSTRACT FROM AUTHOR]

While the majority of methods for community detection produce disjoint communities of nodes, most real-world networks naturally involve overlapping communities. In this paper, a scalable method for the detection of overlapping communities in large networks is proposed. The method is based on an extension of the notion of normalized cut to cope with overlapping communities. A spectral clustering algorithm is formulated to solve the related cut minimization problem. When available, the algorithm may take into account prior information about the likelihood for each node to belong to several communities. This information can either be extracted from the available metadata or from node centrality measures. We also introduce a hierarchical version of the algorithm to automatically detect the number of communities. In addition, a new benchmark model extending the stochastic blockmodel for graphs with overlapping communities is formulated. Our experiments show that the proposed spectral method outperforms the state-of-the-art algorithms in terms of computational complexity and accuracy on our benchmark graph model and on five real-world networks, including a lexical network and large-scale social networks. The scalability of the proposed algorithm is also demonstrated on large synthetic graphs with millions of nodes and edges. [ABSTRACT FROM AUTHOR]

Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data. [ABSTRACT FROM AUTHOR]

Networks are prevalent and often collected from multiple sources in many high-impact domains, which facilitate many emerging applications that require the connections across multiple networks. Network alignment (i.e., to find the node correspondence between different networks) has become the very first step in many applications and thus has been studied in decades. Although some existing works can use the attribute information as part of the alignment process, they still have certain limitations. For example, some existing network alignment methods can use node attribute similarities as part of the prior alignment information, whereas most of them solely explore the topology consistency without the consistency among attributes of the underlying networks. On the other hand, traditional graph matching methods encode both the node and edge attributes (and possibly the topology) into an affinity matrix and formulate it as a constrained nonconvex quadratic maximization problem. However, these methods cannot scale well to the large-scale networks. In this paper, we propose a family of network alignment algorithms (FINAL) to efficiently align the attributed networks. The key idea is to leverage the node/edge attribute information to guide the (topology-based) alignment process. We formulate this problem as a convex quadratic optimization problem, and develop effective and efficient algorithms to solve it. Moreover, we derive FINAL On-Query, an online variant of FINAL that can find similar nodes for the query nodes across networks. We perform extensive evaluations on real networks to substantiate the superiority of our proposed approaches. [ABSTRACT FROM AUTHOR]

There is always a lack of a cluster validity function and optimization strategy to find out clusters and catch the evolution trend of cluster structures on a categorical data stream. Therefore, this paper presents an optimization model for clustering categorical data streams. In the model, a cluster validity function is proposed as the objective function to evaluate the effectiveness of the clustering model while each new input data subset is flowing. It simultaneously considers the certainty of the clustering model and the continuity with the last clustering model in the clustering process. An iterative optimization algorithm is proposed to solve an optimal solution of the objective function with some constraints. Furthermore, we strictly derive a detection index for drifting concepts from the optimization model. We propose a detection method that integrates the detection index and the optimization model to catch the evolution trend of cluster structures on a categorical data stream. The new method can effectively avoid ignoring the effect of the clustering validity on the detection result. Finally, using the experimental studies on several real data sets, we illustrate the effectiveness of the proposed algorithm in clustering categorical data streams, compared with existing data-streams clustering algorithms. [ABSTRACT FROM AUTHOR]

Feature selection is a challenging problem for high dimensional data processing, which arises in many real applications such as data mining, information retrieval, and pattern recognition. In this paper, we study the problem of unsupervised feature selection. The problem is challenging due to the lack of label information to guide feature selection. We formulate the problem of unsupervised feature selection from the viewpoint of graph regularized data reconstruction. The underlying idea is that the selected features not only preserve the local structure of the original data space via graph regularization, but also approximately reconstruct each data point via linear combination. Therefore, the graph regularized data reconstruction error becomes a natural criterion for measuring the quality of the selected features. By minimizing the reconstruction error, we are able to select the features that best preserve both the similarity and discriminant information in the original data. We then develop an efficient gradient algorithm to solve the corresponding optimization problem. We evaluate the performance of our proposed algorithm on text clustering. The extensive experiments demonstrate the effectiveness of our proposed approach. [ABSTRACT FROM PUBLISHER]

Rank similarity measures provide a method for quantifying differences between search engine results without the need for relevance judgments. For example, the providers of a search service might use such measures to estimate the impact of a proposed algorithmic change across a large number of queries—perhaps millions—identifying those queries where the impact is greatest. In this paper, we propose and validate a family of rank similarity measures, each derived from an associated effectiveness measure. Each member of the family is based on the maximization of effectiveness difference under this associated measure. Computing this maximized effectiveness difference (MED) requires the solution of an optimization problem that varies in difficulty, depending on the associated measure. We present solutions for several standard effectiveness measures, including nDCG, AP, and ERR. Through an experimental validation, we show that MED reveals meaningful differences between retrieval runs. Mathematically, MED is a metric, regardless of the associated measure. Prior work has established a number of other desiderata for rank similarity in the context of search, and we demonstrate that MED satisfies these requirements. Unlike previous proposals, MED allows us to directly translate assumptions about user behavior from any established effectiveness measure to create a corresponding rank similarity measure. In addition, MED cleanly accommodates partial relevance judgments, and if complete relevance information is available, it reduces to a simple difference between effectiveness values. [ABSTRACT FROM PUBLISHER]

The discovery of evolving communities in dynamic networks is an important research topic that poses challenging tasks. Evolutionary clustering is a recent framework for clustering dynamic networks that introduces the concept of temporal smoothness inside the community structure detection method. Evolutionary-based clustering approaches try to maximize cluster accuracy with respect to incoming data of the current time step, and minimize clustering drift from one time step to the successive one. In order to optimize both these two competing objectives, an input parameter that controls the preference degree of a user towards either the snapshot quality or the temporal quality is needed. In this paper the detection of communities with temporal smoothness is formulated as a multiobjective problem and a method based on genetic algorithms is proposed. The main advantage of the algorithm is that it automatically provides a solution representing the best trade-off between the accuracy of the clustering obtained, and the deviation from one time step to the successive. Experiments on synthetic data sets show the very good performance of the method when compared with state-of-the-art approaches. [ABSTRACT FROM PUBLISHER]

Clustering by integrating multiview representations has become a crucial issue for knowledge discovery in heterogeneous environments. However, most prior approaches assume that the multiple representations share the same dimension, limiting their applicability to homogeneous environments. In this paper, we present a novel tensor-based framework for integrating heterogeneous multiview data in the context of spectral clustering. Our framework includes two novel formulations; that is multiview clustering based on the integration of the Frobenius-norm objective function (MC-FR-OI) and that based on matrix integration in the Frobenius-norm objective function (MC-FR-MI). We show that the solutions for both formulations can be computed by tensor decompositions. We evaluated our methods on synthetic data and two real-world data sets in comparison with baseline methods. Experimental results demonstrate that the proposed formulations are effective in integrating multiview data in heterogeneous environments. [ABSTRACT FROM AUTHOR]

The new time series data mining framework proposed in this paper applies Reconstructured Phase Space (RPS) to identify temporal patterns that are characteristic and predictive of significant events in a complex time series. The new framework utilizes the fuzzy set and the Gaussian-shaped membership function to define temporal patterns in the time-delay embedding phase space. The resulting objective function represents not only the overall value of the event function, but also the weight of the vector in the temporal pattern cluster to which it contributes. Also, the new objective function is continuously differentiable so the gradient descent optimization such as quasi-Newton's method can be applied to search the optimal temporal patterns with much faster speed of convergence. The computational stability is significantly improved over the genetic algorithm originally used in our early framework. A new simple but effective two-step optimization strategy is proposed which further improves the search performance. Another significant contribution is the use of mutual information and false neighbors methods to estimate the time delay and the phase space dimension. We also implemented two experimental applications to demonstrate the effectiveness of the new framework with comparisons to the original framework and to the neural network prediction approach. [ABSTRACT FROM AUTHOR]

Network communities refer to groups of vertices within which their connecting links are dense but between which they are sparse. A network community mining problem (or NCMP for short) is concerned with the problem of finding all such communities from a given network. A wide variety of applications can be formulated as NCMPs, ranging from social and/or biological network analysis to web mining and searching. So far, many algorithms addressing NCMPs have been developed and most of them fall into the categories of either optimization based or heuristic methods. Distinct from the existing studies, the work presented in this paper explores the notion of network communities and their properties based on the dynamics of a stochastic model naturally introduced. In the paper, a relationship between the hierarchical community structure of a network and the local mixing properties of such a stochastic model has been established with the large-deviation theory. Topological information regarding to the community structures hidden in networks can be inferred from their spectral signatures. Based on the above-mentioned relationship, this work proposes a general framework for characterizing, analyzing, and mining network communities. Utilizing the two basic properties of metastability, i.e., being locally uniform and temporarily fixed, an efficient implementation of the framework, called the LM algorithm, has been developed that can scalably mine communities hidden in large-scale networks. The effectiveness and efficiency of the LM algorithm have been theoretically analyzed as well as experimentally validated. [ABSTRACT FROM PUBLISHER]

In many information processing tasks, labels are usually expensive and the unlabeled data points are abundant. To reduce the cost on collecting labels, it is crucial to predict which unlabeled examples are the most informative, i.e., improve the classifier the most if they were labeled. Many active learning techniques have been proposed for text categorization, such as SVMActive and Transductive Experimental Design. However, most of previous approaches try to discover the discriminant structure of the data space, whereas the geometrical structure is not well respected. In this paper, we propose a novel active learning algorithm which is performed in the data manifold adaptive kernel space. The manifold structure is incorporated into the kernel space by using graph Laplacian. This way, the manifold adaptive kernel space reflects the underlying geometry of the data. By minimizing the expected error with respect to the optimal classifier, we can select the most representative and discriminative data points for labeling. Experimental results on text categorization have demonstrated the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

Mean-Shift (MS) is a powerful nonparametric clustering method. Although good accuracy can be achieved, its computational cost is particularly expensive even on moderate data sets. In this paper, for the purpose of algorithmic speedup, we develop an agglomerative MS clustering method along with its performance analysis. Our method, namely Agglo-MS, is built upon an iterative query set compression mechanism which is motivated by the quadratic bounding optimization nature of MS algorithm. The whole framework can be efficiently implemented in linear running time complexity. We then extend Agglo-MS into an incremental version which performs comparably to its batch counterpart. The efficiency and accuracy of Agglo-MS are demonstrated by extensive comparing experiments on synthetic and real data sets. [ABSTRACT FROM PUBLISHER]

Given a data point set D, a query point set Q, and an integer k, the Group Nearest Group (GNG) query finds a subset \omega (\vert \omega \vert \le k) of points from D such that the total distance from all points in Q to the nearest point in \omega is not greater than any other subset \omega^\prime (\vert \omega^\prime \vert \le k) of points in D. GNG query is a partition-based clustering problem which can be found in many real applications and is NP-hard. In this paper, Exhaustive Hierarchical Combination (EHC) algorithm and Subset Hierarchial Refinement (SHR) algorithm are developed for GNG query processing. While EHC is capable to provide the optimal solution for k=2, SHR is an efficient approximate approach that combines database techniques with local search heuristic. The processing focus of our approaches is on minimizing the access and evaluation of subsets of cardinality k in D since the number of such subsets is exponentially greater than \vert D\vert. To do that, the hierarchical blocks of data points at high level are used to find an intermediate solution and then refined by following the guided search direction at low level so as to prune irrelevant subsets. The comprehensive experiments on both real and synthetic data sets demonstrate the superiority of SHR in terms of efficiency and quality. [ABSTRACT FROM PUBLISHER]

In this paper, we address the problem of mining large maximal bicliques from a three-dimensional Boolean symmetric adjacency matrix. We propose CubeMiner-MBC algorithm which enumerates all the maximal bicliques satisfying the user-specified size constraints. Our algorithm enumerates all bicliques with less memory in depth first manner and does not store the previously computed patterns in the main memory for duplicate detection. To efficiently prune duplicate patterns, we have proposed a subtree pruning technique which reduces the total number of nodes that are processed and also reduces the total number of duplicate patterns that are generated. We have also incorporated several optimizations for efficient cutter generation and closure checking. Experiments involving several synthetic data sets show that our algorithm takes less running time than CubeMiner algorithm. [ABSTRACT FROM PUBLISHER]