Search

As an algorithmic framework, message passing is extremely powerful and has wide applications in the context of different disciplines including communications, coding theory, statistics, signal processing, artificial intelligence and combinatorial optimization. In this paper, we investigate the performance of a message-passing algorithm called min-sum belief propagation (BP) for the vertex-disjoint shortest $k$ -path problem ($k$ -VDSP) on weighted directed graphs, and derive the iterative message-passing update rules. As the main result of this paper, we prove that for a weighted directed graph $G$ of order $n$ , BP algorithm converges to the unique optimal solution of $k$ -VDSP on $G$ within $O(n^{2}w_{max})$ iterations, provided that the weight $w_{e}$ is nonnegative integral for each arc $e\in E(G)$ , where $w_{max}=\max \{w_{e}: e\in E(G)\}$. To the best of our knowledge, this is the first instance where BP algorithm is proved correct for NP-hard problems. Additionally, we establish the extensions of $k$ -VDSP to the case of multiple sources or sinks. [ABSTRACT FROM AUTHOR]

Gossip protocols are popular methods for average consensus problems in distributed computing. We prove new convergence guarantees for a variety of such protocols, including path, clique, and synchronous pairwise gossip. These arise by exploiting the connection between these protocols and the block randomized Kaczmarz method for solving linear systems. Moreover, we extend existing convergence results for block randomized Kaczmarz to allow for a more general choice of blocks, rank-deficient systems, and provide a tighter convergence rate guarantee. Previous results for block randomized Kaczmarz assumed blocks formed a paving of the systems; our analysis, contrary to the title of our paper, generalizes to a broader class of blocks that we call coverings. We furthermore apply this analysis to inconsistent consensus models and obtain similar guarantees. An extensive empirical analysis of these methods is provided for a variety of synthetic and real networks. [ABSTRACT FROM AUTHOR]

This paper studies distributed algorithms for (strongly convex) composite optimization problems over mesh networks, subject to quantized communications. Instead of focusing on a specific algorithmic design, a black-box model is proposed, casting linearly convergent distributed algorithms in the form of fixed-point iterates. The algorithmic model is equipped with a novel random or deterministic Biased Compression (BC) rule on the quantizer design, and a new Adaptive encoding Non-uniform Quantizer (ANQ) coupled with a communication-efficient encoding scheme, which implements the BC-rule using a finite number of bits (below machine precision). This fills a gap existing in most state-of-the-art quantization schemes, such as those based on the popular compression rule, which rely on communication of some scalar signals with negligible quantization error (in practice quantized at the machine precision). A unified communication complexity analysis is developed for the black-box model, determining the average number of bits required to reach a solution of the optimization problem within a target accuracy. It is shown that the proposed BC-rule preserves linear convergence of the unquantized algorithms, and a trade-off between convergence rate and communication cost under ANQ-based quantization is characterized. Numerical results validate our theoretical findings and show that distributed algorithms equipped with the proposed ANQ have more favorable communication cost than algorithms using state-of-the-art quantization rules. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the convergence speed of the Arimoto-Blahut algorithm. For many channel matrices, the convergence speed is exponential, but for some channel matrices it is slower than exponential. By analyzing the Taylor expansion of the defining function of the Arimoto-Blahut algorithm, we will make the conditions clear for the exponential or slower convergence. The analysis of the slow convergence in this paper is new. Based on this analysis, we will compare the convergence speeds of the Arimoto-Blahut algorithm numerically with the values obtained in our theorems for several channel matrices. The purpose of this paper is to obtain a complete understanding of the convergence speed of the Arimoto-Blahut algorithm. [ABSTRACT FROM AUTHOR]

In real-world applications, it is important for machine learning algorithms to be robust against data outliers or corruptions. In this paper, we focus on improving the robustness of a large class of learning algorithms that are formulated as low-rank semi-definite programming (SDP) problems. Traditional formulations use the square loss, which is notorious for being sensitive to outliers. We propose to replace this with more robust noise models, including the $\ell _1$ ℓ 1 -loss and other nonconvex losses. However, the resultant optimization problem becomes difficult as the objective is no longer convex or smooth. To alleviate this problem, we design an efficient algorithm based on majorization-minimization. The crux is on constructing a good optimization surrogate, and we show that this surrogate can be efficiently obtained by the alternating direction method of multipliers (ADMM). By properly monitoring ADMM's convergence, the proposed algorithm is empirically efficient and also theoretically guaranteed to converge to a critical point. Extensive experiments are performed on four machine learning applications using both synthetic and real-world data sets. Results show that the proposed algorithm is not only fast but also has better performance than the state-of-the-arts. [ABSTRACT FROM AUTHOR]

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

It is common for communities in real-world networks to possess hierarchical and overlapping structures, which make community detection even more challenging. In this paper, by investigating consensus process of the classical DeGroot model in opinion dynamics, we propose a novel method based on the cumulative opinion distance (COD) to discover hierarchical and overlapping communities. It is shown that this method is different from those classical algorithms relying on static fitness metrics that depict the inhomogeneous connectivity across the network. The proposed method is validated from two aspects. First, by estimating the eigenvectors of adjacency matrices, we investigate the detectability limit of our algorithms on random networks, which together with the results concerning the convergence speed of consensus guarantees the performance of our method theoretically. Second, experiments on both large scale real-world networks and artificial benchmarks show that our method is very effective and competitive on hierarchical modular graphs. In particular, it outperforms the state-of-the-art algorithms on overlapping community detection. [ABSTRACT FROM AUTHOR]

Dual Coordinate Descent (DCD) is one of the most popular optimization methods. The parallelization of DCD is difficult, as DCD is sequential in nature. As such, simultaneously running multiple DCD threads on batches of data elements causes result inaccuracy and slow convergence, due to the concurrent updates of multiple coordinates. Some parallelization methods adopt separable approximate functions that depend on the degree of parallelism. Such dependencies result in both poor scalability and slow-convergence. To address these challenges, in this paper we present a new parallel primal-dual algorithm for DCD, called PPD. In PPD, the block data distribution is utilized to obtain a new approximate function that is independent from the parallelism. Moreover, PPD is designed with a novel primal-dual acceleration scheme to approach the optimal solution closely and quickly. We demonstrate the advantages of PPD in terms of scalability and efficiency through experiments. [ABSTRACT FROM AUTHOR]

We consider a class of nonconvex regularized optimization problems, which appear frequently in machine learning and data processing. Due to the structure of the problems, the iteratively reweighted algorithm was developed and applied to the consensus optimization. In this paper, we propose the acceleration of this scheme by adding an inertial term in each iteration. The proposed algorithms inherit the advantages of classical decentralized algorithms: they can be implemented over a connected network, in which the agents communicate with their neighbors and perform local computations. We also employ the diminishing stepsizes technique for the iteratively reweighted algorithm and consider its acceleration. In specific cases, our algorithms reduce to existing decentralized schemes and also indicate novel ones. Mathematically, we prove the convergence for both algorithms with several assumptions on the objective functions. With Kurdyka-Łojasiewicz property, convergence rates can be derived for constant stepsize case. Numerical results demonstrate the efficiency of the algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel Newton method-based distributed algorithm (NMDA), which is also effective in solving the general single-area EDP (SAEDP), to deal with the multi-area economic dispatch problem (MAEDP), of which the focus is to minimize the total generation cost in the presence of system and generator constraints. To develop the NMDA, we first introduce a virtual SAEDP formulation to fit the framework of Newton method (NM), and then employ the average consensus protocol to obtain the global information needed to execute the NM and backtracking line search algorithm in a distributed manner. Compared with the centralized methods that can yield the optimal solution, the proposed NMDA provides a suboptimal solution with a very small relative error. The NMDA ensures the instantaneous system power balance throughout the iteration process while the centralized methods compared in this paper cannot do so. We also provide a rigorous theoretical analysis for the convergence of NMDA. Moreover, the advantage of NMDA in terms of the convergence speed is validated by comparing with other distributed methods such as the gradient-based ADMM (G-ADMM) and quasi Newton-based primal dual interior point (QN-PDIP) method. Finally, case studies demonstrate the effectiveness and scalability of the proposed distributed algorithm. [ABSTRACT FROM AUTHOR]

In this paper, we propose to extend the standard Convolutional Dictionary Learning problem to a tensor representation where the activations are constrained to be “low-rank” through a Canonical Polyadic decomposition. We show that this additional constraint increases the robustness of the CDL with respect to noise and improve the interpretability of the final results. In addition, we discuss in detail the advantages of this representation and introduce two algorithms, based on ADMM or FISTA, that efficiently solve this problem. We show that by exploiting the low rank property of activations, they achieve lower complexity than the main CDL algorithms. Finally, we evaluate our approach on a wide range of experiments, highlighting the modularity and the advantages of this tensorial low-rank formulation. [ABSTRACT FROM AUTHOR]

The use of a large-scale antenna array (LSAA) has become an important characteristic of multi-antenna communication systems to achieve beamforming gains such as in designing millimeter-wave (mmWave) systems to combat severe propagation losses. In such applications, each antenna element has to be driven by a radio frequency (RF) chain for the implementation of fully-digital beamformers, significantly increasing the hardware cost, complexity, and power consumption. Therefore, constant-modulus analog beamforming (CMAB) becomes a viable solution. In this paper, we consider the scaled analog beamforming (SAB) or constant-modulus analog beamforming (CMAB) architecture and design the system parameters by solving two variants of beampattern matching problem. In the first case, both the magnitude and phase of the beampattern are matched to the given desired beampattern whereas in the second case, only the magnitude of the beampattern is matched. Both the beampattern matching problems are cast as a variant of the constant-modulus least-squares (CLS) problem. We provide efficient algorithms based on the alternating majorization-minimization (AMM) framework that combines the alternating minimization and the MM frameworks and the conventional-cyclic coordinate descent (C-CCD) algorithms to solve the problem in each case. We also propose algorithms based on a new modified-CCD (M-CCD) based approach. For all the developed algorithms we prove convergence to a Karush-Kuhn-Tucker (KKT) point (or a stationary point). Numerical results demonstrate that the proposed algorithms converge faster than the state-of-the-art solutions. Among all the algorithms, the M-CCD-based algorithms have faster convergence when evaluated in terms of the number of iterations and the AMM-based algorithms offer lower complexity. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

This paper studies the convergence properties of the well-known message-passing algorithm for convex optimisation. Under the assumption of pairwise separability and scaled diagonal dominance, asymptotic convergence is established and a simple bound for the convergence rate is provided for message-passing. In comparison with previous results, our results do not require the given convex program to have known convex pairwise components and that our bound for the convergence rate is tighter and simpler. When specialised to quadratic optimisation, we generalise known results by providing a very simple bound for the convergence rate. [ABSTRACT FROM AUTHOR]

In this paper, we consider the policy evaluation problem in reinforcement learning with agents on a decentralized and directed network. In order to evaluate the quality of a fixed policy in this decentralized setting, one option is for agents to run decentralized temporal-difference (TD) collaboratively. To account for the practical scenarios where the state and action spaces are large and malicious attacks emerge, we focus on the decentralized TD learning with linear function approximation in the presence of malicious agents (often termed as Byzantine agents). We propose a trimmed mean-based Byzantine-resilient decentralized TD algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error that depends on the number of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper proposes a two-level distributed algorithmic framework for solving the AC optimal power flow (OPF) problem with convergence guarantees. The presence of highly nonconvex constraints in OPF poses significant challenges to distributed algorithms based on the alternating direction method of multipliers (ADMM). In particular, convergence is not provably guaranteed for nonconvex network optimization problems like AC OPF. In order to overcome this difficulty, we propose a new distributed reformulation for AC OPF and a two-level ADMM algorithm that goes beyond the standard framework of ADMM. We establish the global convergence and iteration complexity of the proposed algorithm under mild assumptions. Extensive numerical experiments over some largest test cases from NESTA and PGLib-OPF (up to 30 000-bus systems) demonstrate advantages of the proposed algorithm over existing ADMM variants in terms of convergence, scalability, and robustness. Moreover, under appropriate parallel implementation, the proposed algorithm exhibits fast convergence comparable to or even better than the state-of-the-art centralized solver. [ABSTRACT FROM AUTHOR]

We consider the problem of minimizing the $\ell _1$ norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms with provable guarantees for tackling this problem. Specifically, our contribution is fourfold. First, we present a manifold proximal point algorithm (ManPPA) for the problem and show that it converges at a global sublinear rate. Furthermore, we show that ManPPA can achieve a local quadratic convergence rate when applied to sharp instances of the problem. Second, we develop a semismooth Newton-based inexact augmented Lagrangian method for computing the search direction in each iteration of ManPPA and show that it has an asymptotic superlinear convergence rate. Third, we propose a stochastic variant of ManPPA called StManPPA, which is well suited for large-scale computation, and establish its sublinear convergence rate. Both ManPPA and StManPPA have provably faster convergence rates than existing subgradient-type methods. Fourth, using ManPPA as a building block, we propose a new heuristic method for solving a matrix analog of the problem, in which the sphere is replaced by the Stiefel manifold. The results from our extensive numerical experiments on the ODL and RSR problems demonstrate the efficiency and efficacy of our proposed methods. [ABSTRACT FROM AUTHOR]

In this paper, we investigate resource allocation for IRS-assisted green multiuser multiple-input single-output (MISO) systems. To minimize the total transmit power, both the beamforming vectors at the access point (AP) and the phase shifts at multiple IRSs are jointly optimized, while taking into account the minimum required quality-of-service (QoS) of multiple users. First, two novel algorithms, namely a penalty-based alternating minimization (AltMin) algorithm and an inner approximation (IA) algorithm, are developed to tackle the non-convexity of the formulated optimization problem when perfect channel state information (CSI) is available. Existing designs employ semidefinite relaxation in AltMin-based algorithms, which, however, cannot ensure convergence. In contrast, the proposed penalty-based AltMin and IA algorithms are guaranteed to converge to a stationary point and a Karush-Kuhn-Tucker (KKT) solution of the design problem, respectively. Second, the impact of imperfect knowledge of the CSI of the channels between the AP and the users is investigated. To this end, a non-convex robust optimization problem is formulated and the penalty-based AltMin algorithm is extended to obtain a stationary solution. Simulation results reveal a key trade-off between the speed of convergence and the achievable total transmit power for the two proposed algorithms. In addition, we show that the proposed algorithms can significantly reduce the total transmit power at the AP compared to various baseline schemes and that the optimal numbers of transmit antennas and IRS reflecting elements, which maximize the system energy efficiency of the considered system, are finite. [ABSTRACT FROM AUTHOR]

Coupled Matrix Tensor Factorization (CMTF) facilitates the integration and analysis of multiple data sources and helps discover meaningful information. Nonnegative CMTF (N-CMTF) has been employed in many applications for identifying latent patterns, prediction, and recommendation. However, due to the added complexity with coupling between tensor and matrix data, existing N-CMTF algorithms exhibit poor computation efficiency. In this paper, a computationally efficient N-CMTF factorization algorithm is presented based on the column-wise element selection, preventing frequent gradient updates. Theoretical and empirical analyses show that the proposed N-CMTF factorization algorithm is not only more accurate but also more computationally efficient than existing algorithms in approximating the tensor as well as in identifying the underlying nature of factors. [ABSTRACT FROM AUTHOR]

In image detection, one problem is to test whether the set, though mainly consisting of uniformly scattered points, also contains a small fraction of points sampled from some (a priori unknown) curve, for example, a curve with Cα-norm bounded by β. One approach is to analyze the data by counting membership in multiscale multianisotropic strips, which involves an algorithm that delves into the length of the path connecting many consecutive “significant” nodes. In this paper, we develop the mathematical formalism of this algorithm and analyze the statistical property of the length of the longest significant run. The rate of convergence is derived. Using percolation theory and random graph theory, we present a novel probabilistic model named, pseudo-tree model. Based on the asymptotic results for the pseudo-tree model, we further study the length of the longest significant run in an “inflating” Bernoulli net. We find that the probability parameter p of significant node plays an important role: there is a threshold pc, such that in the cases of p < pc and p > pc, very different asymptotic behaviors of the length of the significant runs are observed. We apply our results to the detection of an underlying curvilinear feature and prove that the test based on our proposed longest run theory is asymptotically powerful. [ABSTRACT FROM AUTHOR]

Incomplete multi-view clustering (IMVC) optimally combines multiple pre-specified incomplete views to improve clustering performance. Among various excellent solutions, the recently proposed multiple kernel $k$ k -means with incomplete kernels (MKKM-IK) forms a benchmark, which redefines IMVC as a joint optimization problem where the clustering and kernel matrix imputation tasks are alternately performed until convergence. Though demonstrating promising performance in various applications, we observe that the manner of kernel matrix imputation in MKKM-IK would incur intensive computational and storage complexities, over-complicated optimization and limitedly improved clustering performance. In this paper, we first propose an Efficient and Effective Incomplete Multi-view Clustering (EE-IMVC) algorithm to address these issues. Instead of completing the incomplete kernel matrices, EE-IMVC proposes to impute each incomplete base matrix generated by incomplete views with a learned consensus clustering matrix. Moreover, we further improve this algorithm by incorporating prior knowledge to regularize the learned consensus clustering matrix. Two three-step iterative algorithms are carefully developed to solve the resultant optimization problems with linear computational complexity, and their convergence is theoretically proven. After that, we theoretically study the generalization bound of the proposed algorithms. Furthermore, we conduct comprehensive experiments to study the proposed algorithms in terms of clustering accuracy, evolution of the learned consensus clustering matrix and the convergence. As indicated, our algorithms deliver their effectiveness by significantly and consistently outperforming some state-of-the-art ones. [ABSTRACT FROM AUTHOR]

With growth in the number of smart devices and advancements in their hardware, in recent years, data-driven machine learning techniques have drawn significant attention. However, due to privacy and communication issues, it is not possible to collect this data at a centralized location. Federated learning is a machine learning setting where the centralized location trains a learning model over remote devices. Federated learning algorithms cannot be employed in the real world scenarios unless they consider unreliable and resource-constrained nature of the wireless medium. In this paper, we propose a federated learning algorithm that is suitable for cellular wireless networks. We prove its convergence, and provide a sub-optimal scheduling policy that improves the convergence rate. We also study the effect of local computation steps and communication steps on the convergence of the proposed algorithm. We prove, in practice, federated learning algorithms may solve a different problem than the one that they have been employed for if the unreliability of wireless channels is neglected. Finally, through numerous experiments on real and synthetic datasets, we demonstrate the convergence of our proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper presents a unified power flow algorithm for large and complex mixed ac/dc power systems based on the partitioning of power grids, a version of the two-level Newton method, and the modified nodal analysis formulation. Partitioning significantly increases numerical efficiency without altering the accuracy of the power flow solution. This approach can be applied to any combination of multiple dc and possibly non-synchronized ac systems. It considers the steady-state interaction of ac and dc systems for a wide range of converter representations and control functions. Simplification of elements within the converter stations is not required. Both dc and ac-side converter losses are adequately accounted for by using a generalized converter loss model. The steady-state converter equations are derived in their most general form, with the ac and/or dc-side power exchange or voltages defined as controlled quantities. [ABSTRACT FROM AUTHOR]

Distributed, consensus-based algorithms have emerged as a promising approach for the coordination of distributed energy resources (DER) due to their communication, computation, privacy and reliability advantages over centralized approaches. However, state-of-the-art consensus-based algorithms address the DER coordination problem in independent time periods and therefore are inherently unable to capture the time-coupling operating characteristics of flexible demand (FD) and energy storage (ES) resources. This paper demonstrates that state-of-the-art algorithms fail to converge when these time-coupling characteristics are considered. In order to address this fundamental limitation, a novel consensus-based algorithm is proposed which includes additional consensus variables. These variables express relative maximum power limits imposed on the FD and ES resources which effectively mitigate the concentration of the FD and ES responses at the same time periods and steer the consensual outcome to a feasible and optimal solution. The convergence and optimality of the proposed algorithm are theoretically proven while case studies numerically demonstrate its convergence, optimality, robustness to initialization and information loss, and plug-and-play adaptability. [ABSTRACT FROM AUTHOR]

In this paper, an intelligent reflecting surface (IRS) assisted spectrum sharing underlay cognitive radio (CR) wiretap channel (WTC) is studied, and we aim at enhancing the secrecy rate of secondary user in this channel subject to total power constraint at secondary transmitter (ST), interference power constraint (IPC) at primary receiver (PR) as well as unit modulus constraint at IRS. Due to extra IPC and eavesdropper (Eve) are considered, all the existing solutions for enhancing secrecy rate of IRS-assisted non-CR WTC as well as enhancing transmission rate in IRS-assisted CR channel without eavesdropper fail in this work. Therefore, we propose new numerical solutions to optimize the secrecy rate of this channel under full primary, secondary users’ channel state information (CSI) and three different cases of Eve’s CSI: full CSI, imperfect CSI with bounded estimation error, and no CSI. To solve the difficult non-convex optimization problem, an efficient alternating optimization (AO) algorithm is proposed to jointly optimize the beamformer at ST and phase shift coefficients at IRS. In particular, when optimizing the phase shift coefficients during each iteration of AO, a Dinkelbach based solution in combination with successive approximation and penalty based solution is proposed under full CSI and a penalty convex-concave procedure solution is proposed under imperfect Eve’s CSI. For no Eve’s CSI case, artificial noise (AN) aided approach is adopted to help enhancing the secrecy rate. Simulation results show that our proposed solutions for the IRS-assisted design greatly enhance the secrecy performance compared with the existing numerical solutions with and without IRS under full and imperfect Eve’s CSI. And positive secrecy rate can be achieved by our proposed AN aided approach given most channel realizations under no Eve’s CSI case so that secure communication also can be guaranteed. All of the proposed AO algorithms are guaranteed to monotonic convergence. [ABSTRACT FROM AUTHOR]

Alternating direction method of multipliers (ADMM) is an efficient implementation of linear programming (LP) decoding for low-density parity-check (LDPC) codes. By adding penalty terms to the objective function of the LP decoding model, ADMM variable node (VN) penalized decoding can suppress the non-integral solutions and improve the frame error rate (FER) performance in the low signal-to-noise ratio (SNR) region. In this paper, we propose a novel ADMM check node (CN) penalized decoding algorithm. Codeword solutions which satisfy all parity-check equations will have smaller penalty values than non-codeword solutions, including the non-integral solutions. We discuss the required properties of CN-penalty functions, propose a few functions that satisfy those properties, and study their performance/complexity trade-offs. We also investigate the convergence properties of the proposed algorithm and prove that its performance is independent of the transmitted codeword. Using Monte Carlo simulations and instanton analysis, we then demonstrate that the proposed CN-penalized decoder outperforms ADMM VN penalized decoders in both waterfall and error floor regions. This comes at the expense of some increase in the decoding complexity. [ABSTRACT FROM AUTHOR]

With the prosperity of vehicular networks and intelligent transport systems, vast amount of data can be easily collected by vehicular devices from their users and widely spread in the vehicular networks for the purpose of solving large-scale machine learning problems. Hence how to preserve the data privacy of users during the learning process has become a public concern. To address this concern, under the celebrated framework of differential privacy (DP), we present in this paper a decentralized parallel stochastic gradient descent (D-PSGD) algorithm, called DP2-SGD, which can offer protection for privacy of users in vehicular networks. With thorough analysis we show that DP2-SGD satisfies (ε, δ)-DP while the learning efficiency is the same as D-PSGD without privacy preservation. We also propose a refined algorithm called EC-SGD by introducing an error-compensate strategy. Extensive experiments show that EC-SGD can further improve the convergence efficiency over DP2-SGD in reality. [ABSTRACT FROM AUTHOR]

With the increase in the number of optimization objectives, balancing the convergence and diversity in evolutionary multiobjective optimization becomes more intractable. So far, a variety of evolutionary algorithms have been proposed to solve many-objective optimization problems (MaOPs) with more than three objectives. Most of the existing algorithms, however, find difficulties in simultaneously counterpoising convergence and diversity during the whole evolutionary process. To address the issue, this paper proposes to solve MaOPs via multistage evolutionary search. To be specific, a two-stage evolutionary algorithm is developed, where the convergence and diversity are highlighted during different search stages to avoid the interferences between them. The first stage pushes multiple subpopulations with different weight vectors to converge to different areas of the Pareto front. After that, the nondominated solutions coming from each subpopulation are selected for generating a new population for the second stage. Moreover, a new environmental selection strategy is designed for the second stage to balance the convergence and diversity close to the Pareto front. This selection strategy evenly divides each objective dimension into a number of intervals, and then one solution having the best convergence in each interval will be retained. To assess the performance of the proposed algorithm, 48 benchmark functions with 7, 10, and 15 objectives are used to make comparisons with five representative many-objective optimization algorithms. [ABSTRACT FROM AUTHOR]

The paper addresses a sequential changepoint detection problem for a general stochastic model, assuming that the observed data may be non-i.i.d. (i.e., dependent and non-identically distributed) and prior distribution of the change point is arbitrary. Tartakovsky and Veeravalli (2005), Baron and Tartakovsky (2006), and, more recently, Tartakovsky (2017) developed a general asymptotic theory of changepoint detection for non-i.i.d. stochastic models, assuming certain stability of the log-likelihood ratio process, in the case of simple hypotheses when both pre-change and post-change models are completely specified. However, in most applications, the post-change distribution is not completely known. In the present paper, we generalize previous results to the case of parametric uncertainty, assuming that the parameter of the post-change distribution is unknown. We introduce two detection rules based on mixtures–the mixture Shiryaev rule and the mixture Shiryaev–Roberts rule–and study their asymptotic properties in the Bayesian context. In particular, we provide sufficient conditions under which these rules are first-order asymptotically optimal, minimizing moments of the delay to detection as the probability of false alarm approaches zero. [ABSTRACT FROM AUTHOR]

In many different applications, it is important to determine the Hilbert transform of a given function. However, it is generally impossible to calculate it in closed form. Therefore Hilbert transform approximations are used. This paper studies the convergence and divergence behavior of general classes of such approximation methods. These classes are characterized by two very natural axioms and they include basically all known traditional numerical algorithms. The convergence of these methods is investigated on a family of signal spaces of continuous functions with finite energy. These spaces are parametrized by a number which measures the energy concentration in the low frequency components of the signal. It is shown that stable methods only exist on signal spaces with a sufficient energy concentration and this paper gives some explicit examples of convergent methods. On all other spaces in the family of signal spaces, every sampling-based Hilbert transform approximation shows a blowup behavior of its peak value, i.e., on these spaces, every sampling-based Hilbert transform approximation diverges. [ABSTRACT FROM AUTHOR]

This paper investigates the resource allocation problem for a group of agents communicating over a strongly connected directed graph, where the total objective function of the problem is composted of the sum of the local objective functions incurred by the agents. With local convex sets, we first design a continuous-time projection algorithm over a strongly connected and weight-balanced directed graph. Our convergence analysis indicates that when the local objective functions are strongly convex, the output state of the projection algorithm could asymptotically converge to the optimal solution of the resource allocation problem. In particular, when the projection operation is not involved, we show the exponential convergence at the equilibrium point of the algorithm. Second, we propose an adaptive continuous-time gradient algorithm over a strongly connected and weight-unbalanced directed graph for the reduced case without local convex sets. In this case, we prove that the adaptive algorithm converges exponentially to the optimal solution of the considered problem, where the local objective functions and their gradients satisfy strong convexity and Lipachitz conditions, respectively. Numerical simulations illustrate the performance of our algorithms. [ABSTRACT FROM AUTHOR]

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

Image acquisition in many biomedical imaging modalities is corrupted by Poisson noise followed by additive Gaussian noise. While total variation and related regularization methods for solving biomedical inverse problems are known to yield high quality reconstructions in most situations, such methods mostly use log-likelihood of either Gaussian or Poisson noise models, and rarely use mixed Poisson-Gaussian (PG) noise model. There is a recent work which deals with exact PG likelihood and total variation regularization. This method adapts the primal-dual approach involving gradients steps on the PG log-likelihood, with step size limited by the inverse of its Lipschitz constant. This leads to limitations in the convergence speed. Although the alternating direction method of multipliers (ADMM) algorithm does not have such step size restrictions, ADMM has never been applied for this problem, for the possible reason that PG log-likelihood is quite complex. In this paper, we develop an ADMM based optimization for roughness minimizing image restoration under PG log-likelihood. We achieve this by first developing a novel iterative method for computing the proximal solution of PG log-likelihood, deriving the termination conditions for this iterative method, and then integrating into a provably convergent ADMM scheme. We experimentally demonstrate that the proposed method outperform primal-dual method in most of the cases. [ABSTRACT FROM AUTHOR]

Recently, reconciling social networks, identifying the accounts belonging to the same individual across social networks, receives significant attention from both academic and industry. Most of the existing studies have limitations in the following three aspects: multiplicity, comprehensiveness and robustness. To address these limitations, we rethink this problem and, for the first time, robustly and comprehensively reconcile multiple social networks. In this paper, we propose two frameworks, MASTER and MASTER+, i.e., across Multiple social networks, integrate Attribute and STructure Embedding for Reconciliation. In MASTER, we first design a novel Constrained Dual Embedding model, simultaneously embedding and reconciling multiple social networks, to formulate this problem into a unified optimization. To address this optimization, we then design an effective NS-Alternating algorithm and prove it converges to KKT points. To further speed up MASTER, we propose a scalable framework, namely MASTER+. The core idea is to group accounts into clusters and then perform MASTER in each cluster in parallel. Specifically, we design an efficient Augmented Pre-Embedding model and Balance-aware Fuzzy Clustering algorithm for the high efficiency and the high accuracy. Extensive experiments demonstrate that both MASTER and MASTER+ outperform the state-of-the-art approaches. Moreover, MASTER+ inherits the effectiveness of MASTER and enjoys higher efficiency. [ABSTRACT FROM AUTHOR]

Numerous tasks at the core of statistics, learning and vision areas are specific cases of ill-posed inverse problems. Recently, learning-based (e.g., deep) iterative methods have been empirically shown to be useful for these problems. Nevertheless, integrating learnable structures into iterations is still a laborious process, which can only be guided by intuitions or empirical insights. Moreover, there is a lack of rigorous analysis about the convergence behaviors of these reimplemented iterations, and thus the significance of such methods is a little bit vague. This paper moves beyond these limits and proposes Flexible Iterative Modularization Algorithm (FIMA), a generic and provable paradigm for nonconvex inverse problems. Our theoretical analysis reveals that FIMA allows us to generate globally convergent trajectories for learning-based iterative methods. Meanwhile, the devised scheduling policies on flexible modules should also be beneficial for classical numerical methods in the nonconvex scenario. Extensive experiments on real applications verify the superiority of FIMA. [ABSTRACT FROM AUTHOR]

Suppose that ${ { y}}= \lvert \text {A} { x}_{0}\rvert +\eta $ where ${ x}_{0}\in {\mathbb R} ^{{d}}$ is the target signal and $\eta \in {\mathbb R}^{{m}}$ is a noise vector. The aim of phase retrieval is to estimate ${x}_{0}$ from ${ { y}}$. A popular model for estimating ${ x}_{0}$ is the nonlinear least squares ${\widehat { {x}}}:={\mathrm{ argmin}}_{ x} \| \lvert \text {A} { x}\rvert - { { y}}\|_{2}$. One has already developed many efficient algorithms for solving the model, such as the seminal error reduction algorithm. In this paper, we present the estimation performance of the model with proving that $\| {\widehat { {x}}}- { x}_{0}\|\lesssim {\|\eta \|_{2}}/{\sqrt {{m}}}$ under the assumption of A being a Gaussian random matrix. We also prove the reconstruction error ${\|\eta \|_{2}}/{\sqrt {{m}}}$ is sharp. For the case where ${ x}_{0}$ is sparse, we study the estimation performance of both the nonlinear Lasso of phase retrieval and its unconstrained version. Our results are non-asymptotic, and we do not assume any distribution on the noise $\eta $. To the best of our knowledge, our results represent the first theoretical guarantee for the nonlinear least squares and for the nonlinear Lasso of phase retrieval. [ABSTRACT FROM AUTHOR]

In this paper, the robust stabilization issues on uncertain system with ${H_\infty }$ performance are investigated by applying impulsive time related control algorithm. A hybrid control scheme is novelly proposed using proportional impulsive plus integral mechanism. This impulse-time-dependent (ITD) feedback controller employs the proportional control action ${K_{p}}$ at impulse instants for stabilizing. Then, intermittently the integral control strategy ${K_{I}(t)}$ is implemented between impulsive time intervals for improving the state convergence. Sufficient conditions are derived to ensure the closed-loop performance. In addition, a memory controller ${K_{D}}$ is introduced to tackle the stabilization problem of system with signal transmission delay. Furthermore, a distributed time-varying delay ${\delta _{k}(t)}$ is defined in the impulsive control-based system. Considering discrete and distributed time-varying delays concurrently, the ITD stability criteria are deduced to guarantee the ${H_\infty }$ performance on uncertain system with exogenous disturbance. Numerical examples are conducted to show the expected stabilization and state convergence response with disturbance attenuation of this development. [ABSTRACT FROM AUTHOR]

This paper considers the problem of data driven iterative learning control (DDILC) for a class of nonaffine nonlinear systems subject to data quantization and sensor saturation. Two novel quantized DDILC (QDDILC) algorithms are proposed based on saturated and quantized information of system outputs. The convergence of the proposed QDDILC algorithms is strictly proved and the effects of output saturation and data quantification are also analyzed. It is shown that sensor saturation does not change the convergence property, thus it causes the convergence rate to slow down. For the QDDILC algorithm, data quantization will cause the tracking error to converge to a bound depending on the quantization level. However, the modified QDDILC algorithm, which using the different quantization scheme from QDDILC algorithm, can ensure that the tracking error converges to zero. Illustrative simulations are exploited to verify the theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a powerful technique based on Leave-One-Out analysis to the study of low-rank matrix completion problems. Using this technique, we develop a general approach for obtaining fine-grained, entrywise bounds for iterative stochastic procedures in the presence of probabilistic dependency. We demonstrate the power of this approach in analyzing two of the most important algorithms for matrix completion: (i) the non-convex approach based on Projected Gradient Descent (PGD) for a rank-constrained formulation, also known as the Singular Value Projection algorithm, and (ii) the convex relaxation approach based on nuclear norm minimization (NNM). Using this approach, we establish the first convergence guarantee for the original form of PGD without regularization or sample splitting, and in particular shows that it converges linearly in the infinity norm. For NNM, we use this approach to study a fictitious iterative procedure that arises in the dual analysis. Our results show that NNM recovers an $d$ -by- $d$ rank- $r$ matrix with $\mathcal {O}(\mu r \log (\mu r) d\log d)$ observed entries. This bound has optimal dependence on the matrix dimension and is independent of the condition number. To the best of our knowledge, none of previous sample complexity results for tractable matrix completion algorithms satisfies these two properties simultaneously. [ABSTRACT FROM AUTHOR]

Graphical data is comprised of a graph with marks on its edges and vertices. The mark indicates the value of some attribute associated to the respective edge or vertex. Examples of such data arise in social networks, molecular and systems biology, and web graphs, as well as in several other application areas. Our goal is to design schemes that can efficiently compress such graphical data without making assumptions about its stochastic properties. Namely, we wish to develop a universal compression algorithm for graphical data sources. To formalize this goal, we employ the framework of local weak convergence, also called the objective method, which provides a technique to think of a marked graph as a kind of stationary stochastic processes, stationary with respect to movement between vertices of the graph. In recent work, we have generalized a notion of entropy for unmarked graphs in this framework, due to Bordenave and Caputo, to the case of marked graphs. We use this notion to evaluate the efficiency of a compression scheme. The lossless compression scheme we propose in this paper is then proved to be universally optimal in a precise technical sense. It is also capable of performing local data queries in the compressed form. [ABSTRACT FROM AUTHOR]

For systems that can only be locally stabilized, control laws and their effective regions are both important. In this paper, invariant policy iteration is proposed to solve the optimal control of discrete-time systems. At each iteration, a given policy is evaluated in its invariantly admissible region, and a new policy and a new region are updated for the next iteration. Theoretical analysis shows the method is regionally convergent to the optimal value and the optimal policy. Combined with sum-of-squares polynomials, the method is able to achieve the near-optimal control of a class of discrete-time systems. An invariant adaptive dynamic programming algorithm is developed to extend the method to scenarios where system dynamics is not available. Online data are utilized to learn the near-optimal policy and the invariantly admissible region. Simulated experiments verify the effectiveness of our method. [ABSTRACT FROM AUTHOR]