Search

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

In this paper, we propose to extend the standard Convolutional Dictionary Learning problem to a tensor representation where the activations are constrained to be “low-rank” through a Canonical Polyadic decomposition. We show that this additional constraint increases the robustness of the CDL with respect to noise and improve the interpretability of the final results. In addition, we discuss in detail the advantages of this representation and introduce two algorithms, based on ADMM or FISTA, that efficiently solve this problem. We show that by exploiting the low rank property of activations, they achieve lower complexity than the main CDL algorithms. Finally, we evaluate our approach on a wide range of experiments, highlighting the modularity and the advantages of this tensorial low-rank formulation. [ABSTRACT FROM AUTHOR]

The use of a large-scale antenna array (LSAA) has become an important characteristic of multi-antenna communication systems to achieve beamforming gains such as in designing millimeter-wave (mmWave) systems to combat severe propagation losses. In such applications, each antenna element has to be driven by a radio frequency (RF) chain for the implementation of fully-digital beamformers, significantly increasing the hardware cost, complexity, and power consumption. Therefore, constant-modulus analog beamforming (CMAB) becomes a viable solution. In this paper, we consider the scaled analog beamforming (SAB) or constant-modulus analog beamforming (CMAB) architecture and design the system parameters by solving two variants of beampattern matching problem. In the first case, both the magnitude and phase of the beampattern are matched to the given desired beampattern whereas in the second case, only the magnitude of the beampattern is matched. Both the beampattern matching problems are cast as a variant of the constant-modulus least-squares (CLS) problem. We provide efficient algorithms based on the alternating majorization-minimization (AMM) framework that combines the alternating minimization and the MM frameworks and the conventional-cyclic coordinate descent (C-CCD) algorithms to solve the problem in each case. We also propose algorithms based on a new modified-CCD (M-CCD) based approach. For all the developed algorithms we prove convergence to a Karush-Kuhn-Tucker (KKT) point (or a stationary point). Numerical results demonstrate that the proposed algorithms converge faster than the state-of-the-art solutions. Among all the algorithms, the M-CCD-based algorithms have faster convergence when evaluated in terms of the number of iterations and the AMM-based algorithms offer lower complexity. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

This paper studies the convergence properties of the well-known message-passing algorithm for convex optimisation. Under the assumption of pairwise separability and scaled diagonal dominance, asymptotic convergence is established and a simple bound for the convergence rate is provided for message-passing. In comparison with previous results, our results do not require the given convex program to have known convex pairwise components and that our bound for the convergence rate is tighter and simpler. When specialised to quadratic optimisation, we generalise known results by providing a very simple bound for the convergence rate. [ABSTRACT FROM AUTHOR]

In this paper, we consider the policy evaluation problem in reinforcement learning with agents on a decentralized and directed network. In order to evaluate the quality of a fixed policy in this decentralized setting, one option is for agents to run decentralized temporal-difference (TD) collaboratively. To account for the practical scenarios where the state and action spaces are large and malicious attacks emerge, we focus on the decentralized TD learning with linear function approximation in the presence of malicious agents (often termed as Byzantine agents). We propose a trimmed mean-based Byzantine-resilient decentralized TD algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error that depends on the number of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

We consider the problem of minimizing the $\ell _1$ norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms with provable guarantees for tackling this problem. Specifically, our contribution is fourfold. First, we present a manifold proximal point algorithm (ManPPA) for the problem and show that it converges at a global sublinear rate. Furthermore, we show that ManPPA can achieve a local quadratic convergence rate when applied to sharp instances of the problem. Second, we develop a semismooth Newton-based inexact augmented Lagrangian method for computing the search direction in each iteration of ManPPA and show that it has an asymptotic superlinear convergence rate. Third, we propose a stochastic variant of ManPPA called StManPPA, which is well suited for large-scale computation, and establish its sublinear convergence rate. Both ManPPA and StManPPA have provably faster convergence rates than existing subgradient-type methods. Fourth, using ManPPA as a building block, we propose a new heuristic method for solving a matrix analog of the problem, in which the sphere is replaced by the Stiefel manifold. The results from our extensive numerical experiments on the ODL and RSR problems demonstrate the efficiency and efficacy of our proposed methods. [ABSTRACT FROM AUTHOR]

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers’ choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms. [ABSTRACT FROM AUTHOR]

This work addresses the robust principal component analysis (PCA) problem using generalized nonoconvex regularization for low-rank and sparsity promotion. While the popular nuclear and $\ell _1$ -norm penalties have a bias problem, nonconvex regularization can alleviate the bias problem and can be expected to achieve better performance. In this paper, a proximal block coordinate descent (BCD) algorithm is used to efficiently solve the nonconvex regularized robust PCA problem. It is globally convergent under weak conditions. Further, for a popular class of penalties having discontinuous threshoding functions, we establish the convergence to a restricted strictly local minimizer and, also, a local linear convergence rate for the proximal BCD algorithm. Moreover, convergence to a local minimizer has been derived for hard-thresholding. Our result is the first on nonconvex robust PCA with established convergence to strictly local minimizer with local linear convergence rate. Numerical experiments have been provided to demonstrate the performance of the new algorithm. [ABSTRACT FROM AUTHOR]

We propose a novel adaptive kernel-based regression method for complex-valued signals: the generalized complex-valued kernel least-mean-square (gCKLMS). We borrow from the new results on widely linear reproducing kernel Hilbert space (WL-RKHS) for nonlinear regression and complex-valued signals, recently proposed by the authors. This paper shows that in the adaptive version of the kernel regression for complex-valued signals we need to include another kernel term, the so-called pseudo-kernel. This new solution is endowed with better representation capabilities in complex-valued fields since it can efficiently decouple the learning of the real and the imaginary part. Also, we review previous realizations of the complex KLMS algorithm and its augmented version to prove that they can be rewritten as particular cases of the gCKLMS. Furthermore, important conclusions on the design of the kernels are drawn that help to greatly improve the convergence of the algorithms. In the experiments, we revisit the nonlinear channel equalization problem to highlight the better convergence of the gCKLMS compared to previous solutions. Also, the flexibility of the proposed generalized approach is tested in a second experiment with non-independent real and imaginary parts. The results illustrate the significant performance improvements of the gCKLMS approach when the complex-valued signals have different properties for the real and imaginary parts. [ABSTRACT FROM AUTHOR]

Sampling from the lattice Gaussian distribution has emerged as an important problem in coding, decoding, and cryptography. In this paper, lattice reduction technique is adopted to Gibbs sampler for lattice Gaussian sampling. First, with respect to lattice Gaussian distribution, we show the convergence rate of systematic scan Gibbs sampling is characterized by the Hirschfeld-Gebelein-Rényi maximal correlation among the multivariate of being sampled. Then, Lattice-reduction-aided Gibbs algorithm is proposed to sample from an equivalent lattice Gaussian distribution but with less correlated multivariate, thus leading to a better Markov mixing. After that, we extend the proposed lattice-reduction-aided Gibbs sampling to lattice decoding, where the choice of the standard deviation for the sampling is fully investigated. A customized solution that suits for each specific decoding case by Euclidean distance is given, which results in a better tradeoff between Markov mixing and sampler decoding. Based on it, a startup mechanism is also proposed for Gibbs sampler decoding, where decoding complexity can be reduced without performance loss. Moreover, the recycling Gibbs sampling that exploits the potential of samples is also considered to improve the decoding performance in lattice decoding. Simulation results based on large-scale uncoded multiple-input multiple-output detection are presented to confirm the performance gain and complexity reduction. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel sparse signal recovery algorithm called the trainable iterative soft thresholding algorithm (TISTA). The proposed algorithm consists of two estimation units: a linear estimation unit and a minimum mean squared error (MMSE) estimator based shrinkage unit. The error variance required in the MMSE shrinkage unit is precisely estimated from a tentative estimate of the original signal. The remarkable feature of the proposed scheme is that TISTA includes adjustable variables that control step size and the error variance for the MMSE shrinkage function. The variables are adjusted by standard deep learning techniques. The number of trainable variables of TISTA is nearly equal to the number of iteration rounds and is much smaller than that of known learnable sparse signal recovery algorithms. This feature leads to highly stable and fast training processes of TISTA. Computer experiments show that TISTA is applicable to various classes of sensing matrices, such as Gaussian matrices, binary matrices, and matrices with large condition numbers. Numerical results also demonstrate that, in many cases, TISTA provides significantly faster convergence than approximate message passing (AMP) and the learned iterative shrinkage thresholding algorithm and also outperforms orthogonal AMP in the NMSE performance. [ABSTRACT FROM AUTHOR]

In CANDECOMP/PARAFAC tensor decomposition, degeneracy often occurs in some difficult scenarios, especially, when the rank exceeds the tensor dimension, or when the loading components are highly collinear in several or all modes, or when CPD does not have an optimal solution. In such cases, norms of some rank-1 tensors become significantly large and cancel each other. This makes algorithms getting stuck in local minima while running a huge number of iterations does not improve the decomposition. In this paper, we propose an error preservation correction method to deal with such problem. Our aim is to seek an alternative tensor, which preserves the approximation error, but norms of rank-1 tensor components of the new tensor are minimized. Alternating and all-at-once correction algorithms have been developed for the problem. In addition, we propose a novel CPD with a bound constraint on the norm of the rank-one tensors. The method can be useful for decomposing tensors that cannot be performed by traditional algorithms. Finally, we demonstrate an application of the proposed method in image denoising and decomposition of the weight tensors in convolutional neural networks. [ABSTRACT FROM AUTHOR]

Phase retrieval is an inverse problem which consists in recovering an unknown signal from a set of absolute squared projections. Recently, gradient descent algorithms have been developed to solve this problem. However, their optimization cost functions are non-convex and non-smooth. To address the non-smoothness of the cost function, some of these methods use truncation thresholds to calculate a truncated step gradient direction. But, the truncation requires designing parameters to obtain a desired performance in the phase recovery, which drastically modifies the search direction update, increasing the sampling complexity. Therefore, this paper develops the Phase Retrieval Smoothing Conjugate Gradient method (PR-SCG) which uses a smoothing function to retrieve the signal. PR-SCG is based on the smooth-ing projected gradient method which is useful for non-convex optimization problems. PR-SCG uses a nonlinear conjugate gradient of the smoothing function as the search direction to accelerate the convergence. Furthermore, the incremental Stochastic Smoothing Phase Retrieval algorithm (SSPR) is developed. SSPR involves a single equation per iteration which results in a simple, scalable, and fast approach useful when the size of the signal is large. Also, it is shown that SSPR converges linearly to the true signal, up to a global unimodular constant. Additionally, the proposed methods do not require truncation parameters. Simulation results are provided to validate the efficiency of PR-SCG and SSPR compared to existing phase retrieval algorithms. It is shown that PR-SCG and SSPR are able to reduce the number of measurements and iterations to recover the phase, compared with recently developed algorithms. [ABSTRACT FROM AUTHOR]

This paper proposes a decentralized algorithm for solving a consensus optimization problem defined in a static networked multi-agent system, where the local objective functions have the smooth+nonsmooth composite form. Examples of such problems include decentralized constrained quadratic programming and compressed sensing problems, as well as many regularization problems arising in inverse problems, signal processing, and machine learning, which have decentralized applications. This paper addresses the need for efficient decentralized algorithms that take advantages of proximal operations for the nonsmooth terms. We propose a proximal gradient exact first-order algorithm (PG-EXTRA) that utilizes the composite structure and has the best known convergence rate. It is a nontrivial extension to the recent algorithm EXTRA. At each iteration, each agent locally computes a gradient of the smooth part of its objective and a proximal map of the nonsmooth part, as well as exchanges information with its neighbors. The algorithm is “exact” in the sense that an exact consensus minimizer can be obtained with a fixed step size, whereas most previous methods must use diminishing step sizes. When the smooth part has Lipschitz gradients, PG-EXTRA has an ergodic convergence rate of O\left(1\over k\right) in terms of the first-order optimality residual. When the smooth part vanishes, PG-EXTRA reduces to P-EXTRA, an algorithm without the gradients (so no “G” in the name), which has a slightly improved convergence rate at o\left(1\over k\right) in a standard (non-ergodic) sense. Numerical experiments demonstrate effectiveness of PG-EXTRA and validate our convergence results [ABSTRACT FROM PUBLISHER]

In this paper, we propose to fit a sparse logistic regression model by a weakly convex regularized nonconvex optimization problem. The idea is based on the finding that a weakly convex function as an approximation of the $\ell _0$ pseudo norm is able to better induce sparsity than the commonly used $\ell _1$ norm. For a class of weakly convex sparsity inducing functions, we prove the nonconvexity of the corresponding problem and study its local optimality conditions and the choice of the regularization parameter. Despite the nonconvexity, a method based on proximal gradient descent is used to solve the general weakly convex sparse logistic regression, and its convergence behavior is studied theoretically. Then, the general framework is applied to a specific weakly convex function, and a local optimality condition and a bound on the logistic loss at a local optimum are provided. The solution method is instantiated in this case as an iterative firm-shrinkage algorithm, and a Nesterov acceleration is used with a convergence guarantee. Its effectiveness is demonstrated in numerical experiments by both randomly generated and real datasets. [ABSTRACT FROM AUTHOR]

A set-membership affine projection algorithm is proposed that can estimate a complex-valued channel matrix using a set of complex-valued pilots in the presence of additive white Gaussian noise. It is shown that the algorithm converges faster than the well-known set-membership normalized least mean square algorithm (SM-NLMS) while it resolves the high steady-state error value and the complexity issues in the regular affine projection algorithm. The fast convergence of the proposed algorithm means that a shorter training sequence inside each data block is required, which in turn improves the effective bit rate. This fast convergence is more pronounced when the pilot vectors are highly correlated. We incorporated the set-membership filtering framework into our study to reduce the computational complexity of the algorithm and preserve energy in the WSNs. Other studies have shown the superiority of the adaptive filtering algorithms, and in particular the NLMS algorithm, over other alternatives in various signal processing areas in WSNs, therefore, our proposed algorithm is a powerful substitute for a variety of algorithms. In other words, the implementation of various signal processing algorithms for different purposes can be replaced with the implementation of the proposed multipurpose algorithm. In this paper, we combine the results of our previous studies and prove the convergence of the algorithm. Furthermore, the steady-state analysis in the output mean square error is presented for two cases of pilot signals, and in the conducted simulations, the MSE performance of the algorithm is compared with the regular affine projection algorithm and the SM-NLMS algorithm. [ABSTRACT FROM AUTHOR]

In this paper, we present an algorithm for the sparse signal recovery problem that incorporates damped Gaussian generalized approximate message passing (GGAMP) into expectation-maximization-based sparse Bayesian learning (SBL). In particular, GGAMP is used to implement the E-step in SBL in place of matrix inversion, leveraging the fact that GGAMP is guaranteed to converge with appropriate damping. The resulting GGAMP-SBL algorithm is much more robust to arbitrary measurement matrix A than the standard damped GAMP algorithm while being much lower complexity than the standard SBL algorithm. We then extend the approach from the single measurement vector case to the temporally correlated multiple measurement vector case, leading to the GGAMP-TSBL algorithm. We verify the robustness and computational advantages of the proposed algorithms through numerical experiments. [ABSTRACT FROM AUTHOR]

This paper examines the mean-square error performance of diffusion stochastic algorithms under a generalized coordinate-descent scheme. In this setting, the adaptation step by each agent is limited to a random subset of the coordinates of its stochastic gradient vector. The selection of coordinates varies randomly from iteration to iteration and from agent to agent across the network. Such schemes are useful in reducing computational complexity at each iteration in power-intensive large data applications. They are also useful in modeling situations where some partial gradient information may be missing at random. Interestingly, the results show that the steady-state performance of the learning strategy is not always degraded, while the convergence rate suffers some degradation. The results provide yet another indication of the resilience and robustness of adaptive distributed strategies. [ABSTRACT FROM AUTHOR]

Stochastic approximation techniques play an important role in solving many problems encountered in machine learning or adaptive signal processing. In these contexts, the statistics of the data are often unknown a priori or their direct computation is too intensive, and they have thus to be estimated online from the observed signals. For batch optimization of an objective function being the sum of a data fidelity term and a penalization (e.g., a sparsity promoting function), Majorize-Minimize (MM) methods have recently attracted much interest since they are fast, highly flexible, and effective in ensuring convergence. The goal of this paper is to show how these methods can be successfully extended to the case when the data fidelity term corresponds to a least squares criterion and the cost function is replaced by a sequence of stochastic approximations of it. In this context, we propose an online version of an MM subspace algorithm and we study its convergence by using suitable probabilistic tools. Simulation results illustrate the good practical performance of the proposed algorithm associated with a memory gradient subspace, when applied to both nonadaptive and adaptive filter identification problems. [ABSTRACT FROM PUBLISHER]

Unit-modulus least squares (ULS) problems arise in many applications, including phase-only beamforming, sensor network localization, synchronization, phase retrieval, and radar code design. ULS formulations can always be recast as unit-modulus quadratic programs, to which semidefinite relaxation (SDR) can be applied, and is often the state-of-the-art approach. However, SDR lifts the problem dimension (i.e., the number of variables) from $N$ to $N^2$, which drastically increases the memory burden and computational cost when the problem size is already large—e.g., when designing phase-only beamformer weights for massive multiple-input-multiple-output systems. This paper focuses on scalable first-order algorithms for the ULS problem and some of its variants. It advocates using simple gradient projection (GP) as a starting point for solving the ULS problem, establishes global convergence of GP to a Karush–Kuhn–Tucker point for this NP-hard problem, and bounds its iteration complexity. Then it proposes ULS extensions tailored to reflect practical beamformer design objectives, bringing in and exploiting new degrees of freedom to improve the beampattern designs. Simple variants of GP are proposed to deal with these extended ULS problems. Simulations are used to showcase the effectiveness of the proposed algorithms in both the plain ULS problem and in the context of phase-only beamforming. [ABSTRACT FROM PUBLISHER]

Particle filters are broadly used to approximate posterior distributions of hidden states in state-space models by means of sets of weighted particles. While the convergence of the filter is guaranteed when the number of particles tends to infinity, the quality of the approximation is usually unknown but strongly dependent on the number of particles. In this paper, we propose a novel method for assessing the convergence of particle filters in an online manner, as well as a simple scheme for the online adaptation of the number of particles based on the convergence assessment. The method is based on a sequential comparison between the actual observations and their predictive probability distributions approximated by the filter. We provide a rigorous theoretical analysis of the proposed methodology and, as an example of its practical use, we present simulations of a simple algorithm for the dynamic and online adaptation of the number of particles during the operation of a particle filter on a stochastic version of the Lorenz 63 system. [ABSTRACT FROM PUBLISHER]

In this paper, we present a novel approach for achieving phase desynchronization in a pulse-coupled oscillator network. Ensuring phase desynchronization is a difficult problem, and existing results are constrained to a completely interconnected network and a fixed number of nodes. Our approach is more robust than previous approaches, removing the constraint of a fixed number of nodes. The removal of this constraint is significant because it allows the network to receive and drop nodes freely without any change to the phase update strategy. Also, to our knowledge, our approach is the first to prove the convergence to the desynchronized state for a topology that is more general than the all-to-all topology. More specifically, our approach is applicable to any circulant, symmetric network topology, including the circulant symmetric ring topology. Rigorous mathematical proofs are provided to support the result that any circulant, symmetric network with ordered phases under our proposed algorithm will converge to uniform phase desynchronization. Simulation results are also presented to demonstrate the algorithm's performance. [ABSTRACT FROM PUBLISHER]

A multiband-structured subband adaptive filter (MSAF) algorithm was introduced to achieve a fast convergence rate for the correlated input signal. The convergence analysis of the adaptive filter algorithm is an important concept because it provides a guideline to design the adaptive filter. However, the convergence analysis of the MSAF algorithm has not been researched as extensively as that of the normalized least-mean-square algorithm. Therefore, it needs to be researched. In this paper, we present a new approach to the mean-square deviation (MSD) analysis of the MSAF algorithm by using the persistently exciting input and the practical assumption that the stopband attenuation of the prototype filter is high. Unlike the previous analysis, the proposed analysis is possible to be applied to the long-length adaptive filter such as the acoustic echo cancellation. The proposed analysis is also applied to a non-stationary model with a random walk of the optimal weight vector. The simulation results match with the theoretical results in both the transient-state and steady-state MSD. [ABSTRACT FROM AUTHOR]

In Part II of this paper, also in this issue, we carried out a detailed mean-square-error analysis of the performance of asynchronous adaptation and learning over networks under a fairly general model for asynchronous events including random topologies, random link failures, random data arrival times, and agents turning on and off randomly. In this Part III, we compare the performance of synchronous and asynchronous networks. We also compare the performance of decentralized adaptation against centralized stochastic-gradient (batch) solutions. Two interesting conclusions stand out. First, the results establish that the performance of adaptive networks is largely immune to the effect of asynchronous events: the mean and mean-square convergence rates and the asymptotic bias values are not degraded relative to synchronous or centralized implementations. Only the steady-state mean-square-deviation suffers a degradation in the order of \nu, which represents the small step-size parameters used for adaptation. Second, the results show that the adaptive distributed network matches the performance of the centralized solution. These conclusions highlight another critical benefit of cooperation by networked agents: cooperation does not only enhance performance in comparison to stand-alone single-agent processing, but it also endows the network with remarkable resilience to various forms of random failure events and is able to deliver performance that is as powerful as batch solutions. [ABSTRACT FROM PUBLISHER]

The success of compressed sensing relies essentially on the ability to efficiently find an approximately sparse solution to an under-determined linear system. In this paper, we developed an efficient algorithm for the sparsity promoting \ell^1-regularized least squares problem by coupling the primal dual active set strategy with a continuation technique (on the regularization parameter). In the active set strategy, we first determine the active set from primal and dual variables, and then update the primal and dual variables by solving a low-dimensional least square problem on the active set, which makes the algorithm very efficient. The continuation technique globalizes the convergence of the algorithm, with provable global convergence under restricted isometry property (RIP). Further, we adopt two alternative methods, i.e., a modified discrepancy principle and a Bayesian information criterion, to choose the regularization parameter automatically. Numerical experiments indicate that our algorithm is very competitive with state-of-the-art algorithms in terms of accuracy and efficiency, without a priori information about the regularization parameter. [ABSTRACT FROM AUTHOR]