Search

Hammerstein model with a static nonlinearity followed by a linear filter is commonly used in numerous applications. This paper focuses on adaptive filtering techniques for parameter identification of Hammerstein systems and output prediction of nonlinear systems. By formulating the underlying filtering problem as a recursive bilinear least-squares optimization with the non-convex feasible region constraint, we develop a recursive non-convex projected least-squares (RncPLS) algorithm based on alternating direction method of multipliers (ADMM). The RncPLS algorithm alternates between implementing ridge regression and projecting on the non-convex feasible set, which successively refines the system parameters. The convergence and accuracy properties of the proposed RncPLS algorithm are theoretically investigated. Moreover, extensive simulation results in the context of system identification, nonlinear predication, and acoustic echo cancellation, are also included to demonstrate the performance characteristics of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Data representation techniques have made a substantial contribution to advancing data processing and machine learning (ML). Improving predictive power was the focus of previous representation techniques, which unfortunately perform rather poorly on the interpretability in terms of extracting underlying insights of the data. Recently, the Kolmogorov model (KM) was studied, which is an interpretable and predictable representation approach to learning the underlying probabilistic structure of a set of random variables. The existing KM learning algorithms using semi-definite relaxation with randomization (SDRwR) or discrete monotonic optimization (DMO) have, however, limited utility to big data applications because they do not scale well computationally.In this paper, we propose a computationally scalable KM learning algorithm, based on the regularized dual optimization combined with enhanced gradient descent (GD) method. To make our method more scalable to large-dimensional problems, we propose two acceleration schemes, namely, the eigenvalue decomposition (EVD) elimination strategy and an approximate EVD algorithm. Furthermore, a thresholding technique by exploiting the error bound analysis and leveraging the normalized Minkowski $\ell _1$ -norm, is provided for the selection of the number of iterations of the approximate EVD algorithm. When applied to big data applications, it is demonstrated that the proposed method can achieve compatible training/prediction performance with significantly reduced computational complexity; roughly two orders of magnitude improvement in terms of the time overhead, compared to the existing KM learning algorithms. Furthermore, it is shown that the accuracy of logical relation mining for interpretability by using the proposed KM learning algorithm exceeds 80%. [ABSTRACT FROM AUTHOR]

Constrained reinforcement learning (CRL), also termed as safe reinforcement learning, is a promising technique enabling the deployment of RL agent in real-world systems. In this paper, we propose a successive convex approximation based off-policy optimization (SCAOPO) algorithm to solve the general CRL problem, which is formulated as a constrained Markov decision process (CMDP) in context of the average cost. The SCAOPO is based on solving a sequence of convex objective/feasibility optimization problems obtained by replacing the objective and constraint functions in the original problem with convex surrogate functions. The proposed SCAOPO enables reuse of experiences from previous updates, thereby significantly reducing implementation cost when deployed in real-world engineering systems that need to online learn the environment. In spite of the time-varying state distribution and the stochastic bias incurred by off-policy learning, the SCAOPO with a feasible initial point can still provably converge to a Karush-Kuhn-Tucker (KKT) point of the original problem almost surely. [ABSTRACT FROM AUTHOR]

One-bit compressive sensing gains its popularity in signal processing and communications due to its low storage costs and low hardware complexity. However, it has been a challenging task to recover the signal only by exploiting the one-bit (the sign) information. In this paper, we appropriately formulate the one-bit compressive sensing into a double-sparsity constrained optimization problem. The first-order optimality conditions for this nonconvex and discontinuous problem are established via the newly introduced $\tau$ -stationarity, based on which, a gradient projection subspace pursuit (GPSP) algorithm is developed. It is proven that GPSP can converge globally and terminate within finite steps. Numerical experiments have demonstrated its excellent performance in terms of a high order of accuracy with a high computational speed. [ABSTRACT FROM AUTHOR]

Optimizing non-convex functions is of primary importance in modern pattern recognition because it underlies the training of deep networks and nonlinear dimensionality reduction. First-order algorithms under suitable randomized perturbations or step-size rules have been shown to be effective for such settings as their limit points can be guaranteed to be local extrema rather than saddle points. However, it is well-known that the practical convergence of first-order methods is slower than those which exploit additional structure. In particular, empirically, successive convex approximation (SCA) converges faster than first-order methods. However, to date, SCA in general non-convex settings converges to first-order stationary points, which could either be local extrema or saddle points whose performance is typically inferior. To mitigate this issue, we propose calibrated randomized perturbations of SCA, which exhibit the improved convergence rate as compared to the gradient descent counter part. In particular, our main technical contributions are to establish the non-asymptotic performance of SCA algorithm and its perturbed variant converges to an approximate second-order stationary point. Experiments on multi-dimensional scaling, a machine learning problem whose training objective is non-convex, substantiate the performance gains associated with employing random perturbations. [ABSTRACT FROM AUTHOR]

We consider stochastic optimization of a smooth non-convex loss function with a convex non-smooth regularizer. In the online setting, where a single sample of the stochastic gradient of the loss is available at every iteration, the problem can be solved using the proximal stochastic gradient descent (SGD) algorithm and its variants. However in many problems, especially those arising in communications and signal processing, information beyond the stochastic gradient may be available thanks to the structure of the loss function. By constructing a strongly convex surrogate of the loss function at every iteration, the stochastic SCA algorithms can exploit the structural properties of the loss function. In this work, we take a closer look at the stochastic SCA algorithm and develop its asynchronous variant which can be used for resource allocation in wireless networks. While the stochastic SCA algorithm is known to converge asymptotically, its iteration complexity has not been well-studied, and is the focus of the current work. The insights obtained from the non-asymptotic analysis allow us to develop a more practical asynchronous variant of the stochastic SCA algorithm which allows the use of surrogate calculated in earlier iterations. We characterize precise bound on the maximum delay the algorithm can tolerate, while still achieving the same convergence rate. We apply the algorithm to the problem of linear precoding in wireless sensor networks, where it can be implemented at low complexity but is shown to perform well in practice. [ABSTRACT FROM AUTHOR]

In this paper, we consider the policy evaluation problem in reinforcement learning with agents on a decentralized and directed network. In order to evaluate the quality of a fixed policy in this decentralized setting, one option is for agents to run decentralized temporal-difference (TD) collaboratively. To account for the practical scenarios where the state and action spaces are large and malicious attacks emerge, we focus on the decentralized TD learning with linear function approximation in the presence of malicious agents (often termed as Byzantine agents). We propose a trimmed mean-based Byzantine-resilient decentralized TD algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error that depends on the number of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

The first moment and second central moments of the portfolio return, a.k.a. mean and variance, have been widely employed to assess the expected profit and risk of the portfolio. Investors pursue higher mean and lower variance when designing the portfolios. The two moments can well describe the distribution of the portfolio return when it follows the Gaussian distribution. However, the real world distribution of assets return is usually asymmetric and heavy-tailed, which is far from being a Gaussian distribution. The asymmetry and the heavy-tailedness are characterized by the third and fourth central moments, i.e., skewness and kurtosis, respectively. Higher skewness and lower kurtosis are preferred to reduce the probability of extreme losses. However, incorporating high-order moments in the portfolio design is very difficult due to their non-convexity and rapidly increasing computational cost with the dimension. In this paper, we propose a very efficient and convergence-provable algorithm framework based on the successive convex approximation (SCA) algorithm to solve high-order portfolios. The efficiency of the proposed algorithm framework is demonstrated by the numerical experiments. [ABSTRACT FROM AUTHOR]

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

Standard Principal-Component Analysis (PCA) is known to be sensitive to outliers among the processed data. On the other hand, L1-norm-based PCA (L1-PCA) exhibits sturdy resistance against outliers, while it performs similar to standard PCA when applied to nominal or smoothly corrupted data. Exact calculation of the K L1-norm Principal Components (L1-PCs) of a rank-r data matrix X ∈ RD × N costs O(N(r −1)K + 1). In this work, we present reduced-rank L1-PCA (RR L1-PCA): a hybrid approach that approximates the K L1-PCs of X by the L1-PCs of its L2-norm-based rank-d approximation (d ≤ r), calculable exactly with reduced complexity O(N(d − 1)K + 1). The proposed method combines the denoising capabilities and low computation cost of standard PCA with the outlier-resistance of L1-PCA. RR L1-PCA is accompanied by formal performance guarantees as well as thorough numerical studies that corroborate its computational and corruption resistance merits. [ABSTRACT FROM AUTHOR]

Sequences with low autocorrelation sidelobes are needed in a diverse set of active sensing applications, including radar, sonar, communications, and biomedicine. The recently proposed and widely adopted methods of the Cyclic Algorithm-New (CAN) and Periodic CAN (PeCAN) are known to be computationally efficient, particularly as they employ fast Fourier transform (FFT) operations to design unimodular (i.e., unit-modulus) sequences with good autocorrelation properties. However, these cyclic algorithms cannot be directly used to design binary sequences with good autocorrelation properties due to the extremely multimodal nature of the associated optimization objectives. In this paper, we combine the notion of converging functions with the CAN and PeCAN frameworks, to considerably enhance their performances for binary sequence synthesis. Moreover, the convergence of these algorithms are established and their convergence properties are thoroughly analyzed. Numerical examples are provided to show that the proposed algorithms can outperform existing approaches for aperiodic and periodic binary sequence synthesis, especially for long sequence designs. [ABSTRACT FROM AUTHOR]

Many resource allocation tasks are challenging global (i.e., non-convex) optimization problems. The main issue is that the computational complexity of these problems grows exponentially in the number of variables instead of polynomially as for many convex optimization problems. However, often the non-convexity stems only from a subset of variables. Conventional global optimization frameworks like monotonic optimization or DC programming treat all variables as global variables and require complicated, problem specific decomposition approaches to exploit the convexity in some variables. To overcome this challenge, we develop an easy-to-use algorithm that inherently differentiates between convex and non-convex variables, preserving the low computational complexity in the number of convex variables. Another issue with these widely used frameworks is that they may suffer from severe numerical problems. We discuss this issue in detail and provide a clear motivating example. The solution to this problem is to replace the traditional approach of finding an $\varepsilon $ -approximate solution by the novel concept of $\varepsilon $ -essential feasibility. The underlying algorithmic approach is called successive incumbent transcending (SIT) algorithm and builds the foundation of our developed algorithm. A further highlight of this algorithm is that it inherently treats fractional objectives making the use of Dinkelbach's iterative algorithm obsolete. Numerical experiments show a speed-up of four orders of magnitude over state-of-the-art algorithms and almost three orders of magnitude of additional speed-up over Dinkelbach's algorithm for fractional programs. [ABSTRACT FROM AUTHOR]

This work addresses the robust principal component analysis (PCA) problem using generalized nonoconvex regularization for low-rank and sparsity promotion. While the popular nuclear and $\ell _1$ -norm penalties have a bias problem, nonconvex regularization can alleviate the bias problem and can be expected to achieve better performance. In this paper, a proximal block coordinate descent (BCD) algorithm is used to efficiently solve the nonconvex regularized robust PCA problem. It is globally convergent under weak conditions. Further, for a popular class of penalties having discontinuous threshoding functions, we establish the convergence to a restricted strictly local minimizer and, also, a local linear convergence rate for the proximal BCD algorithm. Moreover, convergence to a local minimizer has been derived for hard-thresholding. Our result is the first on nonconvex robust PCA with established convergence to strictly local minimizer with local linear convergence rate. Numerical experiments have been provided to demonstrate the performance of the new algorithm. [ABSTRACT FROM AUTHOR]

In this paper, we consider the (global and sum) energy efficiency optimization problem in downlink multi-input multi-output multi-cell systems, where all users suffer from multi-user interference. This is a challenging problem due to several reasons: First, it is a nonconvex fractional programming problem; second, the transmission rate functions are characterized by (complex-valued) transmit covariance matrices; and third the processing-related power consumption may depend on the transmission rate. We tackle this problem by the successive pseudoconvex approximation approach, and we argue that pseudoconvex optimization plays a fundamental role in designing novel iterative algorithms, not only because every locally optimal point of a pseudoconvex optimization problem is also globally optimal but also because a descent direction is easily obtained from every optimal point of a pseudoconvex optimization problem. The proposed algorithms have the following advantages: First, fast convergence as the structure of the original optimization problem is preserved as much as possible in the approximate problem solved in each iteration; second, easy implementation as each approximate problem is suitable for parallel computation and its solution has a closed-form expression; and third, guaranteed convergence to a stationary point or a Karush–Kuhn–Tucker point. The advantages of the proposed algorithm are also illustrated numerically. [ABSTRACT FROM AUTHOR]

The optimal mean-reverting portfolio (MRP) design problem is an important task for statistical arbitrage, also known as pairs trading, in the financial markets. The target of the problem is to construct a portfolio of the underlying assets (possibly with an asset selection target) that can exhibit a satisfactory mean reversion property and a desirable variance property. In this paper, the optimal MRP design problem is studied under an investment leverage constraint representing the total investment positions on the underlying assets. A general problem formulation is proposed by considering the design targets subject to a leverage constraint. To solve the problem, a unified optimization framework based on the successive convex approximation method is developed. The superior performance of the proposed formulation and the algorithms are verified through numerical simulations on both synthetic data and real market data. [ABSTRACT FROM AUTHOR]

In this paper, four iterative algorithms for learning analysis operators are presented. They are built upon the same optimization principle underlying both Analysis K-SVD and Analysis SimCO. The forward and sequential analysis operator learning (AOL) algorithms are based on projected gradient descent with optimally chosen step size. The implicit AOL algorithm is inspired by the implicit Euler scheme for solving ordinary differential equations and does not require to choose a step size. The fourth algorithm, singular value AOL, uses a similar strategy as Analysis K-SVD while avoiding its high computational cost. All algorithms are proven to decrease or preserve the target function in each step and a characterization of their stationary points is provided. Further they are tested on synthetic and image data, compared to Analysis SimCO and found to give better recovery rates and faster decay of the objective function, respectively. In a final denoising experiment the presented algorithms are again shown to perform similar to or better than the state-of-the-art algorithm ASimCO. [ABSTRACT FROM AUTHOR]

This paper considers the reconstruction of a discrete-valued random vector from possibly underdetermined linear measurements using sum-of-absolute-value (SOAV) optimization. The proposed algorithm, referred to as discreteness-aware approximate message passing (DAMP), is based on the idea of approximate message passing (AMP), which has been originally proposed for compressed sensing. The DAMP algorithm has low computational complexity and its performance in the large system limit can be predicted analytically via state evolution framework, where we provide a condition for the exact reconstruction with DAMP in the noise-free case. From the analysis, we also propose a method to determine the parameters of the SOAV optimization. Moreover, based on the state evolution, we provide Bayes optimal DAMP, which has the minimum mean-square-error at each iteration of the algorithm. Simulation results show that the DAMP algorithms can reconstruct the discrete-valued vector from underdetermined linear measurements and the empirical performance agrees with our theoretical results in large-scale systems. When the problem size is not large enough, the SOAV optimization with the proposed parameters can achieve better performance than the DAMP algorithms for high signal-to-noise ratio. [ABSTRACT FROM AUTHOR]

Two-stage stochastic optimization, in which a long-term master problem is coupled with a family of short-term subproblems, plays a critical role in various application areas. However, most existing algorithms for two-stage stochastic optimization only work for special cases, and/or are based on the batch method, which requires huge memory and computational complexity. To the best of our knowledge, there still lack efficient and general two-stage online stochastic optimization algorithms. This paper proposes a two-stage online successive convex approximation (TOSCA) algorithm for general two-stage nonconvex stochastic optimization problems. At each iteration, the TOSCA algorithm first solves one short-term subproblem associated with the current realization of the system state. Then, it constructs a convex surrogate function for the objective of the long-term master problem. Finally, the long-term variables are updated by solving a convex approximation problem obtained by replacing the objective function in the long-term master problem with the convex surrogate function. We establish the almost sure convergence of the TOSCA algorithm and customize the algorithmic framework to solve three important application problems. Simulations show that the TOSCA algorithm can achieve superior performance over existing solutions. [ABSTRACT FROM AUTHOR]

Numerical optimization has played a central role in addressing key signal processing (SP) problems. Highly effective methods have been developed for a large variety of SP applications such as communications, radar, filter design, and speech and image analytics, just to name a few. However, optimization algorithms often entail considerable complexity, which creates a serious gap between theoretical design/analysis and real-time processing. In this paper, we aim at providing a new learning-based perspective to address this challenging issue. The key idea is to treat the input and output of an SP algorithm as an unknown nonlinear mapping and use a deep neural network (DNN) to approximate it. If the nonlinear mapping can be learned accurately by a DNN of moderate size, then SP tasks can be performed effectively—since passing the input through a DNN only requires a small number of simple operations. In our paper, we first identify a class of optimization algorithms that can be accurately approximated by a fully connected DNN. Second, to demonstrate the effectiveness of the proposed approach, we apply it to approximate a popular interference management algorithm, namely, the WMMSE algorithm. Extensive experiments using both synthetically generated wireless channel data and real DSL channel data have been conducted. It is shown that, in practice, only a small network is sufficient to obtain high approximation accuracy, and DNNs can achieve orders of magnitude speedup in computational time compared to the state-of-the-art interference management algorithm. [ABSTRACT FROM AUTHOR]

In this paper, we propose a tensor train neighborhood preserving embedding (TTNPE) to embed multidimensional tensor data into low-dimensional tensor subspace. Novel approaches to solve the optimization problem in TTNPE are proposed. For this embedding, we evaluate a novel tradeoff gain among classification, computation, and dimensionality reduction (storage) for supervised learning. It is shown that compared to the state-of-the-arts tensor embedding methods, TTNPE achieves superior tradeoff in classification, computation, and dimensionality reduction in MNIST handwritten digits, Weizmann face datasets, and financial market datasets. [ABSTRACT FROM AUTHOR]

Phase retrieval has been mainly considered in the presence of Gaussian noise. However, the performance of the algorithms proposed under the Gaussian noise model severely degrades when grossly corrupted data, i.e., outliers, exist. This paper investigates techniques for phase retrieval in the presence of heavy-tailed noise, which is considered a better model for situations where outliers exist. An $\ell _p$ -norm ($0

Quadratically constrained quadratic programming (QCQP) is NP-hard in its general non-convex form, yet it frequently arises in various engineering applications. Several polynomial-time approximation algorithms exist for non-convex QCQP problems (QCQPs), but their success hinges upon the ability to find at least one feasible point—which is also hard for a general problem instance. In this paper, we present a heuristic framework for computing feasible points of general non-convex QCQPs using simple first-order methods. Our approach features low computational and memory requirements, which makes it well suited for application to large-scale problems. While a priori it may appear that these benefits come at the expense of technical sophistication, rendering our approach too simple to even merit consideration for a non-convex and NP-hard problem, we provide compelling empirical evidence to the contrary. Experiments on synthetic as well as real-world instances of non-convex QCQPs reveal the surprising effectiveness of first-order methods compared to more established and sophisticated alternatives. [ABSTRACT FROM PUBLISHER]

This paper considers the downlink transmission of cloud-radio access networks (C-RANs) with limited fronthaul capacity. We formulate a joint design of remote radio head (RRH) selection, RRH-user association, and transmit beamforming for simultaneously optimizing the achievable sum rate and total power consumption, using the multiobjective optimization concept. Due to the nonconvexity of perfronthaul capacity constraints and introduced binary selection variables, the formulated problem lends itself to a mixed-integer nonconvex program, which is generally non-deterministic polynomial-time hard. Motivated by powerful computing capability of C-RAN and for benchmarking purposes, we propose a branch and reduce and bound-based algorithm to attain a globally optimal solution. For more practically appealing approaches, we then propose three iterative low-complexity algorithms. In the first method, we iteratively approximate the continuous nonconvex constraints by convex conic ones using successive convex approximation framework. More explicitly, the problem obtained at each iteration is a mixed-integer second-order cone program (MI-SOCP) for which dedicated solvers are available. In the second method, we first relax the binary variables to be continuous to arrive at a sequence of SOCPs and then perform a postprocessing procedure on the relaxed variables to search for a high-performance solution. In the third method, we solve the considered problem in view of sparsity-inducing regularization. Numerical results show that our proposed algorithms converge rapidly and achieve near-optimal performance as well as outperform the known algorithms. [ABSTRACT FROM PUBLISHER]

This paper investigates the optimal power allocation that maximizes the system utility for a full-duplex (FD) amplify-and-forward (AF) two-way relay network (TWRN) with a rate outage and a power constraint. Both cases of an individual and a sum power constraint are considered. To formulate the corresponding optimization problem, closed-form expressions of outage probabilities are needed. With FD transceivers, the derivation of such closed-form expressions become too involved. Thus, approximate closed-form expressions were derived instead. The resulting optimization problem is still nonconvex and difficult to solve. Via solving a series of approximate convex problems, a successive convex approximation (SCA) algorithm was proposed. Our simulation results demonstrate the accuracy of the approximate closed-form expressions of outage probabilities and that the proposed SCA algorithm achieves near-optimal performance and significantly outperforms the full power allocation under the individual power constraint and uniform power allocation under the sum power constraint both in system utility and in power consumption. Our results further show that simultaneous power transmission for all relay nodes with judicious power allocation generated by the SCA algorithm achieves $174\%$ of performance gain under the individual power constraint, and $85 \%$ of performance gain under the sum power constraint over the relay selection scheme. A tradeoff between the FD system and the half-duplex (HD) system with respect to the residual interference was also observed. We can also observe that not only the residual self-interference but also the cross-link interference is a factor that degrades the system performance for an FD system. [ABSTRACT FROM PUBLISHER]

Stochastic approximation techniques play an important role in solving many problems encountered in machine learning or adaptive signal processing. In these contexts, the statistics of the data are often unknown a priori or their direct computation is too intensive, and they have thus to be estimated online from the observed signals. For batch optimization of an objective function being the sum of a data fidelity term and a penalization (e.g., a sparsity promoting function), Majorize-Minimize (MM) methods have recently attracted much interest since they are fast, highly flexible, and effective in ensuring convergence. The goal of this paper is to show how these methods can be successfully extended to the case when the data fidelity term corresponds to a least squares criterion and the cost function is replaced by a sequence of stochastic approximations of it. In this context, we propose an online version of an MM subspace algorithm and we study its convergence by using suitable probabilistic tools. Simulation results illustrate the good practical performance of the proposed algorithm associated with a memory gradient subspace, when applied to both nonadaptive and adaptive filter identification problems. [ABSTRACT FROM PUBLISHER]

Gaussian and quadratic approximations of message passing algorithms on graphs have attracted considerable recent attention due to their computational simplicity, analytic tractability, and wide applicability in optimization and statistical inference problems. This paper presents a systematic framework for incorporating such approximate message passing (AMP) methods in general graphical models. The key concept is a partition of dependencies of a general graphical model into strong and weak edges, with the weak edges representing small, linearizable couplings of variables. AMP approximations based on the central limit theorem can be readily applied to aggregates of many weak edges and integrated with standard message passing updates on the strong edges. The resulting algorithm, which we call hybrid generalized approximate message passing (HyGAMP), can yield significantly simpler implementations of sum-product and max-sum loopy belief propagation. By varying the partition of strong and weak edges, a performance-complexity tradeoff can be achieved. Group sparsity and multinomial logistic regression problems are studied as examples of the proposed methodology. [ABSTRACT FROM PUBLISHER]

A number of important problems in engineering can be formulated as non-convex quadratically constrained quadratic programming (QCQP). The general QCQP problem is NP-Hard. In this paper, we consider a class of non-convex QCQP problems that are expressible as the maximization of the point-wise minimum of homogeneous convex quadratics over a “simple” convex set. Existing approximation strategies for such problems are generally incapable of achieving favorable performance-complexity tradeoffs. They are either characterized by good performance but high complexity and lack of scalability, or low complexity but relatively inferior performance. This paper focuses on bridging this gap by developing high performance, low complexity successive non-smooth convex approximation algorithms for problems in this class. Exploiting the structure inherent in each subproblem, specialized first-order methods are used to efficiently compute solutions. Multicast beamforming is considered as an application example to showcase the effectiveness of the proposed algorithms, which achieve a very favorable performance-complexity tradeoff relative to the existing state of the art. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a successive pseudoconvex approximation algorithm to efficiently compute stationary points for a large class of possibly nonconvex optimization problems. The stationary points are obtained by solving a sequence of successively refined approximate problems, each of which is much easier to solve than the original problem. To achieve convergence, the approximate problem only needs to exhibit a weak form of convexity, namely, pseudoconvexity. We show that the proposed framework not only includes as special cases a number of existing methods, for example, the gradient method and the Jacobi algorithm, but also leads to new algorithms that enjoy easier implementation and faster convergence speed. We also propose a novel line search method for nondifferentiable optimization problems, which is carried out over a properly constructed differentiable function with the benefit of a simplified implementation as compared to the state-of-the-art line search techniques that directly operate on the original nondifferentiable objective function. The advantages of the proposed algorithm are shown, both theoretically and numerically, by several example applications, namely, multi-input multi-output (MIMO) broadcast channel capacity computation, energy efficiency maximization in massive MIMO systems, and $\ell_1$-regularized least square problems in sparse signal recovery. [ABSTRACT FROM PUBLISHER]

This paper investigates the problem of distributed stochastic approximation in multiagent systems. The algorithm under study consists of two steps: A local stochastic approximation step and a diffusion step, which drives the network to a consensus. The diffusion step uses row-stochastic matrices to weight the network exchanges. As opposed to previous works, exchange matrices are not supposed to be doubly stochastic, and may also depend on the past estimate. We prove that nondoubly stochastic matrices generally influence the limit points of the algorithm. Nevertheless, the limit points are not affected by the choice of the matrices provided that the latter are doubly stochastic in expectation. This conclusion legitimates the use of broadcast-like diffusion protocols, which are easier to implement. Next, by means of a central limit theorem, we prove that doubly stochastic protocols perform asymptotically as well as centralized algorithms and we quantify the degradation caused by the use of nondoubly stochastic matrices. Throughout this paper, a special emphasis is put on the special case of distributed nonconvex optimization as an illustration of our results. [ABSTRACT FROM PUBLISHER]

The calculation of a sparse approximate solution to a linear system of equations is often performed using either L1-norm regularization and convex optimization or nonconvex regularization and nonconvex optimization. Combining these principles, this paper describes a type of nonconvex regularization that maintains the convexity of the objective function, thereby allowing the calculation of a sparse approximate solution via convex optimization. The preservation of convexity is viable in the proposed approach because it uses a regularizer that is nonseparable. The proposed method is motivated and demonstrated by the calculation of sparse signal approximation using tight frames. Examples of denoising demonstrate improvement relative to L1 norm regularization. [ABSTRACT FROM PUBLISHER]

We introduce the decentralized Broyden–Fletcher–Goldfarb–Shanno (D-BFGS) method as a variation of the BFGS quasi-Newton method for solving decentralized optimization problems. Decentralized quasi-Newton methods are of interest in problems that are not well conditioned, making first-order decentralized methods ineffective, and in which second-order information is not readily available, making second-order decentralized methods impossible. D-BFGS is a fully distributed algorithm in which nodes approximate curvature information of themselves and their neighbors through the satisfaction of a secant condition. We additionally provide a formulation of the algorithm in asynchronous settings. Convergence of D-BFGS is established formally in both the synchronous and asynchronous settings and strong performance advantages relative to existing methods are shown numerically. [ABSTRACT FROM PUBLISHER]

There is a big trend nowadays to use event-triggered proximity report for indoor positioning. This paper presents a generic received-signal-strength (RSS) threshold optimization framework for generating informative proximity reports. The proposed framework contains five main building blocks, namely the deployment information, RSS model, positioning metric selection, optimization process and management. Among others, we focus on Gaussian process regression (GPR)-based RSS models and positioning metric computation. The optimal RSS threshold is found through minimizing the best achievable localization root-mean-square-error formulated with the aid of fundamental lower bound analysis. Computational complexity is compared for different RSS models and different fundamental lower bounds. The resulting optimal RSS threshold enables enhanced performance of new fashioned low-cost and low-complex proximity report-based positioning algorithms. The proposed framework is validated with real measurements collected in an office area where bluetooth-low-energy (BLE) beacons are deployed. [ABSTRACT FROM PUBLISHER]

In this two-part paper, we propose a general algorithmic framework for the minimization of a nonconvex smooth function subject to nonconvex smooth constraints, and also consider extensions to some structured, nonsmooth problems. The algorithm solves a sequence of (separable) strongly convex problems and maintains feasibility at each iteration. Convergence to a stationary solution of the original nonconvex optimization is established. Our framework is very general and flexible and unifies several existing successive convex approximation (SCA)-based algorithms. More importantly, and differently from current SCA approaches, it naturally leads to distributed and parallelizable implementations for a large class of nonconvex problems. This Part I is devoted to the description of the framework in its generality. In Part II, we customize our general methods to several (multiagent) optimization problems in communications, networking, and machine learning; the result is a new class of centralized and distributed algorithms that compare favorably to existing ad-hoc (centralized) schemes. [ABSTRACT FROM PUBLISHER]

In this paper, based on a successively accuracy-increasing approximation of the $\ell _0$ norm, we propose a new algorithm for recovery of sparse vectors from underdetermined measurements. The approximations are realized with a certain class of concave functions that aggressively induce sparsity and their closeness to the $\ell _0$ norm can be controlled. We prove that the series of the approximations asymptotically coincides with the $\ell _1$ and $\ell _0$ norms when the approximation accuracy changes from the worst fitting to the best fitting. When measurements are noise-free, an optimization scheme is proposed that leads to a number of weighted $\ell _1$ minimization programs, whereas, in the presence of noise, we propose two iterative thresholding methods that are computationally appealing. A convergence guarantee for the iterative thresholding method is provided, and, for a particular function in the class of the approximating functions, we derive the closed-form thresholding operator. We further present some theoretical analyses via the restricted isometry, null space, and spherical section properties. Our extensive numerical simulations indicate that the proposed algorithm closely follows the performance of the oracle estimator for a range of sparsity levels wider than those of the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

This paper considers the problem of minimizing the expected value of a (possibly nonconvex) cost function parameterized by a random (vector) variable, when the expectation cannot be computed accurately (e.g., because the statistics of the random variables are unknown and/or the computational complexity is prohibitive). Classical stochastic gradient methods for solving this problem may suffer from slow convergence. In this paper, we propose a stochastic parallel Successive Convex Approximation-based (best-response) algorithm for general nonconvex stochastic sum-utility optimization problems, which arise naturally in the design of multi-agent networks. The proposed novel decomposition approach enables all users to update their optimization variables in parallel by solving a sequence of strongly convex subproblems, one for each user. Almost sure convergence to stationary points is proved. We then customize the algorithmic framework to solve the stochastic sum rate maximization problem over single-input-single-output (SISO) frequency-selective interference channels, multiple-input-multiple-output (MIMO) interference channels, and MIMO multiple-access channels. Numerical results corroborate that the proposed algorithms can converge faster than state-of-the-art stochastic gradient schemes while achieving the same (or better) sum-rates. [ABSTRACT FROM AUTHOR]

Sparse approximation addresses the problem of approximately fitting a linear model with a solution having as few non-zero components as possible. While most sparse estimation algorithms rely on suboptimal formulations, this work studies the performance of exact optimization of \ell_0-norm-based problems through Mixed-Integer Programs (MIPs). Nine different sparse optimization problems are formulated based on \ell_1, \ell_2 or \ell_\infty data misfit measures, and involving whether constrained or penalized formulations. For each problem, MIP reformulations allow exact optimization, with optimality proof, for moderate-size yet difficult sparse estimation problems. Algorithmic efficiency of all formulations is evaluated on sparse deconvolution problems. This study promotes error-constrained minimization of the \ell_0 norm as the most efficient choice when associated with \ell_1 and \ell_\infty misfits, while the \ell_2 misfit is more efficiently optimized with sparsity-constrained and sparsity-penalized problems. Exact \ell_0-norm optimization is shown to outperform classical methods in terms of solution quality, both for over- and underdetermined problems. Numerical simulations emphasize the relevance of the different \ell_p fitting possibilities as a function of the noise statistical distribution. Such exact approaches are shown to be an efficient alternative, in moderate dimension, to classical (suboptimal) sparse approximation algorithms with \ell_2 data misfit. They also provide an algorithmic solution to less common sparse optimization problems based on \ell_1 and \ell_\infty misfits. For each formulation, simulated test problems are proposed where optima have been successfully computed. Data and optimal solutions are made available as potential benchmarks for evaluating other sparse approximation methods. [ABSTRACT FROM PUBLISHER]

Sparse signal restoration is usually formulated as the minimization of a quadratic cost function \Vert \mbi y - \mbi A \mbi x \Vert2^{2} where \mbi A is a dictionary and \mbi x is an unknown sparse vector. It is well-known that imposing an \ell 0 constraint leads to an NP-hard minimization problem. The convex relaxation approach has received considerable attention, where the \ell 0-norm is replaced by the \ell 1-norm. Among the many effective \ell 1 solvers, the homotopy algorithm minimizes \Vert \mbi y - \mbi A \mbi x \Vert2^{2}+\lambda \Vert \mbi x \Vert 1 with respect to \mbi x for a continuum of \lambda ’s. It is inspired by the piecewise regularity of the \ell 1-regularization path, also referred to as the homotopy path. In this paper, we address the minimization problem \Vert \mbi y - \mbi A \mbi x \Vert2^{2}+\lambda \Vert \mbi x \Vert 0 for a continuum of \lambda ’s and propose two heuristic search algorithms for \ell 0-homotopy. Continuation Single Best Replacement is a forward–backward greedy strategy extending the Single Best Replacement algorithm, previously proposed for \ell 0-minimization at a given \lambda . The adaptive search of the \lambda -values is inspired by \ell 1-homotopy. \ell 0 Regularization Path Descent is a more complex algorithm exploiting the structural properties of the \ell 0-regularization path, which is piecewise constant with respect to \lambda . Both algorithms are empirically evaluated for difficult inverse problems involving ill-conditioned dictionaries. Finally, we show that they can be easily coupled with usual methods of model order selection. [ABSTRACT FROM PUBLISHER]

Many natural signals exhibit a sparse representation, whenever a suitable describing model is given. Here, a linear generative model is considered, where many sparsity-based signal processing techniques rely on such a simplified model. As this model is often unknown for many classes of the signals, we need to select such a model based on the domain knowledge or using some exemplar signals. This paper presents a new exemplar based approach for the linear model (called the dictionary) selection, for such sparse inverse problems. The problem of dictionary selection, which has also been called the dictionary learning in this setting, is first reformulated as a joint sparsity model. The joint sparsity model here differs from the standard joint sparsity model as it considers an overcompleteness in the representation of each signal, within the range of selected subspaces. The new dictionary selection paradigm is examined with some synthetic and realistic simulations. [ABSTRACT FROM PUBLISHER]

This paper seeks to combine linear time-invariant (LTI) filtering and sparsity-based denoising in a principled way in order to effectively filter (denoise) a wider class of signals. LTI filtering is most suitable for signals restricted to a known frequency band, while sparsity-based denoising is suitable for signals admitting a sparse representation with respect to a known transform. However, some signals cannot be accurately categorized as either band-limited or sparse. This paper addresses the problem of filtering noisy data for the particular case where the underlying signal comprises a low-frequency component and a sparse or sparse-derivative component. A convex optimization approach is presented and two algorithms derived: one based on majorization-minimization (MM), and the other based on the alternating direction method of multipliers (ADMM). It is shown that a particular choice of discrete-time filter, namely zero-phase noncausal recursive filters for finite-length data formulated in terms of banded matrices, makes the algorithms effective and computationally efficient. The efficiency stems from the use of fast algorithms for solving banded systems of linear equations. The method is illustrated using data from a physiological-measurement technique (i.e., near infrared spectroscopic time series imaging) that in many cases yields data that is well-approximated as the sum of low-frequency, sparse or sparse-derivative, and noise components. [ABSTRACT FROM PUBLISHER]

We consider the data-driven dictionary learning problem. The goal is to seek an over-complete dictionary from which every training signal can be best approximated by a linear combination of only a few codewords. This task is often achieved by iteratively executing two operations: sparse coding and dictionary update. The focus of this paper is on the dictionary update step, where the dictionary is optimized with a given sparsity pattern. We propose a novel framework where an arbitrary set of codewords and the corresponding sparse coefficients are simultaneously updated, hence the term simultaneous codeword optimization (SimCO). The SimCO formulation not only generalizes benchmark mechanisms MOD and K-SVD, but also allows the discovery that singular points, rather than local minima, are the major bottleneck of dictionary update. To mitigate the problem caused by the singular points, regularized SimCO is proposed. First and second order optimization procedures are designed to solve regularized SimCO. Simulations show that regularization substantially improves the performance of dictionary learning. [ABSTRACT FROM PUBLISHER]

Mixed-scaling-rotation (MSR) coordinate rotation digital computer (CORDIC) is an attractive approach to synthesizing complex rotators. This paper presents the fixed-point error analysis and parameter selections of MSR-CORDIC with applications to the fast Fourier transform (FFT). First, the fixed-point mean squared error of the MSR-CORDIC is analyzed by considering both the angle approximation error and signal round-off error incurred in the finite precision arithmetic. The signal to quantization noise ratio (SQNR) of the output of the FFT synthesized using MSR-CORDIC is thereafter estimated. Based on these analyses, two different parameter selection algorithms of MSR-CORDIC are proposed for general and dedicated MSR-CORDIC structures. The proposed algorithms minimize the number of adders and word-length when the SQNR of the FFT output is constrained. Design examples show that the FFT designed by the proposed method exhibits a lower hardware complexity than existing methods. [ABSTRACT FROM PUBLISHER]

We revisit the problem of computing submatrices of the Cramér-Rao bound (CRB), which lower bounds the variance of any unbiased estimator of a vector parameter \mbi θ. We explore iterative methods that avoid direct inversion of the Fisher information matrix, which can be computationally expensive when the dimension of \mbi θ is large. The computation of the bound is related to the quadratic matrix program, where there are highly efficient methods for solving it. We present several methods, and show that algorithms in prior work are special instances of existing optimization algorithms. Some of these methods converge to the bound monotonically, but in particular, algorithms converging nonmonotonically are much faster. We then extend the work to encompass the computation of the CRB when the Fisher information matrix is singular and when the parameter \mbi θ is subject to constraints. As an application, we consider the design of a data streaming algorithm for network measurement. [ABSTRACT FROM PUBLISHER]

Real-time data delivery over best-effort error-prone packet networks has invigorated the study of robust coding schemes, such as scalable multiple description coding (SMDC). In this context, the paper introduces a novel generic symmetric scalable multiple description quantizer (SSMDSQ) which generates perfectly balanced source descriptions. Novel embedded index assignments are proposed which are used to realize high, as well as medium-to-low redundancy SSMDSQs. Compared to existing designs, it is shown that the proposed quantizer constructions exhibit superior distortion-rate (D-R) performance. Moreover, this paper describes an innovative extension of the Lloyd-Max algorithm in order to optimize symmetric and asymmetric scalable multiple description quantizers. For a family of Generalized Gaussian (GG) source distributions, the proposed optimization algorithm yields on average a significant D-R performance gain over unoptimized quantizers. Furthermore, anchored in the designed SSMDSQs, an SMDC framework is established to realize packet-based transmission over erasure channels. In this framework, transmission strategies are determined for scenarios wherein the average packet loss rate over the transmission link is (a) unknown and (b) can be estimated at the encoder. For both scenarios, SMDC packetized transmission is simulated for a family of GG distributions. Experimental results confirm that, compared to contemporary schemes, the designed quantizer constructions (with or without optimization) account for a significant average gain in signal-to-noise ratio (SNR) for a wide range of packet loss rates. [ABSTRACT FROM PUBLISHER]

The performance of multiuser systems is both difficult to measure fairly and to optimize. Most resource allocation problems are nonconvex and NP-hard, even under simplifying assumptions such as perfect channel knowledge, homogeneous channel properties among users, and simple power constraints. We establish a general optimization framework that systematically solves these problems to global optimality. The proposed branch-reduce-and-bound (BRB) algorithm handles general multicell downlink systems with single-antenna users, multiantenna transmitters, arbitrary quadratic power constraints, and robustness to channel uncertainty. A robust fairness-profile optimization (RFO) problem is solved at each iteration, which is a quasiconvex problem and a novel generalization of max–min fairness. The BRB algorithm is computationally costly, but it shows better convergence than the previously proposed outer polyblock approximation algorithm. Our framework is suitable for computing benchmarks in general multicell systems with or without channel uncertainty. We illustrate this by deriving and evaluating a zero-forcing solution to the general problem. [ABSTRACT FROM AUTHOR]

Throughput-optimal multihop wireless network operation entails a key physical-layer optimization problem: maximizing a weighted sum of link rates, with weights given by the differential queue backlogs. This emerges in joint back-pressure routing and power control, which is central in cross-layer wireless networking. We begin by showing that the core problem is not only nonconvex, but also NP-hard. This is a negative result, which however comes with a positive flip side: drawing from related developments in the digital subscriber line (DSL) literature, we propose effective ways to approximate it. Exploiting quasi-periodicity of the power allocation in stable setups due to the push-pull nature of the solution, we derive two custom algorithms that offer excellent throughput performance at reasonable, worst-case polynomial complexity. Judicious simulations illustrate the merits of the proposed algorithms. [ABSTRACT FROM PUBLISHER]

Intercell interference is the major limiting factor in wireless multicell networks. Recently, it has been shown that significant performance gains can be achieved by cooperation among base stations. Different degrees of cooperation are possible. In this paper, cooperation in the form of intercell interference management is considered. The base stations are equipped with multiple antennas, while the mobile terminals only have a single antenna. The base stations jointly coordinate beamforming and scheduling, and thus perform a weaker form of cooperation among base stations than suggested in recently proposed joint transmission techniques. The terminals treat intercell interference as noise. The corresponding resource allocation problem is cast as a utility maximization problem, which includes common performance objectives such as the arithmetic mean, the geometric mean, and the max-min operation of achievable user rates. The resulting utility maximization problem is a nonconvex optimization problem. After a suitable reformulation, the problem can be solved to global optimality using the framework of monotonic optimization. Although, the numerical complexity of the proposed method is exponential, the reformulation step leads to an optimization problem which only scales in the number of mobile terminals instead of the entire set of physical layer parameters. In essence, the proposed framework represents a powerful tool for computing benchmarks for certain scenarios and utility functions under jointly optimal beamforming and scheduling. [ABSTRACT FROM PUBLISHER]

Intercell interference is the major limitation to the performance of future cellular systems. Despite the joint detection and joint transmission techniques aiming at interference cancelation which suffer from the limited possible cooperation among the nodes, power allocation is a promising approach for optimizing the system performance. If the interference is treated as noise, the power allocation optimization problem aiming at maximizing the sum rate with a total power constraint is nonconvex and up to now an open problem. In the present paper, the solution is found by reformulating the nonconvex objective function of the sum rate as a difference of two concave functions. A globally optimum power allocation is found by applying a branch-and-bound algorithm to the new formulation. In principle, the algorithm partitions the feasible region recursively into subregions where for every subregion the objective function is upper and lower bounded. For each subregion, a linear program is applied for estimating the upper bound of the sum rate which is derived from a convex maximization formulation of the problem with piecewise linearly approximated constraints. The performance is investigated by system-level simulations. The results show that the proposed algorithm outperforms the known conventional suboptimum schemes. Furthermore, it is shown that the algorithm asymptotically converges to a globally optimum power allocation. [ABSTRACT FROM PUBLISHER]

The problem of finding the least squares solution \bf s to a system of equations \bf Hs = \bf y is considered, when \bf s is a vector of binary variables and the coefficient matrix \bf H is unknown but of bounded uncertainty. Similar to previous approaches to robust binary least squares, we explore the potential of a min-max design with the aim to provide solutions that are less sensitive to the uncertainty in \bf H. We concentrate on the important case of ellipsoidal uncertainty, i.e., the matrix \bf H is assumed to be a deterministic unknown quantity which lies in a given uncertainty ellipsoid. The resulting problem is NP-hard, yet amenable to convex approximation techniques: Starting from a convenient reformulation of the original problem, we propose an approximation algorithm based on semidefinite relaxation that explicitly accounts for the ellipsoidal uncertainty in the coefficient matrix. Next, we show that it is possible to construct a tighter relaxation by suitably changing the description of the feasible region of the problem, and formulate an approximation algorithm that performs better in practice. Interestingly, both relaxations are derived as Lagrange bidual problems corresponding to the two equivalent problem reformulations. The strength of the proposed tightened relaxation is demonstrated by pertinent simulations. [ABSTRACT FROM AUTHOR]

In this article, we show that when targets are closely spaced, traditional tracking algorithms can be adjusted to perform better under a performance measure that disregards identity. More specifically, we propose an adjusted version of the joint probabilistic data association (JPDA) filter, which we call set JPDA (SJPDA). Through examples and theory we motivate the new approach, and show its possibilities. To decrease the computational requirements, we further show that the SJPDA filter can be formulated as a continuous optimization problem which is fairly easy to handle. Optimal approximations are also discussed, and an algorithm, Kullback–Leibler SJPDA (KLSJPDA), which provides optimal Gaussian approximations in the Kullback–Leibler sense is derived. Finally, we evaluate the SJPDA filter on two scenarios with closely spaced targets, and compare the performance in terms of the mean optimal subpattern assignment (MOSPA) measure with the JPDA filter, and also with the Gaussian-mixture cardinalized probability hypothesis density (GM-CPHD) filter. The results show that the SJPDA filter performs substantially better than the JPDA filter, and almost as well as the more complex GM-CPHD filter. [ABSTRACT FROM PUBLISHER]

This paper deals with the reconstruction of T1-T2 correlation spectra in nuclear magnetic resonance relaxometry. The ill-posed character and the large size of this inverse problem are the main difficulties to tackle. While maximum entropy is retained as an adequate regularization approach, the choice of an efficient optimization algorithm remains a challenging task. Our proposal is to apply a truncated Newton algorithm with two original features. First, a theoretically sound line search strategy suitable for the entropy function is applied to ensure the convergence of the algorithm. Second, an appropriate preconditioning structure based on a singular value decomposition of the forward model matrix is used to speed up the algorithm convergence. Furthermore, we exploit the specific structures of the observation model and the Hessian of the criterion to reduce the computation cost of the algorithm. The performances of the proposed strategy are illustrated by means of synthetic and real data processing. [ABSTRACT FROM AUTHOR]