Search

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research. A list of important papers discussed in this survey, corresponding codes, and additional resources on BLOs are publicly available at: https://github.com/vis-opt-group/BLO. [ABSTRACT FROM AUTHOR]

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research. A list of important papers discussed in this survey, corresponding codes, and additional resources on BLOs are publicly available at: https://github.com/vis-opt-group/BLO. [ABSTRACT FROM AUTHOR]

Variance reduction is popular in accelerating gradient descent and stochastic gradient descent for optimization problems defined on both euclidean space and Riemannian manifold. This paper further improves on existing variance reduction methods for non-convex Riemannian optimization, including R-SVRG and R-SRG/R-SPIDER by providing a unified framework for batch size adaptation. Such framework is more general than the existing works by considering retraction and vector transport and mini-batch stochastic gradients. We show that the adaptive-batch variance reduction methods require lower gradient complexities for both general non-convex and gradient dominated functions, under both finite-sum and online optimization settings. Moreover, under the new framework, we complete the analysis of R-SVRG and R-SRG, which is currently missing in the literature. We prove convergence of R-SVRG with much simpler analysis, which leads to curvature-free complexity bounds. We also show improved results for R-SRG under double-loop convergence, which match the optimal complexities as the R-SPIDER. In addition, we prove the first online complexity results for R-SVRG and R-SRG. Lastly, we discuss the potential of adapting batch size for non-smooth, constrained and second-order Riemannian optimizers. Extensive experiments on a variety of applications support the analysis and claims in the paper. [ABSTRACT FROM AUTHOR]

As an algorithmic framework, message passing is extremely powerful and has wide applications in the context of different disciplines including communications, coding theory, statistics, signal processing, artificial intelligence and combinatorial optimization. In this paper, we investigate the performance of a message-passing algorithm called min-sum belief propagation (BP) for the vertex-disjoint shortest $k$ -path problem ($k$ -VDSP) on weighted directed graphs, and derive the iterative message-passing update rules. As the main result of this paper, we prove that for a weighted directed graph $G$ of order $n$ , BP algorithm converges to the unique optimal solution of $k$ -VDSP on $G$ within $O(n^{2}w_{max})$ iterations, provided that the weight $w_{e}$ is nonnegative integral for each arc $e\in E(G)$ , where $w_{max}=\max \{w_{e}: e\in E(G)\}$. To the best of our knowledge, this is the first instance where BP algorithm is proved correct for NP-hard problems. Additionally, we establish the extensions of $k$ -VDSP to the case of multiple sources or sinks. [ABSTRACT FROM AUTHOR]

This paper proposes a momentum algorithm based on Borges difference and an adaptive momentum (Adam) algorithm based on Borges difference to update parameters, which can adjust the momentum information more flexibly. The Borges difference is proposed from the definition of Borges derivative to be combined with the gradient algorithm in convolutional neural networks. The proposed momentum algorithm based on Borges difference and Adam algorithm based on Borges difference can be adjusted more flexibly in order to speed up the convergence. The parameter optimization algorithm with the Borges difference presents a better performance compared with the integer-order momentum algorithm and integer-order Adam algorithm, with the proposed nonlinear adjustment method for the parameter tuning of convolutional neural networks. By analyzing experimental results of Fashion-MNIST dataset and CIFAR-10 dataset, the optimization algorithms based on Borges difference proposed in this paper gain better effects on the optimization model compared with the corresponding ones based on the integer-order difference, and can speed up the convergence speed and recognition accuracy of the image recognition. [ABSTRACT FROM AUTHOR]

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD($\lambda$ λ ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an $\epsilon$ ε accuracy, the number of iterations needed is $\tilde{O}(\epsilon ^{-2}\ln ^4\frac{1}{\epsilon }/\ln ^4\frac{1}{\rho })$ O ˜ (ε - 2 ln 4 1 ε / ln 4 1 ρ) in the general case, where $\rho$ ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD($\lambda$ λ ), which is referred to as AdaTD($\lambda$ λ ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD($\lambda$ λ ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches. [ABSTRACT FROM AUTHOR]

With the increasing of the data size and the development of multi-core computers, asynchronous parallel stochastic optimization algorithms such as KroMagnon have gained significant attention. In this paper, we propose a new Sparse approximation and asynchronous parallel Stochastic Variance Reduced Gradient (SSVRG) method for sparse and high-dimensional machine learning problems. Unlike standard SVRG and its asynchronous parallel variant, KroMagnon, the snapshot point of SSVRG is set to the average of all the iterates in the previous epoch, which allows it to take much larger learning rates and also makes it more robust to the choice of learning rates. In particular, we use the sparse approximation of the popular SVRG estimator to perform completely sparse updates at all iterations. Therefore, SSVRG has a much lower per-iteration computational cost than its dense counterpart, SVRG++, and is very friendly to asynchronous parallel implementation. Moreover, we provide the convergence guarantees of SSVRG for both strongly convex and non-strongly convex problems, while existing asynchronous algorithms (e.g., KroMagnon and ASAGA) only have convergence guarantees for strongly convex problems. Finally, we extend SSVRG to non-smooth and asynchronous parallel settings. Numerical experimental results demonstrate that SSVRG converges significantly faster than the state-of-the-art asynchronous parallel methods, e.g., KroMagnon, and is usually more than three orders of magnitude faster than SVRG++. [ABSTRACT FROM AUTHOR]

The increasing application of voltage source converter (VSC) based high voltage direct current (VSC-HVDC) technology in power grids has raised the importance of incorporating DC grids and converters into the existing transmission network. This poses significant challenges in dealing with the resulting optimal power flow (OPF) problem. In this paper, a recently proposed nonconvex distributed optimization algorithm—Augmented Lagrangian based Alternating Direction Inexact Newton method (aladin), is tailored to solve the nonconvex AC/DC OPF problem for emerging voltage source converter (VSC) based multiterminal high voltage direct current (VSC-MTDC) meshed AC/DC hybrid systems. The proposed scheme decomposes this AC/DC hybrid OPF problem and handles it in a fully distributed way. Compared to the existing state-of-art Alternating Direction Method of Multipliers (admm), which is in general, not applicable for nonconvex problems, aladin has a theoretical convergence guarantee. Applying these two approaches to VSC-MTDC coupled with an IEEE benchmark AC power system illustrates that the tailored aladin outperforms admm in convergence speed and numerical robustness. [ABSTRACT FROM AUTHOR]

This paper studies distributed algorithms for (strongly convex) composite optimization problems over mesh networks, subject to quantized communications. Instead of focusing on a specific algorithmic design, a black-box model is proposed, casting linearly convergent distributed algorithms in the form of fixed-point iterates. The algorithmic model is equipped with a novel random or deterministic Biased Compression (BC) rule on the quantizer design, and a new Adaptive encoding Non-uniform Quantizer (ANQ) coupled with a communication-efficient encoding scheme, which implements the BC-rule using a finite number of bits (below machine precision). This fills a gap existing in most state-of-the-art quantization schemes, such as those based on the popular compression rule, which rely on communication of some scalar signals with negligible quantization error (in practice quantized at the machine precision). A unified communication complexity analysis is developed for the black-box model, determining the average number of bits required to reach a solution of the optimization problem within a target accuracy. It is shown that the proposed BC-rule preserves linear convergence of the unquantized algorithms, and a trade-off between convergence rate and communication cost under ANQ-based quantization is characterized. Numerical results validate our theoretical findings and show that distributed algorithms equipped with the proposed ANQ have more favorable communication cost than algorithms using state-of-the-art quantization rules. [ABSTRACT FROM AUTHOR]

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID, imbalanced (statistical heterogeneity) and distribution shifted training data of FL is distributed in the federated network, which will increase the divergences between the local models and the global model, further degrading performance. In this paper, we propose a flexible clustered federated learning (CFL) framework named FlexCFL, in which we 1) group the training of clients based on the similarities between the clients’ optimization directions for lower training divergence; 2) implement an efficient newcomer device cold start mechanism for framework scalability and practicality; 3) flexibly migrate clients to meet the challenge of client-level data distribution shift. FlexCFL can achieve improvements by dividing joint optimization into groups of sub-optimization and can strike a balance between accuracy and communication efficiency in the distribution shift environment. The convergence and complexity are analyzed to demonstrate the efficiency of FlexCFL. We also evaluate FlexCFL on several open datasets and made comparisons with related CFL frameworks. The results show that FlexCFL can significantly improve absolute test accuracy by $+10.6\%$ + 10. 6 % on FEMNIST compared with FedAvg, $+3.5\%$ + 3. 5 % on FashionMNIST compared with FedProx, $+8.4\%$ + 8. 4 % on MNIST compared with FeSEM, $+4.7\%$ + 4. 7 % on Sentiment140 compare with IFCA. The experiment results show that FlexCFL is also communication efficient in the distribution shift environment. [ABSTRACT FROM AUTHOR]

This paper presents novel numerical approaches to finding the secrecy capacity of the multiple-input multiple-output (MIMO) wiretap channel subject to multiple linear transmit covariance constraints, including sum power constraint, per antenna power constraints and interference power constraint. An analytical solution to this problem is not known and existing numerical solutions suffer from slow convergence rate and/or high per-iteration complexity. Deriving computationally efficient solutions to the secrecy capacity problem is challenging since the secrecy rate is expressed as a difference of convex functions (DC) of the transmit covariance matrix, for which its convexity is only known for some special cases. In this paper we propose two low-complexity methods to compute the secrecy capacity along with a convex reformulation for degraded channels. In the first method we capitalize on the accelerated DC algorithm which requires solving a sequence of convex subproblems, for which we propose an efficient iterative algorithm where each iteration admits a closed-form solution. In the second method, we rely on the concave-convex equivalent reformulation of the secrecy capacity problem which allows us to derive the so-called partial best response algorithm to obtain an optimal solution. Notably, each iteration of the second method can also be done in closed form. The simulation results demonstrate a faster convergence rate of our methods compared to other known solutions. We carry out extensive numerical experiments to evaluate the impact of various parameters on the achieved secrecy capacity. [ABSTRACT FROM AUTHOR]

In real-world applications, it is important for machine learning algorithms to be robust against data outliers or corruptions. In this paper, we focus on improving the robustness of a large class of learning algorithms that are formulated as low-rank semi-definite programming (SDP) problems. Traditional formulations use the square loss, which is notorious for being sensitive to outliers. We propose to replace this with more robust noise models, including the $\ell _1$ ℓ 1 -loss and other nonconvex losses. However, the resultant optimization problem becomes difficult as the objective is no longer convex or smooth. To alleviate this problem, we design an efficient algorithm based on majorization-minimization. The crux is on constructing a good optimization surrogate, and we show that this surrogate can be efficiently obtained by the alternating direction method of multipliers (ADMM). By properly monitoring ADMM's convergence, the proposed algorithm is empirically efficient and also theoretically guaranteed to converge to a critical point. Extensive experiments are performed on four machine learning applications using both synthetic and real-world data sets. Results show that the proposed algorithm is not only fast but also has better performance than the state-of-the-arts. [ABSTRACT FROM AUTHOR]

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

Globally,wind power plants (WPPs) installations are dramatically increasing, leading to a large-scale integration into the power grids. Huge endeavors are devoted to achieving an efficient operation of WPPs. This paper offers a novel chaotic-billiards optimizer (C-BO) algorithm to properly design the fuzzy logic control (FLC) strategy for stability enhancement of grid-integrated WPPs. The C-BO algorithm has distinct features such as simple construction, low numbers of parameters required to design, high convergence speed, and low computational burden. The permanent-magnet synchronous generator (PMSG)-based WPP is interlinked into the electric power grid through power converters. A cascaded FLC approach, which is optimally fine-tuned by the C-BO, is offered as the control methodology for these converters. The C-BO algorithm fine-tunes the gain factors of multiple FLCs at a rapid convergence speed. A simulation-based optimization approach is applied, in which the integrated square error criterion is chosen as an objective function. For the sake of preciseness, the PMSG-based WPP is connected to the IEEE 39-bus New England test system. The effectiveness of the optimal FLC using the C-BO algorithm is compared with that achieved using the water cycle algorithm-based optimal FLC, the genetic algorithm-based optimal FLC, and the marine predator algorithm-based optimal proportional-integral controller, taking into account severe grid disturbances. Moreover, real wind speed measured data captured from Zaafarana Station in Egypt are extensively performed in the system studies. The validity of the offered control approach is widely verified by the simulation analyses using the MATLAB/Simulink environment. [ABSTRACT FROM AUTHOR]

Unit Commitment (UC) is an important problem in power system operations. It is traditionally planned for 24 hours with one-hour time intervals. To accommodate the increasing net-load variability, sub-hourly UC has been suggested for improved system flexibility. Such a problem is larger and more complicated than hourly UC because of the increased number of periods and reduced unit ramping capabilities per period. The computational burden is further exacerbated for systems with large numbers of virtual transactions leading to dense transmission constraint matrices. Consequently, the state-of-the-art and practice method, branch-and-cut (B&C), suffers from poor performance. In this paper, our recent Surrogate Absolute-Value Lagrangian Relaxation (SAVLR) is enhanced by embedding ordinal-optimization concepts for a drastic reduction in subproblem solving time. Rather than formally solving subproblems by using B&C, subproblem solutions satisfying SAVLR's convergence condition are obtained by modifying solutions from previous iterations or solving crude subproblems. All virtual transactions are included in each subproblem to reduce major changes in solutions across iterations. A parallel version is also developed to further reduce the computation time. Testing on MISO's large cases demonstrates that our ordinal-optimization embedded approach obtains near-optimal solutions efficiently, is robust, and provides a new way of solving other MILP problems. [ABSTRACT FROM AUTHOR]

Kernel-based methods for support vector machines (SVM) have shown highly advantageous performance in various applications. However, they may incur prohibitive computational costs for large-scale sample datasets. Therefore, data reduction (reducing the number of support vectors) appears to be necessary, which gives rise to the topic of the sparse SVM. Motivated by this problem, the sparsity constrained kernel SVM optimization has been considered in this paper in order to control the number of support vectors. Based on the established optimality conditions associated with the stationary equations, a Newton-type method is developed to handle the sparsity constrained optimization. This method is found to enjoy the one-step convergence property if the starting point is chosen to be close to a local region of a stationary point, thereby leading to a super-high computational speed. Numerical comparisons with several powerful solvers demonstrate that the proposed method performs exceptionally well, particularly for large-scale datasets in terms of a much lower number of support vectors and shorter computational time. [ABSTRACT FROM AUTHOR]

This paper explores the non-convex composition optimization consisting of inner and outer finite-sum functions with a large number of component functions. This problem arises in important applications such as nonlinear embedding and reinforcement learning. Although existing approaches such as stochastic gradient descent (SGD) and stochastic variance reduced gradient (SVRG) descent can be applied to solve this problem, their query complexities tend to be high, especially when the number of inner component functions is large. Therefore, to significantly improve the query complexity of current approaches, we have devised the stochastic composition via variance reduction (SCVR). What's more, we analyze the query complexity under different numbers of inner function and outer function. Based on different kinds of estimation of inner component function, we also present the SCVRII algorithm, though the order of query complexities are the same with SCVR. Additionally, we propose an extension to handle the mini-batch cases, which improve the query complexity under the optimal mini-batch size. The experimental results validate our proposed algorithms and theoretical analyses. [ABSTRACT FROM AUTHOR]

Partial multi-view clustering has attracted various attentions from diverse fields. Most existing methods adopt separate steps to obtain unified representations and extract clustering indicators. This separate manner prevents two learning processes to negotiate to achieve optimal performance. In this paper, we propose the Joint Representation Learning and Clustering (JRLC) framework to address this issue. The JRLC framework employs representation matrices to extract view-specific clustering information directly from the presence of partial similarity matrices, and rotates them to learn a common probability label matrix simultaneously, which connects representation learning and clustering seamlessly to achieve better clustering performance. Under the guidance of JRLC framework, several new incomplete multi-view clustering methods can be developed by extending existing single-view graph-based representation learning methods. For illustration, within the framework, we propose two specific methods, JRLC with spectral embedding (JRLC-SE) and JRLC via integrating nonnegative embedding and spectral embedding (JRLC-NS). Two iterative algorithms with guaranteed convergence are designed to solve the resultant optimization problems of JRLC-SE and JRLC-NS. Experimental results on various datasets and news topic clustering application demonstrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Kernel methods have achieved tremendous success in the past two decades. In the current big data era, data collection has grown tremendously. However, existing kernel methods are not scalable enough both at the training and predicting steps. To address this challenge, in this paper, we first introduce a general sparse kernel learning formulation based on the random feature approximation, where the loss functions are possibly non-convex. In order to reduce the scale of random features required in experiment, we also use that formulation based on the orthogonal random feature approximation. Then we propose a new asynchronous parallel doubly stochastic algorithm for large scale sparse kernel learning (AsyDSSKL). To the best our knowledge, AsyDSSKL is the first algorithm with the techniques of asynchronous parallel computation and doubly stochastic optimization. We also provide a comprehensive convergence guarantee to AsyDSSKL. Importantly, the experimental results on various large-scale real-world datasets show that, our AsyDSSKL method has the significant superiority on the computational efficiency at the training and predicting steps over the existing kernel methods. [ABSTRACT FROM AUTHOR]

The iterative closest point (ICP) algorithm and its variants are a fundamental technique for rigid registration between two point sets, with wide applications in different areas from robotics to 3D reconstruction. The main drawbacks for ICP are its slow convergence, as well as its sensitivity to outliers, missing data, and partial overlaps. Recent work such as Sparse ICP achieves robustness via sparsity optimization at the cost of computational speed. In this paper, we propose a new method for robust registration with fast convergence. First, we show that the classical point-to-point ICP can be treated as a majorization-minimization (MM) algorithm, and propose an Anderson acceleration approach to speed up its convergence. In addition, we introduce a robust error metric based on the Welsch’s function, which is minimized efficiently using the MM algorithm with Anderson acceleration. On challenging datasets with noises and partial overlaps, we achieve similar or better accuracy than Sparse ICP while being at least an order of magnitude faster. Finally, we extend the robust formulation to point-to-plane ICP, and solve the resulting problem using a similar Anderson-accelerated MM strategy. Our robust ICP methods improve the registration accuracy on benchmark datasets while being competitive in computational time. [ABSTRACT FROM AUTHOR]

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

Many signal processing and machine learning problems can be formulated as consensus optimization problems which can be solved efficiently via a cooperative multi-agent system. However, the agents in the system can be unreliable due to a variety of reasons: noise, faults and attacks. Providing erroneous updates leads the optimization process in a wrong direction, and degrades the performance of distributed machine learning algorithms. This paper considers the problem of decentralized learning using ADMM in the presence of unreliable agents. First, we rigorously analyze the effect of erroneous updates (in ADMM learning iterations) on the convergence behavior of the multi-agent system. We show that the algorithm linearly converges to a neighborhood of the optimal solution under certain conditions and characterize the neighborhood size analytically. Next, we provide guidelines for network design to achieve a faster convergence to the neighborhood. We also provide conditions on the erroneous updates for exact convergence to the optimal solution. Finally, to mitigate the influence of unreliable agents, we propose ROAD, a robust variant of ADMM, and show its resilience to unreliable agents with an exact convergence to the optimum. [ABSTRACT FROM AUTHOR]

In this paper, we propose a cost-efficient beamforming optimization algorithm for multi-user wireless communication systems associated with the intelligent reflecting surface (IRS). From the baseline successive refinement algorithm, which gives a sub-optimal solution for the power minimization problem under the signal-to-interference-plus-noise-ratio (SINR) constraint at each user, several optimization techniques are proposed to reduce the computation complexity while maintaining the algorithm-level performance. To reduce the number of required multiply-accumulate (MAC) operations, we first simplify the complicated matrix inversion by utilizing the channel hardening effect. We also present the two-phase refinement process for the group-level optimization of phase-shift elements, further relaxing the computation complexity as well as the processing latency. Applying the proposed optimization techniques, as a result, numerical results show that the fully-optimized algorithm can reduce the computational costs by up to 89.4% while showing less than 1 dB power loss, leading to the practical solution for the next-generation IRS-aided communication. [ABSTRACT FROM AUTHOR]

In this paper, we propose two communication-efficient decentralized optimization algorithms over a general directed multi-agent network. The first algorithm, termed Compressed Push-Pull (CPP), combines the gradient tracking Push-Pull method with communication compression. We show that CPP is applicable to a general class of unbiased compression operators and achieves linear convergence rate for strongly convex and smooth objective functions. The second algorithm is a broadcast-like version of CPP (B-CPP), and it also achieves linear convergence rate under the same conditions on the objective functions. B-CPP can be applied in an asynchronous broadcast setting and further reduce communication costs compared to CPP. Numerical experiments complement the theoretical analysis and confirm the effectiveness of the proposed methods. [ABSTRACT FROM AUTHOR]

Federated learning (FL) has been considered as a promising paradigm for enabling distributed training/learning in many machine-learning services without revealing users’ local data. Driven by the growing interests in exploiting FL in wireless networks, this paper studies the Non-orthogonal Multiple Access (NOMA) assisted FL in which a group of end-devices (EDs) form a NOMA cluster to send their locally trained models to the cellular base station (BS) for model aggregation. In particular, we consider that the BS adopts wireless power transfer (WPT) to power the EDs (for their data transmission and local training) in each round of FL iteration, and formulate a joint optimization of the BS’s WPT for different EDs, the EDs’ NOMA-transmission for sending the local models to the BS, the BS’s broadcasting of the aggregated model to all EDs, the processing-rates of the BS and EDs, as well as the training-accuracy of the FL, with the objective of minimizing the system-wise cost accounting for the total energy consumption as well as the FL convergence latency. In spite of the strict non-convexity of the joint optimization problem, we analytically characterize the BS’s and all EDs’ optimal processing-rates, based on which we propose a layered algorithm for finding the optimal solutions for the joint optimization problem via exploiting monotonic optimization. Numerical results validate that our algorithm can achieve the optimal solution as LINGO’s global-solver (i.e., a commercial optimization package) while significantly reducing the computation-time. Moreover, the results also demonstrate that our NOMA assisted FL can reduce the system cost compared to the benchmark FL scheme with the fixed local training-accuracy by more than 70% and the conventional frequency division multiple access (FDMA) based FL by 78%. [ABSTRACT FROM AUTHOR]

The integration of mobile edge computing (MEC) into an ultra-dense network (UDN) can provide ubiquitous task offloading services to computation-demanding users leveraging densely deployed micro base stations. The conventional multi-user task offloading strategies are performed centrally, where a central node makes global task offloading decisions on server selection and resource allocation. In practice, the deployment becomes prohibitively complex with the increasing number of users as it involves high communication overhead and complex global optimization operations. In this paper, we develop a novel decentralized task offloading strategy in UDN, enabling users to independently make local task offloading decisions. We formulate the associated optimization problem to minimize the long-term average task delay among all users. On this basis, we develop a novel calibrated contextual bandit learning (CCBL) algorithm, where users can learn the computational delay functions of micro base stations and predict the task offloading decisions of other users in a decentralized manner. The convergence of the proposed CCBL algorithm is verified via the approachability theory. Moreover, we transfer the target of calibrated learning from all micro base stations to a single user and propose a user-oriented CCBL algorithm to further decrease the computational complexity and increase the convergence rate. Simulation results illustrate that our proposed algorithm outperforms the existing decentralized algorithms and approaches the centralized one. [ABSTRACT FROM AUTHOR]

Dual Coordinate Descent (DCD) is one of the most popular optimization methods. The parallelization of DCD is difficult, as DCD is sequential in nature. As such, simultaneously running multiple DCD threads on batches of data elements causes result inaccuracy and slow convergence, due to the concurrent updates of multiple coordinates. Some parallelization methods adopt separable approximate functions that depend on the degree of parallelism. Such dependencies result in both poor scalability and slow-convergence. To address these challenges, in this paper we present a new parallel primal-dual algorithm for DCD, called PPD. In PPD, the block data distribution is utilized to obtain a new approximate function that is independent from the parallelism. Moreover, PPD is designed with a novel primal-dual acceleration scheme to approach the optimal solution closely and quickly. We demonstrate the advantages of PPD in terms of scalability and efficiency through experiments. [ABSTRACT FROM AUTHOR]

We consider a class of nonconvex regularized optimization problems, which appear frequently in machine learning and data processing. Due to the structure of the problems, the iteratively reweighted algorithm was developed and applied to the consensus optimization. In this paper, we propose the acceleration of this scheme by adding an inertial term in each iteration. The proposed algorithms inherit the advantages of classical decentralized algorithms: they can be implemented over a connected network, in which the agents communicate with their neighbors and perform local computations. We also employ the diminishing stepsizes technique for the iteratively reweighted algorithm and consider its acceleration. In specific cases, our algorithms reduce to existing decentralized schemes and also indicate novel ones. Mathematically, we prove the convergence for both algorithms with several assumptions on the objective functions. With Kurdyka-Łojasiewicz property, convergence rates can be derived for constant stepsize case. Numerical results demonstrate the efficiency of the algorithms. [ABSTRACT FROM AUTHOR]

Most recent works in device-to-device (D2D) underlay communications focus on the optimization of either power or channel allocation to improve the spectral efficiency, and typically consider uplink and downlink separately. Further, several of them also assume perfect knowledge of channel-state-information (CSI). In this paper, we formulate a joint uplink and downlink resource allocation scheme, which assigns both power and channel resources to D2D pairs and cellular users in an underlay network scenario. The objective is to maximize the overall network rate while maintaining fairness among the D2D pairs. In addition, we also consider imperfect CSI, where we guarantee a certain outage probability to maintain the desired quality-of-service (QoS). The resulting problem is a mixed integer non-convex optimization problem and we propose both centralized and decentralized algorithms to solve it, using convex relaxation, fractional programming, and alternating optimization. In the decentralized setting, the computational load is distributed among the D2D pairs and the base station, keeping also a low communication overhead. Moreover, we also provide a theoretical convergence analysis, including also the rate of convergence to stationary points. The proposed algorithms have been experimentally tested in a simulation environment, showing their favorable performance, as compared with the state-of-the-art alternatives. [ABSTRACT FROM AUTHOR]

Constrained reinforcement learning (CRL), also termed as safe reinforcement learning, is a promising technique enabling the deployment of RL agent in real-world systems. In this paper, we propose a successive convex approximation based off-policy optimization (SCAOPO) algorithm to solve the general CRL problem, which is formulated as a constrained Markov decision process (CMDP) in context of the average cost. The SCAOPO is based on solving a sequence of convex objective/feasibility optimization problems obtained by replacing the objective and constraint functions in the original problem with convex surrogate functions. The proposed SCAOPO enables reuse of experiences from previous updates, thereby significantly reducing implementation cost when deployed in real-world engineering systems that need to online learn the environment. In spite of the time-varying state distribution and the stochastic bias incurred by off-policy learning, the SCAOPO with a feasible initial point can still provably converge to a Karush-Kuhn-Tucker (KKT) point of the original problem almost surely. [ABSTRACT FROM AUTHOR]

One-bit compressive sensing gains its popularity in signal processing and communications due to its low storage costs and low hardware complexity. However, it has been a challenging task to recover the signal only by exploiting the one-bit (the sign) information. In this paper, we appropriately formulate the one-bit compressive sensing into a double-sparsity constrained optimization problem. The first-order optimality conditions for this nonconvex and discontinuous problem are established via the newly introduced $\tau$ -stationarity, based on which, a gradient projection subspace pursuit (GPSP) algorithm is developed. It is proven that GPSP can converge globally and terminate within finite steps. Numerical experiments have demonstrated its excellent performance in terms of a high order of accuracy with a high computational speed. [ABSTRACT FROM AUTHOR]

Over the decades, multiple approaches have been proposed to solve convex programs. The development of interior-point methods allowed solving a more general set of convex programs known as semi-definite and second-order cone programs. However, these methods are excessively slow for high dimensions. On the other hand, optimization algorithms on manifolds have shown great abilities in finding solutions to non-convex problems in a reasonable time. This paper suggests using a Riemannian optimization approach to solve a subset of convex optimization problems wherein the optimization variable is a doubly stochastic matrix. Optimization over the set of doubly stochastic matrices is crucial for multiple communications and signal processing applications, especially graph-based clustering. The paper introduces and investigates the geometries of three convex manifolds, namely the doubly stochastic, the symmetric, and the definite multinomial manifolds which generalize the simplex, also known as the multinomial manifold. Theoretical complexity analysis and numerical simulation results testify the efficiency of the proposed method over state-of-the-art algorithms. In particular, they reveal that the proposed framework outperforms conventional generic and specialized approaches, especially in high dimensions. [ABSTRACT FROM AUTHOR]

This paper presents a preconditioned Krasnoselskii-Mann (KM) algorithm with an improved EM preconditioner (IEM-PKMA) for higher-order total variation (HOTV) regularized positron emission tomography (PET) image reconstruction. The PET reconstruction problem can be formulated as a three-term convex optimization model consisting of the Kullback–Leibler (KL) fidelity term, a nonsmooth penalty term, and a nonnegative constraint term which is also nonsmooth. We develop an efficient KM algorithm for solving this optimization problem based on a fixed-point characterization of its solution, with a preconditioner and a momentum technique for accelerating convergence. By combining the EM precondtioner, a thresholding, and a good inexpensive estimate of the solution, we propose an improved EM preconditioner that can not only accelerate convergence but also avoid the reconstructed image being “stuck at zero.” Numerical results in this paper show that the proposed IEM-PKMA outperforms existing state-of-the-art algorithms including, the optimization transfer descent algorithm and the preconditioned L-BFGS-B algorithm for the differentiable smoothed anisotropic total variation regularized model, the preconditioned alternating projection algorithm, and the alternating direction method of multipliers for the nondifferentiable HOTV regularized model. Encouraging initial experiments using clinical data are presented. [ABSTRACT FROM AUTHOR]

The distributed Volt/Var control solution is a promising one for active distribution networks (ADNs), but a critical issue is how to improve the distributed algorithm's convergence with less communication burden. In this paper, we propose a bi-level consensus alternation direction multiplier method-based fully distributed Volt/Var optimization algorithm (B-FADMM) for ADNs by exploiting inverter-based rapid control devices. In the first level, the Volt/Var optimal solution of subpartitions is obtained in parallel, and the second level completes the variable increment of each partition to correct the search direction of the first level. In the algorithm, firstly, an adaptive matrix $\omega $ initialization strategy is proposed to adapt to the different magnitude of variables, which avoids non-convergence or slow convergence of the algorithm. Secondly, the null-space method is used in the second level of the algorithm to obtain the null-space basis matrix on the active constraint of the subpartition, which reduces the dimensions of the problem and the computational complexity. In addition, the conjugate gradient (CG) algorithm is used to update the dual multiplier of each partition only by exchanging coupling information among partitions, which protects information privacy. Experiments showed the proposed approach attains the same result as the centralized, the computational performance of the proposed method is far superior to some popular other distributed methods in terms of accuracy, computational efficiency, and scalability. [ABSTRACT FROM AUTHOR]

This paper investigates a wireless federated learning (FL) network with limited communication bandwidth, where multiple mobile clients train their individual models with the help of one central server. We consider the practical communication scenarios, where the clients should complete the local computation and model upload within a defined latency. By jointly exploiting the dynamic characteristics of wireless channels and computational capability at the clients, we optimize the federated learning network by maximizing the number of active clients under the constraints of both latency and bandwidth. Specifically, we propose two bandwidth allocation (BA) schemes, where scheme I is based on the instantaneous channel state information (CSI), while scheme II employs the particle swarm optimization (PSO) method, based on the statistical CSI. Simulation results on the test accuracy and convergence rate are finally provided to demonstrate the advantages of the proposed optimization schemes for the considered FL network. [ABSTRACT FROM AUTHOR]

Distributed primal-dual methods have been widely used for solving large-scale constrained optimization problems. The majority of existing results focus on the problems with decoupled constraints. Some recent works have studied the problems subject to separable globally coupled constraints. This paper considers the distributed optimization problems with globally coupled constraints over networks without requiring the separability of the globally coupled constraints. This is made possible by the local estimates of the constraint violations. For solving such a problem, we propose a primal-dual algorithm in the augmented Lagrangian framework, combining the average consensus technique. We first establish a non-ergodic convergence rate of $\mathcal {O}(1/k)$ in terms of the objective residual for solving a distributed constrained convex optimization problem, where $k$ is the iteration counter. Specifically, the global objective function is the aggregate of the local convex and possibly non-smooth costs, and the coupled constraint is the sum of the local linear equality constraints. The numerical results illustrate the performance of the proposed method. [ABSTRACT FROM AUTHOR]

The use of a large-scale antenna array (LSAA) has become an important characteristic of multi-antenna communication systems to achieve beamforming gains such as in designing millimeter-wave (mmWave) systems to combat severe propagation losses. In such applications, each antenna element has to be driven by a radio frequency (RF) chain for the implementation of fully-digital beamformers, significantly increasing the hardware cost, complexity, and power consumption. Therefore, constant-modulus analog beamforming (CMAB) becomes a viable solution. In this paper, we consider the scaled analog beamforming (SAB) or constant-modulus analog beamforming (CMAB) architecture and design the system parameters by solving two variants of beampattern matching problem. In the first case, both the magnitude and phase of the beampattern are matched to the given desired beampattern whereas in the second case, only the magnitude of the beampattern is matched. Both the beampattern matching problems are cast as a variant of the constant-modulus least-squares (CLS) problem. We provide efficient algorithms based on the alternating majorization-minimization (AMM) framework that combines the alternating minimization and the MM frameworks and the conventional-cyclic coordinate descent (C-CCD) algorithms to solve the problem in each case. We also propose algorithms based on a new modified-CCD (M-CCD) based approach. For all the developed algorithms we prove convergence to a Karush-Kuhn-Tucker (KKT) point (or a stationary point). Numerical results demonstrate that the proposed algorithms converge faster than the state-of-the-art solutions. Among all the algorithms, the M-CCD-based algorithms have faster convergence when evaluated in terms of the number of iterations and the AMM-based algorithms offer lower complexity. [ABSTRACT FROM AUTHOR]

Over-the-air computation (AirComp) is a disruptive technique for fast wireless data aggregation in Internet of Things (IoT) networks via exploiting the waveform superposition property of multiple-access channels. However, the performance of AirComp is bottlenecked by the worst channel condition among all links between the IoT devices and the access point. In this paper, a reconfigurable intelligent surface (RIS) assisted AirComp system is proposed to boost the received signal power and thus mitigate the performance bottleneck by reconfiguring the propagation channels. With an objective to minimize the AirComp distortion, we propose a joint design of AirComp transceivers and RIS phase-shifts, which however turns out to be a highly intractable non-convex programming problem. To this end, we develop a novel alternating minimization framework in conjunction with the successive convex approximation technique, which is proved to converge monotonically. To reduce the computational complexity, we transform the subproblem in each alternation as a smooth convex-concave saddle point problem, which is then tackled by proposing a Mirror-Prox method that only involves a sequence of closed-form updates. Simulations show that the computation time of the proposed algorithm can be two orders of magnitude smaller than that of the state-of-the-art algorithms, while achieving a similar distortion performance. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

This paper studies the convergence properties of the well-known message-passing algorithm for convex optimisation. Under the assumption of pairwise separability and scaled diagonal dominance, asymptotic convergence is established and a simple bound for the convergence rate is provided for message-passing. In comparison with previous results, our results do not require the given convex program to have known convex pairwise components and that our bound for the convergence rate is tighter and simpler. When specialised to quadratic optimisation, we generalise known results by providing a very simple bound for the convergence rate. [ABSTRACT FROM AUTHOR]

Sparse coding is a class of unsupervised methods for learning a sparse representation of the input data in the form of a linear combination of a dictionary and a sparse code. This learning framework has led to state-of-the-art results in various signal processing tasks. However, classical methods learn the dictionary and the sparse code based on alternating optimizations, usually without theoretical guarantees for either optimality or convergence due to the non-convexity of the problem. Recent works on sparse coding with a complete dictionary provide strong theoretical guarantees thanks to the development of non-convex optimization. However, initial non-convex approaches learned the dictionary in the sparse coding problem sequentially in an atom-by-atom manner, which led to a long execution time. More recent works  have sought to directly learn the entire dictionary at once, which substantially reduces the execution time. However, the associated recovery performance is degraded with a finite number of data samples. In this paper, we propose an efficient sparse coding scheme with a two-stage optimization. The proposed scheme leverages the global and local Riemannian geometry of the two-stage optimization problem and facilitates fast implementation for superb dictionary recovery performance by a finite number of samples. We further prove that, with high probability, the proposed scheme can exactly recover any atom in the target dictionary with a finite number of samples. Experiments on both synthetic and real-world data verify the efficiency and robustness of the proposed scheme.1 [ABSTRACT FROM AUTHOR]

In this paper, we consider the policy evaluation problem in reinforcement learning with agents on a decentralized and directed network. In order to evaluate the quality of a fixed policy in this decentralized setting, one option is for agents to run decentralized temporal-difference (TD) collaboratively. To account for the practical scenarios where the state and action spaces are large and malicious attacks emerge, we focus on the decentralized TD learning with linear function approximation in the presence of malicious agents (often termed as Byzantine agents). We propose a trimmed mean-based Byzantine-resilient decentralized TD algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error that depends on the number of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

We consider the problem of minimizing the $\ell _1$ norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms with provable guarantees for tackling this problem. Specifically, our contribution is fourfold. First, we present a manifold proximal point algorithm (ManPPA) for the problem and show that it converges at a global sublinear rate. Furthermore, we show that ManPPA can achieve a local quadratic convergence rate when applied to sharp instances of the problem. Second, we develop a semismooth Newton-based inexact augmented Lagrangian method for computing the search direction in each iteration of ManPPA and show that it has an asymptotic superlinear convergence rate. Third, we propose a stochastic variant of ManPPA called StManPPA, which is well suited for large-scale computation, and establish its sublinear convergence rate. Both ManPPA and StManPPA have provably faster convergence rates than existing subgradient-type methods. Fourth, using ManPPA as a building block, we propose a new heuristic method for solving a matrix analog of the problem, in which the sphere is replaced by the Stiefel manifold. The results from our extensive numerical experiments on the ODL and RSR problems demonstrate the efficiency and efficacy of our proposed methods. [ABSTRACT FROM AUTHOR]

In this paper, we consider minimizing a sum of local convex objective functions in a distributed setting, where the cost of communication and/or computation can be expensive. We extend and generalize the analysis for a class of nested gradient-based distributed algorithms [NEAR-DGD, (Berahas et al., 2019)] to account for multiple gradient steps at every iteration. We show the effect of performing multiple gradient steps on the rate of convergence and on the size of the neighborhood of convergence, and prove R-Linear convergence to the exact solution with a fixed number of gradient steps and increasing number of consensus steps. We test the performance of the generalized method on quadratic functions and show the effect of multiple consensus and gradient steps in terms of iterations, number of gradient evaluations, number of communications and cost. [ABSTRACT FROM AUTHOR]

Stochastic compositional optimization generalizes classic (non-compositional) stochastic optimization to the minimization of compositions of functions. Each composition may introduce an additional expectation. The series of expectations may be nested. Stochastic compositional optimization is gaining popularity in applications such as reinforcement learning and meta learning. This paper presents a new Stochastically Corrected Stochastic Compositional gradient method (SCSC). SCSC runs in a single-time scale with a single loop, uses a fixed batch size, and guarantees to converge at the same rate as the stochastic gradient descent (SGD) method for non-compositional stochastic optimization. This is achieved by making a careful improvement to a popular stochastic compositional gradient method. It is easy to apply SGD-improvement techniques to accelerate SCSC. This helps SCSC achieve state-of-the-art performance for stochastic compositional optimization. In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional stochastic optimization. We test SCSC using the model-agnostic meta-learning tasks. [ABSTRACT FROM AUTHOR]

As for Generative Adversarial Networks (GANs), its interpretability may be closely related to optimization objective functions, that is, information metrics play important roles in networks training and data generation. In terms of original GAN, the objective function based on Kullback-Leibler (KL) divergence has limitations on the performance of data training and generation. Therefore, it is significant to investigate objective functions for the optimization in GANs to bring gains on the efficiency of network learning from the perspective of metrics. In this paper, the objective function with exponential form, referred from the Message Importance Measure (MIM), is adapted to replace that with logarithm form in the optimization for adversarial networks. This approach named MIM-based GAN, may provide more hidden information in terms of interpretability on training process and probability events generation. Specifically, we first analyze the intrinsic relationship between the proposed approach and other classical GANs. Moreover, compared with the original GAN, LSGAN and WGAN, we discuss its advantages on training performance in theory including sensitivity, convergence rate and so on. In addition, we do simulations on the datasets to confirm why the MIM-based GAN achieves state-of-the-art performance on training process and data generation. [ABSTRACT FROM AUTHOR]

Phase-only beamforming (POB) plays an important role in modern radar and communication systems. The NP-hard nature of POB makes it difficult to be solved for large-scale antenna arrays. This paper studies two fast and simple GP algorithms for the problem of POB. First we give a new interpretation of the GP method under the framework of sequential quadratic programming (SQP). Then we prove that the GP operation gives a strict global minimizer of the subproblem of SQP. Moreover, the larger the step size in GP operation, the smaller the minimum value of the subproblem. Based on this observation we design two variable step size GP (VSGP) algorithms that improve the convergence speed of the GP method significantly. Furthermore, the VSGP algorithms have a simple structure, making them easy to be implemented in chips. Finally, we derive a necessary and sufficient condition for the Karush-Kuhn-Tucker (KKT) points of the POB problem. We also prove that, for any initialization, all limit points of the iterates generated by the VSGP algorithms converge to the KKT points. Simulation results show that the proposed algorithms outperform the state-of-the-art algorithms in terms of convergence speed and computational cost. [ABSTRACT FROM AUTHOR]

In this paper, we investigate resource allocation for IRS-assisted green multiuser multiple-input single-output (MISO) systems. To minimize the total transmit power, both the beamforming vectors at the access point (AP) and the phase shifts at multiple IRSs are jointly optimized, while taking into account the minimum required quality-of-service (QoS) of multiple users. First, two novel algorithms, namely a penalty-based alternating minimization (AltMin) algorithm and an inner approximation (IA) algorithm, are developed to tackle the non-convexity of the formulated optimization problem when perfect channel state information (CSI) is available. Existing designs employ semidefinite relaxation in AltMin-based algorithms, which, however, cannot ensure convergence. In contrast, the proposed penalty-based AltMin and IA algorithms are guaranteed to converge to a stationary point and a Karush-Kuhn-Tucker (KKT) solution of the design problem, respectively. Second, the impact of imperfect knowledge of the CSI of the channels between the AP and the users is investigated. To this end, a non-convex robust optimization problem is formulated and the penalty-based AltMin algorithm is extended to obtain a stationary solution. Simulation results reveal a key trade-off between the speed of convergence and the achievable total transmit power for the two proposed algorithms. In addition, we show that the proposed algorithms can significantly reduce the total transmit power at the AP compared to various baseline schemes and that the optimal numbers of transmit antennas and IRS reflecting elements, which maximize the system energy efficiency of the considered system, are finite. [ABSTRACT FROM AUTHOR]

Coupled Matrix Tensor Factorization (CMTF) facilitates the integration and analysis of multiple data sources and helps discover meaningful information. Nonnegative CMTF (N-CMTF) has been employed in many applications for identifying latent patterns, prediction, and recommendation. However, due to the added complexity with coupling between tensor and matrix data, existing N-CMTF algorithms exhibit poor computation efficiency. In this paper, a computationally efficient N-CMTF factorization algorithm is presented based on the column-wise element selection, preventing frequent gradient updates. Theoretical and empirical analyses show that the proposed N-CMTF factorization algorithm is not only more accurate but also more computationally efficient than existing algorithms in approximating the tensor as well as in identifying the underlying nature of factors. [ABSTRACT FROM AUTHOR]