Search

In this paper, we study a class of nonconvex and nonsmooth fractional optimization problem, where the numerator of which is the sum of a nonsmooth and nonconvex function and a relative smooth nonconvex function, while the denominator is relative weakly convex nonsmooth function. We propose a Bregman proximal subgradient algorithm for solving this type of fractional optimization problems. Under moderate conditions, we prove that the subsequence generated by the proposed algorithm converges to a critical point, and the generated sequence globally converges to a critical point when the objective function satisfies the Kurdyka-Łojasiewicz property. We also obtain the convergence rate of the proposed algorithm. Finally, two numerical experiments illustrate the effectiveness and superiority of the algorithm. Our results give a positive answer to an open problem proposed by Bot et al. [14]. [ABSTRACT FROM AUTHOR]

In this paper, a novel non-convex fractional-order image denoising model is proposed to suppress the staircase effect produced by the TV model while maintaining a neat contour. The model combines ℓ q (0 < q < 1) quasi-norm and fractional-order regularization, and employs a diffusion coefficient with a faster convergence rate to preserve more image edges and details. Additionally, an adaptive regularization parameter is designed to adjust the denoising performance of the algorithm. To obtain the optimal approximate solution of the model, an enhanced primal-dual algorithm is adopted and the complexity and convergence of the algorithm are theoretically analyzed. Finally, the effectiveness of the proposed method is demonstrated through numerical experiments. • A new non-convex FOTV model is proposed. • A new diffusion coefficient is introduced to retain more edge details of the image. • The existing primal-dual algorithm is improved, and the convergence of this algorithm is analyzed. • An adaptive regularization parameter is designed. [ABSTRACT FROM AUTHOR]

We introduce a bundle method for the unconstrained minimization of nonsmooth difference-of-convex (DC) functions, and it is based on the calculation of a special type of descent direction called descent–ascent direction. The algorithm only requires evaluations of the minuend component function at each iterate, and it can be considered as a parsimonious bundle method as accumulation of information takes place only in case the descent–ascent direction does not provide a sufficient decrease. No line search is performed, and proximity control is pursued independent of whether the decrease in the objective function is achieved. Termination of the algorithm at a point satisfying a weak criticality condition is proved, and numerical results on a set of benchmark DC problems are reported. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms – Continuous. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information (https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0142) as well as from the IJOC GitHub software repository (https://github.com/INFORMSJoC/2023.0142). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/. [ABSTRACT FROM AUTHOR]

The concept of jump system, introduced by Bouchet and Cunningham (1995), is a set of integer points satisfying a certain exchange property. We consider the minimization of a separable convex function on a jump system. It is known that the problem can be solved by a greedy algorithm. In this paper, we are interested in whether the greedy algorithm has the geodesic property, which means that the trajectory of the solutions generated by the algorithm is a geodesic from the initial solution to a nearest optimal solution. We show that a special implementation of the greedy algorithm enjoys the geodesic property, while the original algorithm does not. As a corollary to this, we present a new greedy algorithm for linear optimization on a delta-matroid and show that the algorithm has the geodesic property. [ABSTRACT FROM AUTHOR]

In this article, we consider and investigate a split convex feasibility problem involving two bounded linear operators in Hilbert spaces. We introduce a self-adaptive CQ-type algorithm by selecting the stepsize which is independent of the operator norms and establish a strong convergence result of the proposed algorithm under some mild control conditions. Moreover, we propose a self-adaptive relaxed CQ-type algorithm for solving the problem constrained by sub-level sets of convex functions. A numerical example and an application in compressed sensing are also given to illustrate the convergence behaviour of our proposed algorithms. Our results in this paper improve and generalize some existing results in the literature. [ABSTRACT FROM AUTHOR]

In this paper, we consider a class of structured nonsmooth difference-of-convex (DC) minimization in which the first convex component is the sum of a smooth and a nonsmooth function, while the second convex component is the supremum of finitely many convex smooth functions. The existing methods for this problem usually have weak convergence guarantees or need to solve lots of subproblems per iteration. Due to this, we propose hybrid algorithms for solving this problem in which we first compute approximate critical points and then check whether these points are approximate D-stationary points. Under suitable conditions, we prove that there exists a subsequence of iterates of which every accumulation point is a D-stationary point. Some preliminary numerical experiments are conducted to demonstrate the efficiency of the proposed algorithms. [ABSTRACT FROM AUTHOR]

By the asymptotic oracle property, non-convex penalties represented by minimax concave penalty (MCP) and smoothly clipped absolute deviation (SCAD) have attracted much attentions in high-dimensional data analysis, and have been widely used in signal processing, image restoration, matrix estimation, etc. However, in view of their non-convex and non-smooth characteristics, they are computationally challenging. Almost all existing algorithms converge locally, and the proper selection of initial values is crucial. Therefore, in actual operation, they often combine a warm-starting technique to meet the rigid requirement that the initial value must be sufficiently close to the optimal solution of the corresponding problem. In this paper, based on the DC (difference of convex functions) property of MCP and SCAD penalties, we aim to design a global two-stage algorithm for the high-dimensional least squares linear regression problems. A key idea for making the proposed algorithm to be efficient is to use the primal dual active set with continuation (PDASC) method to solve the corresponding sub-problems. Theoretically, we not only prove the global convergence of the proposed algorithm, but also verify that the generated iterative sequence converges to a d-stationary point. In terms of computational performance, the abundant research of simulation and real data show that the algorithm in this paper is superior to the latest SSN method and the classic coordinate descent (CD) algorithm for solving non-convex penalized high-dimensional linear regression problems. [ABSTRACT FROM AUTHOR]

We consider the incremental subgradient algorithm employing dynamic step sizes for minimizing the sum of a large number of component convex functions. The dynamic step size rule was firstly introduced by Goffin and Kiwiel [Math. Program., 1999, 85(1): 207‐211] for the subgradient algorithm, soon later, for the incremental subgradient algorithm by Nedic and Bertsekas in [SIAM J. Optim., 2001, 12(1): 109‐138]. It was observed experimentally that the incremental approach has been very successful in solving large separable optimizations and that the dynamic step sizes generally have better computational performance than others in the literature. In the present paper, we propose two modified dynamic step size rules for the incremental subgradient algorithm and analyse the convergence and complexity properties of them. At last, the assignment problem is considered and the incremental subgradient algorithms employing different kinds of dynamic step sizes are applied to solve the problem. The computational experiments show that the two modified ones converges dramatically faster and more stable than the corresponding one in [SIAM J. Optim., 2001, 12(1): 109‐138]. Particularly, for solving large separable convex optimizations, we strongly recommend the second one (see Algorithm 3.3 in the paper) since it has interesting computational performance and is the simplest one. [ABSTRACT FROM AUTHOR]

Conjugate gradient methods are much effective and widely used for large-scale unconstrained optimization problems by their simple computation, low memory requirement and strong global convergence property. Spectral gradient methods are also effective for large-scale problems. In this paper, a class of new descent spectral three-term conjugate gradient algorithms are proposed which automatically have the sufficient descent property and satisfy the Dai-Liao conjugate condition. Under the Wolfe line search technique and some standard conditions, the proposed methods are globally convergent for strongly convex functions and general nonlinear functions with the help of the modified secant equations. In numerical part, 732 problems with dimensions varying from 1500 to 150000 and three image restoration problems with three noise levels are considered. Numerical results indicate that the proposed algorithms are more efficient, reliable and robust than the other methods for the testing problems. • A class of spectral descent three-term conjugate gradient methods are proposed. • The proposed methods own the sufficient descent property and satisfy the Dai-Liao conjugate condition automatically. • The proposed methods are globally convergent for general functions without convex assumptions. • The proposed methods are applied to large-scale unconstrained optimization problems and some image restoration problems. [ABSTRACT FROM AUTHOR]

In this paper, we focus on a descent algorithm for solving nonsmooth nonconvex optimization problems. The proposed method is based on the proximal bundle algorithm and the gradient sampling method and uses the advantages of both. In addition, this algorithm has the ability to handle inexact information, which creates additional challenges. The global convergence is proved with probability one. More precisely, every accumulation point of the sequence of serious iterates is either a stationary point if exact values of gradient are provided or an approximate stationary point if only inexact information of the function and gradient values is available. The performance of the proposed algorithm is demonstrated using some academic test problems. We further compare the new method with a general nonlinear solver and two other methods specifically designed for nonconvex nonsmooth optimization problems. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a three-operator splitting algorithm with deviations for solving the minimization problem composed of the sum of two convex functions minus a convex and smooth function in a real Hilbert space. The main feature of the proposed method is that two per-iteration deviation vectors provide additional degrees of freedom. We propose one-step and two step inertial three-operator splitting algorithms by selecting the deviations along a momentum direction. A numerical experiment for DC regularized sparse recovery problems shows that the proposed algorithms have better performance than the original three-operator splitting algorithm. [ABSTRACT FROM AUTHOR]

In this paper, an algorithm for maximizing a convex function over a convex feasible set is proposed. The algorithm, called CoMax, consists of two phases: in phase 1, a feasible starting point is obtained that is used in a gradient ascent algorithm in phase 2. The main contribution of the paper is connected to phase 1; five different methods are used to approximate the original NP-hard problem of maximizing a convex function (MCF) by a tractable convex optimization problem. All the methods use the minimizer of the convex objective function in their construction. In phase 2, the gradient ascent algorithm yields stationary points to the MCF problem. The performance of CoMax is tested on a wide variety of MCF problems, demonstrating its efficiency. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek [Grant 406.17.511]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplementary Information [https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1238] or is available from the IJOC GitHub software repository (https://github.com/INFORMSJoC) at [http://dx.doi.org/10.5281/zenodo.6884872]. [ABSTRACT FROM AUTHOR]

This paper considers the problem of minimizing a convex expectation function with a set of inequality convex expectation constraints. We propose a stochastic augmented Lagrangian-type algorithm—namely, the stochastic linearized proximal method of multipliers—to solve this convex stochastic optimization problem. This algorithm can be roughly viewed as a hybrid of stochastic approximation and the traditional proximal method of multipliers. Under mild conditions, we show that this algorithm exhibits O (K − 1 / 2) expected convergence rates for both objective reduction and constraint violation if parameters in the algorithm are properly chosen, where K denotes the number of iterations. Moreover, we show that, with high probability, the algorithm has a O (log (K) K − 1 / 2) constraint violation bound and O (log 3 / 2 (K) K − 1 / 2) objective bound. Numerical results demonstrate that the proposed algorithm is efficient. History: Accepted byAntonio Frangioni, Area Editor for Design & Analysis ofAlgorithms—Continuous. Funding: This work was supported by the National Natural Science Foundation of China [Grants 11971089, 11731013, 12071055, and 11871135]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplementary Information [https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1228] or is available from the IJOC GitHub software repository (https://github.com/INFORMSJoC) at http://dx.doi.org/10.5281/zenodo.6818229. [ABSTRACT FROM AUTHOR]

In this paper, a new Modified Meta-Heuristic algorithm is proposed. This method contains some modifications to improve the performance of the simulated-annealing algorithm (SA). Most authors who deal with improving the SA algorithm presented some improvements and modifications to one or more of the five standard features of the SA algorithm. In this paper, we improve the SA algorithm by presenting some suggestions and modifications to all five standard features of the SA algorithm. Through these suggestions and modifications, we obtained a new algorithm that finds the approximate solution to the global minimum of a non-convex function. The new algorithm contains novel parameters, which are updated at each iteration. Therefore, the variety and alternatives in choosing these parameters demonstrated a noticeable impact on the performance of the proposed algorithm. Furthermore, it has multiple formulas by which the candidate solutions are generated. Diversity in these formulas helped the proposed algorithm to escape a local point while finding the global minimizer of a non-convex function. The efficiency of the proposed algorithm is reported through extensive numerical experiments on some well-known test problems. The performance profiles are used to evaluate and compare the performance of our proposed algorithm against the other five meta-heuristic algorithms. The comparison results between the performance of our suggested algorithm and the other five algorithms indicate that the proposed algorithm is competitive with, and in all cases superior to, the five algorithms in terms of the efficiency, reliability, and effectiveness for finding the global minimizers of non-convex functions. This superiority of the new proposed algorithm is due to those five modified standard features. [ABSTRACT FROM AUTHOR]

This paper considers a stochastic optimization problem over the fixed point sets of quasinonexpansive mappings on Riemannian manifolds. The problem enables us to consider Riemannian hierarchical optimization problems over complicated sets, such as the intersection of many closed convex sets, the set of all minimizers of a nonsmooth convex function, and the intersection of sublevel sets of nonsmooth convex functions. We focus on adaptive learning rate optimization algorithms, which adapt step-sizes (referred to as learning rates in the machine learning field) to find optimal solutions quickly. We then propose a Riemannian stochastic fixed point optimization algorithm, which combines fixed point approximation methods on Riemannian manifolds with the adaptive learning rate optimization algorithms. We also give convergence analyses of the proposed algorithm for nonsmooth convex and smooth nonconvex optimization. The analysis results indicate that, with small constant step-sizes, the proposed algorithm approximates a solution to the problem. Consideration of the case in which step-size sequences are diminishing demonstrates that the proposed algorithm solves the problem with a guaranteed convergence rate. This paper also provides numerical comparisons that demonstrate the effectiveness of the proposed algorithms with formulas based on the adaptive learning rate optimization algorithms, such as Adam and AMSGrad. [ABSTRACT FROM AUTHOR]

In this paper, we propose two novel inertial forward–backward splitting methods for solving the constrained convex minimization of the sum of two convex functions, φ 1 + φ 2 , in Hilbert spaces and analyze their convergence behavior under some conditions. For the first method (iFBS), we use the forward–backward operator. The step size of this method depends on a constant of the Lipschitz continuity of ∇ φ 1 , hence a weak convergence result of the proposed method is established under some conditions based on a fixed point method. With the second method (iFBS-L), we modify the step size of the first method, which is independent of the Lipschitz constant of ∇ φ 1 by using a line search technique introduced by Cruz and Nghia. As an application of these methods, we compare the efficiency of the proposed methods with the inertial three-operator splitting (iTOS) method by using them to solve the constrained image inpainting problem with nuclear norm regularization. Moreover, we apply our methods to solve image restoration problems by using the least absolute shrinkage and selection operator (LASSO) model, and the results are compared with those of the forward–backward splitting method with line search (FBS-L) and the fast iterative shrinkage-thresholding method (FISTA). [ABSTRACT FROM AUTHOR]

In this paper, we present difference of convex algorithms for solving bilevel programs in which the upper level objective functions are difference of convex functions, and the lower level programs are fully convex. This nontrivial class of bilevel programs provides a powerful modelling framework for dealing with applications arising from hyperparameter selection in machine learning. Thanks to the full convexity of the lower level program, the value function of the lower level program turns out to be convex and hence the bilevel program can be reformulated as a difference of convex bilevel program. We propose two algorithms for solving the reformulated difference of convex program and show their convergence to stationary points under very mild assumptions. Finally we conduct numerical experiments to a bilevel model of support vector machine classification. [ABSTRACT FROM AUTHOR]

In this paper, a subgradient-based neurodynamic algorithm is presented to solve the nonsmooth nonconvex interval-valued optimization problem with both partial order and linear equality constraints, where the interval-valued objective function is nonconvex, and interval-valued partial order constraint functions are convex. The designed neurodynamic system is constructed by a differential inclusion with upper semicontinuous right-hand side, whose calculation load is reduced by relieving penalty parameters estimation and complex matrix inversion. Based on nonsmooth analysis and the extension theorem of the solution of differential inclusion, it is obtained that the global existence and boundedness of state solution of neurodynamic system, as well as the asymptotic convergence of state solution to the feasible region and the set of L U -critical points of interval-valued nonconvex optimization problem. Several numerical experiments and the applications to emergency supplies distribution and nondeterministic fractional continuous static games are solved to illustrate the applicability of the proposed neurodynamic algorithm. [ABSTRACT FROM AUTHOR]

This paper considers the decentralized convex optimization problem, which has a wide range of applications in large-scale machine learning, sensor networks, and control theory. We propose novel algorithms that achieve optimal computation complexity and near optimal communication complexity. Our theoretical results give affirmative answers to the open problem on whether there exists an algorithm that can achieve a communication complexity (nearly) matching the lower bound depending on the global condition number instead of the local one. Furthermore, the linear convergence of our algorithms only depends on the strong convexity of global objective and it does not require the local functions to be convex. The design of our methods relies on a novel integration of well-known techniques including Nesterov's acceleration, multi-consensus and gradient-tracking. Empirical studies show the outperformance of our methods for machine learning applications. [ABSTRACT FROM AUTHOR]

Dynamic networks are popularly used to describe networks that change with time. Although there have been a large number of research works on understanding dynamic networks using link prediction, node classification and community detection, there is rare work that is specially designed to address the challenge of big network size of dynamic networks. To this end, we study in this paper an emerging and challenging problem of network coarsening in dynamic networks. Network coarsening refers to a class of network "zoom-out" operations where node pairs and edges are grouped together for efficient analysis on big networks. However, existing network coarsening approaches can only handle static networks where network structure weights have been predefined before the coarsening calculation. Under the observation that big networks are highly dynamic and naturally change over time, we consider in this paper to embed information diffusion data which reflect the dynamics of networks for network coarsening. Specifically, we present a new Semi-NetCoarsen approach that jointly maximizes the likelihood of observing the information diffusion data and minimizes the network regularization with respect to the predefined network structural data. The learning function is convex and we use the accelerated proximal gradient algorithm to obtain the global optimal solution. We conduct experiments on two synthetic and five real-world data sets to validate the performance of the proposed method. [ABSTRACT FROM AUTHOR]

In the development of algorithms for convex optimization problems, symmetry plays a very important role in the approximation of solutions in various real-world problems. In this paper, based on a fixed point algorithm with the inertial technique, we proposed and study a new accelerated algorithm for solving a convex bilevel optimization problem for which the inner level is the sum of smooth and nonsmooth convex functions and the outer level is a minimization of a smooth and strongly convex function over the set of solutions of the inner level. Then, we prove its strong convergence theorem under some conditions. As an application, we apply our proposed algorithm as a machine learning algorithm for solving some data classification problems. We also present some numerical experiments showing that our proposed algorithm has a better performance than the five other algorithms in the literature, namely BiG-SAM, iBiG-SAM, aiBiG-SAM, miBiG-SAM and amiBiG-SAM. [ABSTRACT FROM AUTHOR]

In this paper, a relaxed variable metric primal-dual fixed-point algorithm is proposed for solving the convex optimization problem involving the sum of two convex functions where one is differentiable with the Lipschitz continuous gradient while the other is composed of a linear operator. Based on the preconditioned forward–backward splitting algorithm, the convergence of the proposed algorithm is proved. At the same time, we show that some existing algorithms are special cases of the proposed algorithm. Furthermore, the ergodic convergence and linear convergence rates of the proposed algorithm are established under relaxed parameters. Numerical experiments on the image deblurring problems demonstrate that the proposed algorithm outperforms some existing algorithms in terms of the number of iterations. [ABSTRACT FROM AUTHOR]

This paper focuses on how to obtain globally optimal multisensor data association results under the condition that the number of targets is unknown. This problem is likely to occur in scenarios where the sensors' fields of view (FoVs) are limited. The authors proved that, ideally, the cost of the association result is a convex function of the assumed target number and with the minimal value only if the target number is correctly assumed. Based on this, a multiple hypothesis S‐dimensional assignment (MHSDA) algorithm is proposed, which replaces the traditional irrevocable onetime assumption with a multiple hypothesis approach to counteract the effect of target number uncertainty. Furthermore, a hybrid implementation architecture based on Union Find plus MHSDA is proposed to ensure the real‐time performance. The proposed algorithm requires neither prior knowledge on the number of targets nor on the FoVs of sensors and is applicable to target dense scenarios. Simulation results under different FoV, target number and measurement noise configurations demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In real-world applications, it is important for machine learning algorithms to be robust against data outliers or corruptions. In this paper, we focus on improving the robustness of a large class of learning algorithms that are formulated as low-rank semi-definite programming (SDP) problems. Traditional formulations use the square loss, which is notorious for being sensitive to outliers. We propose to replace this with more robust noise models, including the $\ell _1$ ℓ 1 -loss and other nonconvex losses. However, the resultant optimization problem becomes difficult as the objective is no longer convex or smooth. To alleviate this problem, we design an efficient algorithm based on majorization-minimization. The crux is on constructing a good optimization surrogate, and we show that this surrogate can be efficiently obtained by the alternating direction method of multipliers (ADMM). By properly monitoring ADMM's convergence, the proposed algorithm is empirically efficient and also theoretically guaranteed to converge to a critical point. Extensive experiments are performed on four machine learning applications using both synthetic and real-world data sets. Results show that the proposed algorithm is not only fast but also has better performance than the state-of-the-arts. [ABSTRACT FROM AUTHOR]

At present, many conjugate gradient methods with global convergence have been proposed in unconstrained optimization, such as MPRP algorithm proposed by Zhang et al. (IMA J. Numer. Anal.26(4):629–640, 2006). Unfortunately, almost all of these methods require gradient Lipschitz continuity condition. As far as we know, how do the current conjugate gradient methods deal with gradient non-Lipschitz continuity problems is basically blank. For gradient non-Lipschitz continuity problems, Algorithm 1 and Algorithm 2 are proposed in this paper based on MPRP algorithm. The proposed algorithms have the following characteristics: (i) Algorithm 1 retains sufficient descent property independent of line search technology in MPRP algorithm; (ii) for nonconvex and gradient non-Lipschitz continuous functions, the global convergence of Algorithm 1 is obtained in combination with the trust region property and the weak Wolfe-Powell line search technique; (iii) based on Algorithm 1, Algorithm 2 is further improved which global convergence can be obtained independently of line search technique; (iv) according to numerical experiments, the proposed algorithms perform competitively with other similar algorithms. [ABSTRACT FROM AUTHOR]

The golden ratio primal-dual algorithm (GRPDA) is a new variant of the classical Arrow-Hurwicz method for solving structured convex optimization problems, in which the objective function consists of the sum of two closed proper convex functions, one of which involves a composition with a linear transform. The same as the Arrow-Hurwicz method and the popular primal-dual algorithm (PDA) of Chambolle and Pock, GRPDA is full-splitting in the sense that it does not rely on solving any subproblems or linear system of equations iteratively. Compared with PDA, an important feature of GRPDA is that it permits larger primal and dual stepsizes. However, the stepsize condition of GRPDA requires that the spectral norm of the linear transform is known, which can be difficult to obtain in some applications. Furthermore, constant stepsizes are usually overconservative in practice. In this paper, we propose a linesearch strategy for GRPDA, which not only does not require the spectral norm of the linear transform but also allows adaptive and potentially much larger stepsizes. Within each linesearch step, only the dual variable needs to be updated, and it is thus quite cheap and does not require any extra matrix-vector multiplications for many special yet important applications such as a regularized least-squares problem. Global iterate convergence and 0(1/N) ergodic convergence rate results, measured by the function value gap and constraint violations of an equivalent optimization problem, are established, where N denotes the iteration counter. When one of the component functions is strongly convex, faster O(l/N²) ergodic convergence rate results, quantified by the same measures, are established by adaptively choosing some algorithmic parameters. Moreover, when the subdifferential operators of the component functions are strongly metric subregular, a condition that is much weaker than strong convexity, we show that the iterates converge R-linearly to the unique solution. Numerical experiments on matrix game and LASSO problems illustrate the effectiveness of the proposed linesearch strategy. [ABSTRACT FROM AUTHOR]

In this paper, we concern with the split feasibility problem (SFP) whenever the convex sets involved are composed of level sets. By applying Gradient-projection algorithm which is used to solve constrained convex minimization problem of a real valued convex function, we construct two new algorithms for the split feasibility problem and prove that both of them are convergent weakly to a solution of the feasibility problem. In the end, as an application, we obtain a new algorithm for solving the split equality problem. [ABSTRACT FROM AUTHOR]

In this paper, we study distributed multiagent optimisation over undirected graphs. The optimisation problem is to minimise a global objective function, which is composed of the sum of a set of local objective functions. Recent researches on this problem have made significant progress by using primal-dual methods. However, the inner link among different algorithms is unclear. This paper shows that some state-of-the-art algorithms differ in that they incorporate the slightly different last dual gradient terms based on the augmented Lagrangian analysis. Then, we propose a distributed Nesterov accelerated optimisation algorithm, where a doubly stochastic matrix is allowed to use, and nonidentical local step-sizes are employed. We analyse the convergence of the proposed algorithm by using the generalised small gain theorem under the assumption that each local objective function is strongly convex and has Lipschitz continuous gradient. We prove that the sequence generated by the proposed algorithm linearly converge to an optimal solution if the largest step-size is positive and less than an explicitly estimated upper bound, and the largest momentum parameter is nonnegative and less than an upper bound determined by the largest step-size. Simulation results further illustrate the efficacy of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In Chirkov et al., (2019), classes of conic and discrete conic functions were introduced. In this paper we use the term convic instead conic. The class of convic functions properly includes the classes of convex functions, strictly quasiconvex functions and the class of quasiconvex polynomials. On the other hand, the class of convic functions is properly included in the class of quasiconvex functions. The discrete convic function is a discrete analogue of the convic function. In Chirkov et al., (2019), the lower bound 3 n − 1 log (2 ρ − 1) for the number of calls to the comparison oracle needed to find the minimum of the discrete convic function defined on integer points of some n -dimensional ball with radius ρ was obtained. But the problem of the existence of a polynomial (in log ρ for fixed n) algorithm for minimizing such functions has remained open. In this paper, we answer positively the question of the existence of such an algorithm. Namely, we propose an algorithm for minimizing discrete convic functions that uses 2 O (n 2 log n) log ρ calls to the comparison oracle and has 2 O (n 2 log n) poly (log ρ) bit complexity. [ABSTRACT FROM AUTHOR]

In some circumstances, the performance of machine learning (ML) tasks are based on the quality of signal (data) that is processed in these tasks. Therefore, the pre-processing techniques, such as reconstruction and denoising methods, are important techniques in ML tasks. In reconstructed (estimated) method, the fused lasso algorithm with non-convex penalty function is an efficient method when the signal corrupted by additive white Gaussian noise (AWGN) is considered. Therefore, this paper proposes new shrinkage functions for non-convex penalty functions, modified arctangent and exponential models, in fused lasso formulation. A lot of works present the shrinkage function for arctangent penalty function. Unfortunately, there is no closed-form solution. The numerical solution is required for shrinkage function of this penalty function. However, the analytical solution is derived in this paper. Moreover, the shrinkage function of modified exponential penalty function is proposed. This shrinkage function obtains from simple iterative method, fixed-point algorithm. We demonstrate the proposed methods through simulations with standard one-dimensional signals contaminated by AWGN. The proposed techniques are compared with traditional estimation methods, such as total variation (TV) and wavelet denoising methods. In experimental results, our proposed methods outperform several exiting methods both visual quality and in terms of root mean square error (RMSE). In fact, the proposed methods can better preserve the feature of noise-free signal than the compared methods. The denoised signals produced by the proposed methods are less smooth than the denoised signals produced by the compared methods. [ABSTRACT FROM AUTHOR]

It is well-known that conjugate gradient algorithms are widely applied in many practical fields, for instance, engineering problems and finance models, as they are straightforward and characterized by a simple structure and low storage. However, challenging problems remain, such as the convergence of the PRP algorithms for nonconvexity under an inexact line search, obtaining a sufficient descent for all conjugate gradient methods, and other theory properties regarding global convergence and the trust region feature for nonconvex functions. This paper studies family conjugate gradient formulas based on the six classic formulas, PRP, HS, CD, FR, LS, and DY, where the family conjugate gradient algorithms have better theory properties than those of the formulas by themselves. Furthermore, this technique of the presented conjugate gradient formulas can be extended to any two-term conjugate gradient formula. This paper designs family conjugate gradient algorithms for nonconvex functions, which have the following features without other conditions: (i) the sufficient descent property holds, (ii) the trust region feature is true, and (iii) the global convergence holds under normal assumptions. Numerical results show that the given conjugate gradient algorithms are competitive with those of normal methods. [ABSTRACT FROM AUTHOR]

In this paper, we consider a class of single-ratio fractional minimization problems, in which the numerator of the objective is the sum of a nonsmooth nonconvex function and a smooth nonconvex function while the denominator is a nonsmooth convex function. In this work, we first derive its first-order necessary optimality condition, by using the first-order operators of the three functions involved. Then we develop first-order algorithms, namely, the proximity-gradientsubgradient algorithm (PGSA), PGSA with monotone line search (PGSA ML), and PGSA with nonmonotone line search (PGSA NL). It is shown that any accumulation point of the sequence generated by them is a critical point of the problem under mild assumptions. Moreover, we establish global convergence of the sequence generated by PGSA or PGSA ML and analyze its convergence rate, by further assuming the local Lipschitz continuity of the nonsmooth function in the numerator, the smoothness of the denominator, and the Kurdyka--Łojasiewicz (KL) property of the objective. The proposed algorithms are applied to the sparse generalized eigenvalue problem associated with a pair of symmetric positive semidefinite matrices, and the corresponding convergence results are obtained according to their general convergence theorems. We perform some preliminary numerical experiments to demonstrate the efficiency of the proposed algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we consider a class of structured fractional minimization problems where the numerator part of the objective is the sum of a convex function and a Lipschitz differentiable (possibly) nonconvex function, while the denominator part is a convex function. By exploiting the structure of the problem, we propose a first-order algorithm, namely, a proximal-gradient-subgradient algorithm with backtracked extrapolation (PGSA_BE) for solving this type of optimization problem. It is worth pointing out that there are a few differences between our backtracked extrapolation and other popular extrapolations used in convex and nonconvex optimization. One of such differences is as follows: if the new iterate obtained from the extrapolated iteration satisfies a backtracking condition, then this new iterate will be replaced by the one generated from the non-extrapolated iteration. We show that any accumulation point of the sequence generated by PGSA_BE is a critical point of the problem regarded. In addition, by assuming that some auxiliary functions satisfy the Kurdyka-Łojasiewicz property, we are able to establish global convergence of the entire sequence, in the case where the denominator is locally Lipschitz differentiable, or its conjugate satisfies the calmness condition. Finally, we present some preliminary numerical results to illustrate the efficiency of PGSA_BE. [ABSTRACT FROM AUTHOR]

This paper proposes and investigates a novel near-optimal finite-control-set model predictive control (NOP-MPC) algorithm to control the grid-connected, pulsewidth-modulator-driven voltage source converters with LCL filters. Exploiting the convex and elliptical paraboloid properties of the cost error, NOP-MPC adopts a systematic iterative algorithm within each control cycle to progressively synthesize finite sets of virtual voltage vectors (VVs) for the control optimization stage. The synthesis has the inherent features of respecting the converter voltage limits and converging the sets of VV candidates toward the global optimal point. The fixed-switching-frequency feature of NOP-MPC is expected to ease the LCL filter design. Effects of computational delay, pulsewidth modulation delay, and deadtime are considered and compensated successfully. A two-vector-variable cost function is used to actively damp the inherent LC resonance through an adjustable, weighting-factor-based damping level. This paper is substantiated by theoretical consideration, simulation and experimental results, parameter sensitivity study, and a comparative study with the standard finite-control-set model predictive control that uses only actual VVs. [ABSTRACT FROM AUTHOR]

First-order optimization algorithms play a major role in large scale machine learning. A new class of methods, called adaptive algorithms, were recently introduced to adjust iteratively the learning rate for each coordinate. Despite great practical success in deep learning, their behavior and performance on more general loss functions are not well understood. In this paper, we derive a non-autonomous system of differential equations, which is the continuous time limit of adaptive optimization methods. We study the convergence of its trajectories and give conditions under which the differential system, underlying all adaptive algorithms, is suitable for optimization. We discuss convergence to a critical point in the non-convex case and give conditions for the dynamics to avoid saddle points and local maxima. For convex loss function, we introduce a suitable Lyapunov functional which allows us to study its rate of convergence. Several other properties of both the continuous and discrete systems are brie y discussed. The differential system studied in the paper is general enough to encompass many other classical algorithms (such as Heavy Ball and Nesterov's accelerated method) and allow us to recover several known results for these algorithms. [ABSTRACT FROM AUTHOR]

In this paper we consider non-smooth convex optimization problems with (possibly) infinite intersection of constraints. In contrast to the classical approach, where the constraints are usually represented as intersection of simple sets, which are easy to project onto, in this paper we consider that each constraint set is given as the level set of a convex but not necessarily differentiable function. For these settings we propose subgradient iterative algorithms with random minibatch feasibility updates. At each iteration, our algorithms take a subgradient step aimed at only minimizing the objective function and then a subsequent subgradient step minimizing the feasibility violation of the observed minibatch of constraints. The feasibility updates are performed based on either parallel or sequential random observations of several constraint components. We analyze the convergence behavior of the proposed algorithms for the case when the objective function is strongly convex and with bounded subgradients, while the functional constraints are endowed with a bounded first-order black-box oracle. For a diminishing stepsize, we prove sublinear convergence rates for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected suboptimality of the function values along the weighted averages. Our convergence rates are known to be optimal for subgradient methods on this class of problems. Moreover, the rates depend explicitly on the minibatch size and show when minibatching helps a subgradient scheme with random feasibility updates. [ABSTRACT FROM AUTHOR]

The paper considers scheduling jobs online to minimize the objective ∑ i ∈ [ n ] w i g (C i - r i) , where w i is the weight of job i, r i is its release time, C i is its completion time and g is any non-decreasing convex function. It is known that the clairvoyant algorithm Highest-Density-First (HDF) is (2 + ϵ) -speed O(1)-competitive for this objective on a single machine for any fixed 0 < ϵ < 1 (Im et al., in: ACM-SIAM symposium on discrete algorithms, pp 1254–1265, 2012). In this paper, we give the first non-trivial results for this problem when g is a non-decreasing convex function and the algorithm must be non-clairvoyant. More specifically, our results include: A (2 + ϵ) -speed O(1)-competitive non-clairovyant algorithm on a single machine for all non-decreasing convex g, matching the performance of HDF for any fixed 0 < ϵ < 1 . A (3 + ϵ) -speed O(1)-competitive non-clairovyant algorithm on multiple identical machines for all non-decreasing convex g for any fixed 0 < ϵ < 1 . The paper gives the first non-trivial upper-bound on multiple machines even if the algorithm is allowed to be clairvoyant. All performance guarantees above hold for all non-decreasing convex functions gsimultaneously. The positive results are supplemented by almost matching lower bounds. We show that any algorithm that is oblivious to g is not O(1)-competitive with speed augmentation less than 2 on a single machine. Further, any non-clairvoyent algorithm that knows the function g cannot be O(1)-competitive with speed augmentation less than 2 on a single machine or (2 - 1 m) on m identical machines. [ABSTRACT FROM AUTHOR]

Abstract This paper suggests a novel continuous-time gradient descent algorithm of a constrained convex unknown function with a stochastic noise in the observed data. The penalty function approach is employed to introduce the restrictions of the system and to provide a successful optimization process. The solution is restricted to a static scheme dealing with the class of strongly convex functions subject to a set of constraints. To estimate the stochastic gradient we employ a modified version of the synchronous detection method. All the parameters of the proposed approach are decreasing in time to both compensate noise effect in the observations and to provide the mean-square convergence of the suggested approach. We present two different numerical examples to validate the contributions of the paper. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high level, the method is an inexact proximal-point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochastic projected subgradient method. The primary contribution of this paper is a simple proof that the proposed algorithm converges at the same rate as the stochastic gradient method for smooth nonconvex problems. This result appears to be the first convergence rate analysis of a stochastic (or even deterministic) subgradient method for the class of weakly convex functions. In addition, a two-phase variant is proposed that significantly reduces the variance of the solutions returned by the algorithm. Finally, preliminary numerical experiments are also provided. [ABSTRACT FROM AUTHOR]

In this paper we present efficient algorithms to compute the radius, diameter, incenter, circumcenter, width and k dimensional enclosing cylinder for convex polyhedral and convex polyhedral offset distance functions in plane and in ℜ d for several types of inputs. We get optimal algorithms when the time complexities are linear. Let the size of input (convex polyhedron P /set S of points/convex polyhedra) be n / N , respectively, size of distance function convex polygon/polyhedron Q be m and size of constraint convex polyhedral region R be r in plane or ℜ d. The radius, incenter and circumcenter of the convex polygon P in ℜ 2 can be optimally computed in linear time, i.e. O (n + m) and in time O (n + m + r) , if the incenter or circumcenter is additionally constrained in a convex polygon/polyhedron R. In ℜ d , the radius, incenter and circumcenter of a convex polyhedron as well as of a set of convex polyhedra, unconstrained or constrained, can be computed in O (N m) and O (N m + r) time respectively, for convex constraint polyhedral region R. We also compute the minimum stabbing sphere for a set of convex polyhedra, unconstrained or constrained, for the above mentioned distance functions in O (N m) -time or O (N m + r) -time respectively for convex constraint polyhedral region R in ℜ d. The diameter of a convex polygon in plane can be computed in O (n + m) time and the diameter of a convex polyhedron in ℜ d can be computed in O (n m) time. The width of a convex polyhedron can be computed in O (n + m) time and O (n 2 m) time, for ℜ 2 and ℜ d , respectively. The diameter and width of a set of convex polyhedra can be computed in O (N m) -time and (N 2 m) , respectively, in any dimensions. We also show how the k dimensional minimum enclosing/stabbing cylinder for the convex polyhedron P can be computed in O (n d − k + 2 m 2) in ℜ d and in O (n d m (n + m)) in ℜ d for k = 1. The k dimensional minimum enclosing/stabbing cylinder for the set of convex polyhedra S in ℜ d can be computed in O (N d − k + 2 m 2) -time. The diameter and width problems can also be solved for a set of points in ℜ d , either in time O (n m) or in time O (m T (n)) , where T (n) is the time complexity of the best convex hull computation algorithm for the given set of points. [ABSTRACT FROM AUTHOR]

This paper proposes a two-level distributed algorithmic framework for solving the AC optimal power flow (OPF) problem with convergence guarantees. The presence of highly nonconvex constraints in OPF poses significant challenges to distributed algorithms based on the alternating direction method of multipliers (ADMM). In particular, convergence is not provably guaranteed for nonconvex network optimization problems like AC OPF. In order to overcome this difficulty, we propose a new distributed reformulation for AC OPF and a two-level ADMM algorithm that goes beyond the standard framework of ADMM. We establish the global convergence and iteration complexity of the proposed algorithm under mild assumptions. Extensive numerical experiments over some largest test cases from NESTA and PGLib-OPF (up to 30 000-bus systems) demonstrate advantages of the proposed algorithm over existing ADMM variants in terms of convergence, scalability, and robustness. Moreover, under appropriate parallel implementation, the proposed algorithm exhibits fast convergence comparable to or even better than the state-of-the-art centralized solver. [ABSTRACT FROM AUTHOR]

This paper proposes a nonconvex model (called LogTVSCAD) for deblurring images with impulsive noises, using the log-function penalty as the regularizer and adopting the smoothly clipped absolute deviation (SCAD) function as the data-fitting term. The proposed nonconvex model can effectively overcome the poor performance of the classical TVL1 model for high-level impulsive noise. A difference of convex functions algorithm (DCA) is proposed to solve the nonconvex model. For the model subproblem, we consider the alternating direction method of multipliers (ADMM) algorithm to solve it. The global convergence is discussed based on Kurdyka–Lojasiewicz. Experimental results show the advantages of the proposed nonconvex model over existing models. [ABSTRACT FROM AUTHOR]

Tensor principal component pursuit (TPCP) is a powerful approach in the tensor robust principal component analysis (TRPCA), where the goal is to decompose a data tensor to a low-tubal-rank part plus a sparse residual. TPCP is shown to be effective under certain tensor incoherence conditions, which can be restrictive in practice. In this paper, we propose a Modified-TPCP, which incorporates the prior subspace information in the analysis. With the aid of prior info, the proposed method is able to recover the low-tubal-rank and the sparse components under a significantly weaker incoherence assumption. We further design an efficient algorithm to implement Modified-TPCP based upon the alternating direction method of multipliers (ADMM). The promising performance of the proposed method is supported by simulations and real data applications. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a new accelerated iterative method for finding a common fixed point of a countable family of nonexpansive mappings in the Hilbert spaces framework. Using our main result, we obtain a new accelerated image restoration iterative method for solving a minimization problem in the form of the sum of two proper lower semi-continuous and convex functions. As applications, we apply our algorithm to solving image restoration problems. [ABSTRACT FROM AUTHOR]

The purpose of this paper is to invent an accelerated algorithm for the convex minimization problem which can be applied to the image restoration problem. Theoretically, we first introduce an algorithm based on viscosity approximation method with the inertial technique for finding a common fixed point of a countable family of nonexpansive operators. Under some suitable assumptions, a strong convergence theorem of the proposed algorithm is established. Subsequently, we utilize our proposed algorithm to solving a convex minimization problem of the sum of two convex functions. As an application, we apply and analyze our algorithm to image restoration problems. Moreover, we compare convergence behavior and efficiency of our algorithm with other well-known methods such as the forward-backward splitting algorithm and the fast iterative shrinkage-thresholding algorithm. By using image quality metrics, numerical experiments show that our algorithm has a higher efficiency than the mentioned algorithms. [ABSTRACT FROM AUTHOR]

With the proliferation of distributed generators and energy storage systems, traditional passive consumers in power systems have been gradually evolving into the so-called “prosumers,” i.e., proactive consumers, which can both produce and consume power. To encourage energy exchange among prosumers, energy sharing is increasingly adopted, which is usually formulated as a generalized Nash game (GNG). In this paper, a distributed approach is proposed to seek the Generalized Nash equilibrium (GNE) of the energy sharing game. To this end, we first prove the strong monotonicity of the game. Then, the GNG is converted into an equivalent optimization problem. An algorithm based on Nesterov's methods is thereby devised to solve the equivalent problem and consequently find the GNE in a distributed manner. The convergence of the proposed algorithm is proved rigorously based on the nonexpansive operator theory. The performance of the algorithm is validated by experiments with three prosumers, and the scalability is tested by simulations using 1888 prosumers. [ABSTRACT FROM AUTHOR]

In this paper we discuss the simple bilevel programming problem (SBP) and its extension, the simple mathematical programming problem under equilibrium constraints (SMPEC). Here we first define both these problems and study their interrelations. Next we study the various types of necessary and sufficient optimality conditions for the (SMPEC) problems, which occur under various reformulations. The optimality conditions for (SBP) are special cases of the results obtained for (SMPEC) when the lower level objective is the gradient of a convex function. Among the various optimality conditions presented in this article are the sequential optimality conditions, which do not need any constraint qualification. We also present a schematic algorithm for (SMPEC), where the sequential optimality conditions play a key role in the convergence analysis. [ABSTRACT FROM AUTHOR]

In linear regression, outliers have a serious effect on the estimation of regression model parameters and the prediction of final results, so outlier detection is one of the key steps in data analysis. In this paper, we use a mean shift model and then we apply the penalty function to penalize the mean shift parameters, which is conducive to get a sparse parameter vector. We choose Sorted L1 regularization (SLOPE), which provides a convex loss function, and shows good statistical properties in parameter selection. We apply an iterative process which using gradient descent method and parameter selection at each step. Our algorithm has higher computational efficiency since the calculation of inverse matrix is avoided. Finally, we use Cross-Validation rules (CV) and Bayesian Information Criterion (BIC) criteria to fine tune the parameters, which helps our program identify outliers and obtain more robust regression coefficients. Compared with other methods, the experimental results show that our program has a fantastic performance in all aspects of outlier detection. [ABSTRACT FROM AUTHOR]

The alternating direction method of multipliers (ADMM) is an efficient method for solving separable problems. However, ADMM may not converge when there is a nonconvex function in the objective. The main contributions of this paper are proposing and analysing an inertial proximal ADMM for a class of nonconvex optimization problems. The proposed algorithm combines the basic ideas of the proximal ADMM and the inertial proximal point method. The global and strong convergence of the proposed algorithm is analysed under mild conditions. Finally, we give some preliminary numerical results to show the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]