Search

Let $({\bf P},{\bf D})$ be a primal-dual pair of SDPs with a nonzero finite duality gap. Under such circumstances, ${\bf P}$ and ${\bf D}$ are weakly feasible and if we perturb the problem data to recover strong feasibility, the (common) optimal value function $v$ as a function of the perturbation is not well-defined at zero (unperturbed data) since there are ``two different optimal values'' $v({\bf P})$ and $v({\bf D})$, where $v({\bf P})$ and $v({\bf D})$ are the optimal values of ${\bf P}$ and ${\bf D}$ respectively. Thus, continuity of $v$ is lost at zero though $v$ is continuous elsewhere. Nevertheless, we show that a limiting version ${v_a}$ of $v$ is a well-defined monotone decreasing continuous bijective function connecting $v({\bf P})$ and $v({\bf D})$ with domain $[0, \pi/2]$ under the assumption that both ${\bf P}$ and ${\bf D}$ have singularity degree one. The domain $[0,\pi/2]$ corresponds to directions of perturbation defined in a certain manner. Thus, ${v_a}$ ``completely fills'' the nonzero duality gap under a mild regularity condition. Our result is tight in that there exists an instance with singularity degree two for which ${v_a}$ is not continuous., Comment: 26 pages. Comments welcome

We study properties of the central path underlying a nonlinear semidefinite optimization problem, called NSDP for short. The latest radical work on this topic was contributed by Yamashita and Yabe (2012): they proved that the Jacobian of a certain equation-system derived from the Karush-Kuhn-Tucker (KKT) conditions of the NSDP is nonsingular at a KKT point under the second-order sufficient condition (SOSC), the strict complementarity condition (SC), and the nondegeneracy condition (NC). This yields uniqueness and existence of the central path through the implicit function theorem. In this paper, we consider the following three assumptions on a KKT point: the strong SOSC, the SC, and the Mangasarian-Fromovitz constraint qualification. Under the absence of the NC, the Lagrange multiplier set is not necessarily a singleton and the nonsingularity of the above-mentioned Jacobian is no longer valid. Nonetheless, we establish that the central path exists uniquely, and moreover prove that the dual component of the path converges to the so-called analytic center of the Lagrange multiplier set. As another notable result, we clarify a region around the central path where Newton's equations relevant to primal-dual interior point methods are uniquely solvable.

We extend the Levenberg-Marquardt method on Euclidean spaces to Riemannian manifolds. Although a Riemannian Levenberg-Marquardt (RLM) method was produced by Peeters in 1993, to the best of our knowledge, there has been no analysis of theoretical guarantees for global and local convergence properties. As with the Euclidean LM method, how to update a specific parameter known as the damping parameter has significant effects on its performances. We propose a trust-region-like approach for determining the parameter. We evaluate the worst-case iteration complexity to reach an epsilon-stationary point, and also prove that it has desirable local convergence properties under the local error-bound condition. Finally, we demonstrate the efficiency of our proposed algorithm by numerical experiments.

Minimizing the sum of a convex function and a composite function appears in various fields. The generalized Levenberg--Marquardt (LM) method, also known as the prox-linear method, has been developed for such optimization problems. The method iteratively solves strongly convex subproblems with a damping term. This study proposes a new generalized LM method for solving the problem with a smooth composite function. The method enjoys three theoretical guarantees: iteration complexity bound, oracle complexity bound, and local convergence under a H\"olderian growth condition. The local convergence results include local quadratic convergence under the quadratic growth condition; this is the first to extend the classical result for least-squares problems to a general smooth composite function. In addition, this is the first LM method with both an oracle complexity bound and local quadratic convergence under standard assumptions. These results are achieved by carefully controlling the damping parameter and solving the subproblems by the accelerated proximal gradient method equipped with a particular termination condition. Experimental results show that the proposed method performs well in practice for several instances, including classification with a neural network and nonnegative matrix factorization.

We consider an identification method for a linear continuous time-invariant autonomous system from noisy state observations. In particular, we focus on the identification to satisfy the asymptotic stability of the system with some prior knowledge. To this end, we propose to model this identification problem as a Riemannian nonlinear optimization (RNLO) problem, where the stability is ensured through a certain Riemannian manifold and the prior knowledge is expressed as nonlinear constraints defined on this manifold. To solve this RNLO, we apply the Riemannian sequential quadratic optimization (RSQO) that was proposed by Obara, Okuno, and Takeda (2022) most recently. RSQO performs quite well with theoretical guarantee to find a point satisfying the Karush-Kuhn-Tucker conditions of RNLO. In this paper, we demonstrate that the identification problem can be indeed solved by RSQO more effectively than competing algorithms., Comment: 8 pages, 4 figures

Strongly motivated from use in various fields including machine learning, the methodology of sparse optimization has been developed intensively so far. Especially, the recent advance of algorithms for solving problems with nonsmooth regularizers is remarkable. However, those algorithms suppose that weight parameters of regularizers, called hyperparameters hereafter, are pre-fixed, and it is a crucial matter how the best hyperparameter should be selected. In this paper, we focus on the hyperparameter selection of regularizers related to the $\ell_p$ function with $0

We propose a primal-dual interior-point method (IPM) with convergence to second-order stationary points (SOSPs) of nonlinear semidefinite optimization problems, abbreviated as NSDPs. As far as we know, the current algorithms for NSDPs only ensure convergence to first-order stationary points such as Karush-Kuhn-Tucker points, but without a worst-case iteration complexity. The proposed method generates a sequence approximating SOSPs while minimizing a primal-dual merit function for NSDPs by using scaled gradient directions and directions of negative curvature. Under some assumptions, the generated sequence accumulates at an SOSP with a worst-case iteration complexity. This result is also obtained for a primal IPM with a slight modification. Finally, our numerical experiments show the benefits of using directions of negative curvature in the proposed method., Comment: 42 pages, 1 figure

We consider optimization problems on Riemannian manifolds with equality and inequality constraints, which we call Riemannian nonlinear optimization (RNLO) problems. Although they have numerous applications, the existing studies on them are limited especially in terms of algorithms. In this paper, we propose Riemannian sequential quadratic optimization (RSQO) that uses a line-search technique with an ell_1 penalty function as an extension of the standard SQO algorithm for constrained nonlinear optimization problems in Euclidean spaces to Riemannian manifolds. We prove its global convergence to a Karush-Kuhn-Tucker point of the RNLO problem by means of parallel transport and the exponential mapping. Furthermore, we establish its local quadratic convergence by analyzing the relationship between sequences generated by RSQO and the Riemannian Newton method. Ours is the first algorithm that has both global and local convergence properties for constrained nonlinear optimization on Riemannian manifolds. Empirical results show that RSQO finds solutions more stably and with higher accuracy compared with the existing Riemannian penalty and augmented Lagrangian methods., Comment: 36 pages, 2 figure

The recent advance of algorithms for nonlinear semi-definite optimization problems, called NSDPs, is remarkable. Yamashita et al. first proposed a primal-dual interior point method (PDIPM) for solving NSDPs using the family of Monteiro-Zhang (MZ) search directions. Since then, various kinds of PDIPMs have been proposed for NSDPs, but, as far as we know, all of them are based on the MZ family. In this paper, we present a PDIPM equipped with the family of Monteiro-Tsuchiya (MT) directions, which were originally devised for solving linear semi-definite optimization problems as were the MZ family. We further prove local superlinear convergence to a Karush-Kuhn-Tucker point of the NSDP in the presence of certain general assumptions on scaling matrices, which are used in producing the MT scaling directions.

A new Levenberg--Marquardt (LM) method for solving nonlinear least squares problems with convex constraints is described. Various versions of the LM method have been proposed, their main differences being in the choice of a damping parameter. In this paper, we propose a new rule for updating the parameter so as to achieve both global and local convergence even under the presence of a convex constraint set. The key to our results is a new perspective of the LM method from majorization-minimization methods. Specifically, we show that if the damping parameter is set in a specific way, the objective function of the standard subproblem in LM methods becomes an upper bound on the original objective function under certain standard assumptions. Our method solves a sequence of the subproblems approximately using an (accelerated) projected gradient method. It finds an $\epsilon$-stationary point after $O(\epsilon^{-2})$ computation and achieves local quadratic convergence for zero-residual problems under a local error bound condition. Numerical results on compressed sensing and matrix factorization show that our method converges faster in many cases than existing methods.

We consider primal-dual pairs of semidefinite programs and assume that they are ill-posed, i.e., both primal and dual are either weakly feasible or weakly infeasible. Under such circumstances, strong duality may break down and the primal and dual might have a nonzero duality gap. Nevertheless, there are arbitrary small perturbations to the problem data which makes the perturbed primal-dual pair strongly feasible thus zeroing the duality gap. In this paper, we conduct an asymptotic analysis of the optimal value as the perturbation is driven to zero. Specifically, we fix two positive definite matrices (typically the identity matrices), and shift the associated affine spaces of the primal and dual slightly in the direction of the two positive definite matrices possibly in a different proportion so that the perturbed problems have interior feasible solutions, and analyze the behavior of the optimal value of the perturbed problem when the perturbation is reduced to zero keeping the proportion. A key feature of our analysis is that no further assumptions such as compactness or constraint qualifications are ever made. It will be shown that the optimal value of the perturbed problem converges to a value between the primal and dual optimal values of the original problem. Finally, the analysis leads us to the relatively surprising consequence that the infeasible interior-point algorithms for SDP generates a sequence converging to a number between the primal and dual optimal values, even in the presence of a nonzero duality gap. We expect that this property might be particularly useful in solving mixed integer SDPs with infeasible interior-point methods., Comment: This is a thoroughly revised version to improve readability. A detailed analysis on interior-point algorithms is added, though the main results are unchanged

In this paper, we propose a new sequential quadratic semidefinite programming (SQSDP) method for solving degenerate nonlinear semidefinite programs (NSDPs), in which we produce iteration points by solving a sequence of stabilized quadratic semidefinite programming (QSDP) subproblems, which we derive from the minimax problem associated with the NSDP. Unlike the existing SQSDP methods, the proposed one allows us to solve those QSDP subproblems inexactly, and each QSDP is feasible. One more remarkable point of the proposed method is that constraint qualifications (CQs) or boundedness of Lagrange multiplier sequences are not required in the global convergence analysis. Specifically, without assuming such conditions, we prove the global convergence to a point satisfying any of the following: the stationary conditions for the feasibility problem, the approximate-Karush-Kuhn-Tucker (AKKT) conditions, and the trace-AKKT conditions. Finally, we conduct some numerical experiments to examine the efficiency of the proposed method.

The LP-Newton method solves the linear programming problem (LP) by repeatedly projecting a current point onto a certain relevant polytope. In this paper, we extend the algorithmic framework of the LP-Newton method to the second-order cone programming problem (SOCP) via a linear semi-infinite programming (LSIP) reformulation of the given SOCP. In the extension, we produce a sequence by projection onto polyhedral cones constructed from LPs obtained by finitely relaxing the LSIP. We show the global convergence property of the proposed algorithm under mild assumptions, and investigate its efficiency through numerical experiments comparing the proposed approach with the primal-dual interior-point method for the SOCP.

In this paper, we propose two algorithms for nonlinear semi-infinite semi-definite programs with infinitely many convex inequality constraints, called SISDP for short. A straightforward approach to the SISDP is to use classical methods for semi-infinite programs such as discretization and exchange methods and solve a sequence of (nonlinear) semi-definite programs (SDPs). However, it is often too demanding to find exact solutions of SDPs. Our first approach does not rely on solving SDPs but on approximately following {a path leading to a solution, which is formed on the intersection of the semi-infinite region and the interior of the semi-definite region. We show weak* convergence of this method to a Karush-Kuhn-Tucker point of the SISDP under some mild assumptions and further provide with sufficient conditions for strong convergence. Moreover, as the second method, to achieve fast local convergence, we integrate a two-step sequential quadratic programming method equipped with Monteiro-Zhang scaling technique into the first method. We particularly prove two-step superlinear convergence of the second method using Alizadeh-Hareberly-Overton-like, Nesterov-Todd, and Helmberg-Rendle-Vanderbei-Wolkowicz/Kojima-Shindoh-Hara/Monteiro scaling directions. Finally, we conduct some numerical experiments to demonstrate the efficiency of the proposed method through comparison with a discretization method that solves SDPs obtained by finite relaxation of the SISDP.

We point out that Chubanov's oracle-based algorithm for linear programming [5] can be applied almost as it is to linear semi-infinite programming (LSIP). In this note, we describe the details and prove the polynomial complexity of the algorithm based on the real computation model proposed by Blum, Shub and Smale (the BSS model) which is more suitable for floating point computation in modern computers. The adoption of the BBS model makes our description and analysis much simpler than the original one by Chubanov [5]. Then we reformulate semidefinite programming (SDP) and second-order cone programming (SOCP) into LSIP, and apply our algorithm to obtain new complexity results for computing interior feasible solutions of homogeneous SDP and SOCP., Comment: 17 pages

In this paper, we consider a nonlinear semi-infinite program that minimizes a function including a log-determinant (logdet) function over positive definite matrix constraints and infinitely many convex inequality constraints, called SIPLOG for short. The main purpose of the paper is to develop an algorithm for computing a Karush-Kuhn-Tucker (KKT) point for the SIPLOG efficiently. More specifically, we propose an interior point sequential quadratic programming-type method that inexactly solves a sequence of semi-infinite quadratic programs approximating the SIPLOG. Furthermore, to generate a search direction in the dual matrix space associated with the semi-definite constraint, we solve scaled Newton equations {that yield} the family of Monteiro-Zhang directions. We prove that the proposed method weakly* converges to a KKT point under some mild assumptions. Finally, we conduct some numerical experiments to demonstrate the efficiency of the proposed method.

We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0

In this paper, we consider a class of mixed integer programming problems (MIPs) whose objective functions are DC functions, that is, functions representable in terms of the difference of two convex functions. These MIPs contain a very wide class of computationally difficult nonconvex MIPs since the DC functions have powerful expressability. Recently, Maehara, Marumo, and Murota provided a continuous reformulation without integrality gaps for discrete DC programs having only integral variables. They also presented a new algorithm to solve the reformulated problem. Our aim is to extend their results to MIPs and give two specific algorithms to solve them. First, we propose an algorithm based on DCA originally proposed by Pham Dinh and Le Thi, where convex MIPs are solved iteratively. Next, to handle nonsmooth functions efficiently, we incorporate a smoothing technique into the first method. We show that sequences generated by the two methods converge to stationary points under some mild assumptions.