Search

The linear conjugate gradient method is an efficient iterative method for the convex quadratic minimization problems $ \mathop {\min }\limits_{x \in { \mathbb R^n}} f(x) =\dfrac{1}{2}x^TAx+b^Tx $, where $ A \in R^{n \times n} $ is symmetric and positive definite and $ b \in R^n $. It is generally agreed that the gradients $ g_k $ are not conjugate with respective to $ A $ in the linear conjugate gradient method (see page 111 in Numerical optimization (2nd, Springer, 2006) by Nocedal and Wright). In the paper we prove the conjugacy of the gradients $ g_k $ generated by the linear conjugate gradient method, namely, $$g_k^TAg_i=0, \; i=0,1,\cdots, k-2.$$ In addition,a new way is exploited to derive the linear conjugate gradient method based on the conjugacy of the search directions and the orthogonality of the gradients, rather than the conjugacy of the search directions and the exact stepsize.

Subspace minimization conjugate gradient (SMCG) methods have become a class of quite efficient iterative methods for unconstrained optimization and have attracted extensive attention recently. Usually, the search directions of SMCG methods are generated by minimizing approximate models with the approximation matrix $ B_k $ of the objective function at the current iterate over the subspace spanned by the current gradient $ g_k $ and the latest search direction. The $ g_k^TB_kg_k $ must be estimated properly in the calculation of the search directions, which is crucial to the theoretical properties and the numerical performance of SMCG methods. It is a great challenge to estimate it properly. The projection technique has been used successfully to generate conjugate gradient directions such as Dai-Kou conjugate gradient direction. Motivated by the above two observations, in the paper we present a new subspace minimization conjugate gradient methods by using a projection technique based on the memoryless quasi-Newton method. More specially, we project the search direction of the memoryless quasi-Newton method into the subspace spanned by the current gradient and the latest search direction and drive a new search direction, which is proved to be descent. Remarkably, the proposed method without any line search enjoys the finite termination property for two dimensional convex quadratic functions, which is helpful for designing algorithm. An adaptive scaling factor in the search direction is given based on the above finite termination property. The proposed method does not need to determine the parameter $ \rho_k $ and can be regarded as an extension of Dai-Kou conjugate gradient method. The global convergence of the proposed method is analyzed. Numerical comparisons indicate the proposed method is very promising.

In this paper, based on the limited memory techniques and subspace minimization conjugate gradient (SMCG) methods, a regularized limited memory subspace minimization conjugate gradient method is proposed, which contains two types of iterations. In SMCG iteration, we obtain the search direction by minimizing the approximate quadratic model or approximate regularization model. In RQN iteration, combined with regularization technique and BFGS method, a modified regularized quasi-Newton method is used in the subspace to improve the orthogonality. Moreover, some simple acceleration criteria and an improved tactic for selecting the initial stepsize to enhance the efficiency of the algorithm are designed. Additionally, an generalized nonmonotone line search is utilized and the global convergence of our proposed algorithm is established under mild conditions. Finally, numerical results show that, the proposed algorithm has a significant improvement over ASMCG_PR and is superior to the particularly well-known limited memory conjugate gradient software packages CG_DESCENT (6.8) and CGOPT(2.0) for the CUTEr library.

The "Inertial Forward-Backward algorithm" (IFB) is a powerful tool for convex nonsmooth minimization problems, it gives the well known "fast iterative shrinkage-thresholding algorithm " (FISTA), which enjoys $O\left( {\frac{1}{{{k^2}}}} \right)$ global convergence rate of function values, however, no convergence of iterates has been proved; by do a small modification, an accelerated IFB called "FISTA\_CD" improves the convergence rate of function values to $o\left( {\frac{1}{{{k^2}}}} \right)$ and shows the weak convergence of iterates. The local error bound condition is extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems, and in practical application, a large number of problems with special structure often satisfy the error bound condition. Naturally, using local error bound condition to derive or improve the convergence rate of IFB is a common means. In this paper, based on the local error bound condition, we exploit an new assumption condition for the important parameter $t_k$ in IFB, and establish the improved convergence rate of function values and strong convergence of the iterates generated by the IFB algorithms with six $t_k$ satisfying the above assumption condition in Hilbert space. It is remarkable that, under the local error bound condition, we establish the strong convergence of the iterates generated by the original FISTA, and prove that the convergence rates of function values for FISTA\_CD is actually related to the value of parameter $a,$ and show that the IFB algorithms with some $t_k$ mentioned above can achieve sublinear convergence rate $o\left( {\frac{1}{{{k^p}}}} \right)$ for any positive integer $p>1$. Some numerical experiments are conducted to illustrate our results.

In this paper, two new subspace minimization conjugate gradient methods based on $p - $regularization models are proposed, where a special scaled norm in $p - $regularization model is analyzed. Different choices for special scaled norm lead to different solutions to the $p - $regularized subproblem. Based on the analyses of the solutions in a two-dimensional subspace, we derive new directions satisfying the sufficient descent condition. With a modified nonmonotone line search, we establish the global convergence of the proposed methods under mild assumptions. $R - $linear convergence of the proposed methods are also analyzed. Numerical results show that, for the CUTEr library, the proposed methods are superior to four conjugate gradient methods, which were proposed by Hager and Zhang (SIAM J Optim 16(1):170-192, 2005), Dai and Kou (SIAM J Optim 23(1):296-320, 2013), Liu and Liu (J Optim Theory Appl 180(3):879-906, 2019) and Li et al. (Comput Appl Math 38(1): 2019), respectively.

A new type of stepsize, which was recently introduced by Liu and Liu (Optimization, 67(3), 427-440, 2018), is called approximately optimal stepsize and is quit efficient for gradient method. Interestingly, all gradient methods can be regarded as gradient methods with approximately optimal stepsizes. In this paper, based on the work (Numer. Algorithms 78(1), 21-39, 2018), we present an improved gradient method with approximately optimal stepsize based on conic model for unconstrained optimization. If the objective function $ f $ is not close to a quadratic on the line segment between the current and latest iterates, we construct a conic model to generate approximately optimal stepsize for gradient method if the conic model can be used; otherwise, we construct some quadratic models to generate approximately optimal stepsizes for gradient method. The convergence of the proposed method is analyzed under suitable conditions. Numerical comparisons with some well-known conjugate gradient software packages such as CG$ \_ $DESCENT (SIAM J. Optim. 16(1), 170-192, 2005) and CGOPT (SIAM J. Optim. 23(1), 296-320, 2013) indicate the proposed method is very promising., Comment: arXiv admin note: text overlap with arXiv:1907.01794

It is widely accepted that the stepsize is of great significance to gradient method. Two efficient gradient methods with approximately optimal stepsizes mainly based on regularization models are proposed for unconstrained optimization. More exactly, if the objective function is not close to a quadratic function on the line segment between the current and latest iterates, regularization models are exploited carefully to generate approximately optimal stepsizes. Otherwise, quadratic approximation models are used. In particular, when the curvature is non-positive, special regularization models are developed. The convergence of the proposed methods is established under the weak conditions. Extensive numerical experiments indicated the proposed method is superior to the BBQ method (SIAM J. Optim. 2021,31(4), 3068-3096) and other efficient gradient methods, and is competitive to two famous and efficient conjugate gradient software packages CG$ \_ $DESCENT (5.0) (SIAM J. Optim. 16(1), 170-192, 2005) and CGOPT (1.0) (SIAM J. Optim. 23(1), 296-320, 2013) for the CUTEr library. Due to the surprising efficiency, we believe that gradient methods with approximately optimal stepsizes can become strong candidates for large-scale unconstrained optimization.