Search

A modified Polak Ribiere Polyak(PRP) conjugate gradient(CG) method is proposed for solving unconstrained optimization problems. The search direction generated by this method satisfies sufficient descent condition at each iteration and this method inherits one remarkable property of the standard PRP method. Under the standard Armijo line search, the global convergence and the linearly convergent rate of the presented method is established. Some numerical results are given to show the effectiveness of the proposed method by comparing with some existing CG methods. [ABSTRACT FROM AUTHOR]

The paper considers the convergence of the complex block Jacobi diagonalization methods under the large set of the generalized serial pivot strategies. The global convergence of the block methods for Hermitian, normal and J -Hermitian matrices is proven. In order to obtain the convergence results for the block methods that solve other eigenvalue problems, such as the generalized eigenvalue problem, we consider the convergence of a general block iterative process which uses the complex block Jacobi annihilators and operators. [ABSTRACT FROM AUTHOR]

We present a novel numerical approach for solving nonlinear constrained optimal control problems (NCOCPs). Instead of directly solving the NCOCPs, we start by linearizing the constraints and dynamic system, which results in a sequence of sub-problems. For each sub-problem, we use finite number of Chebyshev polynomials to estimate the control and state vectors. To eliminate the errors at non-collocation points caused by conventional collocation methods, we additionally estimate the coefficient functions involved in the linear constraints and dynamic system by Chebyshev polynomials. By leveraging the characteristics of Chebyshev polynomials, the approximate sub-problem is changed into an equivalent nonlinear optimization problem with linear equality constraints. Consequently, any feasible point of the approximate sub-problem will satisfy the constraints and dynamic system throughout the entire time scale. To validate the efficacy of the new method, we solve three examples and assess the accuracy of the method through the computation of its approximation error. Numerical results obtained show that our approach achieves lower approximation error when compared to the Chebyshev pseudo-spectral method. The proposed method is particularly suitable for scenarios that require high-precision approximation, such as aerospace and precision instrument production. • A sequence of sub-problems is constructed by linearizing the dynamic system and constraints. • Each sub-problem can be transformed into an equivalent nonlinear optimization problem with linear equality constraints through Chebyshev polynomials. • Any feasible solution of the approximate sub-problem will satisfy its dynamic system and constraints over the entire time horizon. • Global convergence of the proposed method is established. [ABSTRACT FROM AUTHOR]

Due to their simplicity and global convergence properties, the conjugate gradient (CG) methods are widely used for solving unconstrained optimization problems, especially those of large scale. To establish the global convergence and to obtain better numerical performance in practice, much effort has been devoted to develop new CG methods or even to modify well- known methods. In 2012, Rivaie et al., have proposed a new CG method, called RMIL which has good numerical results and globally convergent under the exact line search. However, in 2016, Dai has pointed out a mistake in the steps of the proof of global convergence of RMIL and hence to guarantee the global convergence he suggested a modified version of RMIL, called RMIL+. In this paper, we present another modified version of RMIL, which is globally convergent via the exact line search. Furthermore, to support the theoretical proof of the global convergence of the modified version in practical computation, a numerical experiment based on comparing it with RMIL, RMIL+, and CG-DESCENT was done. [ABSTRACT FROM AUTHOR]

From Newton's method, we construct a Newton-type iterative method that allows studying a class of nonlinear Hammerstein-type integral equations. This method is reduced to Newton's method if the kernel of the integral equation is separable and, unlike Newton's method, can be applied to approximate a solution if the kernel is nonseparable. In addition, from an analysis of the global convergence of the method, we can locate and separate solutions of the nonlinear Hammerstein-type integral equations involved. For this study of the global convergence, we use auxiliary functions and obtain restricted global convergence domains that are usually balls. [ABSTRACT FROM AUTHOR]

Based on the convex combination technique, we propose a projection-based hybrid conjugate gradient algorithm for solving nonlinear equations with convex constraints in this paper. The conjugate parameter of the proposed algorithm is a convex combination of the modified Polak-Ribière-Polyak and Dai-Yuan type conjugate parameters, and the search direction has the sufficient descent property without the use of a line search strategy. The proposed hybrid algorithm's global convergence is established under appropriate assumptions. The numerical experiments demonstrate that the proposed algorithm is more efficient and competitive than existing methods under some benchmark test problems. Furthermore, it is also extended to solve the sparse signal and impulse noise image restoration problem that arises in compressive sensing. [ABSTRACT FROM AUTHOR]

In this paper, a truncated three-term conjugate gradient method is proposed for nonconvex unconstrained optimization. At each iteration, the search direction generated by this method is sufficiently descent independent of any line search. We investigate its global convergence with Wolfe line search under the standard assumptions. Moreover, we present its complexity analysis under the objective function with the Lipschitz continuously gradient and Armijo line search which implies that the proposed method is also globally convergent with Armijo line search. A remarkable point about its complexity is that the proposed method possesses the same complexity order as the classical gradient descent method without any restart strategy. Finally, the proposed method is applied to solve the nonconvex robust regression problems. Numerical results show that the proposed method is efficient. [ABSTRACT FROM AUTHOR]

• We propose a new plug-and-play deep neural network method to remove Rician noise. • We prove some mathematical properties and the convergence of the proposed method. • Experimental results show that the proposed method outperforms existing methods. Restoring images corrupted by Rician noise is a challenging issue in medical image processing. In the existing methods, the model-driven method can not recover the images well, and the learning-based methods lack good interpretability. In this paper, we propose a plug-and-play (PnP) method to remove Rician noise. Due to the statistical properties of Rician distribution and the implicit deep image priors, the problem is non-convex. We present a convergent PnP method to address these issues by an adaptively relaxed alternating direction method of multipliers. Theoretically, we give some useful mathematical properties and the global linear convergence of the proposed method by an adaptive relaxation strategy. Experimental results show that the proposed method outperforms the existing state-of-art traditional and learning-based methods. [ABSTRACT FROM AUTHOR]

In this paper, an adaptive three-term conjugate gradient method is proposed based on the Perry family by using a modified secant condition. The new method depends on a positive parameter which has a critical role in the efficiency of the algorithm. We propose an adaptive choice for this parameter to control the condition number of the direction matrix by minimizing the distance between the smallest and largest eigenvalue. Global convergence of the new algorithm is proved for general functions, under standard assumptions. Experimental results verify the expected numerical behavior of our algorithm. [ABSTRACT FROM AUTHOR]

The inertial proximal method is extended to minimize the sum of a series of separable nonconvex and possibly nonsmooth objective functions and a smooth nonseparable function (possibly nonconvex). Here, we propose two new algorithms. The first one is an inertial proximal coordinate subgradient algorithm, which updates the variables by employing the proximal subgradients of each separable function at the current point. The second one is an inertial proximal block coordinate method, which updates the variables by using the subgradients of the separable functions at the partially updated points. Global convergence is guaranteed under the Kurdyka–Łojasiewicz (KŁ) property and some additional mild assumptions. Convergence rate is derived based on the Łojasiewicz exponent. Two numerical examples are given to illustrate the effectiveness of the algorithms. [ABSTRACT FROM AUTHOR]

Recently, a global non-smooth optimization method was developed for efficiently eliminating specular highlights from single color image. Although this method was reported to be effective on both natural and medical images, its algorithm's speed is slow due to the non-smoothness of the objective function. This paper proposes a global smooth optimization method, where two gradient regularization terms are used to describe diffuse and specular component features for texture details. A simplified optimality condition is established and thus a new iteration algorithm for specular highlight removal is proposed. By employing the gradient regularization of diffuse reflection, image texture detail can be preserved during specular highlight removal. Due to the smoothness of the objective function, the proposed smooth optimization algorithm has lower computational complexity and less storage requirement than the global non-smooth optimization algorithm. Furthermore, the proposed iteration algorithm is guaranteed to converge to a global optimal solution under a fixed step length. Experimental results demonstrate that the proposed smooth optimization algorithm is much faster than the non-smooth optimization algorithm. Moreover, the proposed highlight removal algorithm is more effective on benchmark natural images and real laparoscopic images than conventional highlight removal algorithms. • A global smooth optimization method with preserving texture detail in highlight removal is proposed. • A matrix iteration algorithm with low computational complexity and less storage requirement is presented. • A simplified optimality condition and algorithm convergence are established. [ABSTRACT FROM AUTHOR]

ℓ 1 regularization has been used for logistic regression to circumvent the overfitting and use the estimated sparse coefficient for feature selection. However, the challenge of such regularization is that the ℓ 1 regularization is not differentiable, making the standard convex optimization algorithm not applicable to this problem. This paper presents a simple projection neural network for ℓ 1 -regularized logistics regression. In contrast to many available solvers in the literature, the proposed neural network does not require any extra auxiliary variable nor smooth approximation, and its complexity is almost identical to that of the gradient descent for logistic regression without ℓ 1 regularization, thanks to the projection operator. We also investigate the convergence of the proposed neural network by using the Lyapunov theory and show that it converges to a solution of the problem with any arbitrary initial value. The proposed neural solution significantly outperforms state-of-the-art methods concerning the execution time and is competitive in terms of accuracy and AUROC. [ABSTRACT FROM AUTHOR]

• Two ZNN-type models are proposed for TVQPPs with equality and inequality constraints, which removes convex condition of activation function. • An universal GNTZNNM-NAF framework is reconstructed from a control-based perspective for TVQPPs with equality and inequality constraints with different noises. • Global convergence and strong robustness of ZNN-type models are analyzed in detail. • Theoretical results demonstrate that the residual errors of ZNN-type models globally converge to zero even under different measurement noises. Time-varying quadratic programming problems (TVQPPs) with equality and inequality constraints often arise in the fields of scientific computation and engineering application. Zeroing neural network (ZNN), being a special kind of recurrent neural network, has shown powerful capabilities to compute a variety of zeroing finding problems with monotonically increasing odd activation function. However, the projection sets of nonconvex activation function are obviously excluded, which means a general conclusion remain unexplored. In addition, noises are always ubiquitous in actual applications. Nevertheless, most existing ZNN-based models usually assume that the solving process is free of noise before the calculation. In this paper, a general zeroing neural network model with nonconvex activated function (GZNNM-NAF) and a general noise-tolerant zeroing neural network model with nonconvex activated function (GNTZNNM-NAF), which are also viewed as ZNN-type models, are developed by the inspiration of the traditional ZNN model from a control-based perspective. The ZNN-type models break the limitation of the traditional ZNN models of activation function, which allows nonconvex sets for projection operations and combines nonlinear complementary function for dealing with inequality constraints arising in TVQPPs. Moreover, theoretical results indicate that the ZNN-type models globally converge to time-varying optimal solution of TVQPPs with equality and inequality constraints under the noise circumstance. According to the different cases of nonconvex activation function, robustness analyses are demonstrated in detail for TVQPPs. It may enlarge the scope of the proposed method, especially in the field of practical application. Finally, a numerical example and an application example to manipulator motion generation are analyzed to verify the superiority and robustness of the developed ZNN-type models for TVQPPs with different measurement noises. [ABSTRACT FROM AUTHOR]

This work proposes a matrix-free algorithm incorporating structured spectral parameters with a nonmonotone line search for solving nonlinear least squares problems. The structured Hessian matrix associated with the Newton's-like direction is approximated with a scalar multiple of identity. The scalar is a convex combination of the spectral parameters such that the structured quasi-Newton condition is approximately satisfied. Using a nonmonotone line search with some appropriate conditions, the global convergence of the sequence generated by the proposed algorithm is presented. The numerical result shows that the algorithm is competitive with the existing algorithms in the literature. [ABSTRACT FROM AUTHOR]

In this paper, we propose an ascent algorithm by exploiting the Riemannian gradient, which is to solve the optimization problems with orthogonality constraints. It applies to orthogonal approximation problems of symmetric tensors. The iteration possesses some nice properties and has the same expression as the Polar decomposition. We establish that the global convergence of our algorithm. Preliminary results show that the computational advantage of our algorithm. [ABSTRACT FROM AUTHOR]

This paper develops two neurodynamic approaches for solving the L 1 -minimization problem with the linear inequality constraints. First, a centralized neurodynamic approach is proposed based on projection operator and nonnegative quadrant. The stability and global convergence of the centralized neurodynamic approach are analyzed by the Lyapunov method in detail. Considering that the distributed optimization problem has the advantages of information protection and scalability, the L 1 -minimization problem with linear inequality constraints is transformed into a distributed sparse optimization problem under mild conditions. Then, using the centralized neurodynamic approach and multi-agent consensus theory, a distributed neurodynamic approach is proposed for the distributed optimization problem. Furthermore, relevant theories show that each agent globally converges to an optimal solution of the distributed optimization problem. Finally, the presented centralized neurodynamic approach is applied to sparse recovery problem with L ∞ -norm noise constraints and the effectiveness of distributed approach is shown by several experiments on sparse signal recovery. [ABSTRACT FROM AUTHOR]

This paper presents a sparse optimization method for the simultaneous orthogonal tensor diagonalization. The model treats off-diagonal elements of tensors as entities requiring sparsity, guided by an ℓ 1 norm regularizer to optimize the diagonalization process. A gradient-based alternating multi-block Jacobi-AMB algorithm is developed to address the optimization problem on the product of orthogonal groups. We establish the global convergence based on the Kurdyka-Łojasiewicz property. Numerical experiments demonstrate that the Jacobi-AMB performs well in efficiency; under certain circumstances, its stability and effectiveness also perform well. • An ℓ 1 norm-induced sparse model is introduced for tensor diagonalization. • A gradient-based alternating multi-block Jacobi-AMB algorithm is developed. • Sufficient descent inequality is established and global convergence is proved. • Numerical experiments demonstrate the algorithm's efficiency and stability. [ABSTRACT FROM AUTHOR]

This paper presents a new modified conjugate gradient (NMCG) algorithm which satisfies the sufficient descent property under any line search for unconstrained optimization problems. We analyze that the algorithm is global convergence under the Wolfe line search. We use the proposed algorithm NMCG to unconstrained optimization problems to prove its effectiveness. Furthermore, we also extend it to solve image restoration and sparse signal recovery problems in compressive sensing, and the results indicate that our algorithm is effective and competitive. [ABSTRACT FROM AUTHOR]

Accurate wind power prediction is highly significant to the safety, stability, and economic operation of power grids. Currently, typical ensemble methods for wind power forecasting are widely designed based on the mean square error (MSE) loss, which are very suitable for the assumption that the error distribution obeys the Gaussian distribution. However, the complex nonlinear conversion of wind energy into wind power may change the statistical characteristics of errors, thus the prediction model based on the traditional MSE loss may lead to the performance degradation of the forecasting model. To address these problems, a probability density function based adaptive ensemble learning with global convergence is proposed for wind power prediction, which comprises three modules: a data preprocessing module, a prediction module, and a combination module. Firstly, an effective feature generation mechanism is employed to extract the multi-mode characteristics of wind data. Then, an auxiliary error based adaptive global convergence model is developed as benchmark predictor in the prediction module, where an adaptive updating algorithm is derived based on the Lyapunov approach to ensure the global convergence of the model weights. Moreover, considering the asymmetric characteristic of modeling error, a probability density function (PDF) based ensemble learning is created to integrate the results of benchmark predictors. Specifically, the ensemble model parameter updating is transformed into the shape control for the modeling error PDF, which can break through the limitation of MSE loss capturing only the second moment information, and emphasize the spatial distribution of errors to make an unbiased estimate of wind power. Experimental results show that the proposed ensemble model has significant advantages over other models involved in this study. • A diversity feature generation mechanism is introduced for extraction of multi-modes. • An auxiliary error based adaptive model is developed as benchmark predictor. • A probability density function based adaptive ensemble learning is created. • A rigorous convergence analysis of the learning algorithm is implemented. [ABSTRACT FROM AUTHOR]

This paper proposes two innovative vector transport operators, leveraging the Cayley transform, for the generalized Stiefel manifold embedded with a non-standard metric. Specifically, it introduces the differentiated retraction and an approximation of the Cayley transform to the differentiated matrix exponential. These vector transports are demonstrated to satisfy the Ring–Wirth non-expansive condition under non-standard metrics, and one of them is also isometric. Building upon the novel vector transport operators, we extend the modified Polak–Ribière–Polyak (PRP) conjugate gradient method to the generalized Stiefel manifold. Under a non-monotone line search condition, we prove our algorithm globally converges to a stationary point. The efficiency of the proposed vector transport operators is empirically validated through numerical experiments involving generalized eigenvalue problems and canonical correlation analysis. [ABSTRACT FROM AUTHOR]

In this paper, we focus on the two-block nonconvex and nonsmooth optimization with linear constraints, where the objective function is the sum of a convex but nonsmooth function and a smooth but nonconvex function. This problem encompasses many important applications in engineering and machine learning. Based on the splitting algorithm, making use of the quadratic approximation of the smooth part, and with the help of Armijo line search technique, we first propose a splitting augmented Lagrangian method with partial quadratic approximation. Under some mild conditions, the global convergence of the proposed method is proved. Moreover, when the gradient of the smooth part in the objective function is Lipschitz continuous, we then propose and analyze a variant of the aforementioned method without performing line search. We report some preliminary numerical results on solving nonconvex quadratic regularization problems to show the feasibility and effectiveness of the two proposed methods. Finally, we show applicability and encouraging efficiency of our methods by applying them to solve sparse signal recovery problems. [ABSTRACT FROM AUTHOR]

In this paper, we propose an efficient trust region algorithm for unconstrained optimization problem. Different from the traditional trust region algorithm, our algorithm obtains the next iteration point by projection after getting the trust region trial step. The main feature of this algorithm is that it generates a bounded iteration sequence automatically, and it retains the excellent convergence properties. The numerical tests for support vector machine(SVM) and some unconstrained optimization test sets problems show the efficiency of the proposed algorithm. [ABSTRACT FROM AUTHOR]

• Biaxial bending strength capacity assessment of composite sections exposed to fire. • A global convergent strain-driven iterative search methodology has been developed. • A computationally effective adaptive plastic centroid (APC) has been introduced. • Non-convexity of nominal moment-capacity contours associated with strain-softening. • A new 3D α N -isogonic vertical interaction diagrams are identified. This paper introduces an advanced computational method for assessing the biaxial bending capacities of arbitrary-shaped reinforced concrete and steel–concrete composite cross-sections under fire conditions. The proposed approach involves a strain-driven iterative method coupled with an adaptive plastic centroid providing a "fail-safe" methodology by combining the bisection and damped Newton methods to improve its global convergence properties. Several key computational issues are addressed: (1) strength assessment criteria and its impact on computational outcomes, (2) the pathological behavior of local convergent iterative methods causing divergence or spurious solutions in stress-resultant space, (3) the softening behavior of concrete in compression affecting solution uniqueness and interaction diagram convexity, (4) non-planar vertical interaction diagrams induced by a mobile centroid, and (5) computational challenges related to solution non-uniqueness or non-existence in M-M stress resultant space when axial force falls outside the iso -load contour. An additional notable feature and novelty of the proposed method lie in its unique capability to assess true plane vertical interaction diagrams enabling also both ultimate and nominal strength assessment. Validation includes comparisons with other numerical results and experimental data from international literature, extending the benchmark results for the strength capacity assessment of composite cross-sections exposed to high temperatures. [ABSTRACT FROM AUTHOR]

The Broyden-type algorithm is one of the most effective quasi-Newton methods for solving unconstrained optimization problems. However, the study of the global convergence of the Broyden-type algorithm is not enough. Motivated by Andrei (J. Comput. Appl. Math. 332: 26–44, 2018), we present a modified two-parameter scaled Broyden-type algorithm. Some approaches are used in the designed method: (1) we introduce a modified Broyden-type formula with two parameters; (2) we use approximated parameter settings to reduce the effect of eigenvalues of sequence { B k } ; (3) we propose a modified weak Wolfe–Powell line search technique with projection technique to avoid using failed search direction. The parameters of our study have the same functionality, that of correcting the structure of the eigenvalues of the approximate Hessian matrix. The difference is in the approach of 4-term Broyden updates instead of BFGS. Numerical performance shows that the modified Broyden-type algorithm is competitive with the classical Broyden approach. [ABSTRACT FROM AUTHOR]

In this article, a hybrid approach technique incorporated with three-term conjugate gradient (CG) method is proposed to solve constrained nonlinear monotone operator equations. The search direction is defined such that it is close to the one obtained by the memoryless Broyden-Fletcher-Goldferb-Shanno (BFGS) method. Independent of the line search, the search direction possesses the sufficient descent and trust region properties. Furthermore, the sequence of iterates generated converge globally under some appropriate assumptions. In addition, numerical experiments are carried out to test the efficiency of the proposed method in contrast with existing methods. Finally, the applicability of the proposed method in compressive sensing is shown. [ABSTRACT FROM AUTHOR]

Nonlinear conjugate gradient methods are often used to solve large-scale unconstrained optimization problems owing to their lower memory requirement and less computation cost. In this paper, we propose a new Polak-Ribiére-Polyak type conjugate gradient method, which satisfies the sufficient descent condition independent of any line search. A remarkable property about this method is its strong global convergence with the standard Wolfe line search conditions as well as the standard Armijo line search strategy without convexity assumption of the objective function. The numerical results demonstrating the efficiency of the proposed method are reported. [ABSTRACT FROM AUTHOR]

The time-varying complex quadratic programming problem plays an important role in scientific research and engineering applications, and has received extensive attention. In this paper, by solving the problem in different number domains, two types of novel anti-noise integral enhanced recurrent neural networks (IERNNs) are proposed and analyzed. Specifically, the IERNN-A model solves the problems in the complex number domain, while the IERNN-B model solves the problems by converting the problems from the complex number domain into the real number domain. Both the IERNN-A and IERNN-B have the same design paradigm, i.e., an integral-type error function is first defined to obtain good anti-noise capability, and then substitute it into a differential-driven design formula to ensure convergence. Theory analysis and simulation results verify the global convergence and robustness of the proposed IERNNs. Furthermore, if a linear activation is used, the exponential convergence rate is obtained. Meanwhile, when time-varying bounded noise exists, the IERNNs are input-to-state stable. In addition, compared with other art-of-the-state methods, the proposed methods have faster convergence speed, stronger noise-suppression ability, and no overshoot, which are verified by the simulation and application of manipulator trajectory tracking. [ABSTRACT FROM AUTHOR]

Conjugate gradient (CG) methods are efficient algorithms for unconstrained optimization. An inertial step strategy is usually used to accelerate iterative methods. In 1995, Yuan and Stoer (1995) introduced a subspace study on CG algorithms and presented some subspace minimization CG (SMCG) methods. Motivated by these approaches, we incorporate a new inertial step strategy in SMCG methods and present an efficient inertial SMCG method for solving constrained nonlinear monotone equations by combining with the projection technique. The proposed inertial step strategy makes use of information from recent iterations. It can be considered as an extension of the known inertial step strategy. The resulting direction always satisfies the sufficient descent property and that of trust region. Indeed, these properties do not depend on line search rules. The global convergence and Q-linear convergence rate of the proposed method are established under mild assumptions, without any Lipschitz continuity. Numerical results indicate that the proposed method is very promising. [ABSTRACT FROM AUTHOR]