Search

Research on low‐light image enhancement focuses on improving image quality in dim conditions. Recently, deep learning has driven significant advancements, with many studies using neural networks to enhance low‐light images. However, most focus on complex network designs to increase nonlinearity, often neglecting the implicit information from local image transformations. This paper introduces an improved U‐net‐based method for low‐light enhancement, retaining the original encoding network and adding branch links in the decoding network. The method uses attention feature fusion to handle image noise and gradients separately, adjusting brightness through a gradient‐adaptive transform. This approach optimizes performance using loss functions like peak signal‐to‐noise ratio and colour consistency. Unlike previous methods, the approach emphasizes extracting implicit information from image gradients, achieving enhancement that aligns with the original brightness distribution. The result is enhanced images with high detail similarity to the original, achieved through end‐to‐end inference in experiments. [ABSTRACT FROM AUTHOR]

Implementation of reconfigurable metasurfaces demands a complex structure. The design methodology for metasurface elements based on PIN diodes requires targeted optimisation tailored to the varying application environments and cannot be directly transplanted from one methodological approach to another. In addition, inclusion of varactors often imposes undesirable high‐power consumption with complex analog drive circuitry. Furthermore, the design method for the reconfigurable metasurface element has not been simplified and unified, hindering the rapid deployment of metasurface applications. The authors propose a general design model based on low‐power embedded circuitry for a multi‐function metasurface element and give the corresponding numerical analysis. The authors present a reconfigurable method based on the general design model and prove that the method achieves independent regulation of amplitude and phase for the reflected electromagnetic wave. Through analysis of the sensitive area at the bottom of the embedded circuit, a gradient descent algorithm is proposed to find the optimised phase points according to device sensitivity of the application scenario and suitable strategy. Finally, the authors demonstrate that the designed metasurface element possesses phase control capability and decent phase consistency obtained via far‐field scanning experiments. The proposed element design method will provide a theoretical reference for the integrated design of metasurface. [ABSTRACT FROM AUTHOR]

The development of innovative materials, based on the modern technologies and processes, is the key factor to improve the energetic sustainability and reduce the environmental impact of electrical equipment. In particular, the modeling of magnetic hysteresis is crucial for the design and construction of electrical and electronic devices. In recent years, additive manufacturing techniques are playing a decisive role in the project and production of magnetic elements and circuits for applications in various engineering fields. To this aim, the use of the deep learning paradigm, integrated with the most common models of the magnetic hysteresis process, has become increasingly present in recent years. The intent of this paper is to provide the features of a wide range of deep learning tools to be applied to magnetic hysteresis context and beyond. The possibilities of building neural networks in hybrid form are innumerable, so it is not plausible to illustrate them in a single paper, but in the present context, several neural networks used in the scientific literature, integrated with various hysteretic mathematical models, including the well-known Preisach model, are compared. It is shown that this hybrid approach not only improves the modeling of hysteresis by significantly reducing computational time and efforts, but also offers new perspectives for the analysis and prediction of the behavior of magnetic materials, with significant implications for the production of advanced devices. [ABSTRACT FROM AUTHOR]

The geometric optimization of crystal structures is a procedure widely used in computational chemistry that changes the geometrical placement of the particles inside a structure. It is called structural relaxation and constitutes a local minimization problem with a non‐convex objective function whose domain complexity increases according to the number of particles involved. This work studies the performance of the two most popular gradient methods in structural relaxation, Steepest Descent and Conjugate Gradient. Although frequently employed, there is a lack of their study in this context from an algorithmic point of view. The algorithms are initially benchmarked on the basis of a constant step size. Three concepts for designing dynamic step size rules are then examined in detail and analyzed. Results show that there is a trade‐off between convergence rate and the possibility of an experiment to succeed. In order to address this, a function is proposed as a formal means for assigning utility to each method based on preference. The function is built according to a recently introduced model of preference indication concerning algorithms with deadline and their run time. It introduces the quantification of the optimization algorithms' performance according to convergence speed and success rate, thus enabling the appointment of a specific algorithmic recipe as the best choice for balanced preferences. [ABSTRACT FROM AUTHOR]

Crystal structures are important formations, bases of materials used in everyday life. A lot of effort is dedicated to the creation of new materials, starting from the study and combination of crystal structures' properties[1] through the crystals' geometric optimization. In the meantime, an equal amount of effort is dedicated to the study and development of methods employed in optimization problems; this, nevertheless, is carried out independently, in the context of computer science. There is a need, therefore, to acknowledge and highlight the overlap between the two fields, whilst creating a common language and establishing the theoretical foundations on which the optimization methods can become physically interpreted in the case of crystals. This article provides a detailed introduction to the theory behind the geometric optimization of crystals from the point of view of computer science, so as to facilitate interdisciplinary cooperation. Its target is to familiarize scientists from different fields with the problem, its parameters, and their derivations, in order to motivate their assistance in the creation, implementation, and application of new methods for the geometric optimization of crystal structures. [ABSTRACT FROM AUTHOR]

We derive efficient algorithms to compute weakly Pareto optimal solutions for smooth, convex and unconstrained multiobjective optimization problems in general Hilbert spaces. To this end, we define a novel inertial gradient-like dynamical system in the multiobjective setting, which trajectories converge weakly to Pareto optimal solutions. Discretization of this system yields an inertial multiobjective algorithm which generates sequences that converge weakly to Pareto optimal solutions. We employ Nesterov acceleration to define an algorithm with an improved convergence rate compared to the plain multiobjective steepest descent method (Algorithm 1). A further improvement in terms of efficiency is achieved by avoiding the solution of a quadratic subproblem to compute a common step direction for all objective functions, which is usually required in first-order methods. Using a different discretization of our inertial gradient-like dynamical system, we obtain an accelerated multiobjective gradient method that does not require the solution of a subproblem in each step (Algorithm 2). While this algorithm does not converge in general, it yields good results on test problems while being faster than standard steepest descent. [ABSTRACT FROM AUTHOR]

In this paper, a method of stroke‐based rendering is proposed for image representation and reconstruction. The proposed method involves compositing a set of ellipses that greatly vary in location, size, rotation angle, color, and opacity. This study proves that alpha compositing is differentiable if a two‐dimensional Gaussian function is used to draw a solid ellipse. The gradient method can then be employed for automatically identifying the parameters of each ellipse such that the difference between the input image and the composited image is minimized. Experimental results indicate that the proposed method can represent various types of images including portraits, landscapes, buildings, street scenes, artificial objects, medical images etc. The proposed method can particularly render the most details and significant features of an input image with fewer strokes compared to other stroke‐based rendering algorithms. This study also demonstrates that the proposed method can be applied in painting style simulation and sparse‐view computed tomography imaging. [ABSTRACT FROM AUTHOR]

The transmit waveform with ambiguity function specifications in the range‐Doppler plane has been of great interest in traditional single antenna radar systems in recent years. This paper considers the design of waveform sets with a desired slow‐time ambiguity function for multiple‐input multiple‐output radar. Specifically, it is desirable that the ambiguity function has low sidelobes where clutter occurs, as this reduces the impact of the clutter on target detection. Instead of unimodular waveforms, the authors consider amplitude modulation signals and introduce the bound constraint on signal amplitudes into the optimisation for the consideration of transmitter efficiency. The design problem is formulated as minimising the weighted integrated sidelobes level of the ambiguity function over the range‐Doppler plane under bound constraints. The resulting problem has a highly non‐linear objective function and is constrained by bound constraints and quadratic equality (constant energy) constraints, making it difficult to solve. The authors develop an efficient algorithm based on the sequential quadratic programming technique and non‐linear conjugate gradient method to solve it. Simulation results show that the proposed algorithm is superior to the existing methods. [ABSTRACT FROM AUTHOR]

An algorithm has been developed for numerically solving the source determination problem in the model of information dissemination in synthetic online social networks, described by reaction–diffusion-type equations, using additional information about the process at fixed time points. The degree of ill-posedness of the source determination problem for a parabolic equation is studied based on the analysis of singular values of the linearized operator of the inverse problem. The algorithm developed is based on a combination of the tensor optimization method and gradient descent supplemented with the A.N. Tikhonov regularization. Numerical calculations demonstrate the smallest relative error in the reconstructed source obtained by the developed algorithm in comparison with classical approaches. [ABSTRACT FROM AUTHOR]

The approximating sequence Riccati equation method is an efficient approach for solving the nonlinear optimal control problems, but its neglect of nonlinear dynamics and necessary optimality condition makes the control law difficult to satisfy the optimality. In this paper, an approximating pseudospectral method with state‐dependent coefficient optimization algorithm is proposed to solve this defect. By introducing the approximating pseudospectral method, the original nonlinear problem is transformed into a sequence of linear subproblems, which preserves the nonlinearity of solution. Then a state‐dependent coefficient optimization algorithm based on the gradient projection technique is proposed, which ensures the optimality of the control law. A double‐layer optimization structure is designed to facilitate the coordination between the approximating method and the optimization algorithm. Theoretical analysis proves the convergence of the proposed method. Comparative case studies illustrate the effectiveness in reducing the performance index and ensuring the optimality of the control law. [ABSTRACT FROM AUTHOR]

For an optimal control problem with constraints on phase coordinates, an iterative method for finding a numerical solution is considered, based on reduction to a finite-dimensional problem and applying to it a sequential linearization algorithm using a modified Lagrange function. To solve linear auxiliary problems at iterations of the method, the reduced gradient method is used. The efficiency of taking into account phase constraints in the calculation of optimal control is illustrated by the numerical solution of problems from the field of aerodynamics and robotics. [ABSTRACT FROM AUTHOR]

We consider distributed optimization over several devices, each sending incremental model updates to a central server. This setting is considered, for instance, in federated learning. Various schemes have been designed to compress the model updates in order to reduce the overall communication cost. However, existing methods suffer from a significant slowdown due to additional variance ω > 0 coming from the compression operator and as a result, only converge sublinearly. What is needed is a variance reduction technique for taming the variance introduced by compression. We propose the first methods that achieve linear convergence for arbitrary compression operators. For strongly convex functions with condition number κ, distributed among n machines with a finite-sum structure, each worker having less than m components, we also (i) give analysis for the weakly convex and the non-convex cases and (ii) verify in experiments that our novel variance reduced schemes are more efficient than the baselines. Moreover, we show theoretically that as the number of devices increases, higher compression levels are possible without this affecting the overall number of communications in comparison with methods that do not perform any compression. This leads to a significant reduction in communication cost. Our general analysis allows to pick the most suitable compression for each problem, finding the right balance between additional variance and communication savings. Finally, we also (iii) give analysis for arbitrary quantized updates. [ABSTRACT FROM AUTHOR]

We study the Unbalanced Optimal Transport (UOT) between two measures of possibly different masses with at most n components, where the marginal constraints of standard Optimal Transport (OT) are relaxed via Kullback-Leibler divergence with regularization factor τ. Although only Sinkhorn-based UOT solvers have been analyzed in the literature with the iteration complexity of O(τ log(n) / ε lof (log(n)/ε)) and per-iteration cost of O(n²) for achieving the desired error ε, their positively dense output transportation plans strongly hinder the practicality. On the other hand, while being vastly used as heuristics for computing UOT in modern deep learning applications and having shown success in sparse OT problem, gradient methods applied to UOT have not been formally studied. In this paper, we propose a novel algorithm based on Gradient Extrapolation Method (GEM-UOT) to find an ε-approximate solution to the UOT problem in O(κ log (τ n/ε)) iterations with O(n²) per-iteration cost, where κ is the condition number depending on only the two input measures. Our proof technique is based on a novel dual formulation of the squared ℓ2-norm UOT objective, which fills the lack of sparse UOT literature and also leads to a new characterization of approximation error between UOT and OT. To this end, we further present a novel approach of OT retrieval from UOT, which is based on GEM-UOT with fine tuned τ and a post-process projection step. Extensive experiments on synthetic and real datasets validate our theories and demonstrate the favorable performance of our methods in practice. We showcase GEM-UOT on the task of color transfer in terms of both the quality of the transfer image and the sparsity of the transportation plan. [ABSTRACT FROM AUTHOR]

The primary aim of this paper is to provide a review of the last few decades of research focused on delayed gradient methods for solving symmetric positive definite linear systems. Recent developments on spectral properties are stressed, as well as their implications for the formulation of new methods. Numerical experiments are conducted with emphasis on the comparison with the conjugate gradient method for solving ill-conditioned problems. [ABSTRACT FROM AUTHOR]

Consider the following distributed optimization scenario. A worker has access to training data that it uses to compute the gradients while a server decides when to stop iterative computation based on its target accuracy or delay constraints. The server receives all its information about the problem instance from the worker via a rate-limited noiseless communication channel. We introduce the principle we call differential quantization (DQ) that prescribes compensating the past quantization errors to direct the descent trajectory of a quantized algorithm towards that of its unquantized counterpart. Assuming that the objective function is smooth and strongly convex, we prove that differentially quantized gradient descent (DQ-GD) attains a linear contraction factor of $\max \{\sigma _{\mathrm {GD}}, \rho _{n} 2^{-R}\}$ , where $\sigma _{\mathrm {GD}}$ is the contraction factor of unquantized gradient descent (GD), $\rho _{n} \geq 1$ is the covering efficiency of the quantizer, and $R$ is the bitrate per problem dimension $n$. Thus at any $R\geq \log _{2} \rho _{n} /\sigma _{\mathrm {GD}}$ bits, the contraction factor of DQ-GD is the same as that of unquantized GD, i.e., there is no loss due to quantization. We show a converse demonstrating that no algorithm within a certain class can converge faster than $\max \{\sigma _{\mathrm {GD}}, 2^{-R}\}$. Since quantizers exist with $\rho _{n} \to 1$ as $n \to \infty $ (Rogers, 1963), this means that DQ-GD is asymptotically optimal. In contrast, naively quantized GD where the worker directly quantizes the gradient barely attains $\sigma _{\mathrm {GD}} + \rho _{n}2^{-R}$. The principle of differential quantization continues to apply to gradient methods with momentum such as Nesterov’s accelerated gradient descent, and Polyak’s heavy ball method. For these algorithms as well, if the rate is above a certain threshold, there is no loss in contraction factor obtained by the differentially quantized algorithm compared to its unquantized counterpart, and furthermore, the differentially quantized heavy ball method attains the optimal contraction achievable among all (even unquantized) gradient methods. Experimental results on least-squares problems validate our theoretical analysis. [ABSTRACT FROM AUTHOR]

We introduce a family of weighted conjugate-gradient-type methods, for strictly convex quadratic functions, whose parameters are determined by a minimization model based on a convex combination of the objective function and its gradient norm. This family includes the classical linear conjugate gradient method and the recently published delayed weighted gradient method as the extreme cases of the convex combination. The inner cases produce a merit function that offers a compromise between function-value reduction and stationarity which is convenient for real applications. We show that each one of the infinitely many members of the family exhibits q-linear convergence to the unique solution. Moreover, each one of them enjoys finite termination and an optimality property related to the combined merit function. In particular, we prove that if the n × n Hessian of the quadratic function has p < n different eigenvalues, then each member of the family obtains the unique global minimizer in exactly p iterations. Numerical results are presented that demonstrate that the proposed family is promising and exhibits a fast convergence behavior which motivates the use of preconditioning strategies, as well as its extension to the numerical solution of general unconstrained optimization problems. [ABSTRACT FROM AUTHOR]

This paper focuses on a time-varying constrained nonconvex optimization problem, and considers the synthesis and analysis of online regularized primal-dual gradient methods to track a Karush-Kuhn-Tucker (KKT) trajectory. The proposed regularized primal-dual gradient method is implemented in a running fashion, in the sense that the underlying optimization problem changes during the execution of the algorithms. In order to study its performance, we first derive its continuous-time limit as a system of differential inclusions. We then study sufficient conditions for tracking a KKT trajectory, and also derive asymptotic bounds for the tracking error (as a function of the time-variability of a KKT trajectory). Further, we provide a set of sufficient conditions for the KKT trajectories not to bifurcate or merge, and also investigate the optimal choice of the parameters of the algorithm. Illustrative numerical results for a time-varying nonconvex problem are provided. [ABSTRACT FROM AUTHOR]

In this paper, we study a fully composite formulation of convex optimization problems, which includes, as a particular case, the problems with functional constraints, max-type minimization problems, and problems with simple nondifferentiable components. We treat all these formulations in a unified way, highlighting the existence of very natural optimization schemes of different order p ≥ 1. As the result, we obtain new high-order (p ≥ 2) optimization methods for composite formulation. We prove the global convergence rates for them under the most general conditions. Assuming that the upper-level component of our objective function is subhomogeneous, we develop efficient modification of the basic fully composite first-order and second-order methods and propose their accelerated variants. [ABSTRACT FROM AUTHOR]

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

In this paper, we consider gradient methods for minimizing smooth convex functions, which employ the information obtained at the previous iterations in order to accelerate the convergence towards the optimal solution. This information is used in the form of a piece-wise linear model of the objective function, which provides us with much better prediction abilities as compared with the standard linear model. To the best of our knowledge, this approach was never really applied in Convex Minimization to differentiable functions in view of the high complexity of the corresponding auxiliary problems. However, we show that all necessary computations can be done very efficiently. Consequently, we get new optimization methods, which are better than the usual Gradient Methods both in the number of oracle calls and in the computational time. Our theoretical conclusions are confirmed by preliminary computational experiments. [ABSTRACT FROM AUTHOR]

In this paper, acceleration of gradient methods for convex optimization problems with weak levels of convexity and smoothness is considered. Starting from the universal fast gradient method which was designed to be an optimal method for weakly smooth problems whose gradients are Hölder continuous, its momentum is modified appropriately so that it can also accommodate uniformly convex and weakly smooth problems. Different from the existing works, fast gradient methods proposed in this paper do not use the restarting technique but use momentums that are suitably designed to reflect both the uniform convexity and weak smoothness information of the target energy function. Both theoretical and numerical results that support the superiority of the proposed methods are presented. [ABSTRACT FROM AUTHOR]

The paper presents a comparative analysis of the numerical solution of the problem of source identification in the diffusion-logistics model from the data on the diffusion process at fixed points in time and space by gradient methods in continuous and discrete formulations. Expressions are obtained for calculating the gradient of the objective functional for two formulations related to the solution of the corresponding adjoint problems. It is shown that, if the discrete functions of the model are approximated by cubic splines, the accuracy of the solutions of the source identification problem has the same order in the case of continuous and discrete calculation of the gradient. Numerical experiments in solving the source identification problem for a discrete model of information dissemination in online social networks have shown that the use of the discrete approach significantly increases the computational time in comparison with the continuous approach. [ABSTRACT FROM AUTHOR]

Optimizing non-convex functions is of primary importance in modern pattern recognition because it underlies the training of deep networks and nonlinear dimensionality reduction. First-order algorithms under suitable randomized perturbations or step-size rules have been shown to be effective for such settings as their limit points can be guaranteed to be local extrema rather than saddle points. However, it is well-known that the practical convergence of first-order methods is slower than those which exploit additional structure. In particular, empirically, successive convex approximation (SCA) converges faster than first-order methods. However, to date, SCA in general non-convex settings converges to first-order stationary points, which could either be local extrema or saddle points whose performance is typically inferior. To mitigate this issue, we propose calibrated randomized perturbations of SCA, which exhibit the improved convergence rate as compared to the gradient descent counter part. In particular, our main technical contributions are to establish the non-asymptotic performance of SCA algorithm and its perturbed variant converges to an approximate second-order stationary point. Experiments on multi-dimensional scaling, a machine learning problem whose training objective is non-convex, substantiate the performance gains associated with employing random perturbations. [ABSTRACT FROM AUTHOR]

It is challenging for machine as well as humans to detect the presence of emotions such as sadness or disgust in a sentence without adequate knowledge about the context. Contextual emotion detection is a challenging problem in natural language processing. As the use of digital agents have increased in text messaging applications, it is essential for these agents to provide sensible responses to its users. The present work demonstrates the effectiveness of Gaussian process detecting contextual emotions present in a sentence. The results obtained are compared with Decision Tree and ensemble models such as Random Forest, AdaBoost and Gradient Boost. Out of the five models built on a small dataset with class imbalance, it has been found that Gaussian Process classifier predicts emotions better than the other classifiers. Gaussian Process classifier performs better by taking predictive variance into account. [ABSTRACT FROM AUTHOR]

Stochastic compositional optimization generalizes classic (non-compositional) stochastic optimization to the minimization of compositions of functions. Each composition may introduce an additional expectation. The series of expectations may be nested. Stochastic compositional optimization is gaining popularity in applications such as reinforcement learning and meta learning. This paper presents a new Stochastically Corrected Stochastic Compositional gradient method (SCSC). SCSC runs in a single-time scale with a single loop, uses a fixed batch size, and guarantees to converge at the same rate as the stochastic gradient descent (SGD) method for non-compositional stochastic optimization. This is achieved by making a careful improvement to a popular stochastic compositional gradient method. It is easy to apply SGD-improvement techniques to accelerate SCSC. This helps SCSC achieve state-of-the-art performance for stochastic compositional optimization. In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional stochastic optimization. We test SCSC using the model-agnostic meta-learning tasks. [ABSTRACT FROM AUTHOR]

We focus on over-the-air (OTA) Federated Learning (FL), which has been suggested recently to reduce the communication overhead of FL due to the repeated transmissions of the model updates by a large number of users over the wireless channel. In OTA FL, all users simultaneously transmit their updates as analog signals over a multiple access channel, and the server receives a superposition of the analog transmitted signals. However, this approach results in the channel noise directly affecting the optimization procedure, which may degrade the accuracy of the trained model. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local stochastic gradient descent (SGD) FL algorithm, introducing precoding at the users and scaling at the server, which gradually mitigates the effect of noise. We analyze the convergence of COTAF to the loss minimizing model and quantify the effect of a statistically heterogeneous setup, i.e. when the training data of each user obeys a different distribution. Our analysis reveals the ability of COTAF to achieve a convergence rate similar to that achievable over error-free channels. Our simulations demonstrate the improved convergence of COTAF over vanilla OTA local SGD for training using non-synthetic datasets. Furthermore, we numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL. [ABSTRACT FROM AUTHOR]

A novel approach to de‐embed the frequency response of filters connected to a 2‐channel multiplexer without the need for unplugging is proposed. The technique to manifold coupled multiplexers with arbitrary number of filters is also theoretically generalised. In multiplexer tuning, there are practically important circumstances in which it is advantageous to de‐embed the frequency response of filters in multiplexers without detaching them. The authors show that using a novel technique and reasonable approximations, all filters of a multiplexer can be modelled by simple microwave networks, and then, the equivalent rational function of the channel filters can be estimated appropriately by the rational H2 approximation; extraction of this model facilitates tuning of the filters using the coupling matrix method. The impact of mechanical tolerances on the accuracy of the proposed technique is also studied. Next, this method is examined on a 2‐channel multiplexer and its application in de‐embedding and tuning of its connected filters is verified. Provided that the mechanical tolerances of the structure are sufficiently small, this technique is a powerful engineering solution towards tuning of manifold‐coupled multiplexers. [ABSTRACT FROM AUTHOR]

In this article, we present a reliable hybrid algorithm for solving convex quadratic minimization problems. At the kth iteration, two points are computed: first, an auxiliary point x˙k is generated by performing a gradient step using an optimal steplength, and second, the next iterate xk + 1 is obtained by means of weighted sum of x˙k with the penultimate iterate xk − 1. The coefficient of the linear combination is computed by minimizing the residual norm along the line determined by the previous points. In particular, we adopt an optimal, nondelayed steplength in the first step and then use a smoothing technique to impose a delay on the scheme. Under a modest assumption, we show that our algorithm is Q‐linearly convergent to the unique solution of the problem. Finally, we report numerical experiments on strictly convex quadratic problems, showing that the proposed method is competitive in terms of CPU time and iterations with the conjugate gradient method. [ABSTRACT FROM AUTHOR]

This article aims at proposing a multistage linear Gauss pseudospectral method (MS-LGPM) for solving the piecewise continuous nonlinear optimal control problem (OCP) with interior-point constraints and terminal constraints. First, the first-order necessary conditions for the multistage nonlinear OCP are derived, and a typical multipoint boundary value problem is obtained. Second, the original problem can be solved iteratively by integral prediction and quasi-linearization. Then Lagrange interpolation polynomials are used to approximate the state, control, and costate variables, to transfer those differential equations into a set of linear algebraic equations. Therefore, the control update coming closer to the optimal solution can be derived in an analytical manner. Additionally, those linear algebraic equations are formulated in regular banded matrix forms to make the code as concise as possible. Finally, the proposed method is applied to the midcourse guidance design for a dual-pulse missile to evaluate its performance. Simulation results show that the proposed method performs well in providing the optimal control with high accuracy and computational efficiency. Furthermore, a comparison with other typical guidance method and Monte Carlo simulations are also provided. Results demonstrate, even with some random uncertainties, the proposed method still has strong robustness and superior performance. [ABSTRACT FROM AUTHOR]

High-intensity focused ultrasound (HIFU) is a widely used technique capable of providing noninvasive heating and ablation for a wide range of applications. However, the major challenges lie in the determination of the position and the amount of heat deposition over a target area. In order to assure that the thermal area is confined to tumor locations, an optimization method should be employed. Sequential quadratic programming and steepest gradient method with closed-form solution have been previously used to solve this kind of problem. However, these methods are complex and computationally inefficient. The goal of this article is to solve and control the solution of inverse problems with partial differential equation (PDE) constraints. Therefore, a distinguishing challenge of this technique is the handling of large numbers of optimization variables in combination with the complexities of discretized PDEs. In our method, the objective function is formulated as the square difference between the actual thermal dose and the desired one. At each iteration of the optimization procedure, we need to develop and solve the variation problem, the adjoint problem, and the gradient of the objective function. The analytical formula for the gradient is derived and calculated based on the solution of the adjoint problem. Several factors have been taken into consideration to demonstrate the robustness and efficiency of the proposed algorithm. The simulation results for all cases indicate the robustness and the computational efficiency of our proposed method compared to the steepest gradient descent method with the closed-form solution. [ABSTRACT FROM AUTHOR]

In this article, we consider a convex optimization problem which minimizes the sum of local agents’ cost functions subject to certain local constraints. Besides, both the local cost function and local constraints are only known by the local agent itself. To solve this problem, a new accelerated distributed gradient-based algorithm is proposed, which is inspired by the “momentum” phenomena in nature and aims to accelerate the convergence speed of conventional distributed gradient algorithms. Sufficient conditions for the stepsizes and the acceleration gains are derived to ensure the convergence of the proposed algorithm. Furthermore, based on this proposed fast distributed algorithm, a new decentralized approach is proposed to solve economic dispatch problem, especially for a large-scale power system. Based on the idea of virtual agent, it is proved that this decentralized algorithm is equivalent to the original fast distributed gradient method. Several case studies implemented on IEEE 30-bus, IEEE 118-bus power systems, and a large-scale power system consisting of 1000 generators are conducted to validate the proposed method. [ABSTRACT FROM AUTHOR]

Optimal control problems with terminal conditions without control constraints, problems with a free right trajectory end and with control constraints, and optimization problems with parameters and parameter and control constraints are considered. For each of these classes of problems, multimethod algorithms involving numerical optimal control methods that are most efficient for the given class are designed. The performance of the proposed algorithms is demonstrated using numerical solutions of complicated real-world problems. [ABSTRACT FROM AUTHOR]

A line of recent works established that when training linear predictors over separable data, using gradient methods and exponentially-tailed losses, the predictors asymptotically converge in direction to the max-margin predictor. As a consequence, the predictors asymptotically do not overfit. However, this does not address the question of whether overfitting might occur non-asymptotically, after some bounded number of iterations. In this paper, we formally show that standard gradient methods (in particular, gradient flow, gradient descent and stochastic gradient descent) never overfit on separable data: If we run these methods for T iterations on a dataset of size m, both the empirical risk and the generalization error decrease at an essentially optimal rate of ~O(1=2T) up till T ß m, at which point the generalization error remains fixed at an essentially optimal level of ~O(1=2m) regardless of how large T is. Along the way, we present non-asymptotic bounds on the number of margin violations over the dataset, and prove their tightness. [ABSTRACT FROM AUTHOR]

This paper optimizes the step coefficients of first-order methods for smooth convex minimization in terms of the worst-case convergence bound (i.e., efficiency) of the decrease in the gradient norm. This work is based on the performance estimation problem approach. The worst-case gradient bound of the resulting method is optimal up to a constant for large-dimensional smooth convex minimization problems, under the initial bounded condition on the cost function value. This paper then illustrates that the proposed method has a computationally efficient form that is similar to the optimized gradient method. [ABSTRACT FROM AUTHOR]

We establish lower bounds on the complexity of finding ϵ -stationary points of smooth, non-convex high-dimensional functions using first-order methods. We prove that deterministic first-order methods, even applied to arbitrarily smooth functions, cannot achieve convergence rates in ϵ better than ϵ - 8 / 5 , which is within ϵ - 1 / 15 log 1 ϵ of the best known rate for such methods. Moreover, for functions with Lipschitz first and second derivatives, we prove that no deterministic first-order method can achieve convergence rates better than ϵ - 12 / 7 , while ϵ - 2 is a lower bound for functions with only Lipschitz gradient. For convex functions with Lipschitz gradient, accelerated gradient descent achieves a better rate, showing that finding stationary points is easier given convexity. [ABSTRACT FROM AUTHOR]

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

Decentralized multi-agent optimization usually relies on information exchange between neighboring agents, which can incur unaffordable communication overhead in practice. To reduce the communication cost, we apply event-triggering technique to the decentralized multi-agent online convex optimization problem, where each agent is associated with a time-varying local loss function and the goal is to minimize the accumulated total loss (the sum of all local loss functions) by choosing appropriate actions sequentially. We first develop an event-triggered decentralized online subgradient descent algorithm for the full information case, where the local loss function is fully revealed to each agent at each time. We establish an upper bound for the regret of each agent in terms of the event-triggering thresholds. It is shown that the regret is sublinear provided that the event-triggering thresholds converge to zero as time goes to infinity. The algorithm and analysis are further extended to the scenario of bandit feedback, where only the values of the local loss function at two random points close to the current action are disclosed to each agent. We show that the two-point bandit feedback does not degrade the performance of the proposed algorithm in order sense and a regret bound similar to the full information case can be established. Finally, numerical results on the problem of decentralized online least squares are presented to validate the proposed algorithms. [ABSTRACT FROM AUTHOR]