Search

Estimating hyperparameters has been a long-standing problem in machine learning. We consider the case where the task at hand is modeled as the solution to an optimization problem. Here the exact gradient with respect to the hyperparameters cannot be feasibly computed and approximate strategies are required. We introduce a unified framework for computing hypergradients that generalizes existing methods based on the implicit function theorem and automatic differentiation/backpropagation, showing that these two seemingly disparate approaches are actually tightly connected. Our framework is extremely flexible, allowing its subproblems to be solved with any suitable method, to any degree of accuracy. We derive a priori and computable a posteriori error bounds for all our methods and numerically show that our a posteriori bounds are usually more accurate. Our numerical results also show that, surprisingly, for efficient bilevel optimization, the choice of hypergradient algorithm is at least as important as the choice of lower-level solver. [ABSTRACT FROM AUTHOR]

We introduce a general framework for large-scale model-based derivative-free optimization based on iterative minimization within random subspaces. We present a probabilistic worst-case complexity analysis for our method, where in particular we prove high-probability bounds on the number of iterations before a given optimality is achieved. This framework is specialized to nonlinear least-squares problems, with a model-based framework based on the Gauss–Newton method. This method achieves scalability by constructing local linear interpolation models to approximate the Jacobian, and computes new steps at each iteration in a subspace with user-determined dimension. We then describe a practical implementation of this framework, which we call DFBGN. We outline efficient techniques for selecting the interpolation points and search subspace, yielding an implementation that has a low per-iteration linear algebra cost (linear in the problem dimension) while also achieving fast objective decrease as measured by evaluations. Extensive numerical results demonstrate that DFBGN has improved scalability, yielding strong performance on large-scale nonlinear least-squares problems. [ABSTRACT FROM AUTHOR]

We present a model-based derivative-free method for optimization subject to general convex constraints, which we assume are unrelaxable and accessed only through a projection operator that is cheap to evaluate. We prove global convergence and a worst-case complexity of O(ε-2) iterations and objective evaluations for nonconvex functions, matching results for the unconstrained case. We introduce new, weaker requirements on model accuracy compared to existing methods. As a result, sufficiently accurate interpolation models can be constructed only using feasible points. We develop a comprehensive theory of interpolation set management in this regime for linear and composite linear models. We implement our approach for nonlinear least-squares problems and demonstrate strong practical performance compared to general-purpose solvers. [ABSTRACT FROM AUTHOR]

We investigate the potential of applying a state-of-the-art, local derivative-free solver, Py-BOBYQA to global optimization problems. In particular, we demonstrate the potential of a restarts procedure – as distinct from multistart methods – to allow Py-BOBYQA to escape local minima (where ordinarily it would terminate at the first local minimum found). We also introduce an adaptive variant of restarts which yields improved performance on global optimization problems. As Py-BOBYQA is a model-based trust-region method, we compare largely with other global optimization methods for which (global) models are important, such as Bayesian optimization and response surface methods; we also consider state-of-the-art representative deterministic and stochastic codes, such as DIRECT and CMA-ES. We find numerically that the restarts procedures in Py-BOBYQA are effective at helping it to escape local minima, when compared to using no restarts in Py-BOBYQA. Additionally, we find that Py-BOBYQA with adaptive restarts has comparable performance with global optimization solvers for all accuracy/budget regimes, in both smooth and noisy settings. In particular, Py-BOBYQA variants are best performing for smooth and multiplicative noise problems in high-accuracy regimes. As a by-product, some preliminary conclusions can be drawn on the relative performance of the global solvers we have tested with default settings. [ABSTRACT FROM AUTHOR]

Ramming paste is a particular type of carbon paste which is used as lining for electric smelting furnaces and Hall-Héroult cells for the production of aluminium. The purpose of this lining is to form an impenetrable barrier, keeping the liquid within the furnace. If the lining has cracks or holes, then liquid can escape, which can lead to safety risks and financial losses, so the integrity of the lining is of great importance. In the present study, we develop a first principles mathematical model for the heat and mass transfer processes occurring during the baking of carbon paste. We then obtain numerical simulations using this model and compare the simulation results to experimental data, demonstrating that the model solutions do indeed describe and predict realistic behaviours of the carbon paste baking process. The simulations indicate a strong pressure buildup during the evaporation of water from fresh paste during the baking process, which is likely to lead to cracking of the paste as it hardens. Furthermore, we are able to show that more gradual heating during the baking process can lower the maximal pressures predicted by the model, which in turn may reduce the prevalence of cracks within the hardened paste. [ABSTRACT FROM AUTHOR]

Prediction of pulp yield of Eucalyptus globulus wood samples based on partial least squares (PLS) regression can be optimized by utilizing specific near infrared (NIR) wavelengths. A critical feature of this approach is the weighting of constraint conditions. Equal weighting balances optimization in terms of calibration and prediction; however, there is a lack of knowledge regarding prediction performance of wood property models when different weight factors are used. In this study, pulp yield models were developed using two E. globulus data sets characterized by narrow (5%) and extreme (22.6%) yield ranges and represented by untreated and second derivative NIR spectra. The global optimization solver pySOT was used to optimize the performance of a PLS regression model in terms of wavelengths selected and number of latent variables. A linear function of R-squares for calibration ( R c 2 ) and prediction ( R p 2 ) sets was utilized as the objective function with the aim of maximizing α R c 2 + 1 - α R p 2 for all values of α between 0 (maximizing R p 2 without concern for R c 2 ) and 1 (only maximizing R c 2 ). Values of α ≤ 0.8 provided good predictive performance, whereas α ≥ 0.9 tended to overfit the calibration data indicating that models are robust for values of α from 0 to 0.8. Representative wavelengths for each data set were identified and assigned to corresponding wood components through a band assignment process. Strong agreement was observed for α ≤ 0.8 ; however, for α ≥ 0.9 , identified wavelengths generally occurred in regions unrelated to vibrations arising from specific wood components. [ABSTRACT FROM AUTHOR]

Uncertainty in climate projections is large as shown by the likely uncertainty ranges in equilibrium climate sensitivity (ECS) of 2.5–4 K and in the transient climate response (TCR) of 1.4–2.2 K. Uncertainty in model projections could arise from the way in which unresolved processes are represented, the parameter values used, or the targets for model calibration. We show that, in two climate model ensembles that were objectively calibrated to minimize differences from observed large-scale atmospheric climatology, uncertainties in ECS and TCR are about 2–6 times smaller than in the CMIP5 or CMIP6 multimodel ensemble. We also find that projected uncertainties in surface temperature, precipitation, and annual extremes are relatively small. Residual uncertainty largely arises from unconstrained sea ice feedbacks. The more than 20-year-old HadAM3 standard model configuration simulates observed hemispheric-scale observations and preindustrial surface temperatures about as well as the median CMIP5 and CMIP6 ensembles while the optimized configurations simulate these better than almost all the CMIP5 and CMIP6 models. Hemispheric-scale observations and preindustrial temperatures are not systematically better simulated in CMIP6 than in CMIP5 although the CMIP6 ensemble seems to better simulate patterns of large-scale observations than the CMIP5 ensemble and the optimized HadAM3 configurations. Our results suggest that most CMIP models could be improved in their simulation of large-scale observations by systematic calibration. However, the uncertainty in climate projections (for a given scenario) likely largely arises from the choice of parameterization schemes for unresolved processes ("structural uncertainty"), with different tuning targets another possible contributor. Significance Statement: Climate models represent unresolved phenomena controlled by uncertain parameters. Changes in these parameters impact how well a climate model simulates current climate and its climate projections. Multiple calibrations of a single climate model, using an objective method, to large-scale atmospheric observations are performed. These models produce very similar climate projections at both global and regional scales. An analysis that combines uncertainties in observations with simulated sensitivity to observations and climate response also has small uncertainty showing that, for this model, current observations constrain climate projections. Recently developed climate models have a broad range of abilities to simulate large-scale climate with only some improvement in their ability to simulate this despite a decade of model development. [ABSTRACT FROM AUTHOR]