Search

In this paper, we develop a unified framework and convergence analysis of descent methods for vector optimization problems (VOPs) from a majorization-minimization perspective. By choosing different surrogate functions, the generic method reduces to some existing descent methods with and without line search, respectively. The unified convergence analysis shows that the slow convergence of the steepest descent method is mainly due to the large gap between surrogate and objective functions. As a result, the performance of descent methods can be improved by narrowing the surrogate gap. Interestingly, we observe that selecting a tighter surrogate function is equivalent to using an appropriate base of the dual cone in the direction-finding subproblem. Furthermore, we use Barzilai-Borwein method to narrow the surrogate gap and devise a Barzilai-Borwein descent method for VOPs with polyhedral cone. By reformulating the subproblem, we provide a new insight into the Barzilai-Borwein descent method and bridge it to the steepest descent method. Finally, several numerical experiments confirm the efficiency of the proposed method.

This paper focuses on developing a conditional gradient algorithm for multiobjective optimization problems with an unbounded feasible region. We employ the concept of recession cone to establish the well-defined nature of the algorithm. The asymptotic convergence property and the iteration-complexity bound are established under mild assumptions. Numerical examples are provided to verify the algorithmic performance.

The vast applications of deep generative models are anchored in three core capabilities -- generating new instances, reconstructing inputs, and learning compact representations -- across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce Generalized Encoding-Decoding Diffusion Probabilistic Models (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models., Comment: ICML 2024 camera-ready. Code is available at https://github.com/guangyliu/EDDPM

The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

This paper addresses an unconstrained multiobjective optimization problem where two or more continuously differentiable functions have to be minimized. We delve into the conjugate gradient methods proposed by Lucambio P\'{e}rez and Prudente (SIAM J Optim, 28(3): 2690--2720, 2018) for this type of problem. Instead of the Wolfe-type line search procedure used in their work, we employ a fixed stepsize formula (or no-line-search scheme), which can mitigate the selection pressure caused by multiple inequalities and avoid the computational cost associated with objective function evaluations in specific applications. The no-line-search scheme is utilized to derive the condition of Zoutendijk's type. Global convergence encompasses the vector extensions of Fletcher-Reeves, conjugate descent, Dai-Yuan, Polak-Ribi\`{e}re-Polyak and Hestenes-Stiefel parameters, subject to certain mild assumptions. Additionally, numerical experiments are conducted to demonstrate the practical performance of the proposed stepsize rule, and comparative analyses are made with the multiobjective steepest descent methods using the Armijo line search and the multiobjective conjugate gradient methods using the Wolfe-type line search.

In this paper, we propose a simple yet efficient strategy for improving the multi-objective steepest descent method proposed by Fliege and Svaiter (Math Methods Oper Res, 2000, 3: 479--494). The core idea behind this strategy involves incorporating a positive modification parameter into the iterative formulation of the multi-objective steepest descent algorithm in a multiplicative manner. This modification parameter captures certain second-order information associated with the objective functions. We provide two distinct methods for calculating this modification parameter, leading to the development of two improved multi-objective steepest descent algorithms tailored for solving multi-objective optimization problems. Under reasonable assumptions, we demonstrate the convergence of sequences generated by the first algorithm toward a critical point. Moreover, for strongly convex multi-objective optimization problems, we establish the linear convergence to Pareto optimality of the sequence of generated points. The performance of the new algorithms is empirically evaluated through a computational comparison on a set of multi-objective test instances. The numerical results underscore that the proposed algorithms consistently outperform the original multi-objective steepest descent algorithm.

The development of artificial intelligence (AI) for science has led to the emergence of learning-based research paradigms, necessitating a compelling reevaluation of the design of multi-objective optimization (MOO) methods. The new generation MOO methods should be rooted in automated learning rather than manual design. In this paper, we introduce a new automatic learning paradigm for optimizing MOO problems, and propose a multi-gradient learning to optimize (ML2O) method, which automatically learns a generator (or mappings) from multiple gradients to update directions. As a learning-based method, ML2O acquires knowledge of local landscapes by leveraging information from the current step and incorporates global experience extracted from historical iteration trajectory data. By introducing a new guarding mechanism, we propose a guarded multi-gradient learning to optimize (GML2O) method, and prove that the iterative sequence generated by GML2O converges to a Pareto critical point. The experimental results demonstrate that our learned optimizer outperforms hand-designed competitors on training multi-task learning (MTL) neural network.

The imbalances and conditioning of the objective functions influence the performance of first-order methods for multiobjective optimization problems (MOPs). The latter is related to the metric selected in the direction-finding subproblems. Unlike single-objective optimization problems, capturing the curvature of all objective functions with a single Hessian matrix is impossible. On the other hand, second-order methods for MOPs use different metrics for objectives in direction-finding subproblems, leading to a high per-iteration cost. To balance per-iteration cost and better curvature exploration, we propose a Barzilai-Borwein descent method with variable metrics (BBDMO\_VM). In the direction-finding subproblems, we employ a variable metric to explore the curvature of all objectives. Subsequently, Barzilai-Borwein's method relative to the variable metric is applied to tune objectives, which mitigates the effect of imbalances. We investigate the convergence behaviour of the BBDMO\_VM, confirming fast linear convergence for well-conditioned problems relative to the variable metric. In particular, we establish linear convergence for problems that involve some linear objectives. These convergence results emphasize the importance of metric selection, motivating us to approximate the trade-off of Hessian matrices to better capture the geometry of the problem. Comparative numerical results confirm the efficiency of the proposed method, even when applied to large-scale and ill-conditioned problems.

In a recent study, Ansary (Optim Methods Softw 38(3):570-590,2023) proposed a Newton-type proximal gradient method for nonlinear multiobjective optimization problems (NPGMO). However, the favorable convergence properties typically associated with Newton-type methods were not established for NPGMO in Ansary's work. In response to this gap, we develop a straightforward framework for analyzing the convergence behavior of the NPGMO. Specifically, under the assumption of strong convexity, we demonstrate that the NPGMO enjoys quadratic termination, superlinear convergence, and quadratic convergence for problems that are quadratic, twice continuously differentiable and twice Lipschitz continuously differentiable, respectively., Comment: arXiv admin note: text overlap with arXiv:2306.09797

When minimizing a multiobjective optimization problem (MOP) using multiobjective gradient descent methods, the imbalances among objective functions often decelerate the convergence. In response to this challenge, we propose two types of the Barzilai-Borwein proximal gradient method for multi-objective composite optimization problems (BBPGMO). We establish convergence rates for BBPGMO, demonstrating that it achieves rates of $O(\frac{1}{\sqrt{k}})$, $O(\frac{1}{k})$, and $O(r^{k})(0

This paper investigates an unsupervised approach towards deriving a universal, cross-lingual word embedding space, where words with similar semantics from different languages are close to one another. Previous adversarial approaches have shown promising results in inducing cross-lingual word embedding without parallel data. However, the training stage shows instability for distant language pairs. Instead of mapping the source language space directly to the target language space, we propose to make use of a sequence of intermediate spaces for smooth bridging. Each intermediate space may be conceived as a pseudo-language space and is introduced via simple linear interpolation. This approach is modeled after domain flow in computer vision, but with a modified objective function. Experiments on intrinsic Bilingual Dictionary Induction tasks show that the proposed approach can improve the robustness of adversarial models with comparable and even better precision. Further experiments on the downstream task of Cross-Lingual Natural Language Inference show that the proposed model achieves significant performance improvement for distant language pairs in downstream tasks compared to state-of-the-art adversarial and non-adversarial models.

This paper presents a piecewise convexification method to approximate the whole approximate optimal solution set of non-convex optimization problems with box constraints. In the process of box division, we first classify the sub-boxes and only continue to divide only some sub-boxes in the subsequent division. At the same time, applying the $\alpha$-based Branch-and-Bound ({\rm$\alpha$BB}) method, we construct a series of piecewise convex relax sub-problems, which are collectively called the piecewise convexification problem of the original problem. Then, we define the (approximate) solution set of the piecewise convexification problem based on the classification result of sub-boxes. Subsequently, we derive that these sets can be used to approximate the global solution set with a predefined quality. Finally, a piecewise convexification algorithm with a new selection rule of sub-box for the division and two new termination tests is proposed. Several instances verify that these techniques are beneficial to improve the performance of the algorithm.

In recent years, by using Bregman distance, the Lipschitz gradient continuity and strong convexity were lifted and replaced by relative smoothness and relative strong convexity. Under the mild assumptions, it was proved that gradient methods with Bregman regularity converge linearly for single-objective optimization problems (SOPs). In this paper, we extend the relative smoothness and relative strong convexity to vector-valued functions and analyze the convergence of an interior Bregman gradient method for vector optimization problems (VOPs). Specifically, the global convergence rates are $\mathcal{O}(\frac{1}{k})$ and $\mathcal{O}(r^{k})(0

The steepest descent method proposed by Fliege et al. motivates the research on descent methods for multiobjective optimization, which has received increasing attention in recent years. However, empirical results show that the Armijo line search often gives a very small stepsize along the steepest direction, which decelerates the convergence seriously. This paper points out that the issue is mainly due to the imbalances among objective functions. To address this issue, we propose a Barzilai-Borwein descent method for multiobjective optimization (BBDMO) that dynamically tunes gradient magnitudes using Barzilai-Borwein's rule in direction-finding subproblem. With monotone and nonmonotone line search techniques, it is proved that accumulation points generated by BBDMO are Pareto critical points, respectively. Furthermore, theoretical results indicate the Armijo line search can achieve a better stepsize in BBDMO. Finally, comparative results of numerical experiments are reported to illustrate the efficiency of BBDMO and verify the theoretical results.