Search

We consider resolvent splitting algorithms for finding a zero of the sum of finitely many maximally monotone operators. The standard approach to solving this type of problem involves reformulating as a two-operator problem in the product-space and applying the Douglas-Rachford algorithm. However, existing results for linear convergence cannot be applied in the product-space formulation due to a lack of appropriate Lipschitz continuity and strong monotonicity. In this work, we investigate a different approach that does not rely on the Douglas-Rachford algorithm or the product-space directly. We establish linear convergence of the "resolvent splitting with minimal lifting" algorithm due to Malitsky and Tam for monotone inclusions with finitely many operators. Our results are then used to derive linear convergence of a primal-dual algorithm for convex minimization problems involving infimal convolutions. The theoretical results are demonstrated on numerical experiments in image denoising.

In this paper, we propose a distributed first-order algorithm with backtracking linesearch for solving multi-agent minimisation problems, where each agent handles a local objective involving nonsmooth and smooth components. Unlike existing methods that require global Lipschitz continuity and predefined stepsizes, our algorithm adjusts stepsizes using distributed linesearch procedures, making it suitable for problems where global constants are unavailable or difficult to compute. The proposed algorithm is designed within an abstract linesearch framework for a primal-dual proximal-gradient method to solve min-max convex-concave problems, enabling the consensus constraint to be decoupled from the optimisation task. Our theoretical analysis allows for gradients of functions to be locally Lipschitz continuous, relaxing the prevalent assumption of globally Lipschitz continuous gradients., Comment: 21 pages

In this work, we consider a nonsmooth minimisation problem in which the objective function can be represented as the maximum of finitely many smooth ``subfunctions''. First, we study a smooth min-max reformulation of the problem. Due to this smoothness, the model provides enhanced capability of exploiting the structure of the problem, when compared to methods that attempt to tackle the nonsmooth problem directly. Then, we present several approaches to identify the set of active subfunctions at a minimiser, all within finitely many iterations of a first order method for solving the smooth model. As is well known, the problem can be equivalently rewritten in terms of these subfunctions, but a key challenge is to identify this set \textit{a priori}. Such an identification is clearly beneficial in an algorithmic sense, since one can apply this knowledge to create an equivalent problem with lower complexity, thus facilitating generally faster convergence. Finally, numerical results comparing the accuracy of each of these approaches are presented, along with the effect they have on reducing the complexity of the original problem., Comment: 30 pages

In this work, we consider a connected network of finitely many agents working cooperatively to solve a min-max problem with convex-concave structure. We propose a decentralised first-order algorithm which can be viewed as a non-trivial combination of two algorithms: PG-EXTRA for decentralised minimisation problems and the forward reflected backward method for (non-distributed) min-max problems. In each iteration of our algorithm, each agent computes the gradient of the smooth component of its local objective function as well as the proximal operator of its nonsmooth component, following by a round of communication with its neighbours. Our analysis shows that the sequence generated by the method converges under standard assumptions with non-decaying stepsize., Comment: 15 pages, 1 figure

Frugal resolvent splittings are a class of fixed point algorithms for finding a zero in the sum of the sum of finitely many set-valued monotone operators, where the fixed point operator uses only vector addition, scalar multiplication and the resolvent of each monotone operator once per iteration. In the literature, the convergence analyses of these schemes are performed in an inefficient, algorithm-by-algorithm basis. In this work, we address this by developing a general framework for frugal resolvent splitting which simultaneously covers and extends several important schemes in the literature. The framework also yields a new resolvent splitting algorithm which is suitable for decentralised implementation on regular networks., Comment: 14 pages

In this work, we consider a class of convex optimization problems in a real Hilbert space that can be solved by performing a single projection, i.e., by projecting an infeasible point onto the feasible set. Our results improve those established for the linear programming setting in Nurminski (2015) by considering problems that: (i) may have multiple solutions, (ii) do not satisfy strict complementary conditions, and (iii) possess non-linear convex constraints. As a by-product of our analysis, we provide a quantitative estimate on the required distance between the infeasible point and the feasible set in order for its projection to be a solution of the problem. Our analysis relies on a "sharpness" property of the constraint set; a new property we introduce here.

Objectives: The present study evaluated the impact of a commercially available explainable AI algorithm in augmenting the ability of clinicians to identify lung cancer on chest X-rays (CXR). Design: This retrospective study evaluated the performance of 11 clinicians for detecting lung cancer from chest radiographs, with and without assistance from a commercially available AI algorithm (red dot, Behold.ai) that predicts suspected lung cancer from CXRs. Clinician performance was evaluated against clinically confirmed diagnoses. Setting: The study analysed anonymised patient data from an NHS hospital; the dataset consisted of 400 chest radiographs from adult patients (18 years and above) who had a CXR performed in 2020, with corresponding clinical text reports. Participants: A panel of readers consisting of 11 clinicians (consultant radiologists, radiologist trainees and reporting radiographers) participated in this study. Main outcome measures: Overall accuracy, sensitivity, specificity and precision for detecting lung cancer on CXRs by clinicians, with and without AI input. Agreement rates between clinicians and performance standard deviation were also evaluated, with and without AI input. Results: The use of the AI algorithm by clinicians led to an improved overall performance for lung tumour detection, achieving an overall increase of 17.4% of lung cancers being identified on CXRs which would have otherwise been missed, an overall increase in detection of smaller tumours, a 24% and 13% increased detection of stage 1 and stage 2 lung cancers respectively, and standardisation of clinician performance. Conclusions: This study showed great promise in the clinical utility of AI algorithms in improving early lung cancer diagnosis and promoting health equity through overall improvement in reader performances, without impacting downstream imaging resources.

Variational inequalities provide a framework through which many optimisation problems can be solved, in particular, saddle-point problems. In this paper, we study modifications to the so-called Golden RAtio ALgorithm (GRAAL) for variational inequalities -- a method which uses a fully explicit adaptive step-size, and provides convergence results under local Lipschitz assumptions without requiring backtracking. We present and analyse two Bregman modifications to GRAAL: the first uses a fixed step-size and converges under global Lipschitz assumptions, and the second uses an adaptive step-size rule. Numerical performance of the former method is demonstrated on a bimatrix game arising in network communication, and of the latter on two problems, namely, power allocation in Gaussian communication channels and $N$-person Cournot completion games. In all of these applications, an appropriately chosen Bregman distance simplifies the projection steps computed as part of the algorithm., Comment: 24 pages

In this work, we propose and analyse forward-backward-type algorithms for finding a zero of the sum of finitely many monotone operators, which are not based on reduction to a two operator inclusion in the product space. Each iteration of the studied algorithms requires one resolvent evaluation per set-valued operator, one forward evaluation per cocoercive operator, and two forward evaluations per monotone operator. Unlike existing methods, the structure of the proposed algorithms are suitable for distributed, decentralised implementation in ring networks without needing global summation to enforce consensus between nodes., Comment: 19 pages

In this work, we study fixed point algorithms for finding a zero in the sum of $n\geq 2$ maximally monotone operators by using their resolvents. More precisely, we consider the class of such algorithms where each resolvent is evaluated only once per iteration. For any algorithm from this class, we show that the underlying fixed point operator is necessarily defined on a $d$-fold Cartesian product space with $d\geq n-1$. Further, we show that this bound is unimprovable by providing a family of examples for which $d=n-1$ is attained. This family includes the Douglas-Rachford algorithm as the special case when $n=2$. Applications of the new family of algorithms in distributed decentralised optimisation and multi-block extensions of the alternation direction method of multipliers (ADMM) are discussed., Comment: 23 pages

A novel approach for solving the general absolute value equation $Ax+B|x| = c$ where $A,B\in \mathbb{R}^{m\times n}$ and $c\in \mathbb{R}^m$ is presented. We reformulate the equation as a feasibility problem which we solve via the method of alternating projections (MAP). The fixed points set of the alternating projections map is characterized under nondegeneracy conditions on $A$ and $B$. Furthermore, we prove linear convergence of the algorithm. Unlike most of the existing approaches in the literature, the algorithm presented here is capable of handling problems with $m\neq n$, both theoretically and numerically., Comment: 38 pages, 1 figure, 3 tables

In this work, we develop a systematic framework for computing the resolvent of the sum of two or more monotone operators which only activates each operator in the sum individually. The key tool in the development of this framework is the notion of the "strengthening" of a set-valued operator, which can be viewed as a type of regularisation that preserves computational tractability. After deriving a number of iterative schemes through this framework, we demonstrate their application to best approximation problems, image denoising and elliptic PDEs.

The method of cyclic projections finds nearest points in the intersection of finitely many affine subspaces. To accelerate convergence, Gearhart and Koshy proposed a modification which, in each iteration, performs an exact line search based on minimising the distance to the solution. When the subspaces are linear, the procedure can be made explicit using feasibility of the zero vector. This work studies an alternative approach which does not rely on this fact, thus providing an efficient implementation in the affine setting., Comment: 8 pages, 1 figure

We introduce a reformulation technique that converts a many-set feasibility problem into an equivalent two-set problem. This technique involves reformulating the original feasibility problem by replacing a pair of its constraint sets with their intersection, before applying Pierra's classical product space reformulation. The step of combining the two constraint sets reduces the dimension of the product spaces. We refer to this as the constraint reduction reformulation and use it to obtain constraint-reduced variants of well-known projection algorithms such as the Douglas--Rachford algorithm and the method of alternating projections, among others. We prove global convergence of constraint-reduced algorithms in the presence of convexity and local convergence in a nonconvex setting. In order to analyse convergence of the constraint-reduced Douglas--Rachford method, we generalize a classical result which guarantees that the composition of two projectors onto subspaces is a projector onto their intersection. Finally, we apply the constraint-reduced versions of Douglas--Rachford and alternating projections to solve the wavelet feasibility problems, and then compare their performance with their usual product variants.

After re-casting the $n$-dimensional wavelet construction problem as a feasibility problem with constraints arising from the requirements of compact support, smoothness and orthogonality, the Douglas--Rachford algorithm is employed in the search for one- and two-dimensional wavelets. New one-dimensional wavelets are produced as well as genuinely non-separable two-dimensional wavelets in the case where the dilation on the plane is the standard $D_af(t)=a^{-1}f(t/a)$ $(t\in{\mathbb R}^n, a>0)$., Comment: 36 pages, 4 figures

In this work, we consider a continuous dynamical system associated with the fixed point set of a nonexpansive operator which was originally studied by Bo\c{t} & Csetnek (2015). Our main results establish convergence rates for the system's trajectories when the nonexpansive operator satisfies an additional regularity property. This setting is the natural continuous-time analogue to discrete-time results obtained in Bauschke, Noll & Phan (2015) and Borwein, Li & Tam (2017) by using the same regularity properties., Comment: 15 pages

In this work, we propose and analyse two splitting algorithms for finding a zero of the sum of three monotone operators, one of which is assumed to be Lipschitz continuous. Each iteration of these algorithms require one forward evaluation of the Lipschitz continuous operator and one resolvent evaluation of each of the other two operators. By specialising to two operator inclusions, we recover the forward-reflected-backward and the reflected-forward-backward splitting methods as particular cases. The inspiration for the proposed algorithms arises from interpretations of the aforementioned reflected splitting algorithms as discretisations of the continuous-time proximal point algorithm., Comment: 15 pages

The Douglas-Rachford method, a projection algorithm designed to solve continuous optimization problems, forms the basis of a useful heuristic for solving combinatorial optimization problems. In order to successfully use the method, it is necessary to formulate the problem at hand as a feasibility problem with constraint sets having efficiently computable nearest points. In this self-contained tutorial, we develop the convergence theory of projection algorithms within the framework of fixed point iterations, explain how to devise useful feasibility problem formulations, and demonstrate the application of the Douglas-Rachford method to said formulations. The paradigm is then illustrated on two concrete problems: a generalization of the "eight queens puzzle" known as the "(m,n)-queens problem", and the problem of constructing a probability distribution with prescribed moments.

In this work, we propose a new algorithm for finding a zero in the sum of two monotone operators where one is assumed to be single-valued and Lipschitz continuous. This algorithm naturally arises from a non-standard discretization of a continuous dynamical system associated with the Douglas--Rachford splitting algorithm. More precisely, it is obtained by performing an explicit, rather than implicit, discretization with respect to one of the operators involved. Each iteration of the proposed algorithm requires the evaluation of one forward and one backward operator.

Convergence of projection-based methods for nonconvex set feasibility problems has been established for sets with ever weaker regularity assumptions. What has not kept pace with these developments is analogous results for convergence of optimization problems with correspondingly weak assumptions on the value functions. Indeed, one of the earliest classes of nonconvex sets for which convergence results were obtainable, the class of so-called super-regular sets introduced by Lewis, Luke and Malick (2009), has no functional counterpart. In this work, we amend this gap in the theory by establishing the equivalence between a property slightly stronger than super-regularity, which we call Clarke super-regularity, and subsmootheness of sets as introduced by Aussel, Daniilidis and Thibault (2004). The bridge to functions shows that approximately convex functions studied by Ngai, Luc and Th\'era (2000) are those which have Clarke super-regular epigraphs. Further classes of regularity of functions based on the corresponding regularity of their epigraph are also discussed., Comment: 15 pages, 2 figures

In this work, we propose a simple modification of the forward-backward splitting method for finding a zero in the sum of two monotone operators. Our method converges under the same assumptions as Tseng's forward-backward-forward method, namely, it does not require cocoercivity of the single-valued operator. Moreover, each iteration only requires one forward evaluation rather than two as is the case for Tseng's method. Variants of the method incorporating a linesearch, relaxation and inertia, or a structured three operator inclusion are also discussed., Comment: 20 pages, 1 figure

In this paper we introduce and study a class of structured set-valued operators which we call union averaged nonexpansive. At each point in their domain, the value of such an operator can be expressed as a finite union of single-valued averaged nonexpansive operators. We investigate various structural properties of the class and show, in particular, that is closed under taking unions, convex combinations, and compositions, and that their fixed point iterations are locally convergent around strong fixed points. We then systematically apply our results to analyze proximal algorithms in situations where union averaged nonexpansive operators naturally arise. In particular, we consider the problem of minimizing the sum two functions where the first is convex and the second can be expressed as the minimum of finitely many convex functions.

In this work, we propose an optimization approach for constructing various classes of circulant combinatorial designs that can be defined in terms of autocorrelations. The problem is formulated as a so-called feasibility problem having three sets, to which the Douglas-Rachford projection algorithm is applied. The approach is illustrated on three different classes of circulant combinatorial designs: circulant weighing matrices, D-optimal matrices, and Hadamard matrices with two circulant cores. Furthermore, we explicitly construct two new circulant weighing matrices, a $CW(126,64)$ and a $CW(198,100)$, whose existence was previously marked as unresolved in the most recent version of Strassler's table.

In this note, we consider a framework for the analysis of iterative algorithms which can described in terms of a structured set-valued operator. More precisely, at each point in the ambient space, we assume that the value of operator can be expressed as a finite union of values of single-valued paracontracting operators. Our main result, which shows that the associated fixed point iteration is locally convergent around strong fixed points, generalises a theorem due to Bauschke and Noll (2014)., Comment: 8 pages

The Douglas-Rachford projection algorithm is an iterative method used to find a point in the intersection of closed constraint sets. The algorithm has been experimentally observed to solve various nonconvex feasibility problems which current theory cannot sufficiently explain. In this paper, we prove convergence of the Douglas-Rachford algorithm in a potentially nonconvex setting. Our analysis relies on the existence of a Lyapunov-type functional whose convexity properties are not tantamount to convexity of the original constraint sets. Moreover, we provide various nonconvex examples in which our framework proves global convergence of the algorithm., Comment: 31 pages, 4 figures

We consider a class of monotone operators which are appropriate for symbolic representation and manipulation within a computer algebra system. Various structural properties of the class (e.g., closure under taking inverses, resolvents) are investigated as well as the role played by maximal monotonicity within the class. In particular, we show that there is a natural correspondence between our class of monotone operators and the subdifferentials of convex functions belonging to a class of convex functions deemed suitable for symbolic computation of Fenchel conjugates which were previously studied by Bauschke & von Mohrenschildt and by Borwein & Hamilton. A number of illustrative examples utilizing the introduced class of operators are provided including computation of proximity operators, recovery of a convex penalty function associated with the hard thresholding operator, and computation of superexpectations, superdistributions and superquantiles with specialization to risk measures., Comment: 17 pages, 2 figures

This paper investigates regularity properties of two non-negative sparsity sets: non-negative sparse vectors, and low-rank positive semi-definite matrices. Novel formulae for their Mordukhovich normal cones are given and used to formulate sufficient conditions for non-convex notions of regularity to hold. Our results provide a useful tool for justifying the application of projection methods to certain rank constrained feasibility problems., Comment: 22 pages

We develop a framework for quantitative convergence analysis of Picard iterations of expansive set-valued fixed point mappings. There are two key components of the analysis. The first is a natural generalization of single-valued averaged mappings to expansive, set-valued mappings that characterizes a type of strong calmness of the fixed point mapping. The second component to this analysis is an extension of the well-established notion of metric subregularity -- or inverse calmness -- of the mapping at fixed points. Convergence of expansive fixed point iterations is proved using these two properties, and quantitative estimates are a natural byproduct of the framework. To demonstrate the application of the theory, we prove for the first time a number of results showing local linear convergence of nonconvex cyclic projections for inconsistent (and consistent) feasibility problems, local linear convergence of the forward-backward algorithm for structured optimization without convexity, strong or otherwise, and local linear convergence of the Douglas--Rachford algorithm for structured nonconvex minimization. This theory includes earlier approaches for known results, convex and nonconvex, as special cases., Comment: 41 pages 70 references. Detailed examples added and more historical commentary (with citations)

In this paper, we establish sublinear and linear convergence of fixed point iterations generated by averaged operators in a Hilbert space. Our results are achieved under a bounded H\"older regularity assumption which generalizes the well-known notion of bounded linear regularity. As an application of our results, we provide a convergence rate analysis for Krasnoselskii-Mann iterations, the cyclic projection algorithm, and the Douglas-Rachford feasibility algorithm along with some variants. In the important case in which the underlying sets are convex sets described by convex polynomials in a finite dimensional space, we show that the H\"older regularity properties are automatically satisfied, from which sublinear convergence follows., Comment: 34 pages, 1 figure

In recent times the Douglas-Rachford algorithm has been observed empirically to solve a variety of nonconvex feasibility problems including those of a combinatorial nature. For many of these problems current theory is not sufficient to explain this observed success and is mainly concerned with questions of local convergence. In this paper we analyze global behavior of the method for finding a point in the intersection of a half-space and a potentially non-convex set which is assumed to satisfy a well-quasi-ordering property or a property weaker than compactness. In particular, the special case in which the second set is finite is covered by our framework and provides a prototypical setting for combinatorial optimization problems.

The Douglas-Rachford reflection method is a general purpose algorithm useful for solving the feasibility problem of finding a point in the intersection of finitely many sets. In this chapter we demonstrate that applied to a specific problem, the method can benefit from heuristics specific to said problem which exploit its special structure. In particular, we focus on the problem of protein conformation determination formulated within the framework of matrix completion, as was considered in a recent paper of the present authors., Comment: 18 pages, 8 figures

We propose a general alternating minimization algorithm for nonconvex optimization problems with separable structure and nonconvex coupling between blocks of variables. To fix our ideas, we apply the methodology to the problem of blind ptychographic imaging. Compared to other schemes in the literature, our approach differs in two ways: (i) it is posed within a clear mathematical framework with practically verifiable assumptions, and (ii) under the given assumptions, it is provably convergent to critical points. A numerical comparison of our proposed algorithm with the current state-of-the-art on simulated and experimental data validates our approach and points toward directions for further improvement., Comment: 32 pages, 3 tables, 5 figures

We provide sufficient conditions for norm convergence of various projection and reflection methods, as well as giving limiting examples regarding convergence rates., Comment: 21 pages, 1 figure, references updated

We analyse the behaviour of the newly introduced cyclic Douglas-Rachford algorithm for finding a point in the intersection of a finite number of closed convex sets. This work considers the case in which the target intersection set is possibly empty., Comment: 13 pages, 2 figures; references updated, figure 2 corrected

In this paper we give general recommendations for successful application of the Douglas-Rachford reflection method to convex and non-convex real matrix-completion problems. These guidelines are demonstrated by various illustrative examples., Comment: 30 pages, 4 figures, 3 tables

We discuss recent positive experiences applying convex feasibility algorithms of Douglas--Rachford type to highly combinatorial and far from convex problems.

In this paper we present two Douglas-Rachford inspired iteration schemes which can be applied directly to N-set convex feasibility problems in Hilbert space. Our main results are weak convergence of the methods to a point whose nearest point projections onto each of the N sets coincide. For affine subspaces, convergence is in norm. Initial results from numerical experiments, comparing our methods to the classical (product-space) Douglas-Rachford scheme, are promising., Comment: 22 pages, 7 figures, 4 tables