Search

We revisit Markowitz's mean-variance portfolio selection model by considering a distributionally robust version, where the region of distributional uncertainty is around the empirical measure and the discrepancy between probability measures is dictated by the so-called Wasserstein distance. We reduce this problem into an empirical variance minimization problem with an additional regularization term. Moreover, we extend recent inference methodology in order to select the size of the distributional uncertainty as well as the associated robust target return rate in a data-driven way., Comment: 20 pages

In this paper, we introduce an unbiased gradient simulation algorithms for solving convex optimization problem with stochastic function compositions. We show that the unbiased gradient generated from the algorithm has finite variance and finite expected computation cost. We then combined the unbiased gradient simulation with two variance reduced algorithms (namely SVRG and SCSG) and showed that the proposed optimization algorithms based on unbiased gradient simulations exhibit satisfactory convergence properties. Specifically, in the SVRG case, the algorithm with simulated gradient can be shown to converge linearly to optima in expectation and almost surely under strong convexity. Finally, for the numerical experiment,we applied the algorithms to two important cases of stochastic function compositions optimization: maximizing the Cox's partial likelihood model and training conditional random fields.

We study sample-path large deviations for L\'evy processes and random walks with heavy-tailed jump-size distributions that are of Weibull type. Our main results include an extended form of an LDP (large deviations principle) in the $J_1$ topology, and a full LDP in the $M_1'$ topology. The rate function can be represented as the solution to a quasi-variational problem. The sharpness and applicability of these results are illustrated by a counterexample proving the nonexistence of a full LDP in the $J_1$ topology, and by an application to a first passage problem.

We model equilibrium allocations in a distribution network as the solution of a linear program (LP) which minimizes the cost of unserved demands across nodes in the network. The constraints in the LP dictate that once a given node's supply is exhausted, its unserved demand is distributed among neighboring nodes. All nodes do the same and the resulting solution is the equilibrium allocation. Assuming that the demands are random (following a jointly Gaussian law), our goal is to study the probability that the optimal cost (i.e. the cost of unserved demands in equilibrium) exceeds a large threshold, which is a rare event. Our contribution is the development of importance sampling and conditional Monte Carlo algorithms for estimating this probability. We establish the asymptotic efficiency of our algorithms and also present numerical results which illustrate strong performance of our procedures., Comment: 27 pages

We provide the first generic exact simulation algorithm for multivariate diffusions. Current exact sampling algorithms for diffusions require the existence of a transformation which can be used to reduce the sampling problem to the case of a constant diffusion matrix and a drift which is the gradient of some function. Such transformation, called Lamperti transformation, can be applied in general only in one dimension. So, completely different ideas are required for exact sampling of generic multivariate diffusions. The development of these ideas is the main contribution of this paper. Our strategy combines techniques borrowed from the theory of rough paths, on one hand, and multilevel Monte Carlo on the other., Comment: 28 pages; Accepted for publication by the Applied Probability Trust (http://www.appliedprobability.org) in 52.4 (December 2020)

We propose a class of strongly efficient rare event simulation estimators for random walks and compound Poisson processes with a regularly varying increment/jump-size distribution in a general large deviations regime. Our estimator is based on an importance sampling strategy that hinges on the heavy-tailed sample path large deviations result recently established in Rhee, Blanchet, and Zwart (2016). The new estimators are straightforward to implement and can be used to systematically evaluate the probability of a wide range of rare events with bounded relative error. They are "universal" in the sense that a single importance sampling scheme applies to a very general class of rare events that arise in heavy-tailed systems. In particular, our estimators can deal with rare events that are caused by multiple big jumps (therefore, beyond the usual principle of a single big jump) as well as multidimensional processes such as the buffer content process of a queueing network. We illustrate the versatility of our approach with several applications that arise in the context of mathematical finance, actuarial science, and queueing theory.

Recently, (Blanchet, Kang, and Murhy 2016, and Blanchet, and Kang 2017) showed that several machine learning algorithms, such as square-root Lasso, Support Vector Machines, and regularized logistic regression, among many others, can be represented exactly as distributionally robust optimization (DRO) problems. The distributional uncertainty is defined as a neighborhood centered at the empirical distribution. We propose a methodology which learns such neighborhood in a natural data-driven way. We show rigorously that our framework encompasses adaptive regularization as a particular case. Moreover, we demonstrate empirically that our proposed methodology is able to improve upon a wide range of popular machine learning estimators.

Data-driven Distributionally Robust Optimization (DD-DRO) via optimal transport has been shown to encompass a wide range of popular machine learning algorithms. The distributional uncertainty size is often shown to correspond to the regularization parameter. The type of regularization (e.g. the norm used to regularize) corresponds to the shape of the distributional uncertainty. We propose a data-driven robust optimization methodology to inform the transportation cost underlying the definition of the distributional uncertainty. We show empirically that this additional layer of robustification, which produces a method we called doubly robust data-driven distributionally robust optimization (DD-R-DRO), allows to enhance the generalization properties of regularized estimators while reducing testing error relative to state-of-the-art classifiers in a wide range of data sets., Comment: arXiv admin note: substantial text overlap with arXiv:1705.07152

Regularized estimators in the context of group variables have been applied successfully in model and feature selection in order to preserve interpretability. We formulate a Distributionally Robust Optimization (DRO) problem which recovers popular estimators, such as Group Square Root Lasso (GSRL). Our DRO formulation allows us to interpret GSRL as a game, in which we learn a regression parameter while an adversary chooses a perturbation of the data. We wish to pick the parameter to minimize the expected loss under any plausible model chosen by the adversary - who, on the other hand, wishes to increase the expected loss. The regularization parameter turns out to be precisely determined by the amount of perturbation on the training data allowed by the adversary. In this paper, we introduce a data-driven (statistical) criterion for the optimal choice of regularization, which we evaluate asymptotically, in closed form, as the size of the training set increases. Our easy-to-evaluate regularization formula is compared against cross-validation, showing good (sometimes superior) performance., Comment: 21 Pages, 1 Figure

We propose a novel method for semi-supervised learning (SSL) based on data-driven distributionally robust optimization (DRO) using optimal transport metrics. Our proposed method enhances generalization error by using the unlabeled data to restrict the support of the worst case distribution in our DRO formulation. We enable the implementation of our DRO formulation by proposing a stochastic gradient descent algorithm which allows to easily implement the training procedure. We demonstrate that our Semi-supervised DRO method is able to improve the generalization error over natural supervised procedures and state-of-the-art SSL estimators. Finally, we include a discussion on the large sample behavior of the optimal uncertainty region in the DRO formulation. Our discussion exposes important aspects such as the role of dimension reduction in SSL.

We introduce a class of unbiased Monte Carlo estimators for the multivariate density of max-stable fields generated by Gaussian processes. Our estimators take advantage of recent results on exact simulation of max-stable fields combined with identities studied in the Malliavin calculus literature and ideas developed in the multilevel Monte Carlo literature. Our approach allows estimating multivariate densities of max-stable fields with precision $\varepsilon $ at a computational cost of order $O\left( \varepsilon ^{-2}\log \log \log \left( 1/\varepsilon \right) \right) $.

We show that several machine learning estimators, including square-root LASSO (Least Absolute Shrinkage and Selection) and regularized logistic regression can be represented as solutions to distributionally robust optimization (DRO) problems. The associated uncertainty regions are based on suitably defined Wasserstein distances. Hence, our representations allow us to view regularization as a result of introducing an artificial adversary that perturbs the empirical distribution to account for out-of-sample effects in loss estimation. In addition, we introduce RWPI (Robust Wasserstein Profile Inference), a novel inference methodology which extends the use of methods inspired by Empirical Likelihood to the setting of optimal transport costs (of which Wasserstein distances are a particular case). We use RWPI to show how to optimally select the size of uncertainty regions, and as a consequence, we are able to choose regularization parameters for these machine learning estimators without the use of cross validation. Numerical experiments are also given to validate our theoretical findings.

We propose a novel framework for analyzing convergence rates of stochastic optimization algorithms with adaptive step sizes. This framework is based on analyzing properties of an underlying generic stochastic process, in particular by deriving a bound on the expected stopping time of this process. We utilize this framework to analyze the bounds on expected global convergence rates of a stochastic variant of a traditional trust region method, introduced in \cite{ChenMenickellyScheinberg2014}. While traditional trust region methods rely on exact computations of the gradient, Hessian and values of the objective function, this method assumes that these values are available up to some dynamically adjusted accuracy. Moreover, this accuracy is assumed to hold only with some sufficiently large, but fixed, probability, without any additional restrictions on the variance of the errors. This setting applies, for example, to standard stochastic optimization and machine learning formulations. Improving upon the analysis in \cite{ChenMenickellyScheinberg2014}, we show that the stochastic process defined by the algorithm satisfies the assumptions of our proposed general framework, with the stopping time defined as reaching accuracy $\|\nabla f(x)\|\leq \epsilon$. The resulting bound for this stopping time is $O(\epsilon^{-2})$, under the assumption of sufficiently accurate stochastic gradient, and is the first global complexity bound for a stochastic trust-region method. Finally, we apply the same framework to derive second order complexity bound under some additional assumptions., Comment: 22 pages. arXiv admin note: text overlap with arXiv:1504.04231

We present the first algorithm that samples $\max_{n\geq0}\{S_{n}-n^{\alpha}\},$ where $S_n$ is a mean zero random walk, and $n^{\alpha}$ with $\alpha\in(1/2,1)$ defines a nonliner boundary. We show that our algorithm has finite expected running time. We also apply the algorithm to construct the first exact simulation method for the steady-state departure process of a $GI/GI/\infty$ queue where the service time distribution has infinite mean.

We consider the random field M(t)=\sup_{n\geq 1}\big\{-\log A_{n}+X_{n}(t)\big\}\,,\qquad t\in T\, for a set $T\subset \mathbb{R}^{m}$, where $(X_{n})$ is an iid sequence of centered Gaussian random fields on $T$ and $00$, samples $M(t_{1}),\ldots ,M(t_{d})$ with complexity $o(c(d)\,d^{\epsilon })$. Moreover, if $X_{n}$ has an a.s. converging series representation, then $M$ can be a.s. approximated with error $\delta $ uniformly over $T$ and with complexity $O(1/(\delta \log (1/\delta ))^{1/\alpha })$, where $\alpha $ relates to the H\"{o}lder continuity exponent of the process $X_{n}$ (so, if $X_{n}$ is Brownian motion, $\alpha =1/2$).

Let $X$ be a L\'evy process with regularly varying L\'evy measure $\nu$. We obtain sample-path large deviations for scaled processes $\bar X_n(t) \triangleq X(nt)/n$ and obtain a similar result for random walks. Our results yield detailed asymptotic estimates in scenarios where multiple big jumps in the increment are required to make a rare event happen; we illustrate this through detailed conditional limit theorems. In addition, we investigate connections with the classical large deviations framework. In that setting, we show that a weak large deviation principle (with logarithmic speed) holds, but a full large deviation principle does not hold.

We present a novel inference approach that we call Sample Out-of-Sample (or SOS) inference. The approach can be used widely, ranging from semi-supervised learning to stress testing, and it is fundamental in the application of data-driven Distributionally Robust Optimization (DRO). Our method enables measuring the impact of plausible out-of-sample scenarios in a given performance measure of interest, such as a financial loss. The methodology is inspired by Empirical Likelihood (EL), but we optimize the empirical Wasserstein distance (instead of the empirical likelihood) induced by observations. From a methodological standpoint, our analysis of the asymptotic behavior of the induced Wasserstein-distance profile function shows dramatic qualitative differences relative to EL. For instance, in contrast to EL, which typically yields chi-squared weak convergence limits, our asymptotic distributions are often not chi-squared. Also, the rates of convergence that we obtain have some dependence on the dimension in a non-trivial way but remain controlled as the dimension increases.

This paper deals with the problem of quantifying the impact of model misspecification when computing general expected values of interest. The methodology that we propose is applicable in great generality, in particular, we provide examples involving path dependent expectations of stochastic processes. Our approach consists in computing bounds for the expectation of interest regardless of the probability measure used, as long as the measure lies within a prescribed tolerance measured in terms of a flexible class of distances from a suitable baseline model. These distances, based on optimal transportation between probability measures, include Wasserstein's distances as particular cases. The proposed methodology is well-suited for risk analysis, as we demonstrate with a number of applications. We also discuss how to estimate the tolerance region non-parametrically using Skorokhod-type embeddings in some of these applications.

We study distributional robustness in the context of Extreme Value Theory (EVT). We provide a data-driven method for estimating extreme quantiles in a manner that is robust against incorrect model assumptions underlying the application of the standard Extremal Types Theorem. Typical studies in distributional robustness involve computing worst case estimates over a model uncertainty region expressed in terms of the Kullback-Leibler discrepancy. We go beyond standard distributional robustness in that we investigate different forms of discrepancies, and prove rigorous results which are helpful for understanding the role of a putative model uncertainty region in the context of extreme quantile estimation. Finally, we illustrate our data-driven method in various settings, including examples showing how standard EVT can significantly underestimate quantiles of interest.

We provide the first perfect sampling algorithm for a Generalized Jackson Network of FIFO queues under arbitrary topology and non-Markovian assumptions on the input of the network. We assume (in addition to stability) that the interarrival and service times of customers have finite moment generating function in a neighborhood of the origin, and the interarrival times have unbounded support.

We provide the first rate of convergence analysis for RBM as the dimension grows under natural uniformity conditions. In particular, if the underlying routing matrix is uniformly contractive, uniform stability of the drift vector holds, and the variances of the underlying Brownian Motion (BM) are bounded, then we show that the RBM converges exponentially fast to stationarity with a relaxation time of order $O(d^4\log(d)^2)$ as $d\to\infty$.

Using a result of Blanchet and Wallwater (2015: Exact sampling of stationary and time-reversed queues. ACM TOMACS, 25, 26) for exactly simulating the maximum of a negative drift random walk queue endowed with independent and identically distributed (iid) increments, we extend it to a multi-dimensional setting and then we give a new algorithm for simulating exactly the stationary distribution of a first-in-first-out (FIFO) multi-server queue in which the arrival process is a general renewal process and the service times are iid; the FIFO GI/GI/c queue with 2 \le c < 1. Our method utilizes dominated coupling from the past (DCFP) as well as the Random Assignment (RA) discipline, and complements the earlier work in which Poisson arrivals were assumed, such as the recent work of Connor and Kendall (2015: Perfect simulation of M/G/c queues. Advances in Applied Probability, 47, 4). We also consider the models in continuous-time and show that with mild further assumptions, the exact simulation of those stationary distributions can also be achieved. We also give, using our FIFO algorithm, a new exact simulation algorithm for the stationary distribution of the infinite server case, the GI/GI/\infty model. Finally, we even show how to handle Fork-Join queues, in which each arriving customer brings c jobs, one for each server., Comment: 32 pages, 4 figures

We introduce the first class of perfect sampling algorithms for the steady-state distribution of multi-server queues with general interarrival time and service time distributions. Our algorithm is built on the classical dominated coupling from the past protocol. In particular, we use a coupled multi-server vacation system as the upper bound process and develop an algorithm to simulate the vacation system backwards in time from stationarity at time zero. The algorithm has finite expected termination time with mild moment assumptions on the interarrival time and service time distributions.

If the inter-arrival time distribution of a renewal process is regularly varying with index $\alpha\in\left( 0,1\right) $ (i.e. the inter-arrival times have infinite mean) and if $A\left( t\right) $ is the associated age process at time $t$. Then we show that if $C\left( t\right) $ is the length of the current cycle at time $t$, \[ A\left( t\right) /C\left( t\right) \Rightarrow U^{1/\alpha}, \] where $U$ is $U\left( 0,1\right) $. This extends a classical result in renewal theory in the finite mean case which indicates that the limit is $U\left( 0,1\right) $., Comment: 3 pages

An algorithm for estimating quasi-stationary distribution of finite state space Markov chains has been proven in a previous paper. Now this paper proves a similar algorithm that works for general state space Markov chains under very general assumptions., Comment: The current version requires substantial re-writing to make it suitable for a stand-alone submission. It would require approximately half a year of work

We obtain asymptotic bounds for the tail distribution of steady-state waiting time in a two server queue where each server processes incoming jobs at a rate equal to the rate of their arrivals (that is, the half-loaded regime). The job sizes are taken to be regularly varying. When the incoming jobs have finite variance, there are basically two types of effects that dominate the tail asymptotics. While the quantitative distinction between these two manifests itself only in the slowly varying components, the two effects arise from qualitatively very different phenomena (arrival of one extremely big job (or) two big jobs). Then there is a phase transition that occurs when the incoming jobs have infinite variance. In that case, only one of these effects dominate the tail asymptotics, the one involving arrival of one extremely big job.

We present the first exact simulation method for multidimensional reflected Brownian motion (RBM). Exact simulation in this setting is challenging because of the presence of correlated local-time-like terms in the definition of RBM. We apply recently developed so-called $\varepsilon-$strong simulation techniques (also known as Tolerance-Enforced Simulation) which allow us to provide a piece-wise linear approximation to RBM with $\varepsilon $ (deterministic) error in uniform norm. A novel conditional acceptance/rejection step is then used to eliminate the error. In particular, we condition on a suitably designed information structure so that a feasible proposal distribution can be applied.

We provide the first algorithm that under minimal assumptions allows to simulate the stationary waiting-time sequence of a single-server queue backwards in time, jointly with the input processes of the queue (inter-arrival and service times). The single-server queue is useful in applications of DCFTP (Dominated Coupling From The Past), which is a well known protocol for simulation without bias from steady-state distributions. Our algorithm terminates in finite time assuming only finite mean of the inter-arrival and service times. In order to simulate the single-server queue in stationarity until the first idle period in finite expected termination time we require the existence of finite variance. This requirement is also necessary for such idle time (which is a natural coalescence time in DCFTP applications) to have finite mean. Thus, in this sense, our algorithm is applicable under minimal assumptions., Comment: 30 pages, 3 figures, Journal

Consider a multidimensional diffusion process $X=\{X\left(t\right) :t\in\lbrack0,1]\}$. Let $\varepsilon>0$ be a \textit{deterministic}, user defined, tolerance error parameter. Under standard regularity conditions on the drift and diffusion coefficients of $X$, we construct a probability space, supporting both $X$ and an explicit, piecewise constant, fully simulatable process $X_{\varepsilon}$ such that \[ \sup_{0\leq t\leq1}\left\Vert X_{\varepsilon}\left(t\right) -X\left(t\right) \right\Vert_{\infty}<\varepsilon \] with probability one. Moreover, the user can adaptively choose $\varepsilon^{\prime}\in\left(0,\varepsilon\right) $ so that $X_{\varepsilon^{\prime}}$ (also piecewise constant and fully simulatable) can be constructed conditional on $X_{\varepsilon}$ to ensure an error smaller than $\varepsilon^{\prime}>0$ with probability one. Our construction requires a detailed study of continuity estimates of the Ito map using Lyon's theory of rough paths. We approximate the underlying Brownian motion, jointly with the L\'{e}vy areas with a deterministic $\varepsilon$ error in the underlying rough path metric.

We study a new technique for the asymptotic analysis of heavy-tailed systems conditioned on large deviations events. We illustrate our approach in the context of ruin events of multidimensional regularly varying random walks. Our approach is to study the Markov process described by the random walk conditioned on hitting a rare target set. We construct a Markov chain whose transition kernel can be evaluated directly from the increment distribution of the associated random walk. This process is shown to approximate the conditional process of interest in total variation. Then, by analyzing the approximating process, we are able to obtain asymptotic conditional joint distributions and a conditional functional central limit theorem of several objects such as the time until ruin, the whole random walk prior to ruin, and the overshoot on the target set. These types of joint conditional limit theorems have been obtained previously in the literature only in the one dimensional case. In addition to using different techniques, our results include features that are qualitatively different from the one dimensional case. For instance, the asymptotic conditional law of the time to ruin is no longer purely Pareto as in the multidimensional case., Comment: Published in at http://dx.doi.org/10.3150/12-BEJ492 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

This paper studies a method, which has been proposed in the Physics literature by [8, 7, 10], for estimating the quasi-stationary distribution. In contrast to existing methods in eigenvector estimation, the method eliminates the need for explicit transition matrix manipulation to extract the principal eigenvector. Our paper analyzes the algorithm by casting it as a stochastic approximation algorithm (Robbins-Monro) [23, 16]. In doing so, we prove its convergence and obtain its rate of convergence. Based on this insight, we also give an example where the rate of convergence is very slow. This problem can be alleviated by using an improved version of the algorithm that is given in this paper. Numerical experiments are described that demonstrate the effectiveness of this improved method.

We present the first class of perfect sampling (also known as exact simulation) algorithms for the steady-state distribution of non-Markovian loss networks. We use a variation of Dominated Coupling From The Past for which we simulate a stationary infinite server queue backwards in time and analyze the running time in heavy traffic. In particular, we are able to simulate stationary renewal marked point processes in unbounded regions. We use the infinite server queue as an upper bound process to simulate loss systems. The running time analysis of our perfect sampling algorithm for loss systems is performed in the Quality-Driven (QD) and the Quality-and-Efficiency-Driven regimes. In both cases, we show that our algorithm achieves sub-exponential complexity as both the number of servers and the arrival rate increase. Moreover, in the QD regime, our algorithm achieves a nearly optimal rate of convergence.

We derive a continuous time model for the joint evolution of the mid price and the bid-ask spread from a multiscale analysis of the whole limit order book (LOB) dynamics. We model the LOB as a multiclass queueing system and perform our asymptotic analysis using stylized features observed empirically. We argue that in the asymptotic regime supported by empirical observations the mid price and bid-ask-spread can be described using only certain parameters of the book (not the whole book itself). Our limit process is characterized by reflecting behavior and state-dependent jumps. Our analysis allows to explain certain characteristics observed in practice such as: the connection between power-law decaying tails in the volumes of the order book and the returns, as well as statistical properties of the long-run spread distribution.

Given a marked renewal point process (assuming that the marks are i.i.d.) we say that an unbounded region is stable if it contains finitely many points of the point process with probability one. In this paper we provide algorithms that allow to sample these finitely many points efficiently. We explain how exact simulation of the steady-state measure valued state descriptor of the infinite server queue follows as a simple corollary of our algorithms. We provide numerical evidence supporting that our algorithms are not only theoretically sound but also practical. Finally, having simulation optimization in mind, we also apply our results to gradient estimation of steady-state performance measures.

Based on a weak convergence argument, we provide a necessary and sufficient condition that guarantees that a nonnegative local martingale is indeed a martingale. Typically, conditions of this sort are expressed in terms of integrability conditions (such as the well-known Novikov condition). The weak convergence approach that we propose allows to replace integrability conditions by a suitable tightness condition. We then provide several applications of this approach ranging from simplified proofs of classical results to characterizations of processes conditioned on first passage time events and changes of measures for jump processes., Comment: 15 pages, added a new example on sampling Ornstein-Uhlenbeck processes, conditioned on rare first passage time events