Search

While permutation tests and bootstraps have very wide-ranging application, both share a common potential drawback: as data-intensive resampling methods, both can be runtime prohibitive when applied to large- or even medium-sized data samples drawn from large datasets. The data explosion over the past few decades has made this a common occurrence, and it highlights the increasing need for faster, and more efficient and scalable, permutation test and bootstrap algorithms. Seven bootstrap and six permutation test algorithms coded in SAS (the largest privately owned software firm globally) are compared herein. The fastest algorithms (‘OPDY’ for the bootstrap, ‘OPDN’ for permutation tests) are new, use no modules beyond Base SAS, and achieve speed increases orders of magnitude faster than the relevant ‘built-in’ SAS procedures (OPDY is over 200× faster than Proc SurveySelect; OPDN is over 240× faster than Proc SurveySelect, over 350× faster than NPAR1WAY (which crashes on datasets less than a 10th the size OPDN can handle), and over 720× faster than Proc Multtest). OPDY also is much faster than hashing, which crashes on datasets smaller—sometimes by orders of magnitude—than OPDY can handle. OPDY is easily generalizable to multivariate regression models, and OPDN, which uses an extremely efficient draw-by-draw random-sampling-without-replacement algorithm, can use virtually any permutation statistic, so both have a very wide range of application. And the time complexity of both OPDY and OPDN is sublinear, making them not only the fastest but also the only truly scalable bootstrap and permutation test algorithms, respectively, in SAS. WIREs Comput Stat 2013. doi: 10.1002/wics.1266 For further resources related to this article, please visit the WIREs website.

Across many types of risk (e.g. insurance risk, credit risk, the risk of catastrophic loss, enterprise-level portfolio risk, and operational risk) researchers and practitioners commonly estimate the size of rare but very impactful loss events using the Compound Loss Distribution (CLD) framework. The CLD combines an estimated frequency distribution, representing the number of losses that can occur during a specified time period, with an estimated severity distribution, representing the magnitudes of these losses, to obtain an estimated compound loss distribution (CLD). The CLD allows researchers to extrapolate beyond sparse extant data to ‘fill in’ the far right tail of the loss distribution to more reliably calculate selected risk metrics such as extreme Value-at-Risk (‘VaR’ is simply the value (quantile) associated with a large percentile of the distribution, such as 99.9%tile or higher). However, under real world conditions where loss data samples often are small to moderately sized and severity distributions typically are heavy-tailed, extreme VaR is a convex function of the key severity parameters, so regardless of the choice of (unbiased) estimators used to estimate these parameters, this convexity will upwardly bias the VaR estimate due to Jensen’s inequality (see Jensen, 1906). The magnitude of this bias often is very material, sometimes even multiples of the true value of VaR, and this effect holds even when the model is based on a ‘single’ rather than a ‘compound’ loss distribution (i.e. assuming just a constant number of losses and no frequency distribution). Surprisingly, this convexity-induced quantile bias has been almost entirely overlooked in the relevant literatures despite an intense focus during the past dozen years on the estimation of very large amounts of operational risk capital in the banking industry, which has used this exact CLD model to estimate extreme VaR. We discuss herein some of the reasons for this oversight, and propose a straightforward estimator that effectively neutralizes the systematic upward bias while simultaneously notably increasing the precision and robustness of the VaR estimate. We do this within a framework that identifies, isolates, and properly treats the distinct effects of each source of error on the ultimate VaR estimate: approximation error, model error, and estimation (sampling) error. Although this has not been done previously in the literature, it is absolutely necessary to achieve our ultimate goal here, which is to obtain reasonably accurate, precise, and robust estimates of extreme VaR when using either compound or single loss distributions.

An original algorithm is presented that generates both restricted integer compositions and restricted integer partitions that can be constrained simultaneously by (a) upper and lower bounds on the number of summands (“parts”) allowed, and (b) upper and lower bounds on the values of those parts. The algorithm can implement each constraint individually, or no constraints to generate unrestricted sets of integer compositions or partitions. The algorithm is recursive, based directly on very fundamental mathematical constructs, and given its generality, reasonably fast with good time complexity. A general, closed form solution to the open problem of counting the number of integer compositions doubly restricted in this manner also is presented; its formulaic link to an analogous solution for counting doubly-restricted integer partitions is shown to mirror the algorithmic link between these two objects.

We present an easily implemented, fast, and accurate method for approximating extreme quantiles of compound loss distributions (frequency and severity) as are commonly used in insurance and operational risk capital models. The Interpolated Single Loss Approximation (ISLA) of Opdyke (2014) is based on the widely used Single Loss Approximation (SLA) of Degen (2010), and maintains two important advantages over its competitors: first, ISLA correctly accounts for a discontinuity in SLA that otherwise can systematically and notably bias the quantile (capital) approximation under conditions of both finite and infinite mean. Secondly, because it is based on a closed-form approximation, ISLA maintains the notable speed advantages of SLA over other methods requiring algorithmic looping (e.g. fast Fourier transform or Panjer recursion). Speed is important when simulating many quantile (capital) estimates, as is so often required in practice, and essential when simulations of simulations are needed (e.g. some power studies). The modified ISLA (MISLA) presented herein increases the range of application across the severity distributions most commonly used in these settings, and it is tested against extensive Monte Carlo simulation (one billion years’ worth of losses) and the best competing method (the perturbative expansion (PE2) of Hernandez et al., 2014) using twelve heavy-tailed severity distributions, some of which are truncated. MISLA is shown to be comparable to PE2 in terms of both speed and accuracy, and it is arguably more straightforward to implement for the majority of Advanced Measurement Approaches (AMA) banks that are already using SLA (and failing to take into account its biasing discontinuity).

Until recently, since Jobson and Korkie (1981), derivations of the asymptotic distribution of the Sharpe ratio that are practically useable for generating confidence intervals or for conducting one- and two-sample hypothesis tests have relied on the restrictive, and now widely refuted, assumption of normally distributed returns. This paper presents an easily implemented formula for the asymptotic distribution that is valid under very general conditions — stationary and ergodic returns — thus permitting time-varying conditional volatilities, serial correlation, and other non-iid returns behaviour. It is consistent with that of Christie (2005), but it is more mathematically tractable and intuitive, and simple enough to be used in a spreadsheet. Also generalised beyond the normality assumption is the small sample bias adjustment presented in Christie (2005). A thorough simulation study examines the finite sample behaviour of the derived one- and two-sample estimators under the realistic returns conditions of concurrent leptokurtosis, asymmetry, and importantly (for the two-sample estimator), strong positive correlation between funds, the effects of which have been overlooked in previous studies. The two-sample statistic exhibits reasonable level control and good power under these real-world conditions. This makes its application to the ubiquitous Sharpe ratio rankings of mutual funds and hedge funds very useful, since the implicit pairwise comparisons in these orderings have little inferential value on their own. Using actual returns data from 20 mutual funds, the statistic yields statistically significant results for many such pairwise comparisons of the ranked funds. It should be useful for other purposes as well, wherever Sharpe ratios are used in performance assessment.

Since the enactment of the Telecommunications Act of 1996 (1996), extensive expert testimony has justified use of the ‘modified’ t statistic (Brownie et al. Biometrics 46 (1990) 259–266) for performing two-sample hypothesis tests on Bell companies’ CLEC and ILEC performance measurement data. This paper demonstrates how key statistical claims made about the ‘modified’ t in this setting are false, leading not only to incorrect inferences as it currently is being used, but also to the possible undermining of the primary stated objective of the Act—the promotion of competition in the newly deregulated local telephone service markets. A simulation study provides strong evidence for the use of several other easily-implemented statistical procedures in this context; they should be useful in other settings as well.

While the distribution-free nature of permutation tests makes them the most appropriate method for hypothesis testing under a wide range of conditions, their computational demands can be runtime prohibitive, especially if samples are not very small and/or many tests must be conducted (e.g. all pairwise comparisons). This paper presents statistical code that performs continuous-data permutation tests under such conditions very quickly – often more than an order of magnitude faster than widely available commercial alternatives when many tests must be performed and some of the sample pairs contain a large sample. Also presented is an efficient method for obtaining a set of permutation samples containing no duplicates, thus maximizing the power of a pairwise permutation test under a conventional Monte Carlo approach with negligible runtime cost (well under 1% when runtimes are greatest). For multiple comparisons, the code is structured to provide an additional speed premium, making permutation-style p-value adjustments practical to use with permutation test p-values (although for relatively few comparisons at a time). “No-replacement” sampling also provides a power gain for such multiple comparisons, with similarly negligible runtime cost.

The largest US banks are required by regulatory mandate to estimate the operational risk capital they must hold using an Advanced Measurement Approach (AMA) as defined by the Basel II/III Accords. Most use the Loss Distribution Approach (LDA) which defines the aggregate loss distribution as the convolution of a frequency and a severity distribution representing the number and magnitude of losses, respectively. Estimated capital is a Value-at-Risk (99.9th percentile) estimate of this annual loss distribution. In practice, the severity distribution drives the capital estimate, which is essentially a very high quantile of the estimated severity distribution. Unfortunately, because the relevant severities are heavy-tailed AND the quantiles being estimated are so high, VaR always appears to be a convex function of the severity parameters, causing all widely-used estimators to generate biased capital estimates (apparently) due to Jensen's Inequality. The observed capital inflation is sometimes enormous, even at the unit-of-measure (UoM) level (even billions USD). Herein I present an estimator of capital that essentially eliminates this upward bias. The Reduced-bias Capital Estimator (RCE) is more consistent with the regulatory intent of the LDA framework than implementations that fail to mitigate this bias. RCE also notably increases the precision of the capital estimate and consistently increases its robustness to violations of the i.i.d. data presumption (which are endemic to operational risk loss event data). So with greater capital accuracy, precision, and robustness, RCE lowers capital requirements at both the UoM and enterprise levels, increases capital stability from quarter to quarter, ceteris paribus, and does both while more accurately and precisely reflecting regulatory intent. RCE is straightforward to implement using any major statistical software package.

The largest US banks and Systemically Important Financial Institutions are required by regulatory mandate to estimate the operational risk capital they must hold using an Advanced Measurement Approach (AMA) as defined by the Basel II/III Accords. Most of these institutions use the Loss Distribution Approach (LDA) which defines the aggregate loss distribution as the convolution of a frequency distribution and a severity distribution representing the number and magnitude of losses, respectively. Capital is a Value-at-Risk estimate of this annual loss distribution (i.e. the quantile corresponding to the 99.9%tile, representing a one-in-a-thousand-year loss, on average). In practice, the severity distribution drives the capital estimate, which is essentially a very large quantile of the estimated severity distribution. Unfortunately, when using LDA with any of the widely used severity distributions (i.e. heavy-tailed, skewed distributions), all unbiased estimators of severity distribution parameters appear to generate biased capital estimates due to Jensen’s Inequality: VaR always appears to be a convex function of these severities’ parameter estimates because the (severity) quantile being estimated is so large and the severities are heavy-tailed. The resulting bias means that capital requirements always will be overstated, and this inflation is sometimes enormous (sometimes even billions of dollars at the unit-of-measure level). Herein I present an estimator of capital that essentially eliminates this upward bias when used with any commonly used severity parameter estimator. The Reduced-bias Capital Estimator (RCE), consequently, is more consistent with regulatory intent regarding the responsible implementation of the LDA framework than other implementations that fail to mitigate, if not eliminate this bias. RCE also notably increases the precision of the capital estimate and consistently increases its robustness to violations of the i.i.d. data presumption (which are endemic to operational risk loss event data). So with greater capital accuracy, precision, and robustness, RCE lowers capital requirements at both the unit-of-measure and enterprise levels, increases capital stability from quarter to quarter, ceteris paribus, and does both while more accurately and precisely reflecting regulatory intent. RCE is straightforward to explain, understand, and implement using any major statistical software package.

In operational risk measurement, the estimation of severity distribution parameters is the main driver of capital estimates, yet this remains a non-trivial challenge for many reasons. Maximum likelihood estimation (MLE) does not adequately meet this challenge because of its well-documented non-robustness to modest violations of idealized textbook model assumptions, specifically that the data are independent and identically distributed (i.i.d.), which is clearly violated by operational loss data. Yet even under i.i.d. data, capital estimates based on MLE are biased upwards, sometimes dramatically, due to Jensen’s inequality. This overstatement of the true risk profile increases as the heaviness of the severity distribution tail increases, so dealing with data collection thresholds by using truncated distributions, which have thicker tails, increases MLE-related capital bias considerably. Truncation also augments correlation between a distribution’s parameters, and this exacerbates the non-robustness of MLE. This paper derives influence functions for MLE for a number of severity distributions, both truncated and not, to analytically demonstrate its non-robustness and its sometimes counterintuitive behavior under truncation. Empirical influence functions are then used to compare MLE against robust alternatives such as the Optimally Bias-Robust Estimator (OBRE) and the Cramer-von Mises (CvM) estimator. The ultimate focus, however, is on the economic and regulatory capital estimates generated by these three estimators. The mean adjusted single-loss approximation (SLA) is used to translate these parameter estimates into Value-at-Risk (VaR) based estimates of regulatory and economic capital. The results show that OBRE estimators are very promising alternatives to MLE for use with actual operational loss event data, whether truncated or not, when the ultimate goal is to obtain accurate (non-biased) and robust capital estimates.