Search

An anytime valid sequential test permits us to peek at observations as they arrive. This means we can stop, continue or adapt the testing process based on the current data, without invalidating the inference. Given a maximum number of observations $N$, one may believe that this benefit must be paid for in terms of power when compared to a conventional test that waits until all $N$ observations have arrived. Our key contribution is to show that this is false: for any valid test based on $N$ observations, we derive an anytime valid sequential test that matches it after $N$ observations. In addition, we show that the value of the sequential test before a rejection is attained can be directly used as a significance level for a subsequent test. We illustrate this for the $z$-test. There, we find that the current state-of-the-art based on log-optimal $e$-values can be obtained as a special limiting case that replicates a $z$-test with level $\alpha \to 0$ as $N \to \infty$.

A fundamental assumption of classical hypothesis testing is that the significance threshold $\alpha$ is chosen independently from the data. The validity of confidence intervals likewise relies on choosing $\alpha$ beforehand. We point out that the independence of $\alpha$ is guaranteed in practice because, in most fields, there exists one standard $\alpha$ that everyone uses -- so that $\alpha$ is automatically independent of everything. However, there have been recent calls to decrease $\alpha$ from $0.05$ to $0.005$. We note that this may lead to multiple accepted standard thresholds within one scientific field. For example, different journals may require different significance thresholds. As a consequence, some researchers may be tempted to conveniently choose their $\alpha$ based on their p-value. We use examples to illustrate that this severely invalidates hypothesis tests, and mention some potential solutions.

Testing has developed into the fundamental statistical framework for falsifying hypotheses. Unfortunately, tests are binary in nature: a test either rejects a hypothesis or not. Such binary decisions do not reflect the reality of many scientific studies, which often aim to present the evidence against a hypothesis and do not necessarily intend to establish a definitive conclusion. We propose a continuous generalization of a test, which we use to continuously measure the evidence against a hypothesis. Such a continuous test can be viewed as a continuous and non-randomized interpretation of the classical `randomized test'. This offers the benefits of a randomized test, without the downsides of external randomization. Another interpretation is as a literal measure, which measures the amount of binary tests that reject the hypothesis. Our work unifies classical testing and the recently proposed $e$-values: $e$-values bounded to $[0, 1/\alpha]$ are continuously interpreted size $\alpha$ randomized tests. Choosing $\alpha = 0$ yields the regular $e$-value, which we use to define a level 0 continuous test. Moreover, we generalize the traditional notion of power by using generalized means. This produces a framework that contains both classical Neyman-Pearson optimal testing and log-optimal $e$-values, as well as a continuum of other options. The traditional $p$-value appears as the reciprocal of a generally invalid continuous test. In an illustration in a Gaussian location model, we find that optimal continuous tests are of a beautifully simple form.

In an optimization problem, the quality of a candidate solution can be characterized by the optimality gap. For most stochastic optimization problems, this gap must be statistically estimated. We show that standard estimators are optimistically biased for risk-averse problems, which compromises the statistical guarantee on the optimality gap. We introduce estimators for risk-averse problems that do not suffer from this bias. Our method relies on using two independent samples, each estimating a different component of the optimality gap. Our approach extends a broad class of methods for estimating the optimality gap from the risk-neutral case to the risk-averse case, such as the multiple replications procedure and its one- and two-sample variants. Our approach can further make use of existing bias and variance reduction techniques.

An unfortunate feature of traditional hypothesis testing is the necessity to pre-specify a significance level $\alpha$ to bound the size of the test: its probability to falsely reject the hypothesis. Indeed, a data-dependent selection of $\alpha$ would generally distort the size, possibly making it larger than the selected level $\alpha$. We develop post-hoc $\alpha$ hypothesis testing, which guarantees that there is no such size distortion in expectation, even if the level $\alpha$ is arbitrarily selected based on the data. Unlike regular $p$-values, resulting 'post-hoc $p$-values' allow us to 'reject at level $p$' and still provide this guarantee. Moreover, they can easily be combined since the product of independent post-hoc $p$-values is also a post-hoc $p$-value. Interestingly, we find that $p$ is a post-hoc $p$-value if and only if $1/p$ is an $e$-value, a recently introduced measure of evidence. This reveals what $e$-values truly are in the context of a hypothesis testing problem. Post-hoc $\alpha$ hypothesis testing eliminates the need for standardized levels such as $\alpha = 0.05$, which takes away incentives for $p$-hacking and contributes to solving the file-drawer problem.

We study post-hoc ($e$-value-based) and post-hoc anytime valid inference for testing exchangeability and general group invariance. Our methods satisfy a generalized Type I error control that permits a data-dependent selection of both the number of observations $n$ and the significance level $\alpha$. We derive a simple analytical expression for all exact post-hoc valid $p$-values for group invariance, which allows for a flexible plug-in of the test statistic. For post-hoc anytime validity, we derive sequential $p$-processes by multiplying post-hoc $p$-values. In sequential testing, it is key to specify how the number of observations may depend on the data. We propose two approaches, and show how they nest existing efforts. To construct good post-hoc $p$-values, we develop the theory of likelihood ratios for group invariance, and generalize existing optimality results. These likelihood ratios turn out to exist in different flavors depending on which space we specify our alternative. We illustrate our methods by testing against a Gaussian location shift, which yields an improved optimality result for the $t$-test when testing sphericity, connections to the softmax function when testing exchangeability, and an improved method for testing sign-symmetry.

It is conventionally believed that a permutation test should ideally use all permutations. If this is computationally unaffordable, it is believed one should use the largest affordable Monte Carlo sample or (algebraic) subgroup of permutations. We challenge this belief by showing we can sometimes obtain dramatically more power by using a tiny subgroup. As the subgroup is tiny, this simultaneously comes at a much lower computational cost. We exploit this to improve the popular permutation-based Westfall & Young MaxT multiple testing method. We study the relative efficiency in a Gaussian location model, and find the largest gain in high dimensions.

Non-parametric tests based on permutation, rotation or sign-flipping are examples of group-invariance tests. These tests test invariance of the null distribution under a set of transformations that has a group structure, in the algebraic sense. Such groups are often huge, which makes it computationally infeasible to test using the entire group. Hence, it is standard practice to test using a randomly sampled set of transformations from the group. This random sample still needs to be substantial to obtain good power and replicability. We improve upon this standard practice by using a well-designed subgroup of transformations instead of a random sample. The resulting subgroup-invariance test is still exact, as invariance under a group implies invariance under its subgroups. We illustrate this in a generalized location model and obtain more powerful tests based on the same number of transformations. In particular, we show that a subgroup-invariance test is consistent for lower signal-to-noise ratios than a test based on a random sample. For the special case of a normal location model and a particular design of the subgroup, we show that the power improvement is equivalent to the power difference between a Monte Carlo $Z$-test and a Monte Carlo $t$-test.

For a high-dimensional parameter of interest, tests based on quadratic statistics are known to have low power against subsets of the parameter space (henceforth, parameter subspaces). In addition, they typically involve an inverse covariance matrix which is difficult to estimate in high-dimensional settings. I simultaneously address these two issues by proposing a novel test statistic that is large in a conic parameter subspace of interest. This test statistic generalizes the Wald statistic and nests many well-known test statistics. For a given parameter subspace, the statistic is free of tuning parameters and suitable for high-dimensional settings if the subspace is sufficiently small. It can be computed using regularized linear regression, where the type of regularization and the regularization parameters are completely determined by the parameter subspace of interest. I illustrate the statistic on subspaces that consist of sparse or nearly-sparse vectors, for which the computation corresponds to $\ell_0$- and $\ell_1$-regularized regression, respectively.

This paper considers sparsity in linear regression under the restriction that the regression weights sum to one. We propose an approach that combines $\ell_0$- and $\ell_1$-regularization. We compute its solution by adapting a recent methodological innovation made by Bertsimas et al. (2016) for $\ell_0$-regularization in standard linear regression. In a simulation experiment we compare our approach to $\ell_0$-regularization and $\ell_1$-regularization and find that it performs favorably in terms of predictive performance and sparsity. In an application to index tracking we show that our approach can obtain substantially sparser portfolios compared to $\ell_1$-regularization while maintaining a similar tracking performance.

This paper considers the problem of testing many moment inequalities, where the number of moment inequalities ($p$) is possibly larger than the sample size ($n$). Chernozhukov et al. (2019) proposed asymptotic tests for this problem using the maximum $t$ statistic. We observe that such tests can have low power if multiple inequalities are violated. As an alternative, we propose novel randomization tests based on a maximum non-negatively weighted combination of $t$ statistics. We provide a condition guaranteeing size control in large samples. Simulations show that the tests control size in small samples ($n = 30$, $p = 1000$), and often has substantially higher power against alternatives with multiple violations than tests based on the maximum $t$ statistic.