Search

Hypothesis testing is a key part of empirical science and multiple testing as well as the combination of evidence from several tests are continued areas of research. In this article we consider the problem of combining the results of multiple hypothesis tests to i) test global hypotheses and ii) make marginal inference while controlling the k-FWER. We propose a new family of combination tests for joint hypotheses, called the 'Too Many, Too Improbable' (TMTI) statistics, which we show through simulation to have higher power than other combination tests against many alternatives. Furthermore, we prove that a large family of combination tests -- which includes the one we propose but also other combination tests -- admits a quadratic shortcut when used in a Closed Testing Procedure, which controls the FWER strongly. We develop an algorithm that is linear in the number of hypotheses for obtaining confidence sets for the number of false hypotheses among a collection of hypotheses and an algorithm that is cubic in the number of hypotheses for controlling the k-FWER for any k greater than one.

Datasets are sometimes divided into distinct subsets, e.g. due to multi-center sampling, or to variations in instruments, questionnaire item ordering or mode of administration, and the data analyst then needs to assess whether a joint analysis is meaningful. The Principal Component Analysis-based Data Structure Comparisons (PCADSC) tools are three new non-parametric, visual diagnostic tools for investigating differences in structure for two subsets of a dataset through covariance matrix comparisons by use of principal component analysis. The PCADCS tools are demonstrated in a data example using European Social Survey data on psychological well-being in three countries, Denmark, Sweden, and Bulgaria. The data structures are found to be different in Denmark and Bulgaria, and thus a comparison of for example mean psychological well-being scores is not meaningful. However, when comparing Denmark and Sweden, very similar data structures, and thus comparable concepts of well-being, are found. Therefore, inter-country comparisons are warranted for these countries.