Search

The article provides perspectives on p values, null hypothesis testing, and alternative techniques in light of modern robust statistical methods. Null hypothesis testing and p values can provide useful information provided they are interpreted in a sound manner, which includes taking into account insights and advances that have occurred during the past 50 years. There are, of course, limitations to what null hypothesis testing and p values reveal about data. But modern advances make it clear that there are serious limitations and concerns associated with conventional confidence intervals, standard Bayesian methods, and commonly used measures of effect size. Many of these concerns can be addressed using modern robust methods.

During the last half century, hundreds of papers published in statistical journals have documented general conditions where reliance on least squares regression and Pearson's correlation can result in missing even strong associations between variables. Moreover, highly misleading conclusions can be made, even when the sample size is large. There are, in fact, several fundamental concerns related to non–normality, outliers, heteroscedasticity, and curvature that can result in missing a strong association. Simultaneously, a vast array of new methods has been derived for effectively dealing with these concerns. The paper (i) reviews why least squares regression and classic inferential methods can fail, (ii) provides an overview of the many modern strategies for dealing with known problems, including some recent advances, and (iii) illustrates that modern robust methods can make a practical difference in our understanding of data. Included are some general recommendations regarding how modern methods might be used. Copyright © 2011 John Wiley & Sons, Ltd.

For two random variables, X and Y, let D = X - Y, and let θ x, θ y, and θ d be the corresponding medians. It is known that the Wilcoxon-Mann-Whitney test and its modern extensions do not test H0: θ x = θ y, but rather, they test H0: θ d = 0. The article deals with an extension of these methods to dependent groups, so the goal is not to make inferences about the median of the pairwise differences or even the median of the marginal distributions. Rather, if an individual is randomly sampled from the first group and another individual is randomly sampled from the second group, the goal is to make inferences about the median value of the typical difference. The goal is not to suggest that alternative methods be abandoned but rather to provide a perspective on how dependent groups compare that it is similar in spirit to the perspective provided by the Wilcoxon-Mann-Whitney test.

Consider the nonparametric regression model Y = m( X)+ τ( X)ε, where X and ε are independent random variables, ε has a median of zero and variance σ2, τ is some unknown function used to model heteroscedasticity, and m( X) is an unknown function reflecting some conditional measure of location associated with Y, given X. This article considers the problem of testing H0:τ = 1, the hypothesis that the error term is homoscedastic. Several methods were considered, two of which were found that control the probability of a Type I error well in simulations. One is fast from a computational point of view, and the other is based in part on a bootstrap method. Neither dominates in terms of power.

A commonly used method for comparing groups of individuals is the analysis of variance (ANOVA) F test. When the assumptions underlying the derivation of this test are true, its power, meaning its probability of detecting true differences among the groups, competes well with all other methods that might be used. But when these assumptions are false, its power can be relatively low. Many new statistical methods have been proposed—ones that are aimed at achieving about the same amount of power when the assumptions of the F test are true but which have the potential of high power in situations where the F test performs poorly. A brief summary of some relevant issues and recent developments is provided. Some related issues are discussed and implications for future research are described.

It is known that nonnormality, a heteroscedastic error term, or a nonlinearassociation can create serious practical problems when using the conventional analysis of covariance (ANCOVA) method. This article describes a simple ANCOVA method that allows heteroscedasticity, nonnormality, nonlinearity, and multiple covariates. When standard assumptions are met, all indications are that the power of the proposed method compares very favorably with that of the conventional technique.

This article considers the problem of comparing two independent groups in terms of some measure of location. It is well known that with Student's two-independent-sample t test, the actual level of significance can be well above or below the nominal level, confidence intervals can have inaccurate probability coverage, and power can be low relative to other methods. A solution to deal with heterogeneity is Welch's (1938) test. Welch's test deals with heteroscedasticity but can have poor power under arbitrarily small departures from normality. Yuen (1974) generalized Welch's test to trimmed means; her method provides improved control over the probability of a Type I error, but problems remain. Transformations for skewness improve matters, but the probability of a Type I error remains unsatisfactory in some situations. We find that a transformation for skewness combined with a bootstrap method improves Type I error control and probability coverage even if sample sizes are small.

Consider the regression model Y i = β0 + β1 X i1 +...+ β p X iP + λ( X i1,..., X ip)ε i, where the function λ is used to model heteroscedasticity. Assume that the ordinary least squares estimator is used. This note addresses the problem of testing H0:β j = 0 for each j ( j =1,..., p), with the goal of controlling the probability of at least one Type I error. Based on published results dealing with the special case p = 1, there is an obvious guess regarding a general strategy aimed at accomplishing this goal: use in part a modification of the percentile bootstrap method. This note reports simulation results indicating that a simpler approach performs well for p ≤ 2. In particular, use a basic (unmodified) percentile boot-strap method in conjunction with the Bonferroni inequality.

The Welch-James and Improved General Approximation tests were examined in between-subjects × within-subjects repeated measures designs for their rates of Type I error when data were nonnormal, nonspherical, and heterogeneous and when group sizes were unequal as well. The tests were computed with either least squares or robust estimators of central tendency and variability and assessed with critical values that were obtained either theoretically or through a bootstrapping method. Prior findings indicated that one could only obtain a robust test of the interaction effect with the Welch-James procedure when sample sizes were very large. This study’s results indicate that a robust test of the interaction effect can be obtained with reasonable sample sizes when the Welch-James test is computed with trimmed means and Winsorized covariance matrices.

This article reports some results on how the Alexander-Govern heteroscedastic analysis of variance procedure performs under non-normality. It is illustrated that this method can provide poor control over the probability of a Type I error and that there are situations in which power decreases as the the differences among the means get large. Included are some results on how the Alexander-Govern procedure might be improved using a particular form of the bootstrap method advocated by Westfall and Young. For comparative purposes, some new results on comparing trimmed means are reported as well.

1. J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. (Erlbaum, Hillsdale, NJ, 1988). This is the source of the system of power analysis described here; the power values and sam ple sizes of the illustrations derive from this book's tables. 2. J. Neyman and E.S. Pearson, On the use and interpretation of certain test criteria for purposes of statistical inference, Biometrika, 20A, 175-240, 263-294 (1928); J. Neyman and E.S. Pearson, On the problem of the most efficient tests of statistical hypotheses, Transactions of the Royal Society of London Series A, 231, 289-337 (1933). 3. J. Cohen, Things I have learned (so far), American Psychologist, 45, 1304-1312 (1990). 4. For an article-length treatment of sample size determination using the .80 convention and a = .01, .05, and .10, see J. Cohen, A power primer, Psychological Bulletin (in press). A useful alternative treatment is offered in H.C. Kraemer and S. Thie mann, How Many Subjects? Statistical Power Anal ysis in Research (Sage, Newbury Park, CA, 1987). 5. J. Cohen, The statistical power of abnormal social psychological research: A review, Journal of Abnormal and Social Psychology, 65, 145-153 (1962). 6. P. Sedlmeier and G. Gigerenzer, Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316 (1989).

Numerous authors suggest that the data gathered by investigators are not normal in shape. Accordingly, methods for assessing pairwise multiple comparisons of means with traditional statistics will frequently result in biased rates of Type I error and depressed power to detect effects. One solution is to obtain a critical value to assess statistical significance through bootstrap methods. The SAS system can be used to conduct step-down bootstrapped tests. The authors investigated this approach when data were neither normal in form nor equal in variability in balanced and unbalanced designs. They found that the step-down bootstrap method resulted in substantially inflated rates of error when variances and group sizes were negatively paired. Based on their results, and those reported elsewhere, the authors recommend that researchers should use trimmed means and Winsorized variances with a heteroscedastic test statistic. When group sizes are equal, the bootstrap procedure effectively controlled Type I error rates.

Several methods have been proposed for estimating the reliability of a criterion-referenced test. This paper describes and compares seven procedures which can be applied to the more general case of proficiency tests that are scored with latent structure models. Results suggest that the predictive estimate is the most accurate of the procedures.

Various school systems are developing proficiency tests which are conceptualized as representing a variety of skills with one or more items per skill. This paper discusses how certain recent technical advances might be extended to examine these tests. In contrast to previous analyses, errors at the item level are included; and it is shown that inclusion of these errors implies that a substantially longer test might be needed. One approach to this problem is described, and directions for future research are suggested.

In many situations in education and psychology it is desired to select from k binomial populations the one having the largest probability of success. For example, if a pass/fail type of test is administered at the end of an instructional program and if we have several methods of instruction from which to choose, it is desirable to choose the one which maximizes the probability of passing the test. This paper describes a two-stage procedure for accomplishing this goal. The solution is conservative in the sense that it is possible to determine the number of observations required so that the best population is chosen with probability at least P*, 2- k < P* < 1.

Under certain circumstances it is possible to apply latent structure models to estimate ζ, the proportion of skills an examinee has acquired among a domain of skills and to estimate β, the probability of guessing the correct response given that the examinee does not know. Models have also been formulated in which ζ represents the proportion of examinees who know a specific skill and again β is the probability of guessing. Macready and Dayton (1976) have used these models to determine the number of items required in deciding whether an examinee has acquired a particular skill and Wilcox (1979) has applied these models to arrive at a formula score which allows guessing to vary over the population of examinees. In this paper we examine a technical problem associated with estimating β and we also indicate situations for which this problem does not occur. It is shown, however, that this problem does not apply when using Wilcox's formula score. In addition, some new, explicit, maxi mum likelihood estimates of ζ and β are derived which may be applied when items are hierarchically related.

In recent years several discrete discriminate analysis procedures have been proposed which might be used in lieu of the more com mon multinomial solution. These procedures have not been com pared to one another and there is little or no information about how they perform on real data sets. Using three sets of real data, a com parison of these solutions are made using the actual versus the opti mal error rate. It is found in all three cases that the kernel method gives the most accurate results.

Consider an n-item multiple-choice test where it is decided that an examinee knows the answer if and only if he/she gives the correct response. The k out of n reliability of the test, Qk, is defined to be the probability that, for a randomly sampled examinee, at least k correct decisions are made about whether the examinee knows the answer to an item. The paper describes and illustrates how an extension of a recently proposed latent structure model can be used in conjunction with results in Sathe, Pradhan, and Shah (1980) to estimate upper and lower bounds on Qk. A method of empirically checking the model is discussed.

When determining how many items to include on a criterion-referenced test, practitioners must re solve various nonstatistical issues before a par ticular solution can be applied. A fundamental problem is deciding which of three true scores should be used. The first is based on the prob ability that an examinee is correct on a "typical" test item. The second is the probability of having acquired a typical skill among a domain of skills, and the third is based on latent trait models. Once a particular true score is settled upon, there are several perspectives that might be used to de termine test length. The paper reviews and critiques these solutions. Some new results are described that apply when latent structure models are used to esti mate an examinee's true score.

When analyzing the distractors of multiple-choice test items, it is sometimes desired to determine which of the distractors has a small probability of being chosen by a typical examinee. At present, this problem is handled in an informal manner. In particular, using an arbitrary number of examinees, the probabilities associated with the distractors are estimated and then sorted according to whether the estimated values are above or below a known constant p0 In this paper a more formal framework for solving this problem is described. The first portion of the paper considers the problem from the point of view of designing an experiment. The solution is based on a procedure similar to an indifference zone formulation of a ranking and election problem. A later section considers methods that might be employed in a retrospective study. Brief consideration is also given to how an analysis might proceed when a test item has been altered in some way.

When analyzing p, the squared multiple correlation coefficient, an investigator may be more interested in determining whether p is above or below a known constant rather than testing the hypothesis that p = 0. This paper gives the exact sample sizes required for answering this question when an indifference zone formulation of the problem is used. The results reported here also give a lower bound to the probability of making a correct decision when n observations are available.

For a specific achievement test item and a randomly selected examinee, let p be the probability of correctly determining whether the examinee knows the correct response. Various techniques have been proposed for estimating p. This paper describes and illustrates how results in the engineering literature on "k out of n system reliability" can be used to study and characterize tests based on the estimated values of p. In particular, we can empirically determine the minimum number of distractors required for multiple-choice tests. If we estimate p with an answer-until-correct scoring procedure, we can also determine the minimum number of examinees needed to be reasonably certain about whether y is less than or greater than some predetermined constant, where y = [UNKNOWN] pi and pi is the value of p for the ith item on an n-item test. In other words, we can determine whether the expected number of correct decisions on an n-item test is reasonably large.

Wilcox (1977) describes three probability models which characterize a single test item in terms of a population of examinees. The purpose of this note is to indicate that similar models can be derived which characterize a single examinee in terms of an item domain. A numerical illustration is given which shows how our models might be used to adjust observed scores when comparing examinees.

For some situations the beta-binomial distribution might be used to describe the marginal distribution of test scores for a particular population of examinees. To use this distribution it is necessary to estimate two parameters which characterize the beta-binomial model. One method of estimating these parameters is to approximate the maximum likelihood estimates using some appropriately chosen iterative technique. When the number of examinees is small, however, it is not clear that this method of estimation is justified since the iterative approximations might not converge to the maximum likelihood estimate. Using Monte Carlo techniques, this paper compares the Newton-Raphson approximation to the maximum likelihood estimate with several other possible procedures. It is found that the Newton-Raphson method should be used when it yields admissible (positive) results.

A problem of considerable importance in certain educational settings is determining how many items to include on a mastery test. Applying ranking and selection procedures, a solution is given which includes as a special case all existing single-stage, non-Bayesian solutions based on a strong true-score model. Illustrations are given for the binomial and compound binomial distributions.

Brennan's B is a generalized upper-lower item discrimination index, which was first published in 1972. It is interesting to observe that early work by Peirce on the relation between a predictor and an outcome has the same structure as Brennan's B.

When comparing k normal populations, an investigator might want to know the probability that the population with the largest population mean will have the largest sample mean. Put another way, what is the probability of correctly identifying the most effective treatment? The paper describes and illustrates methods of approximating this probability when the variances are unknown and possibly unequal. The results described here can also be used to measure the extent to which the populations differ for one another.

The paper considers the problem of selecting the t best of k normal populations and simultaneously determining whether the selected populations have a mean larger than a known standard. Illustrations are given for selecting the t best of k examinees when the binomial error model applies.

Recently it was suggested that the Bayes risk might be used to characterize tests. To conform to common practices about indexes, a rescaling of the Bayes risk was proposed. The motivation for this new coefficient, d, was to provide an index that has a large value when the Bayes risk is small and that has a value in the closed interval [0, 1]. However, since d might have a value outside this range, a modification of d is described which yields an index that always has a value between zero and one.

In various situations, a multiple-choice test item may have more than one correct alternative, and the goal is to determine how many correct alternatives an examinee actually knows. For a randomly sampled ex aminee, the validity of an item is defined as the prob ability of deciding that the examinee knows i correct alternatives, when in fact exactly i correct alternatives are known. This article describes how latent class models can be used to estimate this probability.

Answer-until-correct (AUC) tests have been in use for some time. Pressey (1950) pointed to their ad vantages in enhancing learning, and Brown (1965) proposed a scoring procedure for AUC tests that appears to increase reliability (Gilman & Ferry, 1972; Hanna, 1975). This paper describes a new scoring procedure for AUC tests that (1) makes it possible to determine whether guessing is at ran dom, (2) gives a measure of how "far away" guess ing is from being random, (3) corrects observed test scores for partial information, and (4) yields a mea sure of how well an item reveals whether an ex aminee knows or does not know the correct re sponse. In addition, the paper derives the optimal linear estimate (under squared-error loss) of true score that is corrected for partial information, as well as another formula score under the assumption that the Dirichlet-multinomial model holds. Once certain parameters are estimated, the latter formula score makes it possible to correct for partial infor mation using only the examinee's usual number- correct observed score. The importance of this for mula score is discussed. Finally, various statistical techniques are described that can be used to check the assumptions underlying the proposed scoring procedure.

The classical estimate of a binomial probability function is to estimate its mean in the usual manner and to substitute the results in the appropriate expression. Two alternative estimation procedures are described and then examined from several points of view. Emphasis is given to the single administration estimate of the reliability of a mastery test.

Let p be the proportion of items in an item domain that an examinee would answer correctly if every item were attempted. This brief note provides unbiased estimates of pt, for any integer t, when a closed sequential testing procedure is used.