Search

The advantages of using person location estimates from the Rasch model over raw scores for the measurement of change using a common test include the linearization of scores and the automatic handling of statistical properties of repeated measurements. However, the application of the model requires that the responses to the items are statistically independent in the sense that the specific responses to the items on the first time of testing do not affect the responses at a second time. This requirement implies that the responses to the items at both times of assessment are governed only by the invariant location parameters of the items at the two times of testing and the location parameters of each person each time. A specific form of dependence that is pertinent when the same items are used is when the observed response to an item at the second time of testing is affected by the response to the same item at the first time, a form of dependence which has been referred to as response dependence. This paper presents the logic of applying the Rasch model to quantify, control and remove the effect of response dependence in the measurement of change when the same items are used on two occasions. The logic is illustrated with four sets of simulation studies with dichotomous items and with a small example of real data. It is shown that the presence of response dependence can reduce the evidence of change, a reduction which may impact interpretations at the individual, research, and policy levels.

Test theories imply statistical, local independence. Where local independence is violated, models of modern test theory that account for it have been proposed. One violation of local independence occurs when the response to one item governs the response to a subsequent item. Expanding on a formulation of this kind of violation between two items in the dichotomous Rasch model, this paper derives three related implications. First, it formalises how the polytomous Rasch model for an item constituted by summing the scores of the dependent items absorbs the dependence in its threshold structure. Second, it shows that as a consequence the unit when the dependence is accounted for is not the same as if the items had no response dependence. Third, it explains the paradox, known, but not explained in the literature, that the greater the dependence of the constituent items the greater the apparent information in the constituted polytomous item when it should provide less information.

Assessments with ratings in ordered categories have become ubiquitous in health, biological and social sciences. Ratings are used when a measuring instrument of the kind found in the natural sciences is not available to assess some property in terms of degree - for example, greater or smaller, better or worse, or stronger or weaker. The handling of ratings has ranged from the very elementary to the highly sophisticated. In an elementary form, and assumed in classical test theory, the ratings are scored with successive integers and treated as measurements; in a sophisticated form, and used in modern test theory, the ratings are characterized by probabilistic response models with parameters for persons and the rating categories. Within modern test theory, two paradigms, similar in many details but incompatible on crucial points, have emerged. For the purposes of this article, these are termed the statistical modeling and experimental measurement paradigms. Rather than reviewing a compendium of available methods and models for analyzing ratings in detail, the article focuses on the incompatible differences between these two paradigms, with implications for choice of model and inferences. It shows that the differences have implications for different roles for substantive researchers and psychometricians in designing instruments with rating scales. To illustrate these differences, an example is provided.

There is a substantial literature on attempts to obtain information on the proficiency of respondents from distractors in multiple choice items. Information in a distractor implies that a person who chooses that distractor has greater proficiency than if the person chose another distractor with no information. A further implication is that the distractor deserves partial credit. However, it immediately follows from the Rasch model that if a distractor deserves partial credit, then the response to that distractor and other distractors should not be pooled into a single response with a single probability of an incorrect response. Using the partial credit parameterization of the polytomous Rasch model, the paper shows how an hypothesis can be formed, and tested, regarding information in a distractor. The hypothesis is formed by studying the shape of the distractor response curves across the continuum, and the hypothesis is tested by scoring the correct response 2, the hypothesized distractor 1, and other distractors 0, and then applying the polytomous Rasch model. Multiple pieces of evidence, including fit of the responses at the two thresholds and the order of the two threshold estimates, are used in deciding if a distractor has information. An example illustrating the theory and its application is provided.

The 'halo effect' may be unique to different raters or common to all raters. When common to all raters, halo is not detectable through standard fit indices of the three-facet Rasch model used to account for differences in rater severities. Using a formulation of halo as a violation of local independence, a halo effect common to all raters is simulated and shown to be diagnosable through contrasts between two-facet stack and rack Rasch analyses. In the former, the thresholds are clustered and the distribution of persons is multimodal; in the latter, all thresholds are close together and the distribution of persons is unimodal. In the former, the scale is stretched, and the person separation inflated, relative to the latter.

By adding items with responses identical to a selected item, Smith (2005) investigated the effect of the response dependence on person and item parameter estimates in the dichotomous Rasch model. By varying the magnitude of response dependence among selected items, rather than their having perfect dependence, this paper provides additional insights into the effects of response dependence on the same estimates in the same model. Two sets of simulations are reported. In the first set, responses to all items except the first were dependent on either the first item or on the immediately preceding item; in the second set, subsets of items were formed first, and then within each of these subsets, responses to all items in a subset except the first were dependent on either the first item or on the immediately preceding item. The effects of dependence were noticeable in all of the statistics reported. In particular, the fit statistics and the parameter estimates showed increasing discrepancies from their theoretical values as a function of the magnitude of the dependence. In some cases, however, two related statistics gave the impression of improvement as a function of increased dependency; first the standard deviation of person estimates showed an increase, and second the index analogous to traditional reliability showed relative increase. In addition to the estimates and depending on the structure and magnitude of the dependence, the person distribution was affected systematically, ranging from becoming skewed to becoming bimodal. The effects on the distribution help explain some of the effects on the statistics reported. In the case of the second set of simulations in which the dependence is within subsets of items, it is possible to take account of the response dependence. This is done by summing the responses of the items within each subset to form a polytomous item and then analyzing the data in terms of a smaller number of polytomous items. This way of accounting for dependence, in which the maximum score for the test as a whole remains the same, gives a more accurate value of the reliability and a more realistic distribution of the person estimates than when the dependence within subsets of items is not taken into account.

Background: As assessment has been shown to direct learning, it is critical that the examinations developed to test clinical competence in medical undergraduates are valid and reliable. The use of extended matching questions (EMQ) has been advocated to overcome some of the criticisms of using multiple-choice questions to test factual and applied knowledge., Methods: We analysed the results from the Extended Matching Questions Examination taken by 4th year undergraduate medical students in the academic year 2001 to 2002. Rasch analysis was used to examine whether the set of questions used in the examination mapped on to a unidimensional scale, the degree of difficulty of questions within and between the various medical and surgical specialties and the pattern of responses within individual questions to assess the impact of the distractor options., Results: Analysis of a subset of items and of the full examination demonstrated internal construct validity and the absence of bias on the majority of questions. Three main patterns of response selection were identified., Conclusion: Modern psychometric methods based upon the work of Rasch provide a useful approach to the calibration and analysis of EMQ undergraduate medical assessments. The approach allows for a formal test of the unidimensionality of the questions and thus the validity of the summed score. Given the metric calibration which follows fit to the model, it also allows for the establishment of items banks to facilitate continuity and equity in exam standards.

A category with a frequency of zero is called a null category. When null categories are present in polytomous responses, then in the Rasch model for such responses, the thresholds that define the categories are inestimable with the commonly used joint maximum likelihood, marginal maximum likelihood, or standard conditional maximum likelihood estimation algorithms. The reason for this situation is that in principle, these estimation algorithms involve frequencies of each category. Andrich and Luo (2003) describe an algorithm in which the thresholds are reparameterized into their principal components and in which the estimate of any threshold is based on a function of the frequencies of all categories of the item rather than the frequency of a particular category. This algorithm works in the presence of null categories. However, in situations where the null categories are at the extremes of a set of categories, the estimates themselves can become too extreme. This paper describes a procedure in which the solution algorithm described by Andrich and Luo is further adapted in the presence of null categories by using their expected frequencies. The procedure is demonstrated with simulated and real data.

In the Rasch model for items with more than two ordered response categories, the thresholds that define the successive categories are an integral part of the structure of each item in that the probability of the response in any category is a function of all thresholds, not just the thresholds between any two categories. This paper describes a method of estimation for the Rasch model that takes advantage of this structure. In particular, instead of estimating the thresholds directly, it estimates the principal components of the thresholds, from which threshold estimates are then recovered. The principal components are estimated using a pairwise maximum likelihood algorithm which specialises to the well known algorithm for dichotomous items. The method of estimation has three advantageous properties. First, by considering items in all possible pairs, sufficiency in the Rasch model is exploited with the person parameter conditioned out in estimating the item parameters, and by analogy to the pairwise algorithm for dichotomous items, the estimates appear to be consistent, though unlike for the dichotomous case, no formal proof has yet been provided. Second, the estimates of each item parameter is a function of frequencies in all categories of the item rather than just a function of frequencies of two adjacent categories. This stabilizes estimates in the presence of low frequency data. Third, the procedure accounts readily for missing data. All of these properties are important when the model is used for constructing variables from large scale data sets which must account for structurally missing data. A simulation study shows that the quality of the estimates is excellent.

The case for the Rasch models, that The comparison between two stimuli should be independent of which particular individuals were instrumental for the comparison; and vice versa (Rasch, 1961), does not depend on the models accounting for any data set. This has two distinctive consequences on the data-model relationship for the Rasch models. First, and this was recognized by Rasch, when there are deviations of one sort or another, it turns upside down the question of whether it is the model or the test that has gone wrong (Rasch, 1960). Second, because the invariance of comparisons among stimuli, and vice versa, is built into the model rather than being merely a requirement of data, further implications of this requirement can be derived mathematically. These implications, too, inevitably turn some questions, and their solutions, upside down. It is argued that having to look at these implications upside down produces substantial psychological and intellectual resistance amongst those schooled in looking at them in the traditional way. It is also argued that in turning the question upside down, Rasch had an insight that goes beyond the mathematical derivations, and that to sustain this insight requires a paradigm shift (Kuhn, 1970) in the data-model relationship. Using an illustrative example, it is suggested that to maintain this paradigm shift, even by those who research the Rasch models, requires the same uncompromising consistency and passion that Rasch displayed in maintaining faith in his insight.