Search

Equating test scores from nonequivalent groups requires to separate the effect of differences in ability from differences in test form difficulty between the test forms. The gold standard for this purpose is to use common items, however, not all testing programs have such items available. There might be other useful information available, and this paper studies the use of covariates in equating as a substitute for common items. To gain further insight about how to best incorporate covariates in equating this paper studies the propensity score, a scalar function of the covariates. The sensitivity of the propensity score as a proxy for ability is studied to investigate if bias is introduced in the estimated equated scores if the propensity score estimation model is misspecified. By considering an incorrect link function, leaving out a covariate and leaving out a second-order term from the estimation model, common misspecifications are studied. The results show that equating with respect to the propensity score is relatively robust to model misspecification, especially modeling nonlinearity with a linear model and leaving out second-order terms.

Mixed-format tests contain items with different formats such as dichotomously scored and polytomously scored items. The aim of this study was to examine the impact of item discrimination, sample size, and proportion of polytomously scored items on item response theory (IRT) kernel equating of mixed-format tests under the equivalent groups design. A simulation study was performed to examine the aim. The results from the simulation study showed that the percent relative errors were low and stable for all conditions, whereas differences in standard errors and equated values where found for the conditions with different sample sizes and item discriminations. Also, the standard errors were higher when the proportion of polytomously scored items in the test where higher.

In the kernel method of test score equating, the first step of the procedure is to presmooth the score distributions. The most common way of doing so is by fitting a log-linear model to the observed-score distributions. In this way, irregularities in the score distributions are smoothed, yielding a more stable estimated equating transformation. Within the kernel equating framework, an alternative way of presmoothing by using item response theory models has recently been suggested. There are furthermore several model selection criteria available for both of these classes of models. Here the model selection criteria are studied for both log-linear and item response theory models. Specifically, the likelihood ratio, AIC, and BIC measures are compared using real admissions data. Results show that the different model selection criteria result in equated scores that have real impact differences.

Test equating methods are widely used in order to make comparable different test forms administered at different occasions to different test takers. Although software for test equating is currently available, in this paper we focus the attention on four different R packages which can facilitate test equating for researchers and test developers. This paper list the different R packages which are available at the moment. Examples are provided for the equate, equateIRT, kequate, and the SNSequate packages. Additional features of these packages are discussed as well.

The aim of this paper is to demonstrate how to use TestGardener to analyze testing data with various item types and explain some main displays. TestGardener is a software designed to aid the development, evaluation, and use of multiple choice examinations, psychological scales, questionnaires, and similar types of data. This software implements the optimal scoring of binary and multi-option items, and uses spline smoothing to obtain item characteristics curves (ICCs) that better fit the real data. Using TestGardner does not require any programming skill or formal statistical knowledge, which will make optimal scoring and item response theory more approachable for test analysts, test developers, researchers, and general public.

This proceedings volume highlights the latest research and developments in psychometrics and statistics. It compiles and expands on selected and peer-reviewed presentations given at the 83rd Annual International Meeting of the Psychometric Society (IMPS), organized by Columbia University and held in New York, USA, July 9th to 13th, 2018. The IMPS is one of the largest international meetings on quantitative measurement in education, psychology, and the social sciences. The last couple of years it has attracted more than 500 participants and more than 250 paper presentations from researchers around the world. Leading experts and promising young researchers from around the world have written the 38 chapters. The chapters address a large variety of topics including but not limited to item response theory, multistage adaptive testing, and cognitive diagnostic models. This volume is the 7th in a series of recent volumes to cover research presented at the IMPS.

This proceedings book highlights the latest research and developments in psychometrics and statistics. Featuring contributions presented at the 82nd Annual Meeting of the Psychometric Society (IMPS ...

When using kernel equating to equate two test forms, a bandwidth needs to be selected. The bandwidth parameter determines the smoothness of the continuized score distributions and has been shown to have a large effect on the kernel density estimate. There are a number of suggested criteria for selecting the bandwidth, and currently four of them have been implemented in kernel equating. In this paper, all four of the existing bandwidth selectors suggested for kernel equating are evaluated and compared against each other using real test data together with a new criterion that implements leave-one-out cross-validation. Although the bandwidth methods generally were similar in terms of equated scores, there were potentially important differences in the upper part of the score scale where critical admission decisions are typically made.

This paper discusses the use of optimal scores as an alternative to sum scores and expected sum scores when analyzing test data. Optimal scores are built on nonparametric methods and use the interaction between the test takers’ responses on each item and the impact of the corresponding items on the estimate of their performance. Both theoretical arguments for optimal score as well as arguments built upon simulation results are given. The paper claims that in order to achieve the same accuracy in terms of mean squared error and root mean squared error, an optimally scored test needs substantially fewer items than a sum scored test. The top-performing test takers and the bottom 5% test takers are by far the groups that benefit most from using optimal scores.

Test equating is used to ensure that test scores from different test forms can be used interchangeably. This paper aims to compare the statistical and computational properties from three equating frameworks: item response theory observed-score equating (IRTOSE), kernel equating and kernel IRTOSE. The real data applications suggest that IRT-based frameworks tend to provide more stable and accurate results than kernel equating. Nonetheless, kernel equating can provide satisfactory results if we can find a good model for the data, while also being much faster than the IRT-based frameworks. Our general recommendation is to try all methods and examine how much the equated scores change, always ensuring that the assumptions are met and that a good model for the data can be found.

The aim of this paper is to reflect around what would be needed in order to replace sum scoring, including technical advances, communication with both test constructors and examinees, and organizational strategy. Sum scoring are proposed to be replaced by smart scoring and a brief description, and some theoretical support for smart scoring and methods for achieving it are given together with an example from a large-scale assessment test.

This chapter provides a general overview of equating and offers a conceptual and formal mathematical definition of equating. The roles of random variables, probability distributions, and parameters in the equating statistical problem are described. Different data collection designs are introduced, and an overview of some of the equating methods that will be described throughout the book is also presented.

One important part of ensuring test quality over consecutive test administrations is to make sure that the equating procedure works as intended, especially when the composition of the test taker groups might change over the administrations. The aim of this study was to examine the equating transformations obtained using one or two previous administrations of a college admissions test that is given twice a year. The test has an external anchor, and thus a nonequivalent group with anchor test design is typically used to equate the test, although other data collection designs are possible. This study examined the use of different equating methods with different data collection designs and different braiding plans. The methods included traditional equating methods and (item response theory) kernel equating methods. We found that different equating methods and different braiding strategies gave somewhat different results, and some reflections on how to proceed in the future are given.

In test score equating, the nonequivalent groups with covariates (NEC) design use covariates with high correlation to the test scores as a substitute for an anchor test when the latter is lacking. However, as the number of covariates increases, the number of observations for each covariate combination decreases. We suggest to use propensity scores instead, which we include in the kernel equating framework using both post-stratification and chained equating. The two approaches are illustrated with data from a large-scale assessment, and the results show an increased precision in comparison with the equivalent groups design and great similarities in comparison with the results when using an anchor test.

This chapter describes traditional equating methods and their implementation in R. The equate package (Albano, J Stat Softw 74(8):1–36, 2016) will be used the most, although the possibility to use SNSequate (Gonzalez, J Stat Softw 59(7):1–30, 2014) for traditional equating methods will also be explored. The methods included in this chapter are mean, linear and equipercentile equating for different data collection designs.

Depending on the equating data collection design, score data will be in the form of either univariate or bivariate distributions. In this chapter, we describe how to prepare the score distributions in order to read them into the different R packages that will be used. Presmoothing the score distributions as a first step in equating is also discussed. Known data sets appearing in the equating literature, as well as real data examples from an admissions test and an achievement test are described. The illustrations will use the three R packages equate (Albano, J Stat Softw 74(8):1–36, 2016), kequate (Andersson et al., J Stat Softw 55(6):1–25, 2013) and SNSequate (Gonzalez, J Stat Softw 59(7):1–30, 2014).

The research articles in this volume cover timely quantitative psychology topics, including new methods in item response theory, computerized adaptive testing, cognitive diagnostic modeling, and psychological scaling. Topics within general quantitative methodology include structural equation modeling, factor analysis, causal modeling, mediation, missing data methods, and longitudinal data analysis. These methods will appeal, in particular, to researchers in the social sciences. The 80th annual meeting took place in Beijing, China, between the 12th and 16th of July, 2015. Previous volumes to showcase work from the Psychometric Society’s Meeting are New Developments in Quantitative Psychology: Presentations from the 77th Annual Psychometric Society Meeting (Springer, 2013), Quantitative Psychology Research: The 78th Annual Meeting of the Psychometric Society (Springer, 2015), and Quantitative Psychology Research: The 79th Annual Meeting of the Psychometric Society, Wisconsin, USA, 2014 (Springer, 2015).