Search

The subjective aspect of standard‐setting is often criticized, yet data‐driven standard‐setting methods are rarely applied. Therefore, we applied a mixture Rasch model approach to setting performance standards across several testing programs of various sizes and compared the results to existing passing standards derived from traditional standard‐setting methods. We found that heterogeneity of the sample is clearly necessary for the mixture Rasch model approach to standard setting to be useful. While possibly not sufficient to determine passing standards on their own, there may be value in these data‐driven models for providing additional validity evidence to support decision‐making bodies entrusted with establishing cut scores. They may also provide a useful tool for evaluating existing cut scores and determining if they continue to be supported or if a new study is warranted. [ABSTRACT FROM AUTHOR]

We address two persistent challenges in large‐scale assessments of the Next Generation Science Standards: (a) the validity of score interpretations that target the standards broadly and (b) how to structure claims for assessments of this complex domain. The NGSS pose a particular challenge for specifying claims about students that evidence from summative assessments can support. As a solution, we propose tiered claims, which explicitly distinguish between claims about what students have done or can do on test items—which are typically easier to support under current test designs—and claims about what students could do in the broader domain of performances described by the standards, for which novel evidence is likely required. We discuss the positive implications of tiered claims for test construction, validation, and reporting of results. [ABSTRACT FROM AUTHOR]

The objectives of this two‐part study were to: (a) investigate English learner (EL) accommodation practices on state accountability assessments of reading/English language arts and mathematics in grades 3–8, and (b) conduct a meta‐analysis of EL accommodation effectiveness on improving test performance. Across all distinct testing programs, we found that at least one EL test accommodation was provided for both test content areas. The most popular accommodations provided were supplying students with word‐to‐word dual language dictionaries, reading aloud test directions and items in English, and allowing flexible time/scheduling. However, we found minimal evidence that testing programs provide practitioners with recommendations on how to assign relevant accommodations to EL test takers' English proficiency level. To evaluate whether accommodations used in practice are supported with evidence of their effectiveness, a meta‐analysis was conducted. On average, across 26 studies and 95 effect sizes (N = 11,069), accommodations improved test performance by.16 standard deviations. Both test content and sampling design were found to moderate accommodation effectiveness; however, none of the accommodations investigated were found to have intervention effects that were statistically different from zero. Overall, these results suggest that currently employed EL test accommodations lack evidence of their effectiveness. [ABSTRACT FROM AUTHOR]

A common belief is that the Bookmark method is a cognitively simpler standard‐setting method than the modified Angoff method. However, a limited amount of research has investigated panelist's ability to perform well the Bookmark method, and whether some of the challenges panelists face with the Angoff method may also be present in the Bookmark method. This article presents results from three experiments where panelists were asked to give Bookmark‐type ratings to separate items into groups based on item difficulty data. Results of the experiments showed, consistent with results often observed with the Angoff method, that panelists typically and paradoxically perceived hard items to be too easy and easy items to be too hard. These perceptions were reflected in panelists often placing their Bookmarks too early for hard items and often placing their Bookmarks too late for easy items. The article concludes with a discussion of what these results imply for educators and policymakers using the Bookmark standard‐setting method. [ABSTRACT FROM AUTHOR]

In this paper we assert that the practice of principled assessment design renders traditional standard‐setting methodology redundant at best and contradictory at worst. We describe the rationale for, and methodological details of, Embedded Standard Setting (ESS; previously, Engineered Cut Scores. Lewis, 2016), an approach to establish performance standards that augments and is embedded within existing principled assessment design practices. Under ESS, achievement level descriptor writing and the targeting of items to specified achievement levels are viewed as the fundamental standard‐setting activities and cut scores are estimated by optimizing the relationship between the item targets and the empirically based grouping of items. If the relationship between item targets and empirical data are sufficiently strong, then the traditional standard‐setting workshop is either unnecessary or altered in nature and significantly reduced in scope. We provide empirical evidence supporting the efficacy of ESS and describe how ESS is embedded throughout the assessment lifecycle. [ABSTRACT FROM AUTHOR]

Evidence of the internal consistency of standard‐setting judgments is a critical part of the validity argument for tests used to make classification decisions. The bookmark standard‐setting procedure is a popular approach to establishing performance standards, but there is relatively little research that reflects on the internal consistency of the resulting judgments. This article presents the results of an experiment in which content experts were randomly assigned to one of two response probability conditions:.67 and.80. If the standard‐setting judgments collected with the bookmark procedure are internally consistent, both conditions should produce highly similar cut scores. The results showed substantially different cut scores for the two conditions; this calls into question whether content experts can produce the type of internally consistent judgments that are required using the bookmark procedure. [ABSTRACT FROM AUTHOR]

In test‐centered standard‐setting methods, borderline performance can be represented by many different profiles of strengths and weaknesses. As a result, asking panelists to estimate item or test performance for a hypothetical group study of borderline examinees, or a typical borderline examinee, may be an extremely difficult task and one that can lead to questionable results in setting cut scores. In this study, data collected from a previous standard‐setting study are used to deduce panelists' conceptions of profiles of borderline performance. These profiles are then used to predict cut scores on a test of algebra readiness. The results indicate that these profiles can predict a very wide range of cut scores both within and between panelists. Modifications are proposed to existing training procedures for test‐centered methods that can account for the variation in borderline profiles. [ABSTRACT FROM AUTHOR]

Extensive research has examined the validity and fairness of standardized tests in academic admissions. However, due to their underrepresentation in higher education, American Indians have gained much less attention in this research. In the present study, we examined for American Indian students (1) group differences on SAT scores, (2) the predictive and incremental validity of SAT over high school grades, (3) the effect of socioeconomic status on SAT validity, (4) differential prediction in the use of SAT scores, and (5) potential omitted variables that could explain differential prediction for American Indian students. Results provided evidence of predictive and incremental validity of SAT scores, and the validity of SAT scores was largely independent of socioeconomic status. Overprediction was found when using SAT scores to predict college performance and it was reduced when including high school grades as an additional predictor. This study provides substantial evidence of the validity and fairness of SAT scores for American Indians. [ABSTRACT FROM AUTHOR]

This 2006 NCME Career Award Address presents observations made over a career in the field of standard setting, summarizes the status of current research, and makes recommendations for future research. [ABSTRACT FROM AUTHOR]

For nearly two decades, virtually all U.S. states and many professional societies have grappled with the challenge of how best to establish test-based performance standards that make sense in terms of both content-related expectations and the score scales reported to stakeholders. Perhaps the most important takeaway from this issue is that it is okay to question standard-setting dogma and adjust methods based on new research, even if that means methods differ within components of an assessment system. The role of standard-setting procedures is central to establishing the validity of the inferences about examinee performance and the correctness of proficiency or competency classifications. [Extracted from the article]