Search

Using item‐response theory to model rater effects provides an alternative solution for rater monitoring and diagnosis, compared to using standard performance metrics. In order to fit such models, the ratings data must be sufficiently connected in order to estimate rater effects. Due to popular rating designs used in large‐scale testing scenarios, there tends to be a large proportion of missing data, yielding sparse matrices and estimation issues. In this article, we explore the impact of different types of connectedness, or linkage, brought about by using a linkage set—a collection of responses scored by most or all raters. We also explore the impact of the properties and composition of the linkage set, the different connectedness yielded from different rating designs, and the role of scores from automated scoring engines. In designing monitoring systems using the rater response version of the generalized partial credit model, the study results suggest use of a linkage set, especially a large one that is comprised of responses representing the full score scale. Results also show that a double‐human‐scoring design provides more connectedness than a design with one human and an automated scoring engine. Furthermore, scores from automated scoring engines do not provide adequate connectedness. We discuss considerations for operational implementation and further study. [ABSTRACT FROM AUTHOR]

The article describes practical suggestions for measurement researchers and psychometricians to respond to calls for social responsibility in assessment. The underlying assumption is that personalizing large‐scale assessment improves the chances that assessment and the use of test scores will contribute to equity in education. This article describes a spectrum of standardization and personalization in large‐scale assessment. Informed by a review of existing theories, models, and frameworks in the context of current and developing technologies and with a social justice lens, we propose steps to take, as part of assessment research and development, to contribute to the science of personalizing large‐scale assessment in technically defensible ways. [ABSTRACT FROM AUTHOR]

Many large‐scale assessments are designed to yield two or more scores for an individual by administering multiple sections measuring different but related skills. Multidimensional tests, or more specifically, simple structured tests, such as these rely on multiple multiple‐choice and/or constructed responses sections of items to generate multiple scores. In the current article, we propose an extension of the hierarchical rater model (HRM) to be applied with simple structured tests with constructed response items. In addition to modeling the appropriate trait structure, the multidimensional HRM (M‐HRM) presented here also accounts for rater severity bias and rater variability or inconsistency. We introduce the model formulation, test parameter recovery with a focus on latent traits, and compare the M‐HRM to other scoring approaches (unidimensional HRMs and a traditional multidimensional item response theory model) using simulated and empirical data. Results show more precise scores under the M‐HRM, with a major improvement in scores when incorporating rater effects versus ignoring them in the traditional multidimensional item response theory model. [ABSTRACT FROM AUTHOR]

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM. [ABSTRACT FROM AUTHOR]

An important consideration of any computer adaptive testing (CAT) program is the criterion used for ending item administration—the stopping rule, which ensures that all examinees are assessed to the same standard. Although various stopping rules exist, none of them have been compared under the generalized partial-credit model (Muraki in Applied Psychological Measurement, 16, 159–176, 1992). In this simulation study we compared the performance of three variable-length stopping rules—standard error (SE), minimum information (MI), and change in theta (CT)—both in isolation and in combination with requirements of minimum and maximum numbers of items, as well as a fixed-length stopping rule. Each stopping rule was examined under two termination criteria—one a more lenient requirement (SE = 0.35, MI = 0.56, CT = 0.05), and one more stringent (SE = 0.30, MI = 0.42, CT = 0.02). The simulation design also included content-balancing and exposure controls, aspects of CAT that have been excluded in previous research comparing variable-length stopping rules. The minimum-information stopping rule produced biased theta estimates and varied greatly in measurement quality across the theta distribution. The absolute-change-in-theta stopping rule had strong performance when paired with a lower criterion and a minimum test length. The standard error stopping rule consistently provided the best balance of measurement precision and operational efficiency and was based on the fewest number of administered items necessary to obtain accurate and precise theta estimates, particularly when it was paired with a maximum-number-of-items stopping rule. [ABSTRACT FROM AUTHOR]

An abstract of the article "A Comparison of Classification Techniques for Diagnosis in an Alzheimer's Study," by Jodi M. Casabianca is presented.

Background: Randomized, controlled trials have shown that nurseled disease management for patients with heart failure can reduce hospitalizations. Less is known about the cost-effectiveness of these interventions. Objective: To estimate the cost-effectiveness of a nurse-led disease management intervention over 12 months, implemented in a randomized, controlled effectiveness trial. Design: Cost-effectiveness analysis conducted alongside a randomized trial. Data Sources: Medical costs from administrative records, and self-reported quality of life and nonmedical costs from patient surveys. Participants: Patients with systolic dysfunction recruited from ambulatory clinics in Harlem, New York. Time Horizon: 12 months. Perspective: Societal and payer. Intervention: 12-month program that involved 1 face-to-face encounter with a nurse and regular telephone follow-up. Outcome Measures: Quality of life as measured by the Health Utilities Index Mark 3 and EuroQol-5D and cost-effectiveness as measured by the incremental cost-effectiveness ratio (ICER). Results of Base-Case Analysis: Costs and quality of life were higher in the nurse-managed group than the usual care group. The ICERs over 12 months were $17 543 per EuroQol-5D-based quality-adjusted life-year (QALY) and $15 169 per Health Utilities Index Mark 3-based QALY (in 2001 U.S. dollars). Results of Sensitivity Analysis: From a payer perspective, the ICER ranged from $3673 to $4495 per QALY. Applying national prices in place of New York City prices yielded a societal ICER of $13 460 to $15 556 per QALY. Cost-effectiveness acceptability curves suggest that the intervention was most likely cost-effective for patients with less severe (New York Heart Association classes I to II) heart failure. Limitation: The trial was conducted in an ethnically diverse, inner-city neighborhood; thus, results may not be generalizable to other communities. Conclusion: Over 12 months, the nurse-led disease management program was a reasonably cost-effective way to reduce the burden of heart failure in this community. [ABSTRACT FROM AUTHOR]