Search

Determining the most appropriate method of scoring an assessment is based on multiple factors, including the intended use of results, the assessment's purpose, and time constraints. Both the dichotomous and partial credit models have their advantages, yet direct comparisons of assessment outcomes from each method are not typical with constructed response items. The present study compared the impact of both scoring methods on the internal structure and consequential validity of a middle-grades problem-solving assessment called the problem solving measure for grade six (PSM6). After being scored both ways, Rasch dichotomous and partial credit analyses indicated similarly strong psychometric findings across models. Student outcome measures on the PSM6, scored both dichotomously and with partial credit, demonstrated strong, positive, significant correlation. Similar demographic patterns were noted regardless of scoring method. Both scoring methods produced similar results, suggesting that either would be appropriate to use with the PSM6.

Assessment and evaluation at all levels of educational systems have become policy priorities for many countries. Two common reasons for this are student learning expectations and accountability. Although much effort has been put into the creation and refinement of content standards, standardized tests, and methods for using testing results, there has been less attention paid to the development of performance standards (proficiency levels) that greatly affect policy decision making. The present study investigates the Objective Standard Setting Model (OSS) as an improved criterion-referenced method for setting multilevel performance standards. To demonstrate how OSS can be employed for performance standard setting, our study used data from preservice teachers completing an assessment literacy test in a pre- and posttest environment. Using OSS, performance standard levels of "proficient" and "excellent" were established with clear content-related descriptions of growth in student content mastery.

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However, access to student-level longitudinal or comparison group data is often scarce, which significantly restricts researchers' choice of research design. Although most state departments of education have school- and district-level data available to the public, the individual student-level data that are often needed to perform appropriate statistical analyses are unavailable. This exploratory study demonstrates the process of and provides evidence that microanalytical simulation methods, conducted using Microsoft Excel, may be a useful research tool in the field of education if adequate school-level modeling information is available. These simulation methods may assist in providing greater opportunities to execute more rigorous methodological designs. (Contains 10 tables.)

Rasch analysis was used to illustrate the usefulness of item-level analyses for evaluating a common therapy outcome measure of general clinical distress, the Symptom Checklist-90-Revised (SCL-90-R; Derogatis, 1994). Using complementary therapy research samples, the instrument's 5-point rating scale was found to exceed clients' ability to make reliable discriminations and could be improved by collapsing it into a 3-point version (combining scale points 1 with 2 and 3 with 4). This revision, in addition to removing 3 misfitting items, increased person separation from 4.90 to 5.07 and item separation from 7.76 to 8.52 (resulting in alphas of 0.96 and 0.99, respectively). Some SCL-90-R subscales had low internal consistency reliabilities; SCL-90-R items can be used to define one factor of general clinical distress that is generally stable across both samples, with two small residual factors.

A new approach based on Rasch measurement theory is described for examining the quality of ratings from standard-setting judges. Ratings of nine judges for 213 items on a nursing examination show that judges vary in their views of the essential items for nursing certification, with statistically significant variability in the judged essentiality of these items. (SLD)

Assessment and evaluation at all levels of educational systems have become policy priorities for many countries. Two common reasons for this are student learning expectations and accountability. Although much effort has been put into the creation and refinement of content standards, standardized tests, and methods for using testing results, there has been less attention paid to the development of performance standards (proficiency levels) that greatly affect policy decision making. The present study investigates the Objective Standard Setting Model (OSS) as an improved criterion-referenced method for setting multilevel performance standards. To demonstrate how OSS can be employed for performance standard setting, our study used data from preservice teachers completing an assessment literacy test in a pre- and posttest environment. Using OSS, performance standard levels of proficient and excellent were established with clear content-related descriptions of growth in student content mastery. [ABSTRACT FROM AUTHOR]

Discusses the fundamental qualities that must be used to judge the merit of performance standards: validity, reliability, and genuineness. The Rasch-based objective model has demonstrated these qualities, but other models should also be evaluated for these characteristics. (SLD)

Copyright of Avances en PsicologÍa Latinoamericana is the property of Colegio Mayor de Nuestra Senora del Rosario and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However, access to student-level longitudinal or comparison group data is often scarce, which significantly restricts researchers' choice of research design. Although most state departments of education have school- and district-level data available to the public, the individual student-level data that are often needed to perform appropriate statistical analyses are unavailable. This exploratory study demonstrates the process of and provides evidence that microanalytical simulation methods, conducted using Microsoft Excel, may be a useful research tool in the field of education if adequate school-level modeling information is available. These simulation methods may assist in providing greater opportunities to execute more rigorous methodological designs. [ABSTRACT FROM AUTHOR]

This paper revisits a half-century long theoretical controversy associated with the use of magnitude estimation scaling (MES) and category rating scaling (CRS) procedures in measurement. The MES procedure in this study involved instructing participants to write a number that matched their impression of difficulty of a test item. Participants were not restricted in the range of numbers they could choose for their scale. They also had the choice of disclosing their individual scale. After the MES task was completed, participants were given a blank copy of the test to rate the perceived difficulty of each item using a researcher-imposed categorical rating scale from 1 (very easy) to 6 (very difficult). The MES and CRS data were both analyzed using Rasch Rating scale model. Additionally, the MES data were examined with Rasch Partial Credit model. Results indicate that knowing each person's scale is associated with smaller errors of measurement. [ABSTRACT FROM AUTHOR]

Colleges and universities conduct student satisfaction studies for many important policy making reasons. However the differences in instrumentation and the use of students' self-reported ratings of satisfaction makes such decisions sample-, instrument-, and institution-dependent. A common metric of student satisfaction would assist decision makers by providing a richness of information not typically obtained. The present study investigated the extent to which two nationally known instruments of student satisfaction could be scaled on the same quantitative metric. Pseudo-common item equating (Fisher, 1997) based on five link items of low and high endorsability enabled comparisons of "similar, but not identical items, from different instruments, calibrated on different samples" (p. 87). Results suggest that both instruments measured similar constructs and could be reasonably used to create a single, common metric. While samples used in the experiment were less than ideal, results clearly demonstrated the usefulness and reasonability of the pseudo-common item equating process.