Search

In this paper, we propose a specialized confirmatory mixture IRT model to analyze complex cognitive assessment data that is designed to evaluate adolescents' developmental stages in deductive reasoning. The model is specified for the following purposes: (1) to measure multiple deductive reasoning traits, (2) to identify adolescents' differential developmental stages based on their ability levels in the multiple dimensions, (3) to quantify the differences in dimension-specific performance between developmental stages, and (4) to examine the difficulty levels of test design factors. A Bayesian estimation of the model is described. The overall goodness-of-fit of the model is assessed as well as its parameter recovery to validate the application of the model to the data.

In this paper, we describe a new method we have developed for setting cut scores between levels of a test. We outline the wide variety of potential methods that have been used for such a process, and emphasize the need for a coherent conceptual framework under which the variety of methods could be understood. We then describe our particular method, based on an item response modeling framework, which uses the Wright Map, a graphical model of item and threshold difficulties, and a piece of computer software that provides probabilities of various responses for scores under consideration as cut scores. Finally, we describe a study we conducted for the Golden State Examination in Chemistry, in which we investigate the classification agreement for two groups using the method, and also investigate the reactions of the committee members to the procedure and the software, and the lessons we learned from this process.

We performed a Rasch analysis of cross-sectional developmental data gathered from children and adults who were presented with a task series derived from Inhelder's and Piaget's balance beam. The partial credit model situates both participants and items along a single hierarchically ordered dimension. As the Model of Hierarchical Complexity predicted, order of hierarchical complexity accurately predicted item difficulty, with notable exceptions at the formal and systematic levels. Gappiness between items was examined using the saltus model. A two level saltus model, which examined the gap between the concrete/abstract and formal/systematic items, was a better predictor of performance than the Rasch analysis (chi square = 71.91, df = 4, p < .01).

Addressing health-related quality of life is considered a desirable component of routine care for patients with multiple sclerosis. However, use of available health-related quality of life surveys is not part of routine multiple sclerosis care, possibly due to administration, scoring, and interpretation challenges presented by available questionnaires. The 25-item Functional Index for Living with Multiple Sclerosis (FILMS) questionnaire was developed and validated to allow providers to monitor and easily apply health-related quality of life information to the patient encounter. The development and pilot test processes and the results of the validation study are reported here. Convergent validity, internal consistency reliability, and test-retest reproducibility were evaluated. Strong correlations of Functional Index for Living with Multiple Sclerosis subscales with control questionnaires were demonstrated by Pearson's correlation coefficients from 0.73 to 0.88. Internal consistency reliabilities ranged from 0.80 to 0.90, demonstrating that items were grouped into the appropriate subscale domains and that the subscale domains and the questionnaire as a whole exhibited good directionality. Test-retest reproducibility was 0.91.

In the past, many assessments, especially standardized assessments, tended to be composed of items with specific right and wrong answers, such as those found in multiple choice, true-false and short response items. Performance-based questions that require students to construct answers rather than select correct responses introduce the complexities of multiple correct answers, dependence on teacher judgment for scoring, and requisite ancillary skills such as language fluency, which are technically difficult to handle, and may even introduce problems such as bias against certain groups of students. Recent developments in assessment design and psychometrics have improved the feasibility of assessing performance-based tasks more efficiently and effectively, thereby providing a rich domain of information from which interpretations can be made about what students know and what they can do when they draw upon that knowledge. We developed the ConstructMap computer program specifically to assist teachers in interpreting and representing this type of performance data. The program accepts as input student scores on items associated with one or multiple performance variables, computes proficiencies using multidimensional item response methods, and produces graphical representations of students' estimated proficiency on each of the variables.

This paper is relevant to measurement educators who are interested in the variability of understanding and use of the four building blocks in the Constructing Measures framework (Wilson, 2005). It proposes a uni-dimensional structure for understanding Wilson's framework, and explores the evidence for and against this conceptualization. Constructed and fixed choice response items are utilized to collect responses from 72 participants who range in experience and expertise with constructing measures. The data was scored by two raters was analyzed with the Rasch partial credit model using ConQuest (1998). Guided by the 1999 Testing Standards, analyses of validity and reliability evidence provide support for the construct theory and limited uses of the instrument pending item design modifications.

These studies examine the relationship between the analytic basis underlying the hierarchies produced by the Model of Hierarchical Complexity and the probabilistic Rasch scales that places both participants and problems along a single hierarchically ordered dimension. A Rasch analysis was performed on data from the balance-beam task series. This yielded scaled stage of performance for each of the items. The items formed a series of clusters along this same dimension, according to their order of hierarchical complexity. We sought to ascertain whether there was a significant relationship between the order of hierarchical complexity (a task property variable) of the tasks and the corresponding Rasch scaled difficulty of those same items (a performance variable). It was found that The Model of Hierarchical Complexity was highly accurate in predicting the Rasch Stage scores of the performed tasks, therefore providing an analytic and developmental basis for the Rasch scaled stages.

This chapter examines an application of the saltus model, a mixture model that was designed for the analysis of developmental data. Some background in the types of research for which such a model might be useful is discussed. The equations defining the model are given, as well as the model's relationship to the Rasch model and to other mixture models. An application of the saltus model to an example data set, collected using Noelting's orange juice mixtures tasks, is examined in detail, along with the control files necessary to run the software, and the output file it produced.