Search

An additive multilevel item structure (AMIS) model with random residuals is proposed. The model includes multilevel latent regressions of item discrimination and item difficulty parameters on covariates at both item and item category levels with random residuals at both levels. The AMIS model is useful for explanation purposes and also for prediction purposes as in an item generation context. The parameters can be estimated with an alternating imputation posterior algorithm that makes use of adaptive quadrature, and the performance of this algorithm is evaluated in a simulation study.

This paper presents a noncompensatory latent trait model, the multicomponent latent trait model for diagnosis (MLTM-D), for cognitive diagnosis. In MLTM-D, a hierarchical relationship between components and attributes is specified to be applicable to permit diagnosis at two levels. MLTM-D is a generalization of the multicomponent latent trait model (MLTM; Whitely in Psychometrika, 45:479-494, 1980; Embretson in Psychometrika, 49:175-186, 1984) to be applicable to measures of broad traits, such as achievement tests, in which component structure varies between items. Conditions for model identification are described and marginal maximum likelihood estimators are presented, along with simulation data to demonstrate parameter recovery. To illustrate how MLTM-D can be used for diagnosis, an application to a large-scale test of mathematics achievement is presented. An advantage of MLTM-D for diagnosis is that it may be more applicable to large-scale assessments with more heterogeneous items than are latent class models.

H. Blanton and J. Jaccard examined the arbitrariness of metrics in the context of 2 current issues: (a) the measurement of racial prejudice and (b) the establishment of clinically significant change. According to Blanton and Jaccard, although research findings are not undermined by arbitrary metrics, individual scores and score changes may not be meaningfully interpreted. The author believes that their points are mostly valid and that their examples were appropriate. However, Blanton and Jaccard's article does not lead directly to solutions, nor did it adequately describe the scope of the metric problem. This article has 2 major goals. First, some prerequisites for nonarbitrary metrics are presented and related to Blanton and Jaccard's issues. Second, the impact of arbitrary metrics on psychological research findings are described. In contrast to Blanton and Jaccard (2006), research findings suggest that metrics have direct impact on statistics for group comparisons and trend analysis., (2006 APA, all rights reserved.)

The continual supply of new items is crucial to maintaining quality for many tests. Automatic item generation (AIG) has the potential to rapidly increase the number of items that are available. However, the efficiency of AIG will be mitigated if the generated items must be submitted to traditional, time-consuming review processes. In two studies, generated mathematics achievement items were subjected to multiple stages of qualitative review for measuring the intended skills, followed by empirical tryout in operational testing. High rates of success were found. Further, items generated from the same item structure had predictable psychometric properties. Thus, the feasibility of a more limited and expedient review processes was supported. Additionally, positive results were obtained on measuring the same skills from item structures with reduced cognitive complexity.

Examinees' thinking processes have become an increasingly important concern in testing. The responses processes aspect is a major component of validity, and contemporary tests increasingly involve specifications about the cognitive complexity of examinees' response processes. Yet, empirical research findings on examinees' cognitive processes are not often available either to provide evidence for validity or to guide the design or selection of items. In this article, studies and developments from the author's research program are presented to illustrate how empirical studies on examinees' thinking processes can impact item and test design.

Recent assessment research joining cognitive psychology and psychometric theory has introduced a new technology, item generation. In algorithmic item generation, items are systematically created based on specific combinations of features that underlie the processing required to correctly solve a problem. Reading comprehension items have been more difficult to model than other item types due to the complexities of quantifying text. However, recent developments in artificial intelligence for text analysis permit quantitative indices to represent cognitive sources of difficulty. The current study attempts to identify generative components for the Graduate Record Examination paragraph comprehension items through the cognitive decomposition of item difficulty. Text comprehension and decision processes accounted for a significant amount of the variance in item difficulties. The decision model variables contributed significantly to variance in item difficulties, whereas the text representation variables did not. Implications for score interpretation and future possibilities for item generation are discussed. (Contains 1 figure and 5 tables.)

The last century was marked by dazzling changes in many areas, such as technology and communications. Predictions into the second century of testing are seemingly difficult in such a context. Yet, looking back to the turn of the last century, Kirkpatrick (1900), in his American Psychological Association presidential address, presented fundamental desiderata for ability testing that in fact guides major testing research even today (normative age standards, emphasis on culture-general tasks, simultaneous measurement of many persons and adult ability measurement). An overview of the last century, presented in this address, shows that most fundamental principles in psychometrics and testing were available by 1930. With a few notable exceptions, the remainder of the last century of testing was devoted to applying or refining these principles. It is predicted that the same pattern will occur in this century of testing. Further developments in model-based measurement and in applying cognitive psychology principles in testing, intermingled with technology, will guide ability testing throughout the next century. These changes, to be elaborated in detail, include fundamental changes in test development procedures, the nature of the measuring tasks, aspects of ability that are measured, and types of interpretations given to ability. (Contains 8 figures and 3 tables.)

Discusses computerized dynamic testing with cues and items presented according to objective algorithms, elaborating on appropriate designs and psychometric models. Presents two studies involving 311 military recruits and 584 recruits that support the psychometric properties of a test measuring the susceptibility of reasoning to stressors. (SLD)

This study examined the influence of processing strategies, and the metacomponents that determine when to apply them, on the construct validity of a verbal reasoning test. A rule-oriented strategy, an association strategy, and a partial rule strategy were examined. All three strategies contributed to individual differences in verbal reasoning. (Author/LMO)

Aptitude test scores are typically interpreted similarly for examinees with the same overall score. However, research has found evidence of examinee differences in strategies, as well as in the continued application of appropriate procedures during testing. Such differences can impact the correlates of test scores, making similar interpretations for equivalent scores questionable. This study presents some item response theory (IRT) models that are relevant to identifying examinee differences in strategies and understanding of test-taking procedures. First, mixture IRT models that identify latent classes of examinees with different patterns of item responses are considered; these models have long been available but unfortunately are not routinely applied. Strategy differences between the classes can then be studied separately by modeling the response patterns with cognitive complexity variables within each class. Secondly, novel psychometric approaches that leverage response time information (in particular, response time residuals) in order to identify both inter and intraindividual variability in response processes are considered. In doing so, a general method for evaluating threats to validity is proposed. The utility of the approach, in terms of providing more interpretable performance estimates and improving the administration of psychological measurement instruments, is then demonstrated with an empirical example. [ABSTRACT FROM AUTHOR]

Understanding the cognitive processes, skills and strategies that examinees use in testing is important for construct validity and score interpretability. Although response processes evidence has long been included as an important aspect of validity (i.e., Standards for Educational and Psychological Tests, 1999), relevant studies are often lacking, especially in large scale educational and psychological testing. An important method for studying response processes involves explanatory mathematical modeling of item responses and item response times from variables that represent sources of cognitive complexity. For many item types, examinees may differ in strategies applied to responding to items. Mixture class item response theory models can identify latent classes of examinees with different processes, skills and strategies based on their pattern of item responses. This study will illustrate the use of response times in conjunction with explanatory item response theory models and mixture models, to provide information relevant to test validity and, hence, to score interpretations. [ABSTRACT FROM AUTHOR]

Heterogeneous item content is prevalent on psychological and educational tests that measure global traits or competencies, such as general intelligence and achievement tests. In such tests, the domain is broadly defined, so as to include many attributes and skills. Items often vary substantially in both the type and the number of attributes or skills that are involved in item solving. A hierarchical organization is often necessary to accommodate the heterogeneity of the test domain. For example, the mathematical achievement tests that are routinely administered in all U.S. states at the end of the school year typically have this hierarchical structure. The multicomponent latent trait model for diagnosis (MLTM-D) was developed for application to heterogeneous tests. MLTM-D is a confirmatory model that permits diagnosis at broad and more specific attribute or skill levels. In the current study, MLTM-D is applied to diagnose mastery at both the broad and specific skill levels in middle school mathematics. MLTM-D is presented, and methods involved in application are described. [ABSTRACT FROM PUBLISHER]

In this article the author discusses the aspects of the Cognitively Based Assessment of, for, and as Learning (CBAL) competency model. He mentions the involvement of critical thinking skills in mathematics associated with problem solving abilities. He says that overemphasis on thinking skills and mathematical problems that demands thinking skills may lead to several consequences including psychometric complexity, effects on phases of validity, and less stress on mathematical skills in assessment.

Cognitive complexity level is important for measuring both aptitude and achievement in largescale testing. Tests for standards-based assessment of mathematics, for example, often include cognitive complexity level in the test blueprint. However, little research exists on how mathematics items can be designed to vary in cognitive complexity level. In fact, determining the cognitive complexity level of items is usually based on correspondence to definitions rather than on empirically and theoretically justifiable variables that can predict item difficulty. In the current study, mathematical problem-solving items were designed for varying cognitive complexity levels based on a cognitive model of item processing. Structural variants of item models were designed to vary on two aspects of the cognitive model, the equation source and the number of subgoals. Participants were randomly assigned to test forms that contained different structural variants of the item models. Results from the linear logistic test model, the twoparameter-logistic-constrained model, and a corresponding linear mixed modeling procedure indicated that the item design variables affected both item difficulty and response time. Implications of the results for using structural variants in item generation and for the plausibility of the hypothesized cognitive model are discussed. [ABSTRACT FROM AUTHOR]

Although deficits in visual attention are often postulated as an important component of many declines in cognitive processing and functional outcomes in older adults, surprisingly little emphasis has been placed on evaluating psychometric instruments with which individual differences in visual attention ability can be assessed. This article reports the development and beginning psychometric evaluation of DriverScan, a change detection measure of attentional search for older adults. A constrained graded response model is used to approximate response speed and accuracy with categories of immediate, delayed, or no response. DriverScan items are shown to have excellent reliability over the studied sample, and the distribution of items is shown to adequately cover the difficulty continuum and to be maximally sensitive at distinguishing individuals with lower than average abilities (i.e., individuals with attention deficits). Item design features representing goal-directed and stimulus-driven attentional processing significantly predict item difficulty as hypothesized. [ABSTRACT FROM AUTHOR]

The article considers the contention of Professor S.E. Embretson that domain-referenced interpretations of ability require both a psychometric and cognitive foundation from the perspective of complex reading comprehension assessment. Research in both experimental cognitive psychology and educational practice indicates the importance of revealing a reader's mental model of the text. This model is likely to include inferences drawn from the text that are not necessarily implied with high certainty by the text. Thus, complex comprehension focuses on thinking and reasoning that integrates text information with information in other texts and with prior knowledge.

Although ability testing relies on static measures, these measures are often viewed as inappropriate for assessing learning potential. Learning processes are measured in contemporary cognitive psychology studies from processing durations while solving tasks and from the responsiveness of performance to changing conditions. This study examines the structure and nature of two learning process measures: response time and cognitive modifiability. Two primary questions are addressed: Do learning process measures represent different constructs than traditional static ability measures? and Do learning process measures rely more on other domains (e.g., personality, motivation) than do static measures? Results indicate that some learning process measures are sensitive to individual differences independent from static learning measures. Furthermore, the learning process specific variance is only partially explained by the other domains. However, the relationship of the learning process measures to static ability and to the other domains depends on the specific ability measured and the testing condition's design. [ABSTRACT FROM AUTHOR]

Contrasts the classical test theory and the model-based version of test theory, called item response theory (IRT). Review of six old rules of measurement that conflict with the new rules; Intuitive explanations of the new rules; Additional informational sources about IRT.

An abstract of the article "Using Cognitive Complexity to Measure the Psychometric Properties of Mathematics Assessment Items," by Kristin M. Morrison and Susan E. Embretson is presented.

Working memory capacity is an important construct in psychology because of its relationship with many higher-order cognitive abilities and psychopathologies. Working memory capacity is often measured using a type of paradigm known as complex span. Some recent work has focused on shortening the administration time of the complex span tasks, resulting in different versions of these tasks being used (Foster et al., 2015; Oswald, McAbee, Redick, & Hambrick, 2015). Variations in the complex span tasks, such as the number of set sizes, can lead to varying power to discriminate individuals at different ability levels. Thus, research findings may be inconsistent across populations due to differing appropriateness for the ability levels. The present study uses a combination of item response theory and correlational analyses to better understand the psychometric properties of the operation span, symmetry span, and rotation span. The findings show that the typical administration of these tasks, particularly the operation span, is not suitable for above average ability samples (Study 1; n = 573). When larger set sizes are added to the tasks (Study 2; n = 351), predictive validity and discriminability is improved for all complex span tasks, however the operation span is still inferior to the spatial tasks. The authors make several conclusions about which tasks and set sizes should be used depending on the intended population, and further suggest avoiding the standard-length operation span for average or higher ability populations. (PsycINFO Database Record [ABSTRACT FROM AUTHOR]