Search

Understanding the cognitive processes, skills and strategies that examinees use in testing is important for construct validity and score interpretability. Although response processes evidence has long been included as an important aspect of validity (i.e., "Standards for Educational and Psychological Tests," 1999), relevant studies are often lacking, especially in large scale educational and psychological testing. An important method for studying response processes involves explanatory mathematical modeling of item responses and item response times from variables that represent sources of cognitive complexity. For many item types, examinees may differ in strategies applied to responding to items. Mixture class item response theory models can identify latent classes of examinees with different processes, skills and strategies based on their pattern of item responses. This study will illustrate the use of response times in conjunction with explanatory item response theory models and mixture models, to provide information relevant to test validity and, hence, to score interpretations.

The Concept Formation subtest of the Woodcock Johnson Tests of Cognitive Abilities represents a dynamic test due to continual provision of feedback from examiner to examinee. Yet, the original scoring protocol for the test largely ignores this dynamic structure. The current analysis applies a dynamic adaptation of an explanatory item response theory model to evaluate the impact of feedback on item difficulty. Additionally, several item features (rule type, number of target shapes) are considered in the item difficulty model. Results demonstrated that all forms of feedback significantly reduced item difficulty, with the exception of corrective feedback that could not be directly applied to the next item in the series. More complex and compound rule types also significantly predicted item difficulty, as did increasing the number of shapes, thereby supporting the response process aspect of validity. Implications for continued use of the Concept Formation subtest for educational programming decisions are discussed.

Examinees' thinking processes have become an increasingly important concern in testing. The responses processes aspect is a major component of validity, and contemporary tests increasingly involve specifications about the cognitive complexity of examinees' response processes. Yet, empirical research findings on examinees' cognitive processes are not often available either to provide evidence for validity or to guide the design or selection of items. In this article, studies and developments from the author's research program are presented to illustrate how empirical studies on examinees' thinking processes can impact item and test design.

Online and on-demand tests are increasingly used in assessment. Although the main focus has been cheating and test security (e.g., Selwyn, 2008) the cross-setting equivalence of scores as a function of contrasting test conditions is also an issue that warrants attention. In this study, the impact of environmental and cognitive distractions, as well as personality, on performance on a computerized mathematical problem-solving test was examined by comparing results in a standardized, proctored (school) setting with a non-standardized, unproctored (home) setting. Science and engineering college students (N = 315) were randomly assigned to one of these two test settings and completed a mathematical reasoning test. As hypothesized, model-based analysis demonstrated that greater environmental and cognitive distractions were reported in the unproctored (home) setting, while performance levels were higher in the proctored (school) setting. Furthermore, the process by which these variables impact performance was found to differ between settings. Namely, a statistical comparison of multiple path models supported the hypothesis that cognitive distractions are primarily responsible for the impact of both test anxiety and self-regulation on test performance, although this effect is stronger in non-standardized setting. Implications of these findings for Web, home-based educational courses, and assessments are discussed. (Contains 1 figure and 2 tables.)

This paper presents a noncompensatory latent trait model, the multicomponent latent trait model for diagnosis (MLTM-D), for cognitive diagnosis. In MLTM-D, a hierarchical relationship between components and attributes is specified to be applicable to permit diagnosis at two levels. MLTM-D is a generalization of the multicomponent latent trait model (MLTM; Whitely in "Psychometrika," 45:479-494, 1980; Embretson in "Psychometrika," 49:175-186, 1984) to be applicable to measures of broad traits, such as achievement tests, in which component structure varies "between" items. Conditions for model identification are described and marginal maximum likelihood estimators are presented, along with simulation data to demonstrate parameter recovery. To illustrate how MLTM-D can be used for diagnosis, an application to a large-scale test of mathematics achievement is presented. An advantage of MLTM-D for diagnosis is that it may be more applicable to large-scale assessments with more heterogeneous items than are latent class models.

Cognitive complexity level is important for measuring both aptitude and achievement in large-scale testing. Tests for standards-based assessment of mathematics, for example, often include cognitive complexity level in the test blueprint. However, little research exists on how mathematics items can be designed to vary in cognitive complexity level. In fact, determining the cognitive complexity level of items is usually based on correspondence to definitions rather than on empirically and theoretically justifiable variables that can predict item difficulty. In the current study, mathematical problem-solving items were designed for varying cognitive complexity levels based on a cognitive model of item processing. Structural variants of item models were designed to vary on two aspects of the cognitive model, the equation source and the number of subgoals. Participants were randomly assigned to test forms that contained different structural variants of the item models. Results from the linear logistic test model, the two-parameter-logistic-constrained model, and a corresponding linear mixed modeling procedure indicated that the item design variables affected both item difficulty and response time. Implications of the results for using structural variants in item generation and for the plausibility of the hypothesized cognitive model are discussed. (Contains 3 figures and 4 tables.)

Spatial ability tasks appear on many intelligence and aptitude tests. Although the construct validity of spatial ability tests has often been studied through traditional correlational methods, such as factor analysis, less is known about the cognitive processes involved in solving test items. This study examines the cognitive processes involved in the assembling objects task found on the Revised Minnesota Paper Form Board Test (RMPFBT), a spatial task in which a test taker must mentally assemble two-dimensional objects. A two-part study was conducted to examine the construct validity of these items and to explain how test takers solve assembling objects tasks. The first part of the study used quantitative methods to examine the contribution of item characteristics to the difficulty of the item and the response time necessary to solve the item. The second part of the study utilized an eye-tracking device to qualitatively examine the steps examinees follow to solve these items. As a result of the second part of this study, a new cognitive processing model is proposed. (Contains 3 figures and 6 tables.)

This article presents the author's comments on Randy Elliot Bennett's article "Cognitively Based Assessment of, for, and as Learning" (CBAL), which appeared in issue 8(2-3). The article presents an integrated approach to assessment in which a theory of action for the intended consequences (i.e., benefits) drives both assessments and instructional practices. In the Cognitively Based Assessment of, for, and as Learning (CBAL) project assessment is the mainstay of instruction. CBAL has several distinctive features: (1) a goal to facilitate learning as well as to assess it, (2) an explicitly formulated student competency model, (3) a basis in scientific learning principles as well as standards-based content, (4) an integration of formative and summative assessments, and (5) an explicitly developed theory of action. The theory of action, formulated by Bennett (2010), is a "depiction of an intervention that explains the cause-effect relationships among input, activities, and intended outcomes." Thus, intended and intermediate outcomes are specified, along with the inputs and activities that are related to these outcomes. Bennett's broad and inclusive conceptualization of assessment, not only provides a strong impact on the activities and ultimate effectiveness of CBAL, but also provides a model for other assessment programs. Combined with a proscribed rigorous research to support the validity claims, this systematic approach seemingly has potential to improve student competency levels. This commentary focuses on mathematics assessments but similar comments may apply to reading and writing assessments.

H. Blanton and J. Jaccard examined the arbitrariness of metrics in the context of 2 current issues: (a) the measurement of racial prejudice and (b) the establishment of clinically significant change. According to Blanton and Jaccard, although research findings are not undermined by arbitrary metrics, individual scores and score changes may not be meaningfully interpreted. The author believes that their points are mostly valid and that their examples were appropriate. However, Blanton and Jaccard's article does not lead directly to solutions, nor did it adequately describe the scope of the metric problem. This article has 2 major goals. First, some prerequisites for nonarbitrary metrics are presented and related to Blanton and Jaccard's issues. Second, the impact of arbitrary metrics on psychological research findings are described. In contrast to Blanton and Jaccard (2006), research findings suggest that metrics have direct impact on statistics for group comparisons and trend analysis.

Recent assessment research joining cognitive psychology and psychometric theory has introduced a new technology, item generation. In algorithmic item generation, items are systematically created based on specific combinations of features that underlie the processing required to correctly solve a problem. Reading comprehension items have been more difficult to model than other item types due to the complexities of quantifying text. However, recent developments in artificial intelligence for text analysis permit quantitative indices to represent cognitive sources of difficulty. The current study attempts to identify generative components for the Graduate Record Examination paragraph comprehension items through the cognitive decomposition of item difficulty. Text comprehension and decision processes accounted for a significant amount of the variance in item difficulties. The decision model variables contributed significantly to variance in item difficulties, whereas the text representation variables did not. Implications for score interpretation and future possibilities for item generation are discussed. (Contains 1 figure and 5 tables.)

"The Second Century of Ability Testing: Some Predictions and Speculations" did not include predictions about the ability construct or the role of fundamental measurement principles. All commentators raised issues about the nature of the ability construct. The diverse viewpoints represented in these comments highlight well the complexity of the construct issues. Measurement principles were also mentioned by several authors. Some of these commentaries have directly opposing concerns about the role of measurement models in ability testing. Although the commentaries had several other valuable points, the construct and measurement issues need summarization and integration. In this paper, the author summarizes and elaborates the issues in these areas and speculates on their place in the second century of ability testing.

The last century was marked by dazzling changes in many areas, such as technology and communications. Predictions into the second century of testing are seemingly difficult in such a context. Yet, looking back to the turn of the last century, Kirkpatrick (1900), in his American Psychological Association presidential address, presented fundamental desiderata for ability testing that in fact guides major testing research even today (normative age standards, emphasis on culture-general tasks, simultaneous measurement of many persons and adult ability measurement). An overview of the last century, presented in this address, shows that most fundamental principles in psychometrics and testing were available by 1930. With a few notable exceptions, the remainder of the last century of testing was devoted to applying or refining these principles. It is predicted that the same pattern will occur in this century of testing. Further developments in model-based measurement and in applying cognitive psychology principles in testing, intermingled with technology, will guide ability testing throughout the next century. These changes, to be elaborated in detail, include fundamental changes in test development procedures, the nature of the measuring tasks, aspects of ability that are measured, and types of interpretations given to ability. (Contains 8 figures and 3 tables.)

Discusses computerized dynamic testing with cues and items presented according to objective algorithms, elaborating on appropriate designs and psychometric models. Presents two studies involving 311 military recruits and 584 recruits that support the psychometric properties of a test measuring the susceptibility of reasoning to stressors. (SLD)