Search

Exploratory cognitive diagnosis models (CDMs) estimate the Q matrix, which is a binary matrix that indicates the attributes needed for affirmative responses to each item. Estimation of Q is an important next step for improving classifications and broadening application of CDMs. Prior research primarily focused on an exploratory version of the restrictive deterministic-input, noisy-and-gate model, and research is needed to develop exploratory methods for more flexible CDMs. We consider Bayesian methods for estimating an exploratory version of the more flexible reduced reparameterized unified model (rRUM). We show that estimating the rRUM Q matrix is complicated by a confound between elements of Q and the rRUM item parameters. A Bayesian framework is presented that accurately recovers Q using a spike-slab prior for item parameters to select the required attributes for each item. We present Monte Carlo simulation studies, demonstrating the developed algorithm improves upon prior Bayesian methods for estimating the rRUM Q matrix. We apply the developed method to the Examination for the Certificate of Proficiency in English data set. The results provide evidence of five attributes with a partially ordered attribute hierarchy.

The Journal of Educational & Behavioral Statistics has published a special section on the use of artificial intelligence (AI) in educational statistics. The growth of AI tools and applications has led to increased interest in its role in educational measurement. Currently, there are no guidelines in place regarding the use of AI in educational measurement, but the Duolingo document titled "Responsible AI Standards" (RAIS) provides a foundation for revising existing standards. The special section includes articles by experts in testing, measurement, psychometrics, and statistics, which offer insights to inform future revisions to the standards and address topics such as construct misspecification, scale discordance, and score bias. [Extracted from the article]

A family of learning models that integrates a cognitive diagnostic model and a higher-order, hidden Markov model in one framework is proposed. This new framework includes covariates to model skill transition in the learning environment. A Bayesian formulation is adopted to estimate parameters from a learning model. The developed methods are applied to a computer-based assessment with a learning intervention. The results show the potential application of the proposed model to track the change of students' skills directly and provide immediate remediation as well as to evaluate the efficacy of different interventions by investigating how different types of learning interventions impact the transitions from nonmastery to mastery.

In the absence of clear incentives, achievement tests may be subject to the effect of slipping where item response functions have upper asymptotes below one. Slipping reduces score precision for higher latent scores and distorts test developers' understandings of item and test information. A multidimensional four-parameter normal ogive model was developed for large-scale assessments and applied to dichotomous items of the 2011 National Assessment of Educational Progress eighth-grade mathematics and reading tests. The results suggest that the probability of slipping exceeded 5% for 47.2% and 51.1% of the dichotomous mathematics and reading items, respectively. Furthermore, allowing for slipping resulted in larger item discrimination parameters, increased information in the lower-to-middle range of the latent trait, and decreased precision for scores one standard deviation above the mean. The results provide evidence that slipping is a factor that should be considered during test development and construction to ensure adequate measurement across the latent continuum.

A latent multivariate regression model is developed that employs a generalized asymmetric Laplace (GAL) prior distribution for regression coefficients. The model is designed for high-dimensional applications where an approximate sparsity condition is satisfied, such that many regression coefficients are near zero after accounting for all the model predictors. The model is applicable to large-scale assessments such as the National Assessment of Educational Progress (NAEP), which includes hundreds of student, teacher, and school predictors of latent achievement. Monte Carlo evidence suggests that employing the GAL prior provides more precise estimation of coefficients that equal zero in comparison to a multivariate normal (MVN) prior, which translates to more accurate model selection. Furthermore, the GAL yielded less biased estimates of regression coefficients in smaller samples. The developed model is applied to mathematics achievement data from the 2011 NAEP for 175,200 eighth graders. The GAL and MVN NAEP estimates were similar, but the GAL was more parsimonious by selecting 12 fewer (i.e., 83 of the 148) variable groups. There were noticeable differences between estimates computed with a GAL prior and plausible value regressions with the AM software (beta version 0.06.00). Implications of the results are discussed for test developers and applied researchers.

A Bayesian model formulation of the deterministic inputs, noisy "and" gate (DINA) model is presented. Gibbs sampling is employed to simulate from the joint posterior distribution of item guessing and slipping parameters, subject attribute parameters, and latent class probabilities. The procedure extends concepts in Béguin and Glas, Culpepper, and Sahu for estimating the guessing and slipping parameters in the three- and four-parameter normal-ogive models. The ability of the model to recover parameters is demonstrated in a simulation study. The technique is applied to a mental rotation test. The algorithm and vignettes are freely available to researchers as the "dina" R package.

The COVID-19 pandemic has had a significant impact on students' educational achievement, leading to a need for robust formative assessments to improve outcomes. The Journal of Educational and Behavioral Statistics is publishing a special issue focused on statistical methods for enhancing educational and behavioral outcomes. The issue includes papers on diagnostic models and restricted latent class models, as well as factor analytic models for identifying and detecting beneficial educational interventions. The goal is to provide decision-makers and users with fine-grained information to support diagnostic educational decisions. [Extracted from the article]

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items. (Contains 7 figures.)

Measurement error significantly biases interaction effects and distorts researchers' inferences regarding interactive hypotheses. This article focuses on the single-indicator case and shows how to accurately estimate group slope differences by disattenuating interaction effects with errors-in-variables (EIV) regression. New analytic findings were presented along with simulation results to compare the relative bias, power, and Type I error rates of EIV, ordinary least squares (OLS), and sparse (i.e., single indicator) multigroup structural equation model (SEM) estimators of interaction effects in the presence of measurement error. The results suggest that EIV was less biased than were OLS and sparse SEM. Furthermore, OLS and sparse SEM were unable to control the Type I error rate for tests of slope differences in circumstances where groups differ in predictor reliability. Additional derivations examined the impact of using Cronbach's alpha, which is typically a lower bound for reliability, with EIV. The results provided evidence that using alpha does result in overcorrected EIV estimates and the bias in EIV estimates associated with using Cronbach's alpha increases with variability in item loadings and bias decreases as either test length or the average loading increases. The bias in EIV estimates when using alpha is not larger than the bias produced by using OLS or sparse SEM. In summary, the results provide compelling evidence that researchers should use EIV instead of OLS and sparse SEM to estimate group slope differences in the presence of measurement error. (Contains 9 figures and 2 notes.)

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by Sorbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error. (Contains 5 figures and 1 table.)

Statistical prediction remains an important tool for decisions in a variety of disciplines. An equally important issue is identifying factors that contribute to more or less accurate predictions. The time series literature includes well developed methods for studying predictability and volatility over time. This article develops distribution-appropriate methods for studying individual differences in predictability for settings in psychological research. Specifically, 3 different approaches are discussed for modeling predictability. The 1st is a bivariate measure of predictability discussed previously in the psychology literature, the squared or absolute valued difference between criterion and predictor, which is shown to follow the gamma distribution. The 2nd method extended limitations of previous research and involved understanding predictability in regression models. The 3rd method used nonlinear multilevel models to study predictability in settings where participants are nested within clusters. An application was presented using SAS NLMIXED to understand the predictability of college grade point average by student demographic characteristics. The findings from the application suggest that the 1st-year college performance of English as a second language students were, on average, less predictable whereas females and Whites tended to demonstrate more predictable academic performance than their male or racial/ethnic minority counterparts. (Contains 2 tables, 1 figure and 2 footnotes.)

This study linked nonlinear profile analysis (NPA) of dichotomous responses with an existing family of item response theory models and generalized latent variable models (GLVM). The NPA method offers several benefits over previous internal profile analysis methods: (a) NPA is estimated with maximum likelihood in a GLVM framework rather than relying on the choice of different dissimilarity measures that produce different results, (b) item and person parameters are computed during the same estimation step with an appropriate distribution for dichotomous variables, (c) the model estimates profile coordinate standard errors, and (d) additional individual-level variables can be included to model relationships with the profile parameters. An application examined experimental differences in topographic map comprehension among 288 subjects. The model produced a measure of overall test performance or comprehension in addition to pattern variables that measured the correspondence between subject response profiles and an item difficulty profile and an item-discrimination profile. The findings suggested that subjects who used 3-dimensional maps tended to correctly answer more items in addition to correctly answering items that were more discriminating indicators of map comprehension. The NPA analysis was also compared with results from a multidimensional item response theory model. (Contains 2 tables and 3 footnotes.)

Researchers continue to develop and advance models for diagnostic research in the social and behavioral sciences. These diagnostic models (DMs) provide researchers with a framework for providing a fine-grained classification of respondents into substantively meaningful latent classes as defined by a multivariate collection of binary attributes. A central concern for DMs is advancing exploratory methods for uncovering the latent structure, which corresponds with the relationship between unobserved binary attributes and observed polytomous items with two or more response options. Multivariate behavioral polytomous data are often collected within a higher-order design where general factors underlying first-order latent variables. This study advances existing exploratory DMs for polytomous data by proposing a new method for inferring the latent structure underlying polytomous response data using a higher-order model to describe dependence among the discrete latent attributes. We report a novel Bayesian formulation that uses variable selection techniques for inferring the latent structure along with a higher-order factor model for attributes. We report evidence of accurate parameter recovery in a Monte Carlo simulation study and present results from an application to the 2012 Programme for International Student Assessment (PISA) problem-solving vignettes to demonstrate the method. [ABSTRACT FROM AUTHOR]

Exploratory factor analysis is a dimension-reduction technique commonly used in psychology, finance, genomics, neuroscience, and economics. Advances in computational power have opened the door for fully Bayesian treatments of factor analysis. One open problem is enforcing rotational identifability of the latent factor loadings, as the loadings are not identified from the likelihood without further restrictions. Nonidentifability of the loadings can cause posterior multimodality, which can produce misleading posterior summaries. The positive-diagonal, lower-triangular (PLT) constraint is the most commonly used restriction to guarantee identifiability, in which the upper m × m submatrix of the loadings is constrained to be a lower-triangular matrix with positive-diagonal elements. The PLT constraint can fail to guarantee identifiability if the constrained submatrix is singular. Furthermore, though the PLT constraint addresses identifiability-related multimodality, it introduces additional mixing issues. We introduce a new Bayesian sampling algorithm that efficiently explores the multimodal posterior surface and addresses issues with PLT-constrained approaches. for this article are available online. [ABSTRACT FROM AUTHOR]