Search

Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals' behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals' test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning.

Using the action sequence data (i.e., log data) from the problem-solving in technology-rich environments assessment on the 2012 Programme for the International Assessment of Adult Competencies survey, the current study examines the associations between adult digital problem-solving behavior and several demographic and cognitive variables. Action sequence features extracted using multidimensional scaling (Tang, Wang, He, Liu, & Ying, 2019) and sequence-to-sequence autoencoders (Tang, Wang, Liu, & Ying, 2019) were used to predict test-taker external characteristics. Features extracted from action sequences were consistently found to contain more information on demographic and cognitive characteristics than final scores. Partial least squares analyses further revealed systematic associations between behavioral patterns and demographic/cognitive characteristics.

Accurate assessment of students' ability is the key task of a test. Assessments based on final responses are the standard. As the infrastructure advances, substantially more information is observed. One of such instances is the process data that is collected by computer-based interactive items, which contain a student's detailed interactive processes. In this paper, we show both theoretically and empirically that appropriately including such information in the assessment will substantially improve relevant assessment precision. The precision is measured empirically by out-of-sample test reliability.

Response process data collected from human-computer interactive items contain rich information about respondents' behavioral patterns and cognitive processes. Their irregular formats as well as their large sizes make standard statistical tools difficult to apply. This paper develops a computationally efficient method for exploratory analysis of such process data. The new approach segments a lengthy individual process into a sequence of short subprocesses to achieve complexity reduction, easy clustering and meaningful interpretation. Each subprocess is considered a subtask. The segmentation is based on sequential action predictability using a parsimonious predictive model combined with the Shannon entropy. Simulation studies are conducted to assess performance of the new methods. We use the process data from PIAAC 2012 to demonstrate how exploratory analysis of process data can be done with the new approach., Comment: 34 pages, 10 figures

In this paper, we develop asymptotic theories for a class of latent variable models for large-scale multi-relational networks. In particular, we establish consistency results and asymptotic error bounds for the (penalized) maximum likelihood estimators when the size of the network tends to infinity. The basic technique is to develop a non-asymptotic error bound for the maximum likelihood estimators through large deviations analysis of random fields. We also show that these estimators are nearly optimal in terms of minimax risk., Comment: 34 pages, 2 figures

Process data refer to data recorded in the log files of computer-based items. These data, represented as timestamped action sequences, keep track of respondents' response processes of solving the items. Process data analysis aims at enhancing educational assessment accuracy and serving other assessment purposes by utilizing the rich information contained in response processes. The R package ProcData presented in this article is designed to provide tools for processing, describing, and analyzing process data. We define an S3 class "proc" for organizing process data and extend generic methods summary and print for class "proc". Two feature extraction methods for process data are implemented in the package for compressing information in the irregular response processes into regular numeric vectors. ProcData also provides functions for fitting and making predictions from a neural-network-based sequence model. These functions call relevant functions in package keras for constructing and training neural networks. In addition, several response process generators and a real dataset of response processes of the climate control item in the 2012 Programme for International Student Assessment are included in the package.

Computer simulations have become a popular tool of assessing complex skills such as problem-solving skills. Log files of computer-based items record the entire human-computer interactive processes for each respondent. The response processes are very diverse, noisy, and of nonstandard formats. Few generic methods have been developed for exploiting the information contained in process data. In this article, we propose a method to extract latent variables from process data. The method utilizes a sequence-to-sequence autoencoder to compress response processes into standard numerical vectors. It does not require prior knowledge of the specific items and human-computers interaction patterns. The proposed method is applied to both simulated and real process data to demonstrate that the resulting latent variables extract useful information from the response processes., Comment: 28 pages, 13 figures

Computer-based interactive items have become prevalent in recent educational assessments. In such items, the entire human-computer interactive process is recorded in a log file and is known as the response process. This paper aims at extracting useful information from response processes. In particular, we consider an exploratory latent variable analysis for process data. Latent variables are extracted through a multidimensional scaling framework and can be empirically proved to contain more information than classic binary responses in terms of out-of-sample prediction of many variables., Comment: 26 pages, 11 figures

In this paper, we aim at solving a class of multiple testing problems under the Bayesian sequential decision framework. Our motivating application comes from binary labeling tasks in crowdsourcing, where the requestor needs to simultaneously decide which worker to choose to provide the label and when to stop collecting labels under a certain budget constraint. We start with the binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under the adaptive sequential probability ratio test (Ada-SPRT) framework. We characterize the structure of the optimal solution, i.e., optimal adaptive sequential design, which minimizes the Bayes risk through log-likelihood ratio statistic. We also develop a dynamic programming algorithm that can efficiently approximate the optimal solution. For the multiple testing problem, we further propose to adopt an empirical Bayes approach for estimating class priors and show that our method has an averaged loss that converges to the minimal Bayes risk under the true model. The experiments on both simulated and real data show the robustness of our method and its superiority in labeling accuracy as compared to several other recently proposed approaches.

The study presents an exploratory graphical modeling approach for evaluating local item dependency within cognitively diagnostic classification models (DCMs). Current approaches to modeling local dependence require known item structure and have limited utility when such information is not available. In this study, we propose an exploratory approach to modeling local dependence so that items' own interactions can be revealed without dependency specification. The new framework is developed by integrating a Markov network into a generalized DCM. The framework unveils item interactions while performing regular cognitive diagnosis within a unified scheme. The inference on the model parameters is made on the regularized pseudo-likelihood and is implemented by an EM algorithm. Numerical experimentation from Monte Carlo simulation suggests that the proposed framework adequately recovers generating parameters and yields reliable standard error estimates. Compared with the regular DCM, the new model produced more accurate item parameter estimates as items exhibit local dependence. The study demonstrates application of the model using two real assessment data and discusses practical benefits of modeling local dependence.

This paper establishes fundamental results for statistical inference of diagnostic classification models (DCM). The results are developed at a high level of generality, applicable to essentially all diagnostic classification models. In particular, we establish identifiability results of various modeling parameters, notably item response probabilities, attribute distribution, and Q-matrix-induced partial information structure. Consistent estimators are constructed. Simulation results show that these estimators perform well under various modeling settings. We also use a real example to illustrate the new method. The results are stated under the setting of general latent class models. For DCM with a specific parameterization, the conditions may be adapted accordingly.

Partial differential equations with random inputs have become popular models to characterize physical systems with uncertainty coming from, e.g., imprecise measurement and intrinsic randomness. In this paper, we perform asymptotic rare event analysis for such elliptic PDEs with random inputs. In particular, we consider the asymptotic regime that the noise level converges to zero suggesting that the system uncertainty is low, but does exists. We develop sharp approximations of the probability of a large class of rare events.

We consider modeling, inference, and computation for analyzing multivariate binary data. We propose a new model that consists of a low dimensional latent variable component and a sparse graphical component. Our study is motivated by analysis of item response data in cognitive assessment and has applications to many disciplines where item response data are collected. Standard approaches to item response data in cognitive assessment adopt the multidimensional item response theory (IRT) models. However, human cognition is typically a complicated process and thus may not be adequately described by just a few factors. Consequently, a low-dimensional latent factor model, such as the multidimensional IRT models, is often insufficient to capture the structure of the data. The proposed model adds a sparse graphical component that captures the remaining ad hoc dependence. It reduces to a multidimensional IRT model when the graphical component becomes degenerate. Model selection and parameter estimation are carried out simultaneously through construction of a pseudo-likelihood function and properly chosen penalty terms. The convexity of the pseudo-likelihood function allows us to develop an efficient algorithm, while the penalty terms generate a low-dimensional latent component and a sparse graphical structure. Desirable theoretical properties are established under suitable regularity conditions. The method is applied to the revised Eysenck's personality questionnaire, revealing its usefulness in item analysis. Simulation results are reported that show the new method works well in practical situations., Comment: 49 pages, 6 figures, and 5 tables

The asymptotic efficiency of a generalized likelihood ratio test proposed by Cox is studied under the large deviations framework for error probabilities developed by Chernoff. In particular, two separate parametric families of hypotheses are considered [Cox, 1961, 1962]. The significance level is set such that the maximal type I and type II error probabilities for the generalized likelihood ratio test decay exponentially fast with the same rate. We derive the analytic form of such a rate that is also known as the Chernoff index [Chernoff, 1952], a relative efficiency measure when there is no preference between the null and the alternative hypotheses. We further extend the analysis to approximate error probabilities when the two families are not completely separated. Discussions are provided concerning the implications of the present result on model selection.

Partial differential equation is a powerful tool to characterize various physics systems. In practice, measurement errors are often present and probability models are employed to account for such uncertainties. In this paper, we present a Monte Carlo scheme that yields unbiased estimators for expectations of random elliptic partial differential equations. This algorithm combines multilevel Monte Carlo [Giles, 2008] and a randomization scheme proposed by [Rhee and Glynn, 2012, Rhee and Glynn, 2013]. Furthermore, to obtain an estimator with both finite variance and finite expected computational cost, we employ higher order approximations.

This work investigates the sequential hypothesis testing problem with online sensor selection and sensor usage constraints. That is, in a sensor network, the fusion center sequentially acquires samples by selecting one "most informative" sensor at each time until a reliable decision can be made. In particular, the sensor selection is carried out in the online fashion since it depends on all the previous samples at each time. Our goal is to develop the sequential test (i.e., stopping rule and decision function) and sensor selection strategy that minimize the expected sample size subject to the constraints on the error probabilities and sensor usages. To this end, we first recast the usage-constrained formulation into a Bayesian optimal stopping problem with different sampling costs for the usage-contrained sensors. The Bayesian problem is then studied under both finite- and infinite-horizon setups, based on which, the optimal solution to the original usage-constrained problem can be readily established. Moreover, by capitalizing on the structures of the optimal solution, a lower bound is obtained for the optimal expected sample size. In addition, we also propose algorithms to approximately evaluate the parameters in the optimal sequential test so that the sensor usage and error probability constraints are satisfied. Finally, numerical experiments are provided to illustrate the theoretical findings, and compare with the existing methods., Comment: 33 pages

This paper considers the sequential composite hypothesis test with multiple sensors. The sensors observe random samples in parallel and communicate with a fusion center, who makes the global decision based on the sensor inputs. On one hand, in the centralized scenario, where local samples are precisely transmitted to the fusion center, the generalized sequential likelihood ratio test (GSPRT) is shown to be asymptotically optimal in terms of the expected sample size as error rates tend to zero. On the other hand, for systems with limited power and bandwidth resources, decentralized solutions that only send a summary of local samples (we particularly focus on a one-bit communication protocol) to the fusion center is of great importance. To this end, we first consider a decentralized scheme where sensors send their one-bit quantized statistics every fixed period of time to the fusion center. We show that such a uniform sampling and quantization scheme is strictly suboptimal and its suboptimality can be quantified by the KL divergence of the distributions of the quantized statistics under both hypotheses. We then propose a decentralized GSPRT based on level-triggered sampling. That is, each sensor runs its own GSPRT repeatedly and reports its local decision to the fusion center asynchronously. We show that this scheme is asymptotically optimal as the local thresholds and global thresholds grow large at different rates. Lastly, two particular models and their associated applications are studied to compare the centralized and decentralized approaches. Numerical results are provided to demonstrate that the proposed level-triggered sampling based decentralized scheme aligns closely with the centralized scheme with substantially lower communication overhead, and significantly outperforms the uniform sampling and quantization based decentralized scheme., Comment: 39 pages

In this paper we study rare events associated to solutions of elliptic partial differential equations with spatially varying random coefficients. The random coefficients follow the lognormal distribution, which is determined by a Gaussian process. This model is employed to study the failure problem of elastic materials in random media in which the failure is characterized by that the strain field exceeds a high threshold. We propose an efficient importance sampling scheme to compute small failure probabilities in the high threshold limit. The change of measure in our scheme is parametrized by two density functions. The efficiency of the importance sampling scheme is validated by numerical examples., Comment: 17 pages, 4 figures, 5 tables

We study a new technique for the asymptotic analysis of heavy-tailed systems conditioned on large deviations events. We illustrate our approach in the context of ruin events of multidimensional regularly varying random walks. Our approach is to study the Markov process described by the random walk conditioned on hitting a rare target set. We construct a Markov chain whose transition kernel can be evaluated directly from the increment distribution of the associated random walk. This process is shown to approximate the conditional process of interest in total variation. Then, by analyzing the approximating process, we are able to obtain asymptotic conditional joint distributions and a conditional functional central limit theorem of several objects such as the time until ruin, the whole random walk prior to ruin, and the overshoot on the target set. These types of joint conditional limit theorems have been obtained previously in the literature only in the one dimensional case. In addition to using different techniques, our results include features that are qualitatively different from the one dimensional case. For instance, the asymptotic conditional law of the time to ruin is no longer purely Pareto as in the multidimensional case., Comment: Published in at http://dx.doi.org/10.3150/12-BEJ492 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

In this paper, we consider a classic problem concerning the high excursion probabilities of a Gaussian random field $f$ living on a compact set $T$. We develop efficient computational methods for the tail probabilities $P(\sup_T f(t) > b)$ and the conditional expectations $E(\Gamma(f) | \sup_T f(t) > b)$ as $b\rightarrow \infty$. For each $\varepsilon$ positive, we present Monte Carlo algorithms that run in \emph{constant} time and compute the interesting quantities with $\varepsilon$ relative error for arbitrarily large $b$. The efficiency results are applicable to a large class of H\"older continuous Gaussian random fields. Besides computations, the proposed change of measure and its analysis techniques have several theoretical and practical indications in the asymptotic analysis of extremes of Gaussian random fields.

In this paper, we consider a stochastic system described by a differential equation admitting a spatially varying random coefficient. The differential equation has been employed to model various static physics systems such as elastic deformation, water flow, electric-magnetic fields, temperature distribution, etc. A random coefficient is introduced to account for the system's uncertainty and/or imperfect measurements. This random coefficient is described by a Gaussian process (the input process) and thus the solution to the differential equation (under certain boundary conditions) is a complexed functional of the input Gaussian process. In this paper, we focus the analysis on the one-dimensional case and derive asymptotic approximations of the tail probabilities of the solution to the equation that has various physics interpretations under different contexts. This analysis rests on the literature of the extreme analysis of Gaussian processes (such as the tail approximations of the supremum) and extends the analysis to more complexed functionals., Comment: supplementary material is included

In this paper, we consider the extreme behavior of a Gaussian random field $f(t)$ living on a compact set $T$. In particular, we are interested in tail events associated with the integral $\int_Te^{f(t)}\,dt$. We construct a (non-Gaussian) random field whose distribution can be explicitly stated. This field approximates the conditional Gaussian random field $f$ (given that $\int_Te^{f(t)}\,dt$ exceeds a large value) in total variation. Based on this approximation, we show that the tail event of $\int_Te^{f(t)}\,dt$ is asymptotically equivalent to the tail event of $\sup_T\gamma(t)$ where $\gamma(t)$ is a Gaussian process and it is an affine function of $f(t)$ and its derivative field. In addition to the asymptotic description of the conditional field, we construct an efficient Monte Carlo estimator that runs in polynomial time of $\log b$ to compute the probability $P(\int_Te^{f(t)}\,dt>b)$ with a prescribed relative accuracy., Comment: Published in at http://dx.doi.org/10.1214/13-AAP960 the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Recent surge of interests in cognitive assessment has led to the developments of novel statistical models for diagnostic classification. Central to many such models is the well-known Q-matrix, which specifies the item-attribute relationship. This paper proposes a principled estimation procedure for the Q-matrix and related model parameters. Desirable theoretic properties are established through large sample analysis. The proposed method also provides a platform under which important statistical issues, such as hypothesis testing and model selection, can be addressed.

Discussion of "A statistical analysis of multiple temperature proxies: Are reconstructions of surface temperatures over the last 1000 years reliable?" by B.B. McShane and A.J. Wyner [arXiv:1104.4002], Comment: Published in at http://dx.doi.org/10.1214/10-AOAS398C the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)