Search

Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents' decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents' behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.

This paper presents a model of recurrent multinomial sequences. Though there exists a quite considerable literature on modeling autocorrelation in numerical data and sequences of categorical outcomes, there is currently no systematic method of modeling patterns of recurrence in categorical sequences. This paper develops a means of discovering recurrent patterns by employing a more restrictive Markov assumption. The resulting model, which I call the recurrent multinomial model, provides a parsimonious representation of recurrent sequences, enabling the investigation of recurrences on longer time scales than existing models. The utility of recurrent multinomial models is demonstrated by applying them to the case of conversational turn-taking in meetings of the Federal Open Market Committee (FOMC). Analyses are effectively able to discover norms around turn-reclaiming, participation, and suppression and to evaluate how these norms vary throughout the course of the meeting.

Understanding the teacher's technology adoption process is essential to comprehend and narrow the digital divide in the post-epidemic age. During the pandemic, the stay-at-home orders not only intervened schooling and teaching but also increased digital accessibility to teachers. This research studies teacher heterogeneity and adoption controls in the epidemic to simultaneously affect teacher's underlying intention and adoption behavior based on a dynamic framework under the theory of planned behavior. We present a quantitative framework for modeling the teachers' adoption behavior of a technology conditioned on intention, defined as latent dynamic processes via a hidden Markov model. This model allows us to examine the effects of three concerned adoption controls: epidemic, community, and experience. We also explicitly characterized teachers' digital traits as the estimated results accounts for teacher's heterogeneity. The implicit quality of digital teaching artifacts is examined to correlate the dynamic analyses with the qualitative supports. We collected data from four primary schools in Shanghai over 173 weeks, using an after-school activity management system. The data collection spanned periods both before and after the school closure caused by the epidemic, providing us with a dynamic view of technology adoption patterns under different circumstances. Our results suggest that the interventions derived from the controls of the epidemic did not significantly narrow the digital gap. In particular, well-prepared teachers may be more sensitive to adjusting their usage to meet the evolving standards. The inexperienced teacher struggles to maintain a high level of adoption once the passive external pressure is eliminated. Even the compulsory policy can temporarily change their adoption behavior. These implications highlight the second-order digital divide problem.

Correct pronunciation significantly increases the intelligibility of communication. However, it is uncertain whether acquiring the pronunciation of the words enhances word retention capability. Therefore, the major purpose of this research is to evaluate whether vocabulary acquisition with the aid of pronouncing with artificial intelligence leads to a longer memory. In this research, a full experimental pattern, and a pre-test and post-test control group design were applied. Furthermore, a total of 56 high school students aged between 14-15 were asked to memorize unknown vocabulary with two pronunciation teaching methods. Prior to the experimental process, the pre-test was applied to both groups, and then, the artificial intelligence-based speech recognition pronunciation teaching process was to the experimental group while the phonetic alphabet pronunciation process was to the control group on the 4th, 8th, and 12th weeks. According to the findings, it was obtained that pronunciation practice via artificial intelligence-enabled the words to remain in memory longer. Additionally, the participants' views were gathered at the end of the research. For further research, this study will benefit other research with a variety of accessible tools to meet objectives by utilizing a new artificial intelligence-supported pronunciation model, through recording and reacting to learners' pronouncing practices in different languages.

Prior programming knowledge of students has a major impact on introductory programming courses. Those with prior experience often seem to breeze through the course. Those without prior experience see others breeze through the course and disengage from the material or drop out. The purpose of this study is to demonstrate that novice student programming behavior can be modeled as a Markov process. The resulting transition matrix can then be used in machine learning algorithms to create clusters of similarly behaving students. We describe in detail the state machine used in the Markov process and how to compute the transition matrix. We compute the transition matrix for 665 students and cluster them using the k-means clustering algorithm. We choose the number of cluster to be three based on analysis of the dataset. We show that the created clusters have statistically different means for student prior knowledge in programming, when measured on a Likert scale of 1-5.

High dropout rate exists universally in massive open online courses (MOOCs) due to the separation of teachers and learners in space and time. Dropout prediction using the machine learning method is an extremely important prerequisite to identify potential at-risk learners to improve learning. It has attracted much attention and there have emerged a few reviews. However, current reviews of MOOC dropout prediction exist some common limitations. Firstly, different definitions of course dropout are not summarized. Secondly, there lacks an overall framework of MOOC dropout prediction. Thirdly, some key challenges are not fully explored. Thus, unlike past reviews, this systematic review concludes with three categories of definitions of course dropout. Then it proposes an overall framework including factors affecting dropout, general feature extraction methods, various machine learning methods and evaluation methods. Finally, the key challenges of interpretability, imbalanced data and semantic learning trajectory modeling are proposed. This study aims to enable researchers to capture a whole picture of dropout prediction from the perspective of machine learning.

Blended learning (BL) has become increasingly popular in higher education institutions. Despite its popularity and the advances in methodologies for the detection of learning tactics and strategies from trace data, little is known about how they apply to BL settings and, therefore, how students use them to plan, organize, monitor, and regulate their learning in these settings. To address this gap, we analyzed the manifestations of learning tactics and strategies of 267 students across three undergraduate-level BL courses with different course designs, instructional activities, and learning contexts. We employed a data-driven method that incorporates hidden Markov models to determine students' learning tactics. Then, we employed optimal matching to identify the students' strategies based on the sequences of tactics they deployed and how they relate to their self-reported self-regulated learning (SRL) skills. Our results indicate that students' tactics and strategies varied significantly depending on the course design and learning context. Tactics with regard to the use of time management resources were common across courses. In contrast, tactics deployed when revisiting old material and interacting with an SRL support tool were course-specific. We identified strategies related to surface and deep learning and found that surface-level strategies manifested consistently across all courses. These findings contribute to a better understanding of student learning mechanisms in BL environments and have implications for instructional design and SRL support.

The testlet design is very popular in educational and psychological assessments. This article proposes a new cognitive diagnosis model, the multiple-choice cognitive diagnostic testlet (MC-CDT) model for tests using testlets consisting of MC items. The MC-CDT model uses the original examinees' responses to MC items instead of dichotomously scored data (i.e., correct or incorrect) to retain information of different distractors and thus enhance the MC items' diagnostic power. The Markov chain Monte Carlo algorithm was adopted to calibrate the model using the WinBUGS software. Then, a thorough simulation study was conducted to evaluate the estimation accuracy for both item and examinee parameters in the MC-CDT model under various conditions. The results showed that the proposed MC-CDT model outperformed the traditional MC cognitive diagnostic model. Specifically, the MC-CDT model fits the testlet data better than the traditional model, while also fitting the data without testlets well. The findings of this empirical study show that the MC-CDT model fits real data better than the traditional model and that it can also provide testlet information.

Recent usage of virtual reality (VR) technology in surgical training has emerged because of its cost-effectiveness, time savings, and cognition-based feedback generation. However, the quantitative evaluation of its effectiveness in training is still not thoroughly studied. This article demonstrates the effectiveness of a VR-based surgical training simulator in laparoscopic surgery and investigates how stochastic modeling, represented as continuous-time Markov chain (CTMC), can be used to explicit determine the training status of the surgeon. By comparing the training in real environments and in VR-based training simulators, the authors also explore the validity of the VR simulator in laparoscopic surgery. The study further aids in establishing learning models for surgeons, supporting continuous evaluation of training processes for the derivation of real-time feedback by CTMC-based modeling.

Questionnaires are by far the most common tool for measuring noncognitive constructs in psychology and educational sciences. Response bias may pose an additional source of variation between respondents that threatens validity of conclusions drawn from questionnaire data. We present a mixture modeling approach that leverages response time data from computer-administered questionnaires for the joint identification and modeling of two commonly encountered response bias that, so far, have only been modeled separately--careless and insufficient effort responding and response styles (RS) in attentive answering. Using empirical data from the Programme for International Student Assessment 2015 background questionnaire and the case of extreme RS as an example, we illustrate how the proposed approach supports gaining a more nuanced understanding of response behavior as well as how neglecting either type of response bias may impact conclusions on respondents' content trait levels as well as on their displayed response behavior. We further contrast the proposed approach against a more heuristic two-step procedure that first eliminates presumed careless respondents from the data and subsequently applies model-based approaches accommodating RS. To investigate the trustworthiness of results obtained in the empirical application, we conduct a parameter recovery study.

Cognitive diagnosis models (CDMs) are the statistical framework for cognitive diagnostic assessment in education and psychology. They generally assume that subjects' latent attributes are dichotomous--mastery or nonmastery, which seems quite deterministic. As an alternative to dichotomous attribute mastery, attention is drawn to the use of a continuous attribute mastery format in recent literature. To obtain subjects' finer-grained attribute mastery for more precise diagnosis and guidance, an equivalent but more explicit form of the partial-mastery-deterministic inputs, noisy "and" gate (DINA) model (termed continuous attribute profile [CAP]-DINA form) is proposed in this article. Its parameters estimation algorithm based on this form using Bayesian techniques with Markov chain Monte Carlo algorithm is also presented. Two simulation studies are conducted then to explore its parameter recovery and model misspecification, and the results demonstrate that the CAP-DINA form performs robustly with satisfactory efficiency in these two aspects. A real data study of the English test also indicates it has a better model fit than DINA.

Capturing evidence for dynamic changes in self-regulated learning (SRL) behaviours resulting from interventions is challenging for researchers. In the current study, we identified students who were likely to do poorly in a biology course and those who were likely to do well. Then, we randomly assigned a portion of the students predicted to perform poorly to a science of learning to learn intervention where they were taught SRL study strategies. Learning outcome and log data (257 K events) were collected from n = 226 students. We used a complex systems framework to model the differences in SRL including the amount, interrelatedness, density and regularity of engagement captured in digital trace data (ie, logs). Differences were compared between students who were predicted to: (1) perform poorly (control, n = 48); (2) perform poorly and received intervention (treatment, n = 95); and (3) perform well (not flagged, n = 83). Results indicated that the regularity of students' engagement was predictive of course grade, and that the intervention group exhibited increased regularity in engagement over the control group immediately after the intervention and maintained that increase over the course of the semester. We discuss the implications of these findings in relation to the future of artificial intelligence and potential uses for monitoring student learning in online environments.

Collaborative learning offers numerous benefits to learners, largely due to the dialogue that is unfolding between them. However, there is still much to learn about the structure of collaborative dialogue, and especially little is known about co-creative dialogues during learning. This paper reports on a study with learners engaged in co-creative tasks where the learners wrote code to create a song and while engaging in textual dialogue as they did so. After gathering the textual dialogue and the actions within the interface, we learned a hidden Markov model (HMM) to reveal co-creative states. The seven-state model revealed four states primarily composed of coding actions that included browsing the curriculum documents, working in the code editor, compiling the code successfully, and receiving a compile error. The remaining three states are primarily composed of dialogue that can be characterized as social, aesthetic, and technical dialogue. Next, we analyzed the relationships between the co-creative states revealed by the HMM and students' partner satisfaction scores from a post-survey. The results reveal the relative frequency of actions in certain states and some transitions between states were predictive of partner satisfaction. For example, partner satisfaction was negatively associated with the "Compilation Error" state and with the relative frequency of transitions from the "Curriculum Browsing" state to the "Code Editing" state. Partner satisfaction was also negatively associated with the relative frequency of transitions from the "Aesthetic Dialogue" state to the "Technical Dialogue" state and the "Code Editing" state. This line of investigation reveals how co-creative processes are associated with partner satisfaction, and holds the potential to inform scaffolding for collaborative learning.

Financial professionals involved in divorce proceedings, whether for a client or an attorney, often use software to project the ability of a dependent spouse to earn income off of her separate estate. These projections have historically relied on static inputs and use a Monte Carlo simulation to illustrate the paths a portfolio might take. Within this study, the effects on dynamic income and expense changes on outcomes were examined. A comparison was made between the traditional Monte Carlo methods and Markov Chain Monte Carlo (MCMC) methods. Results using MCMC methods more closely approximated investment return distribution, and illustrated investable assets were the primary driver of long-term success, and not items such as spousal or child support. Practical implications for financial professionals, family law attorneys, judges, and clients are discussed as well as opportunities for future research.

When working with multimodal Bayesian posterior distributions, Markov chain Monte Carlo (MCMC) algorithms have difficulty moving between modes, and default variational or mode-based approximate inferences will understate posterior uncertainty. And, even if the most important modes can be found, it is difficult to evaluate their relative weights in the posterior. Here we propose an approach using parallel runs of MCMC, variational, or mode-based inference to hit as many modes or separated regions as possible and then combine these using Bayesian stacking, a scalable method for constructing a weighted average of distributions. The result from stacking efficiently samples from multimodal posterior distribution, minimizes cross validation prediction error, and represents the posterior uncertainty better than variational inference, but it is not necessarily equivalent, even asymptotically, to fully Bayesian inference. We present theoretical consistency with an example where the stacked inference approximates the true data generating process from the misspecified model and a non-mixing sampler, from which the predictive performance is better than full Bayesian inference, hence the multimodality can be considered a blessing rather than a curse under model misspecification. We demonstrate practical implementation in several model families: latent Dirichlet allocation, Gaussian process regression, hierarchical regression, horseshoe variable selection, and neural networks.

In order to evaluate the effect of a policy or treatment with pre- and post-treatment outcomes, we propose an approach based on a transition model, which may be applied with multivariate outcomes and accounts for unobserved heterogeneity. This model is based on potential versions of discrete latent variables representing the individual characteristic of interest and may be cast in the hidden (latent) Markov literature for panel data. Therefore, it can be estimated by maximum likelihood in a relatively simple way. The approach extends the difference-in-difference method as it is possible to deal with multivariate outcomes. Moreover, causal effects may be expressed with respect to transition probabilities. The proposal is validated through a simulation study, and it is applied to evaluate educational programs administered to pupils in the sixth and seventh grades during their middle school period. These programs are carried out in an Italian region to improve non-cognitive skills (CSs). We study if they impact also on students' CSs in Italian and Mathematics in the eighth grade, exploiting the pretreatment test scores available in the fifth grade. The main conclusion is that the educational programs aimed to develop noncognitive abilities help the best students to maintain their higher cognitive abilities over time.

The present paper introduces a general multidimensional model to measure individual differences in learning within a single administration of a test. Learning is assumed to result from practicing the operations involved in solving the items. The model accounts for the possibility that the ability to learn may manifest differently for correct and incorrect responses, which allows for distinguishing different types of learning effects in the data. Model estimation and evaluation is based on a Bayesian framework. A simulation study is presented that examines the performance of the estimation and evaluation methods. The results show accuracy in parameter recovery as well as good performance in model evaluation and selection. An empirical study illustrates the applicability of the model to data from a logical ability test.

This study proposed a dynamic model of organizational technology adoption within a school institute culture. We described an implementation of a nonhomogeneous hidden Markov model based on a downscaling scheme that can project the cultural factors of the institute onto a teacher's implementation behavior. To reveal the dynamics of cultural evolution, we modeled the interactions within the institute's organization by incorporating extra dependencies and downscaling variables into the underlying process of cultural change. We applied an analysis scheme to the nine-semester e-textbook usage of a primary school and gain insight into teachers' technology adoption from a cultural perspective. We identified three states that represented the collective adoption contexts to examine how exogenous variables influence both the organization-scale context dynamics and the individual-scale implementation changes. The results showed that the effect of exogenous variables, especially external factors, varied between contexts and scales. The school's norm was shown to affect organizational adoption culture. Teachers were sensitive to the adopted context, as they are inclined to adjust their usage to meet preexisting norms. Interventions such as seminars initiated by the teachers' community were observed to cause a more long-term effect on the movement of the culture to an active context. Moreover, our research provided evidence that an organization can achieve more efficacy from teachers' involvement in a high degree of cooperation in in-class experiences.

The adaptive learning problem concerns how to create an individualized learning plan (also referred to as a learning policy) that chooses the most appropriate learning materials based on a learner's latent traits. In this article, we study an important yet less-addressed adaptive learning problem--one that assumes continuous latent traits. Specifically, we formulate the adaptive learning problem as a Markov decision process. We assume latent traits to be continuous with an unknown transition model and apply a model-free deep reinforcement learning algorithm--the deep Q-learning algorithm--that can effectively find the optimal learning policy from data on learners' learning process without knowing the actual transition model of the learners' continuous latent traits. To efficiently utilize available data, we also develop a transition model estimator that emulates the learner's learning process using neural networks. The transition model estimator can be used in the deep Q-learning algorithm so that it can more efficiently discover the optimal learning policy for a learner. Numerical simulation studies verify that the proposed algorithm is very efficient in finding a good learning policy. Especially with the aid of a transition model estimator, it can find the optimal learning policy after training using a small number of learners.

Process Analysis is an emerging approach to discover meaningful knowledge from temporal educational data. The study presented in this paper shows how we used Process Analysis methods on the National Assessment of Educational Progress (NAEP) test data for modeling and predicting student test-taking behavior. Our process-oriented data exploration gave us insightful findings of how students were interacting with the digital assessment system over time. To discover what processes students were following during the NAEP Digital Assessment, we first developed an innovative set of research questions. Then, we used Process Analysis methods to answer these questions and created a set of features that described student behavior over time. These features were used to create an ensemble model that aimed to accurately predict the digital test-taking efficiency of the students taking NAEP. Our model emerged as one of the most successful models in the 2019 NAEP Data Mining Competition, scoring second place out of 89 teams.

Causal search algorithms have been effectively applied in different fields including biology, genetics, climate science, medicine, and neuroscience. However, there have been scant applications of these methods in social and behavioral sciences. This article provides an illustrative example of how causal search algorithms can shed light on important social and behavioral problems by using these algorithms to find the proximal mechanisms of academic achievement. Using a nationally representative data set with a wide range of relevant contextual and psychological factors, I implement four causal search procedures that varied important dimensions in the algorithms. Consistent with previous research, the algorithms identified prior achievement, executive functions (in particular, working memory, cognitive flexibility, and attentional focusing), and motivation as direct causes of academic achievement. I discuss the advantages and limitations of graphical models in general and causal search algorithms in particular for understanding social and behavioral problems.

To solve a new problem, people spontaneously engage multiple cognitive processes. Previous work has identified a diverse set of oscillatory components critical at different stages of creative problem solving. In this project, we use hidden state modeling to untangle the roles of oscillation processes over time as people solve puzzles. Building on earlier work, we further developed analytical methods, such as incorporating source separating techniques and identifying the optimal number of states using cross-validation. We extracted brain states characterized by spatio-spectral topographies from time-resolved EEG spectral powers. The data driven approach allowed us to infer the dynamic, trial-by-trial, state sequences, and provided a comprehensive depiction of how various oscillation components interact recurrently throughout the trial. The properties of the states suggest their dissociable cognitive functions. For example, we identified three states with dominant activation in alpha bands but having distinct spatial distributions. People were differentially engaged in these states depending on the stages (e.g., onset or response) and outcomes of the trials (solved with insight or analysis). The current approach, applicable to many tasks requiring extended trial duration, can potentially reconcile findings from previous EEG studies and drive new hypotheses to further our understanding of the complex creative process.

Graduates in biology or genetics learn Mendel's laws and Hardy-Weinberg equilibrium as students, and they know, use, and sometimes teach these concepts. However, it is unusual to learn about stochastic processes during the graduate studies of these topics, although the applications of Markov chains in the fields of genetics or biology are increasing. In this paper, and from Mendel's Laws, we are going to obtain an example of the Hardy-Weinberg Equilibrium that can be used to teach it. In the demonstration, stochastic processes theory and Markov chains will be used, in order to develop the transition matrix. In this way, we obtain an example that can be used to illustrate an explanation that involves, through Mendel's laws, Hardy-Weinberg equilibrium for degree students, and stochastic processes for advanced students in biology or genetic sciences (predoctoral students at university). In this example, we are going to use the Scientific Workplace version 6.0 software.

Learning interaction activities are the key part of tracking and evaluating learning behaviors, that plays an important role in data-driven autonomous learning and optimized learning in interactive learning environments. In this study, a big data set of learning behaviors with multiple learning periods is selected. According to the instance characteristics of data attributes and learning behaviors, the analysis of learning interaction activities and their relationships is mapped to the optimized design of Markov random fields. Appropriate models and multi-level algorithms are constructed to demonstrate the maximal cliques that affect the learning behaviors and mine the critical paths, then cooperative inference strategies of learning interaction activities are deduced. In order to verify the effectiveness of the random fields and analysis technology of learning interaction activities, several approximate algorithms are introduced to fully compare multiple statistical indexes. To verify the feasibility of collaborative recommendation strategies, the improvement effect of learning behaviors and learning effect is tested by applying it to the actual teaching process. Research and analysis show that the random fields of learning interaction activities and the collaborative inference strategies have advantages in data analysis, topological relationship construction and teaching practice guidance.

We propose a new item response theory growth model with item-specific learning parameters, or ISLP, and two variations of this model. In the ISLP model, either items or blocks of items have their own learning parameters. This model may be used to improve the efficiency of learning in a formative assessment. We show ways that the ISLP model's learning parameters can be estimated in simulation using Markov chain Monte Carlo (MCMC), demonstrate a way that the model could be used in the context of adaptive item selection to increase the rate of learning, and estimate the learning parameters in an empirical data analysis using the ISLP. In the simulation studies, the one-parameter logistic model was used as the measurement model to generate random response data with various test lengths and sample sizes. Ability growth was modeled with a few variations of the ISLP model, and it was verified that the parameters were accurately recovered. Secondly, we generated data using the linear logistic test model with known Q-matrix structure for the item difficulties. Using a two-step procedure gave very comparable results for the estimation of the learning parameters even when item difficulties were unknown. The potential benefit of using an adaptive selection method in conjunction with the ISLP model was shown by comparing total improvement in the examinees' ability parameter to two other methods of item selection that do not utilize this growth model. If the ISLP holds, adaptive item selection consistently led to larger improvements over the other methods. A real data application of the ISLP was given to illustrate its use in a spatial reasoning study designed to promote learning. In this study, interventions were given after each block of ten items to increase ability. Learning parameters were estimated using MCMC.

Understanding whether or not different types of students master various attributes can aid future learning remediation. In this study, two-level diagnostic classification models (DCMs) were developed to represent the probabilistic relationship between external latent classes and attribute mastery patterns. Furthermore, variational Bayesian (VB) inference and Gibbs sampling Markov chain Monte Carlo methods were developed for parameter estimation of the two-level DCMs. The results of a parameter recovery simulation study show that both techniques appropriately recovered the true parameters; Gibbs sampling in particular was slightly more accurate than VB, whereas VB performed estimation much faster than Gibbs sampling. The two-level DCMs with the proposed Bayesian estimation methods were further applied to fourth-grade data obtained from the Trends in International Mathematics and Science Study 2007 and indicated that mathematical activities in the classroom could be organized into four latent classes, with each latent class connected to different attribute mastery patterns. This information can be employed in educational intervention to focus on specific latent classes and elucidate attribute patterns.

A key goal in cognitive training research is understanding whether cognitive training enhances general cognitive capacity or provides only task-specific improvements. Here, we developed a quantitative model for describing the temporal dynamics of these two processes. We analyzed data from 1300 children enrolled in an 8 week working memory training program that included 5 transfer test sessions. Factor analyses suggested two separate processes: an early task-specific improvement, accounting for 44% of the total increase, and a slower capacity improvement. A hidden Markov model was then applied to individual training data, revealing that the task-specific improvement plateaued on the third day of training on average. Thus, training is not only task specific or transferable but a combination of the two. The models provide methods for quantifying and separating these processes, which is crucial for studying the effects of cognitive training and relating these effects to neural correlates.

Response time models (RTMs) are of increasing interest in educational and psychological testing. This article focuses on the lognormal model for response times, which is one of the most popular RTMs. Several existing statistics for testing normality and the fit of factor analysis models are repurposed for testing the fit of the lognormal model. A simulation study and two real data examples demonstrate the usefulness of the statistics. The Shapiro-Wilk test of normality and a "z"-test for factor analysis models were the most powerful in assessing the misfit of the lognormal model. [For the corresponding grantee submission, see ED603049.]

Intelligent tutoring systems (ITSs) teach skills using learning-by-doing principles and provide learners with individualized feedback and materials adapted to their level of understanding. Given a learner's history of past interactions with an ITS, a learner performance model estimates the current state of a learner's knowledge and predicts her future performance. The advent of increasingly large scale datasets has turned deep learning models for learner performance prediction into competitive alternatives to classical Markov process and logistic regression models. In an extensive empirical comparison on nine real-world datasets, we ask which approach makes the most accurate predictions and in what conditions. Logistic regression--with the right set of features--leads on datasets of moderate size or containing or containing a very large number of interactions per student, whereas Deep Knowledge Tracing leads on datasets of large size or where precise temporal information matters most. Markov process methods, like Bayesian Knowledge Tracing, lag behind other approaches. We follow this analysis with ablation studies to determine what components of leading algorithms explain their performance and a discussion of model calibration (reliability), which is crucial for downstream applications of learner performance prediction models.

American universities use a procedure based on a rolling six-year graduation rate to calculate statistics regarding their students' final educational outcomes (graduating or not graduating). As an alternative to the six-year graduation rate method, many studies have applied absorbing Markov chains for estimating graduation rates. In both cases, a frequentist approach is used. For the standard six-year graduation rate method, the frequentist approach corresponds to counting the number of students who finished their program within six years and dividing by the number of students who entered that year. In the case of absorbing Markov chains, the frequentist approach is used to compute the underlying transition matrix, which is then used to estimate the graduation rate. In this paper, we apply a sensitivity analysis to compare the performance of the standard six-year graduation rate method with that of absorbing Markov chains. Through the analysis, we highlight significant limitations with regards to the estimation accuracy of both approaches when applied to small sample sizes or cohorts at a university. Additionally, we note that the Absorbing Markov chain method introduces a significant bias, which leads to an underestimation of the true graduation rate. To overcome both these challenges, we propose and evaluate the use of a regularly updating multi-level absorbing Markov chain (RUML-AMC) in which the transition matrix is updated year to year. We empirically demonstrate that the proposed RUML-AMC approach nearly eliminates estimation bias while reducing the estimation variation by more than 40%, especially for populations with small sample sizes.