Search

Adaptive learning is an area of educational technology that consists in delivering personalized learning experiences to address the unique needs of each learner. An important subfield of adaptive learning is learning path personalization: it aims at designing systems that recommend sequences of educational activities to maximize students' learning outcomes. Many machine learning approaches have already demonstrated significant results in a variety of contexts related to learning path personalization. However, most of them were designed for very specific settings and are not very reusable. This is accentuated by the fact that they often rely on non-scalable models, which are unable to integrate new elements after being trained on a specific set of educational resources. In this paper, we introduce a flexible and scalable approach towards the problem of learning path personalization, which we formalize as a reinforcement learning problem. Our model is a sequential recommender system based on a graph neural network, which we evaluate on a population of simulated learners. Our results demonstrate that it can learn to make good recommendations in the small-data regime. [For the complete proceedings, see ED630829.]

Online educational technologies facilitate pedagogical experimentation, but typical experimental designs assign a fixed proportion of students to each condition, even if early results suggest some are ineffective. Experimental designs using multi-armed bandit (MAB) algorithms vary the probability of condition assignment for a new student based on prior results, placing more students in more effective conditions. While stochastic MAB algorithms have been used for educational experiments, they collect data that decreases power and increases false positive rates [22]. Instead, we propose using adversarial MAB algorithms, which are less exploitative and thus may exhibit more robustness. Through simulations involving data from 20+ educational experiments [29], we show data collected using adversarial MAB algorithms does not have the statistical downsides of that from stochastic MAB algorithms. Further, we explore how differences in condition variability (e.g., performance gaps between students being narrowed by an intervention) impact MAB versus uniform experimental design. Data from stochastic MAB algorithms systematically reduce power when the better arm is less variable, while increasing it when the better arm is more variable; data from the adversarial MAB algorithms results in the same statistical power as uniform assignment. Overall, these results demonstrate that adversarial MAB algorithms are a viable "off-the-shelf" solution for researchers who want to preserve the statistical power of standard experimental designs while also benefiting student participants. [For the full proceedings, see ED623995.]

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all new and currently considered computer-based tests. In addition to developing new models, researchers will need to give some attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This paper introduces an adaptive testing algorithm that balances maximum differentiation among test takers with stable estimation at each stage of testing, and compares this algorithm with a traditional one using IRT and maximum information. The adaptive testing algorithm introduced is based on the classification and regression tree approach described in L. Breiman, J. Friedman, R. Olshen, and C. Stone (1984) and J. Chambers and T. Hastie (1992). Simulation results from the regression tree approach were compared with simulation results from three parameter logistic model IRT. Simulation results show that the nonparametric tree-based approach to adaptive testing may be superior to conventional IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. It clearly outperformed one-dimensional IRT when the pool was strongly two-dimensional. A technical appendix describes the algorithm. (Contains three figures and six references.) (SLD)

Two-stage and multistage test designs provide a way of roughly adapting item difficulty to test-taker ability. All test takers take a parallel stage-one test, and, based on their scores, they are routed to tests of different difficulty levels in subsequent stages. These designs provide some of the benefits of standard computerized adaptive testing (CAT), such as increased precision of ability estimates over a paper-and-pencil test design. The item selection and scoring algorithms in two-stage and multistage designs may also be easier for test takers and test score users to understand--an important feature for gaining public acceptance of new test designs. This study incorporates testlets (bundles of items) into two-stage and multistage designs, and compares the precision of the ability estimates derived from these designs with those derived from a standard CAT design and from paper-and-pencil test designs. For the group that was used to establish the cutoffs for the two-stage and multistage testlet designs, 50,000 simulated test takers were created randomly. The group of simulated test takers used to simulate all test designs totaled 25,000. Results indicate that all testlet-based designs resulted in improved precision over the same-length paper-and-pencil test, and almost as much precision as the paper-and-pencil test of double length. Given the many other (nonpsychometric) advantages of these designs, they may be viable options for computer-administered tests. (Contains six figures and nine references.) (Author/SLD)

This paper examines the effect of using unidimensional item response theory (IRT) item parameter estimates of multidimensional items to create weakly parallel test forms using target information curves. To date, all computer-based algorithms that have been devised to create parallel test forms assume that the items are unidimensional. This paper focuses on one such algorithm, which was developed by R. M. Luecht and T. M. Hirsch. Unidimensional item parameter estimates were obtained by calibrating response data generated from two-dimensional item parameters. Using these unidimensional estimates, three sets of test items from a pool of 200 multidimensional items were selected for each of two different test lengths for three differently shaped target information functions. The item parameter estimates were obtained by calibrating five forms of the EAAP Mathematics usage test using the multidimensional IRT calibration program NOHARM. Response data were generated for 2,000 abilities. Observed score differences for each triad, based on the multidimensional item parameters, were then compared. Despite the multidimensionality of the selected items, the created forms appear to be quite parallel both unidimensionally and multidimensionally. Two tables and seven figures are included. (SLD)

This paper proposes a new Information-Time index as the basis for item selection in computerized classification testing (CCT) and investigates how this new item selection algorithm can help improve test efficiency for item pools with mixed item types. It also investigates how practical constraints such as item exposure rate control, test difficulty (i.e., cut point), test length constraint, and test time constraint affect the effectiveness of this new item selection algorithm. Monte Carlo simulation techniques were used, and item parameters from the 1996 National Assessment of Educational Progress science assessment were used to build the data pool for an assessment combining 3 grades and consisting of 246 dichotomous items and 266 polytomous items. To create an Information-Time index, a response-time distribution was simulated. The Information-Time index was found feasible and effective in CCT with a mixed item pool. The Information-Time index for item selection worked well with other testing variables like item exposure control, test difficulty, and test constraints. (Contains 2 tables and 19 references.) (SLD)

This discussion of new methods for calibrating item response theory (IRT) models looks into new optimization procedures, such as the Genetic Algorithm (GA) to improve on the use of the Newton-Raphson procedure. The advantages of using a global optimization procedure like GA is that this kind of procedure is not easily affected by local optima and saddle points. Because these procedures do not use gradient information, they can be implemented easily to higher dimensional data, even though they converge more slowly than the Newton-Raphson approach. However, the two approaches can be combined to exploit the advantages of both. That is, GA can be used to find a suitable starting point close to the global optima, and then Newton-Raphson can be used to speed up the convergence. The focus in this paper is on calibrating the unidimensional three-parameter logistic model (3PL) because that is the model most widely used in large-scale standardized tests. Using recent 3PL model estimates from recent Test of English as a Foreign Language administrations to generate examinee responses, the effectiveness of the new method is demonstrated using simulated data. How to implement the new methods with multidimensional data is discussed. (Contains 3 tables, 2 figures, and 10 references.) (SLD)

Although computerized adaptive tests acquire their efficiency by successively selecting items that provide optimal measurement at each examinee's estimated level of ability, operational testing programs will typically consider additional factors in item selection. In practice, items are generally selected with regard to at least three, often conflicting, goals: (1) to maximize test efficiency by measuring examinees as quickly and accurately as possible; (2) to protect the security of the item pool by controlling the rates at which popular items can be administered; and (3) to assure that the test measures the same composite of multiple traits for each examinee by balancing the rates at which items with different content properties are administered. This paper focuses on the goals of maximizing test efficiency and controlling item exposure rates, avoiding a discussion of content balance. Problems in existing algorithms for accomplishing these goals are outlined and illustrated, and some alternative algorithms that offer at least a partial solution are presented. Posterior weighted information is suggested as a new item selection method, and its usefulness is demonstrated through a simulation. Conditional exposure control is suggested to control exposure rate, and a similar simulation is presented to demonstrate its usefulness. (Contains one table, seven figures, and seven references.) (SLD)

One set of approaches to the problem of clustering with dichotomous data in cluster analysis (CA) was studied. The techniques developed for clustering with binary data involve calculating distances between observations based on the variables and then applying one of the standard CA algorithms to these distances. One of the groups of distances that are designed for binary data is known collectively as matching coefficients. There are several incarnations of matching coefficients, but all take as their main goal the measurement of response similarity between any two observations. Thus, distance and similarity come to express the same concept with respect to the observations. This study examined four measures of association that are common to four previous studies. Using Monte Carlo simulation, cluster analysis was conducted using the four distance measures. Under the conditions of this study, the four measures performed very much the same in terms of correctly classifying individuals into two clusters based on dichotomous variables. Another interesting result is that clustering solutions were virtually identical for samples of size 240 and 1,000. (Contains 6 tables, 6 figures, and 12 references.) (SLD)

A study was conducted to extend the sequential probability ratio testing (SPRT) procedure with the polytomous model under some practical constraints in computerized classification testing (CCT), such as methods to control item exposure rate, and to study the effects of other variables, including item information algorithms, test difficulties, item pool sizes, and widths of the indifference region in SPRT. SPRT was applied for polytomous items under the generalized partial credit model of E. Muraki (1992). Monte Carlo simulation technique was used. Independent variables manipulated were: (1) item information algorithm; (2) item exposure control methods; (3) location of the theta cut point (test difficulty); (4) item pool size; and (5) width of indifference region in SPRT. Item parameters from the 1996 National Assessment of Educational Progress were used to build the item pool, and item response data were generated for 10,000 simulated examinees. Polytomous items were found to work well with the SPRT procedure in CCT. Considerations in using the procedure are discussed. (Contains 5 tables and 18 references.) (SLD)

This study investigated and compared the properties of five methods of item exposure control within the purview of estimating examinees' abilities in a computerized adaptive testing (CAT) context. Each of the exposure control algorithms was incorporated into the item selection procedure and the adaptive testing progressed based on the CAT design established for this study. Using Monte Carlo simulation, the merits and shortcomings of these strategies were considered under different item pool sizes and different maximum exposure rates and were evaluated in light of test security, item overlap rate, and the conditional standard errors of measurement. The ultimate goal of the study was to provide guidelines for choosing appropriate methods of controlling item exposure so that test security concern in the CAT environment could be lessened. Among the five exposure control algorithms investigated in this study, the conditional procedure developed by M. Stocking and C. Lewis (1995) best served the purpose of controlling the observed exposure rates to the desired values as well as producing the lowest test overlap rates. The trade-off was that measurement precision was sacrificed to some extent. (Contains 4 tables, 9 figures, and 28 references.) (Author/SLD)

A Monte Carlo study compared the usefulness of six variable weighting methods for cluster analysis. Data were 100 bivariate observations from 2 subgroups, generated according to a finite normal mixture model. Subgroup size, within-group correlation, within-group variance, and distance between subgroup centroids were manipulated. Of the clustering methods examined, the flexible average algorithm with beta equal to -.15 or -.20 gave the best recovery. Of the remaining methods, that of J. H. Ward yielded the best recovery, followed closely by beta-flexible linkage and the EML algorithm of the Statistical Analysis System. In the absence of variable weights, negative within-group correlation resulted in much poorer recovery for all clustering algorithms. The ACE weighing method of D. Art, R. Gnanadesikan, and J. R. Kettenring proved preferable overall. Clustering with Mahalanobis distance based on the pooled within-group covariance matrix indicated that knowing the correct covariance method would yield improved recovery over the ACE method approximately 10% of the time. Two appendix figures provide weighting data. (Contains 8 figures, 7 tables, 2 appendix figures, and 40 references.) (Author/SLD)

When examinees are allowed to review responses on an adaptive test, can they "cheat" the adaptive algorithm in order to take an easier test and improve their performance? Theoretically, deliberately answering items incorrectly will lower the examinee ability estimate and easy test items will be administered. If review is then allowed, examinees can change answers from wrong to right, thereby raising their initial ability estimate. Following this strategy, examinees can take an easy test, rather than a test targeted to their ability. The consequences of following such a strategy, for the examinee and for the testing agency, are explored. Results of a simulation using data based on the Rasch model indicate that cheating is a risky business. If an examinee makes a mistake and fails to change even one answer from wrong to right, the consequences may be dire. When an item bank has very easy items and test length is short, highly able examinee ability is severely underestimated. In addition, "cheating" can be detected and prevented during test administration by altering test targeting. (Contains 4 figures, 4 tables, and 22 references.) (Author/SLD)

This paper describes a student project in digital simulation techniques that is part of a graduate systems analysis course entitled Biosimulation. The students chose different simulation techniques to solve a problem related to the neuron model. (MLH)

This paper describes an investigation of problem sequencing as part of a symposium presenting theoretical concerns and research findings regarding several diverse types of strategies for designing and presenting instructional materials. The first step involved analyzing a set of problems, to determine the knowledge (rules) required for solving the problems. A set of geometric construction problems was analyzed using a method for deriving and simplifying rules or algorithms for solving the problems. Next, assumptions were adopted concerning how subjects learn and apply rules to solve problems. A computer was programmed according to these assumptions, and given some of the simplified rules identified during the previous step, in order to simulate some aspects of subjects' problem-solving and learning. Various problem sequences were then given to the program, to identify those that were solvable and "learnable." Finally, for initial evaluation of the approach, sequences that were learnable for the program were given to subjects, whose performance was compared with the performance of subjects given random or learner-controlled sequences. (Author/MLF)