Search

Psychometric testing is a valuable educational tool for the assessment and monitoring of students’ abilities in core subjects. However, the manual development of these tests is a tedious process requiring test specialists to produce and curate large volumes of high-quality items. In this paper, we consider whether automating test item generation with modern machine learning methods is a feasible solution for obtaining strong psychometric test items at low cost in the domain of sentence reading fluency. We assess the ability of the large neural language model GPT-3 to produce items ``few-shot’’--- from a short prompt with only a handful of examples. Our results show that generated items closely resemble standardized test items in terms of their factual ambiguity, content appropriateness, and complexity. Furthermore, after filtering for correct answer-labeling these generated items possess similar latent psychometric properties to standardized test items, even capturing subtle grade-level variation.

Student retention is a key outcome measure for post-secondary education, but data show relatively stagnant retention rates over the past decade. Longstanding interventions such as counseling, academic advising, and mentoring provide crucial student services, but little change in retention rates suggest there is still need for additional student supports. Within the landscape of higher education, academic coaching is a relatively new, yet burgeoning intervention designed to increase student retention and success. Despite rapid growth of the intervention, little empirical work has been done to systematically describe and evaluate such programs. In this study, we provide a rich description of one academic coaching program and use a quasi-experimental design to evaluate the program's effects on student outcomes. We investigate two research questions: (1) how does academic coaching influence key student outcomes?; and (2) to what extent do these effects vary by amount of coaching received? On average, we found that students with prior semester grade point averages from 1.0-2.0 who participate in the academic coaching program earn grade point averages about 0.4 points higher during the coaching semester, are about 10% more likely to enroll in the semester following coaching, and earn about two more credits in the semester following coaching than students who choose not participate in the program. Outcomes varied minimally based on the number of coaching appointments students attended.

For some students, standardized tests serve as a conduit to disclose sensitive issues of harm or distress that may otherwise go unreported. By detecting this writing, known as "crisis papers," testing programs have a unique opportunity to assist in mitigating the risk of harm to these students. The use of machine learning to automatically detect such writing is necessary in the context of online tests and automated scoring. To achieve a detection system that is accurate, humans must first consistently label the data that are used to train the model. This paper argues that the existing guidelines are not sufficient for this task and proposes a three-level rubric to guide the collection of the training data. In showcasing the fundamental machine learning procedures for creating an automatic text classification system, the following evidence emerges as support of the operational use of this rubric. First, hand-scorers largely agree with one another in assigning labels to text according to the rubric. Additionally, when this labeled data are used to train a baseline classifier, the model exhibits promising performance. Recommendations are made for improving the hand-scoring training process, with the ultimate goal of quickly and accurately assisting students in crisis.

The process of setting and evaluating student learning objectives (SLOs) has become increasingly popular as an example where classroom assessment is intended to fulfill the dual purpose use of informing instruction and holding teachers accountable. A concern is that the high-stakes purpose may lead to distortions in the inferences about students and teachers that SLOs can support. This concern is explored in the present study by contrasting student SLO scores in a large urban school district to performance on a common objective external criterion. This external criterion is used to evaluate the extent to which student growth scores appear to be inflated. Using 2 years of data, growth comparisons are also made at the teacher level for teachers who submit SLOs and have students that take the state-administered large-scale assessment. Although they do show similar relationships with demographic covariates and have the same degree of stability across years, the two different measures of growth are weakly correlated.

In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.

For some students, standardized tests serve as a conduit to disclose sensitive issues of harm or distress that may otherwise go unreported. By detecting this writing, known as crisis papers, testing programs have a unique opportunity to assist in mitigating the risk of harm to these students. The use of machine learning to automatically detect such writing is necessary in the context of online tests and automated scoring. To achieve a detection system that is accurate, humans must first consistently label the data that are used to train the model. This paper argues that the existing guidelines are not sufficient for this task and proposes a three‐level rubric to guide the collection of the training data. In showcasing the fundamental machine learning procedures for creating an automatic text classification system, the following evidence emerges as support of the operational use of this rubric. First, hand‐scorers largely agree with one another in assigning labels to text according to the rubric. Additionally, when this labeled data are used to train a baseline classifier, the model exhibits promising performance. Recommendations are made for improving the hand‐scoring training process, with the ultimate goal of quickly and accurately assisting students in crisis. [ABSTRACT FROM AUTHOR]