Search

Text comprehension is an essential skill in today's information-rich world, and self-explanation practice helps students improve their understanding of complex texts. This study was centered on leveraging open-source Large Language Models (LLMs), specifically FLAN-T5, to automatically assess the comprehension strategies employed by readers while understanding Science, Technology, Engineering, and Mathematics (STEM) texts. The experiments relied on a corpus of three datasets (N = 11,833) with self-explanations annotated on 4 dimensions: 3 comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and overall quality. Besides FLAN-T5, we also considered GPT3.5-turbo to establish a stronger baseline. Our experiments indicated that the performance improved with fine-tuning, having a larger LLM model, and providing examples via the prompt. Our best model considered a pretrained FLAN-T5 XXL model and obtained a weighted F1-score of 0.721, surpassing the 0.699 F1-score previously obtained using smaller models (i.e., RoBERTa).?[This is the online first version of an article published in "Information."]

Writing is essential for success in academics and everyday tasks, but the development of writing skills depends on consistent access to high-quality instruction, extended practice, and personalized feedback. To address these demands and meet students' needs, educators and researchers have turned to technology-based writing tools. Ideally, these tools integrate the core components of intelligent tutoring, including a domain model, student model, tutor model, and interface model to engage students with individualized feedback that is linked to adaptive writing instruction. However, the landscape of writing tools still has much room for improvement in terms of incorporating advanced artificial intelligence-enabled features to better approximate intelligent tutoring systems (ITSs). This chapter describes the key elements of ITS technologies and how they can be integrated to further develop ITS tools for writing. To this end, this chapter: (1) summarizes evidence-based aspects of successful ITSs and how they might be integrated into computer-based tools for writing; (2) reviews how existing systems have leveraged intelligent tutoring approaches; and (3) articulates how future technology-based writing tools could implement advanced intelligent tutoring features to better meet students' needs. The chapter concludes with the implications and future directions of intelligent tutoring for the teaching and learning of writing. [This chapter was published in: "Digital Writing Technologies in Higher Education," edited by O. Kruse et al., Springer, Cham, 2023, pp. 365-383.]

Age of acquisition (AoA) is a measure of word complexity which refers to the age at which a word is typically learned. AoA measures have shown strong correlations with reading comprehension, lexical decision times, and writing quality. AoA scores based on both adult and child data have limitations that allow for error in measurement, and increase the cost and effort to produce. In this paper, we introduce Age of Exposure (AoE) version 2, a proxy for human exposure to new vocabulary terms that expands AoA word lists through training regressors to predict AoA scores. Word2vec word embeddings are trained on cumulatively increasing corpora of texts, word exposure trajectories are generated by aligning the word2vec vector spaces, and features of words are derived for modeling AoA scores. Our prediction models achieve low errors (from 13% with a corresponding R[superscript 2] of 0.35 up to 7% with an R[superscript 2] of 0.74), can be uniformly applied to different AoA word lists, and generalize to the entire vocabulary of a language. Our method benefits from using existing readability indices to define the order of texts in the corpora, while the performed analyses confirm that the generated AoA scores accurately predicted the difficulty of texts (R[superscript 2] of 0.84, surpassing related previous work). Further, we provide evidence of the internal reliability of our word trajectory features, demonstrate the effectiveness of the word trajectory features when contrasted with simple lexical features, and show that the exclusion of features that rely on external resources does not significantly impact performance. [This is the online first version of an article published in "Behavior Research Methods."]

Automated scoring of student language is a complex task that requires systems to emulate complex and multi-faceted human evaluation criteria. Summary scoring brings an additional layer of complexity to automated scoring because it involves two texts of differing lengths that must be compared. In this study, we present our approach to automate summary scoring by evaluating a corpus of approximately 5,000 summaries based on 103 source texts, each summary being scored on a 4-point Likert scale for seven different evaluation criteria. We train and evaluate a series of Machine Learning models that use a combination of independent textual complexity indices from the ReaderBench framework and Deep Learning models based on the Transformer architecture in a multitask setup to predict concurrently all criteria. Our models achieve significantly lower errors than previous work using a similar dataset, with MAE ranging from 0.10-0.16 and corresponding R[superscript 2] values of up to 0.64. Our findings indicate that Longformer-based models are adequate for contextualizing longer text sequences and effectively scoring summaries according to a variety of human-defined evaluation criteria using a single Neural Network. [This paper was published in: "AIED 2022, LNCS 13355," edited by M. M. Rodrigo et al., Springer Nature Switzerland, 2022, pp. 756-761.]

Readers often struggle to identify the main ideas in expository texts. Existing research and instruction provide some guidance on how to encourage readers to identify main ideas. However, there is substantial variability in how main ideas are operationalized and how readers are prompted to identify main ideas. This variability hinders identification of best practices for instruction and intervention. The goal of the current series of experiments was to systematically examine the extent to which different tasks (e.g., selecting main ideas vs. deleting details) and different operationalizations of main ideas (e.g., "important ideas" vs. "main ideas") influenced adult readers' identification of sentences containing main ideas as they read 11 expository texts. Across experiments, the results showed that readers were generally unreliable in identifying main idea sentences; however, they were more reliable when they were instructed to select main idea sentences compared to when they were instructed to delete sentences comprised of details, and more skilled readers were more reliable than less skilled readers. The findings from the current experiments challenge existing instructional approaches and call for additional research to better understand readers' main idea selection. [This is the online version of an article published in "Reading and Writing."]

Prior knowledge is one of the strongest contributors to comprehension, but there is little specificity about different aspects of prior knowledge and how they impact comprehension. This article introduces the Multidimensional Knowledge in Text Comprehension framework, which conceptualizes prior knowledge along four intersecting dimensions: amount, accuracy, specificity, and coherence. "Amount" refers to how many relevant concepts the reader knows. "Accuracy" refers to the extent to which the reader's knowledge is correct. "Specificity" refers the degree to which the knowledge is related to information in the target text. "Coherence" refers to the interconnectedness of prior knowledge. Conceptualizing prior content knowledge along these dimensions deepens understanding of the construct and lends to more specific predictions about how learners process information. Considering knowledge across multiple dimensions is crucially important to the development and selection of prior knowledge assessments and, in turn, educators' ability to capitalize on learners' strengths across various comprehension tasks. [This is the online version of an article published in "Educational Psychologist." For the final published version of this article, see?EJ1305845.]

We compared high school students' performance in a traditional comprehension assessment requiring them to identify key information and draw inferences from single texts, and a scenario-based assessment (SBA) requiring them to integrate, evaluate and apply information across multiple sources. Both assessments focused on a non-academic topic. Performance on the two assessments were moderately correlated (r=0.57), but the SBA was more difficult (Study 1; n=342). The two assessments similarly depended on basic reading skills but diverged in the relation to academic knowledge and (non-academic) topic knowledge (Study 2; n= 1,107). Academic knowledge was highly predictive of traditional comprehension, but less so for SBA. Topic knowledge was more predictive of SBA than traditional comprehension. Thus, the two assessments tap into similar constructs related to comprehension; however, the level of topic knowledge is more important for performance on scenario-based, multiple-source reading tasks, whereas academic knowledge is more important for traditional reading comprehension tasks. [This paper was published in "Learning and Instruction" v73 Article 101462 2021.]

The objective in the current paper is to examine the processes of how our research team negotiated meaning using an iterative design approach as we established, developed, and refined a rubric to capture comprehension processes and strategies evident in students' verbal protocols. The overarching project comprises multiple data sets, multiple scientists across (distant) institutions, and multiple teams of discourse analysts who are tasked with scoring over 20,000 verbal protocols (i.e., think aloud, self-explanation) collected in studies conducted in the last decade. Here, we describe the iterative modifications, negotiations, and realizations while coding our first subset comprising 7,559 individual verbal protocols. Drawing upon work in design research, we describe a process through which the research team has negotiated meaning around theory-driven codes and how this work has influenced our own ways of conceptualizing comprehension research, theory, and practice. [This paper was published in: "ICLS 2021 Proceedings," ISLS, 2021, pp. 139-146.]

Artificial Intelligence (AI) based assessments are commonly used in a variety of settings including business, healthcare, policing, manufacturing, and education. In education, AI-based assessments undergird intelligent tutoring systems as well as many tools used to evaluate students and, in turn, guide learning and instruction. This chapter provides a framework for AI-based Assessment (AIBA) in Education that describes systems and tools. The framework comprises five interrelated dimensions that broadly address the purposes and procedures of AIBA for educational technologies: purposes (i.e., summative feedback, formative feedback, training, and instruction), constructs (e.g., knowledge, skills, strategies, and emotion), data sources (e.g., task performance, behaviors, language, and multimodal data), computational methods (e.g., Bayesian statistics, machine learning, and natural language processing), and visibility (e.g., task-related vs. stealth). This chapter describes systems and tools used to provide AIBA and instantiate these tools within these dimensions. Framing systems within these five dimensions of AIBA enables understanding the landscape of tools and systems, and, in turn, pointing to gaps in the literature and future directions. [This chapter was published in: "Handbook of Artificial Intelligence in Education," edited by Benedict du Boulay et al., Edward Elgar Publishing Limited, 2023, pp. 485-504.]

Assessments of natural language can provide vast information about individuals' thoughts and cognitive process, but they often rely on time-intensive human scoring, deterring researchers from collecting these sources of data. Natural language processing (NLP) gives researchers the opportunity to implement automated textual analyses across a variety of psychological domains. NLP techniques can be used to score written language and provide instantaneous feedback. For example, the Reading Strategy Assessment Tool (RSAT) can score short constructed responses produced while students read texts for presence of various comprehension processes. Another application of NLP is automated essay scoring (AES), which has been used in high-stakes tests. AES systems are developed using machine learning techniques: a corpus of essays is scored by human raters and split into a training and test set. A model is trained to predict human ratings of essays using linguistic features derived using NLP. AES is now often as accurate as expert human raters. While AES systems simply provide scores, automated writing evaluation (AWE) systems also provide feedback on student writing. A continued challenge in both AES and AWE systems is ensuring accurate scoring across a variety of genres, prompts, topics, modes of writing, and writer characteristics. Novel approaches suggest that NLP techniques can be used not only for assessment, but also to better understand how natural language can reveal individual characteristics. For instance, the cohesion of self-explanations can be used to predict the coherence of a reader's mental representation of a text. NLP techniques have also been used to examine emotions and other psychological states from written responses, using tools such as the Linguistic Inquiry and Word Count (LIWC) and WordNet-Affect. Additionally, NLP has been used to increase student learning through implementation in a variety of intelligent tutoring systems (ITSs). When users provide open-ended responses to an ITS, NLP techniques are used to score the responses and provide automated feedback. AutoTutor is an ITS that provides instruction on technical topics such as computer literacy. After being asked a question, students engage in a dialogue of approximately 20 to 100 conversational turns with AutoTutor. The tutor uses NLP to determine how closely a student's response matches the expected answer; if the match is not sufficiently close, the tutor may pump the student for information, provide hints and prompts, or provide the answer, depending on the student's response. Another example is the Interactive Strategy Trainer for Active Reading and Thinking (iSTART), an ITS that helps high school and adult students learn and practice reading comprehension strategies. Students write self-explanations of texts as they read, and their self-explanations are assessed using a series of NLP algorithms. A pedagogical agent provides feedback based on various features of the student's response. These various applications of NLP indicate its great potential for use in learning, assessment, and exploring the relations between language and psychological processes. [This chapter was published in: "APA Handbook of Research Methods in Psychology, Second Edition: Vol. 1. Foundations, Planning, Measures, and Psychometrics," edited by H. Cooper et al., American Psychological Association, 2023, pp. 361-380.]

When students learn, they activate, use, revise, and acquire knowledge. As such, knowledge is a fundamental asset. We advocate for an asset-based approach which capitalizes on students' knowledge through prompts and activities that invite learners to leverage what they already know. Considering knowledge as an asset means that educators must consider multiple aspects of knowledge rather than just the amount of knowledge related to a target domain. This article explores how knowledge can be conceptualized across multiple types (e.g., explicit vs. implicit, declarative vs. procedural, metacognitive, epistemological, disciplinary, formal vs. informal) and dimensions (i.e., amount, accuracy, specificity, coherence) and how acknowledging and responding to the multi-faceted nature of knowledge supports learning. Effective learning comes from capitalizing on what a student already knows and engaging the student in active, knowledge building which results in more meaningful and long-term learning. [This chapter was published in: "International Encyclopedia of Education, 4th Edition, Volume 6," Elsevier, 2023, pp. 209-218.]

This chapter offers an integrated review and discussion of the major advances in theory and practice in the area of literacy, with particular focus on the work since 2014 onward. For the purposes of this review, it approaches Literacy as the ability to read and write. For reading, the chapter highlights theoretical advances that expand the view of reading and its relation to prior knowledge and digital context and practical advances on the impact of language and content-area instruction, assessment, as well as technological innovations. For writing, it also highlights theoretical advances that expand the view of writing and its socio-cultural context, and practical advances on the impact of self-regulated strategy instruction and technological innovations. It then highlights that despite the reciprocal relation between reading and writing, these two areas of work have largely evolved independently of one another, with only recent work directly integrating the two theoretically and practically. While in this review we focus on recent advances in literacy research, it is important to acknowledge the current context in which this effort has been undertaken. National (NAEP, 2022 Reading Assessment) and international assessments (e.g., PISA) continue to underscore the prevalence of inadequate reading and writing performance in students of all ages. There are also wide and persistent opportunity gaps for students living in poverty, as well as for students from marginalized racial and ethnic groups and multilingual students, raising concerns about systemic inequities in both assessment and instruction (Patton-Terry, 2021). Addressing these opportunity gaps is an educational imperative that should be central in ongoing literacy research efforts (Burns et al., 2023). [This chapter is published in: "Handbook of Educational Psychology, 4th Ed.," Routledge New York, 2023, pp. 553-575.]

This chapter provides an overview of research in computer-based writing instruction (CBWI). CBWI may entail any type of writing instruction with computers and may refer specifically to technologies that provide computer-generated feedback and instruction. The purpose of CBWI is to support students in improving their written products and writing processes, and to also optimize teacher labor regarding instruction and feedback. We argue in this chapter that the benefits afforded by CBWI are enhanced within the context of writing instruction that emphasizes principles of evidence-based writing practices and is situated within authentic writing communities. We conclude with a discussion about potential affordances and threats to writing instruction practices and communities from generative artificial intelligence tools. Future research directions for CBWI are discussed. [This chapter was published in: "Handbook of Writing Research (3rd Ed.)," Guilford, 2023.]

Writing an integrated essay based on multiple-documents requires students to both comprehend the documents and integrate the documents into a coherent essay. In the current study, we examined the effects of summarization as a potential reading strategy to enhance participants' multiple-document comprehension and integrated essay writing. Participants (n= 295) were randomly assigned to either summarize or reread five texts on sun exposure and radiation. They produced an integrated essay based on the texts that they read, which were scored by expert raters. Finally, the participants completed three knowledge assessments (topic, domain, general). Readers who summarized texts had lower essay scores than readers who reread the texts. However, within the summary group, summary quality was positively correlated with essay score. These findings are discussed within the context of multiple-document comprehension and writing skill. [This paper will be published in "Contemporary Educational Psychology."]

Computer-based learning environments (CBLEs) provide unprecedented opportunities for personalized learning at scale. One such system, iSTART (Interactive Strategy Training for Active Reading and Thinking) is an adaptive, game-based tutoring system for reading comprehension. This paper describes how efforts to increase personalized learning have improved the system. It also provides results of a recent implementation of an adaptive logic that increases or decreases text difficulty based on students' performance rather than presenting texts randomly. High school students who received adaptive text selection showed increased sense of learning. Adaptive text selection also resulted in greater pre-training to post-training comprehension test gains, especially for less-skilled readers. The findings demonstrate that system-driven, just-in-time support consistent with the goals of personalized learning benefit the efficacy of computer-based learning environments. [This paper will be published in "Journal of Research on Technology in Education."]

Theories of discourse argue that comprehension depends on the coherence of the learner's mental representation. Our aim is to create a reliable automated representation to estimate readers' level of comprehension based on different productions, namely self-explanations and answers to open-ended questions. Previous work relied on Cohesion Network Analysis to model a cohesion graph composed of semantic links between multiple reference texts and student productions. From this graph, a set of features was derived and used to build machine learning models to predict student comprehension scores. In this paper, we build on top of the previous study by: (1) extending the CNA graph by adding new semantic links targeting specific sentences that should have been captured within the learner's productions; and (2) cleaning the self-explanations by eliminating frozen expression, as well as entries which seemed nearly identical to the source text. The results are in line with the conclusions of the previous study regarding the importance of both self-explanations and question answers in predicting the students' reading comprehension level. They also outline the limitations of our feature generation approach, in which no substantial improvements were detected, despite adding more fine-grained features. [This paper was published in: I. I. Bittencourt et al. (Eds.), "AIED 2020" (pp. 228-233). Switzerland: Springer Nature.]

The Design Implementation Framework, or DIF, is a design approach that evaluates learner and user experience at multiple points in the development of intelligent tutoring systems. In this chapter, we explore how DIF was used to make system modifications to iSTART, a game-based intelligent tutoring system for reading comprehension. Using DIF as a guide, we conducted internal testing, focus groups, and usability walk-throughs to develop iSTART-3, the latest iteration of iSTART. In addition to these evaluations, DIF highlights the need for experimental evaluation. With this in mind, we describe an experimental evaluation of iSTART-3 as compared to its predecessor, iSTART-ME2. Analyses revealed an interesting tension between system usability and user preference that has important implications for instructional designers. [This chapter was published in: M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), "Learner and User Experience Research: An Introduction for the Field of Learning Design & Technology." EdTech Books.]

Objective: To develop novel, scalable, and valid literacy profiles for identifying limited health literacy patients by harnessing natural language processing. Data Source: With respect to the linguistic content, we analyzed 283 216 secure messages sent by 6941 diabetes patients to physicians within an integrated system's electronic portal. Sociodemographic, clinical, and utilization data were obtained via questionnaire and electronic health records. Study Design: Retrospective study used natural language processing and machine learning to generate five unique "Literacy Profiles" by employing various sets of linguistic indices: Flesch-Kincaid (LP_FK); basic indices of writing complexity, including lexical diversity (LP_LD) and writing quality (LP_WQ); and advanced indices related to syntactic complexity, lexical sophistication, and diversity, modeled from self-reported (LP_SR), and expert-rated (LP_Exp) health literacy. We first determined the performance of each literacy profile relative to self-reported and expert-rated health literacy to discriminate between high and low health literacy and then assessed Literacy Profiles' relationships with known correlates of health literacy, such as patient sociodemographics and a range of health-related outcomes, including ratings of physician communication, medication adherence, diabetes control, comorbidities, and utilization. Principal Findings: LP_SR and LP_Exp performed best in discriminating between high and low self-reported (C-statistics: 0.86 and 0.58, respectively) and expert-rated health literacy (C-statistics: 0.71 and 0.87, respectively) and were significantly associated with educational attainment, race/ethnicity, Consumer Assessment of Provider and Systems (CAHPS) scores, adherence, glycemia, comorbidities, and emergency department visits. Conclusions: Since health literacy is a potentially remediable explanatory factor in health care disparities, the development of automated health literacy indicators represents a significant accomplishment with broad clinical and population health applications. Health systems could apply literacy profiles to efficiently determine whether quality of care and outcomes vary by patient health literacy; identify at-risk populations for targeting tailored health communications and self-management support interventions; and inform clinicians to promote improvements in individual-level care. [This is the online version of an article published in "Health Services Research."]

This article provides a commentary within the special issue, Integration: The Keystone of Comprehension. According to most contemporary frameworks, a driving force in comprehension is the reader's ability to generate the links among the words and sentences (ideas) in the texts and between the ideas in the text and what the readers already know. As such, the key to successful comprehension is generating inferences: inferences about what a word means, inferences to integrate ideas within the sentences, and inferences to integrate ideas within and outside the text. Accordingly, inferences provide the glue to hold it all together. While there is general agreement in this regard, the factors reported to be key to comprehension also depend on the focus of the research. Some studies focus on surface level and relatively brief measures of reading comprehension. These studies might conclude that comparable measures are the key to comprehension. Other studies use inference-level questions with a focus on comprehension and learning in the context of informational text, even multiple documents. These studies point toward the crucial importance of inference generation. The four articles in this special issue each tackle the question of how readers comprehend text from different angles, using different types of dependent measures, and with foci on different individual differences. Each of these studies provides a glimpse into different experimental paradigms used to investigate comprehension processes as well differing questions that are currently faced by discourse processes researchers. [This is the online version of an article published in "Discourse Processes." For the final published version of this article, see?EJ1286275.]