Search

The American Association of Physicists in Medicine began the Medical Physics Leadership Academy Journal Club in the fall of 2020. The initiative was launched to provide a forum for medical physicists to learn about leadership topics using published material, discuss and reflect on the material, and consider incorporating the discussed skills into their professional practice. This report presents the framework for the MPLA Journal Club program, describes the lessons learned over the last 2 years, summarizes the data collected from attendees, and highlights the roadmap for the program moving forward.

This study evaluated the construct validity of six scoring traits of an automated writing evaluation (AWE) system called "MI Write." Persuasive essays (N = 100) written by students in grades 7 and 8 were randomized at the sentence-level using a script written with Python's NLTK module. Each persuasive essay was randomized 30 times (n = 3000 total randomizations), and the mean trait scores for each set of randomized iterations were compared to those of the control text across all traits. We were specifically interested in evaluating the effects of randomization on the high-level traits of "idea development" and "organization." Given the rubrics and qualitative feedback provided by MI Write, we hypothesized that these high-level traits ought to be sensitive to sentence-level randomization (i.e., scores should decrease). Overall, complete randomizations did not consistently significantly impact trait scoring for these high-level writing traits. In fact, more than a third of the essays saw significant increases in one or both high-level traits despite randomization, indicating a disconnect between MI Write's formative feedback and its underlying constructs. Findings have implications for consumers and developers of AWE.

Purpose: This study presents results from a pilot intervention that integrated self-regulation through reflection and goal setting with automated writing evaluation (AWE) technology to improve students' writing outcomes. Methods: We employed a single-group pretest-posttest design. All students in Grades 5-8 (N = 56) from one urban, all female, public-charter middle school completed pretest and posttest measures of writing beliefs and writing performance. In between pretest and posttest, students completed monthly goal-setting activities via a Qualtrics survey and monthly persuasive writing practice via prompts completed within an AWE system. Findings: Students improved their self-regulation as indicated by improved goal calibration and confidence to achieve their goals over time. They also improved their self-efficacy for writing self-regulation and writing performance between pre and posttest. Students also perceived the intervention to be usable, useful, and desirable. Originality: This is a unique study because we integrated AWE and goal-setting instruction, which has not previously been done. Positive findings indicate the promise of this innovative, feasible, and scalable technology-based writing intervention.

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3-5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

This study investigated fourth and fifth graders' metacognitive knowledge about writing and its relationship to writing performance to help identify areas that might be leveraged when designing effective writing instruction. Students' metacognitive knowledge was probed using a 30-minute informative writing prompt requiring students to teach their reader how to be a good writer (i.e., a metawriting task). The metawriting task was coded for eight dimensions of metacognitive knowledge. Students' writing performance was assessed via additional 30-minute prompts--two narrative, one informative, two persuasive--and evaluated for quality and length using automated essay scoring. Students were most aware of general characteristics of writing quality and production procedures, but they were less aware of substantive processes, genre, or other dimensions. Multiple regression analysis showed that, after controlling for demographics and literacy skills, only knowledge of general characteristics of writing and production procedures uniquely predicted outcomes. Overall, metacognitive writing knowledge inconsistently predicted writing performance.

We investigated the promise of a novel approach to formative writing assessment at scale that involved an automated writing evaluation (AWE) system called MI Write. Specifically, we investigated elementary teachers' perceptions and implementation of MI Write and changes in students' writing performance in three genres from Fall to Spring associated with this implementation. Teachers in Grades 3-5 (n = 14) reported that MI Write was usable and acceptable, useful, and desirable; however, teachers tended to implement MI Write in a limited manner. Multilevel repeated measures analyses indicated that students in Grades 3-5 (n = 570) tended not to increase their performance from Fall to Spring except for third graders in all genres and fourth graders' narrative writing. Findings illustrate the importance of educators utilising scalable formative assessments to evaluate and adjust core instruction.

Automated Writing Evaluation (AWE) provides automatic writing feedback and scoring to support student writing and revising. The purpose of the present study was to analyze a statewide implementation of an AWE software (n = 114,582) in grades 4-11. The goals of the study were to evaluate: (1) to what extent AWE features were used; (2) if equity and access issues influenced AWE usage; and (3) if AWE usage was associated with writing performance on a large-scale state writing assessment. Descriptive statistics and hierarchical linear modeling were used to answer the research questions. Results indicated that the main feature of AWE (i.e., writing and revising essays) were used but some features (peer review and independent lessons) were underutilized. School and student level demographic variables explained little variance in AWE usage. AWE usage was statistically and positively associated with performance on a large-scale state writing assessment when controlling for prior performance and demographics. The study presents evidence that AWE can positively influence writing on a distal measure when implemented at-scale. Implications for large-scale AWE implementation are discussed.

This study examined a naturalistic, districtwide implementation of an automated writing evaluation (AWE) software program called "MI Write" in elementary schools. We specifically examined the degree to which aspects of MI Write were implemented, teacher and student attitudes towards MI Write, and whether MI Write usage along with other predictors like demographics and writing self-efficacy explained variability in students' performance on a proximal and distal measure of writing performance. The participants included 1935 students in Grades 3-5 and 135 writing teachers from 14 elementary schools in a mid-Atlantic school district. Findings indicated that though MI Write was somewhat under-utilized, teachers and students held positive attitudes towards the AWE system. Usage of MI Write had a mixed and limited predictive effect on outcomes: The number of essays written had a small predictive effect on state test performance for Grades 3 and 5; gain on revision had a moderate predictive effect on posttest writing quality and a small predictive effect for Grade 5 state test performance. Students' average AWE scores showed consistently moderate to large predictive effects for all outcomes. Interpreted in light of the underlying architecture of MI Write, findings have implications for other school districts considering implementing AWE as well as the design of AWE systems intended to support the teaching and learning of writing.

Automated writing evaluation (AWE) is a popular form of educational technology designed to supplement writing instruction and feedback, yet research on the effectiveness of AWE has observed mixed findings. The current study considered how students' perceptions of automated essay scoring and feedback influenced their writing performance, revising behaviors, and future intentions toward the technology. The manner in which the software was presented--claims about the accuracy and quality of the automated scoring and feedback--were modestly related to students' expectations and perceptions. However, students' direct experiences with the software were most strongly associated with their perceptions. Importantly, students' perceptions seemed to have minimal impact on their "in the moment" use of the software to write and revise successfully. Students revised and improved their essays regardless of their positive or negative views of the system. However, positive and negative perceptions significantly predicted future intentions to use the software again or to recommend the software to a friend. Implications for AWE design, implementation, and evaluation are discussed.

Many students do not meet expected standards of writing performance, despite the need for writing competence in and out of school. As policy instruments, writing content standards have an impact on what is taught and how students perform. This study reports findings from an evaluation of the content of a sample of seven diverse states' current writing standards compared to content of the Common Core State Standards for writing and language (CCSS-WL). Standards were evaluated for breadth of content coverage (range), how often content was referenced (frequency), the degree of emphasis placed on varied content elements (balance), and the degree of overlap between one set of standards and another (alignment). The study addressed two research questions: (1) What is the nature of the CCSS-WL and the sample states' standards for writing with respect to content breadth, frequency, and balance? (2) To what degree do the states' writing standards align with the CCSS-WL? Results indicated that CCSS-WL are succinct and balanced, with breadth of coverage in some aspects of writing but not others. The seven states' standards represented varying degrees of breadth, frequency, and balance with few patterns across states. None of the states' standards had strong alignment with CCSS-WL, indicating a potential mismatch between prior curricular materials and instructional methods developed with former standards as guides to help students meet grade-level writing expectations in the new CCSS. [This article was published in "Reading Horizons," v55 n3 Article 4 p98-141 Oct 2016, see EJ1117076.]

Educators need actionable information about student progress during the school year. This paper explores an approach to this problem in the writing domain that combines three measurement approaches intended for use in interim-assessment fashion: scenario-based assessments (SBAs), to simulate authentic classroom tasks, automated writing evaluation (AWE) features to track changes in performance and writing process traits derived from a keystroke log. Our primary goal is to determine if SBAs designed to measure English Language Arts skills, supplemented by richer measurement of the writing task, function well as interim assessments that are sensitive to differences in performance related to differences in quality of instruction. We calibrated these measures psychometrically using data from a prior study and then applied them to evaluate changes in performance in one suburban and two urban middle schools that taught argument writing. Of the three schools, only School A (the suburban school, with the strongest overall performance) showed significant score increases on an essay task, accompanied by distinctive patterns of improvement. A general, unconditioned growth pattern was also evident. These results demonstrate an approach that can provide richer, more actionable information about student status and changes in student performance over the course of the school year.

Many students do not meet expected standards of writing performance, despite the need for writing competence in and out of school. As policy instruments, writing content standards have an impact on what is taught and how students perform. This study reports findings from an evaluation of the content of a sample of seven diverse states' current writing standards compared to content of the Common Core State Standards for writing and language (CCSS-WL). Standards were evaluated for breadth of content coverage (range), how often content was referenced (frequency), the degree of emphasis placed on varied content elements (balance), and the degree of overlap between one set of standards and another (alignment). The study addressed two research questions: (1) What is the nature of the CCSS-WL and the sample states' standards for writing with respect to content breadth, frequency, and balance? (2) To what degree do the states' writing standards align with the CCSS-WL? Results indicated that CCSS-WL are succinct and balanced, with breadth of coverage in some aspects of writing but not others. The seven states' standards represented varying degrees of breadth, frequency, and balance with few patterns across states. None of the states' standards had strong alignment with CCSS-WL, indicating a potential mismatch between prior curricular materials and instructional methods developed with former standards as guides to help students meet grade-level writing expectations in the new CCSS.

Though writing plays an important role in academic, social, and economic success, typical writing instruction generally does not reflect evidence-based practices (EBPs). One potential reason for this is limited signposting of EBPs in standards. We analyzed the content of writing standards from a representative sample of states and the Common Core State Standards (CCSS) for writing and language to determine to what degree EBPs were signposted, variability of this signposting, and the overlap of practices signposted in states' standards and the CCSS. We found a few practices signposted fairly consistently (e.g., isolated components of writing process instruction) and others rarely so (e.g., use of text models), as well as great variability across standards, with some covering almost half of the EBPs and others far fewer. Only a few states' writing standards overlapped considerably with the CCSS. We discuss the implications of these findings for teacher professional development and for evaluating standards. [This article was published in "Elementary School Journal," v116 n2 p291-321 Dec 2015 (EJ1084264).]

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used "Project Essay Grade Writing" (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.