Search

Generally, need-based financial aid improves students' academic outcomes (Nguyen, Kramer & Evans, 2019). However, the largest source of need-based grant aid in the United States, the Federal Pell Grant Program (Pell), has a mixed evaluation record (Bettinger, 2004; Rubin, 2011; Marx & Turner, 2018; Park & Scott-Clayton, 2018; Carruthers & Welch, 2019; Denning, Marx, & Turner, 2019; Liu, 2020; Eng & Matsudaira, 2021). We assess the minimum Pell Grant in a regression discontinuity framework in a sample of first-time, first-year students who enroll and file a FAFSA at Kentucky public colleges and universities between 2011 and 2017. We focus on whether and how year-to-year changes in aid eligibility and interactions with other sources of aid attenuate Pell's estimated effects on post-secondary outcomes. This evaluation complements past work by assessing explanations for the null or muted impacts found in our analysis and other Pell evaluations. We also discuss the limitations of using regression discontinuity methods to evaluate Pell--or other interventions with dynamic eligibility criteria--with respect to generalizability and construct validity. Between 1992 and 2022, published tuition and fees at public four-year U.S. colleges and universities more than doubled in real terms (Ma & Pender, 2022). For cohorts facing high direct costs of college, student financial aid has become an essential resource to facilitate access to post-secondary training. Indeed, an extensive literature documents that students respond positively to the availability of need-based financial aid (Nguyen, Kramer & Evans, 2019) and other policies that reduce college costs (Deming & Dynarski, 2010; Page & Scott-Clayton, 2016). Consistent with past studies of the minimum Pell grant (e.g., Marx & Turner, 2018; Park & Scott-Clayton, 2018), we find no evidence that initial Pell eligibility confers academic advantages to marginally eligible students. We also find that initial Pell eligibility significantly decreases the amount a student borrows in their first year; and, for some groups of students, this reduction in borrowing grows over time. We also build upon past studies to consider the ways annual changes in Pell eligibility ("churn") may affect our estimates by assessing the impact of initial Pell eligibility in the population of FAFSA filers who are disproportionately exposed to instability in Pell aid over time. Our findings suggest important interactions between financial aid programs both in the short- and medium-run. In the full sample (as shown in Table 1) as well as within most subgroups, the Pell advantage in grant aid is offset by reduced borrowing, such that we cannot reject the null hypothesis of no difference in total aid (all grants plus all loans) at the Pell eligibility threshold. These null estimated differences in total aid correspond with null estimated impacts on academic outcomes (as shown in Table 2). However, within subgroups that experience the strongest contrasts in aid stability, we find correspondingly large differences in borrowing effects (with smaller, less negative borrowing effects for subgroups that experience more aid stability), which exceed the Pell advantage in grant aid. Within these groups, we find suggestive evidence connecting aid instability and reduced borrowing to worse academic outcomes. In our context, marginal Pell eligibility in one year is a poor predictor of future Pell receipt. This lack of predictability in Pell eligibility emphasizes the barriers complex aid applications and annual renewal processes can create for students (Bettinger et al., 2012; Baum & Scott-Clayton, 2013; Dynarski, Page & Scott-Clayton, 2022). As shown in Figure 1, among re-enrolling students in our analytic sample who did re-apply for federal aid after receiving a Pell grant in their first year, less than 65% maintained any Pell aid, while nearly 16% of initially ineligible students received a Pell grant in their second year of enrollment. In other words, aid recipients often lose Pell grant eligibility from year-to-year, while non-aid recipients can gain eligibility. This "churn" between Pell-eligibility and ineligibility means that cross-sectional estimates of the longer-term effects of receiving a Pell Grant in one year may be attenuated by a narrowing of differences in Pell aid received over time, especially near the Pell eligibility threshold, where year-to-year changes in Pell eligibility are most common. Since Pell aid is least stable for students with EFCs near the minimum Pell threshold, and these students provide identifying variation in aid eligibility for evaluations of the minimum Pell grant that rely on a regression discontinuity (RD) design, attending to year-to-year variation in aid receipt is central to understanding the character of the treatment-comparison contrast in policy exposure in this type of quasi-experimental evaluation of the Pell grant. We emphasize that researchers and policymakers should thus interpret our findings as estimating the effect of receiving a [approximately]$300 award for a single year (about half the minimum Pell Grant award for a full-time student during our sample period) in contrast to reflecting aid impacts for "typical" or "average" Pell recipient (as shown in Figure 2). This is because the majority of Pell students receive larger awards and are more likely to maintain year-to-year Pell eligibility than students with a first-year EFC near the Pell-eligibility threshold, like those in our sample. These facts emphasize the limitations of RD evaluations of the minimum Pell Grant with respect to external validity (i.e., how the results should be expected to generalize) and construct validity (in this case, with respect to what "treatment" the treatment-comparison contrast represents), despite maintaining a strong claim as an internally valid method to estimate the total impact of marginal Pell eligibility in a student's first-year of college. The impact of Pell "churn" on our results thus has important implications for future design and evaluations of broad-reaching student aid programs or any repeated intervention with time-varying, dynamic eligibility criteria. Researchers and policymakers should exercise caution when extrapolating results from this study (or any RD evaluation of the minimum Pell grant) to other groups of students, other amounts of Pell aid, or other grant aid programs.

Background/Context: Political divisiveness, racial reckoning, climate change, and mental health challenges represent just a few contemporary issues facing today's youth. Addressing these challenges now and in the future requires deep interpersonal understanding and collective action--neither of which can be achieved without empathy or prosocial behavior. Fortunately, we have decades of research on both of these skills (Bar-Tal, 1976; Eisenberg et al., 2009). However, the contours of this research call attention to three realities: (a) both skills can be cultivated in schools (Durlak et al., 2011); (b) empathy and prosocial behavior are strongly correlated (Barr & Higgins-D'Alessandro, 2007); (c) we have surprisingly little information about how these complementary skills can be cultivated together. Purpose/Objective/Research Questions: The current study was designed to not only extend our understanding of effectiveness of EL Education's model to develop character (EL Education, n.d., 2018), but to expand our toolkit of teaching practices that help to promote empathy and empathy-motivated prosocial responding (i.e., prosocial behavior motivated by empathy). Our research questions are: 1. How do students at EL Education and Comparison schools define empathy? 2. In what ways, if any, does empathy-motivated prosocial responding appear within students' definitions of empathy? 3. How do educators at EL Education and Comparison schools describe their practices for teaching empathy? 4. In what ways, if any, does empathy-motivated prosocial responding appear within educator's empathy teaching practices? Setting & Population/Participants/Subjects: Students attended nine middle schools (five EL Education and four Comparison) in four cities across the United States. Of the 23 student participants (16 EL Education, 7 Comparison), thirty-five percent identified as Black, 57% as White, and 52% as female. (See Table 1 for profiles.) Of the 18 educators interviewed (12 EL Education, 6 Comparison), eleven percent identified as Black, while the remaining 89% identified as White. Fifty-six percent identified as female and 44% identified as male. (See Table 2 for profiles). Research Design: A part of a larger quasi-experimental study, our primary approach was descriptive-interpretive design (Elliott & Timulak, 2021). This allowed for both surface level (descriptive) and deeper (interpretive) analysis. Guided by this approach, the research team developed open-ended and exploratory research questions and analysis strategies that aimed at better understanding how study participants defined and cultivated empathy across EL Education and Comparison schools. Data Collection and Analysis: All student interviews were conducted in-person at schools December 2019 February 2020. Educator phone interviews began in March 2020 and were suspended due to the pandemic. Some additional educator interviews were conducted in May 2020. A semi-structured interview protocol was used (Cohen & Crabtree, 2006). Interviews lasted 15-30 minutes and were audio recorded for future transcription. Questions used in this analysis included: "What do you think empathy means?" for students and "How do you teach empathy in your classroom?" for educators. The analytic approach followed the descriptive-interpretive design (Elliott & Timulak, 2021). First, following verbatim transcription, the coding team gained familiarity with the transcripts, memoing initial thoughts or potential themes. Next, the team divided the data into distinctive meaning units and developed an organizational structure for the data (i.e., a consensus-built codebook; Glaser & Strauss, 1967). The final step was to iteratively abstract the main findings. The first author defined categories, built prevalence tables, and established categorization thresholds. Rigor was ensured through maintaining an audit trail (Schwandt & Halpern, 1988) and engaging in regular internal and external audits. Credibility was addressed following the guidelines of descriptive-interpretive design, consulting with an experienced qualitative researcher throughout the process (Holloway, 1997), and using the participants' own words (Thomas & Magilvy, 2011). Findings/Results: Results showed three themes in student definitions of empathy: (a) "empathy" (i.e., affective empathy, cognitive empathy, and other empathy-related responses), (b) "prosocial behavior" (i.e., no mention of empathy, only prosocial behavior), and (c) "empathy-motivated prosocial responding." EL Education students and Comparison students were nearly equally likely to provide definitions as exclusively "empathy" (57% and 38%, respectively) and exclusively "prosocial behavior" (13% and 14%, respectively). EL Education students were more likely to define empathy as "empathy-motivated prosocial responding" (50% and 29%, respectively). (See Table 3 for details.) Results also showed that educators at both EL Education and Comparison schools utilized (a) "coaching," (b) "restorative practices," (c) "modeling," (d) "classroom opportunities," (e) "curriculum integration," and (f) "community spaces" to teach empathy. While both were nearly equally likely to use "coaching" (50% each) and "restorative practices" (8% and 17%, respectively), EL Education educators were more likely to use "modeling" (33% and 17%, respectively), "classroom opportunities" (33% and 17%, respectively), and "community spaces" (50% and 17%, respectively). On the other hand, Comparison school educators were more likely to use "curriculum integration" (50% and 17%, respectively). (See Table 4 for details.) Conclusions: Results show that "empathy-motivated prosocial responding" was a stronger theme in EL Education students' definitions, and teaching practices that promote both empathy and empathy-motivated prosocial responding were a stronger theme in EL Education educators' responses. EL Education students' responses indicated not a misunderstanding of what empathy is, but a belief that the practice of empathy is inextricably tied to prosocial action. Similarly, EL Education educators described practices that encouraged empathy and empathy-motivated prosocial responding from their students. One key limitation is our that sample sizes from EL Education and Comparison schools are not equivalent, even with extensive recruitment efforts and school partnerships. To address this, we report both counts and percentages, as well as the thresholds used for categorization for transparency. The teaching practices shared by both EL Education and Comparison school educators are important: they can be powerful as standalone opportunities, but as a larger community commitment, they have the potential to make interpersonal understanding and transformative collective action possible. We believe that supporting the development of empathy and empathy-motivated prosocial responding is critical to the wellness and success of young people--especially with the unique challenges facing their generation. By integrating these practices into our own classrooms and school communities, we can not only imagine, but create a better--and more empathetic--world.

Background: Mission US is a series of interactive first-person role-playing history games and curricular materials that address a critical problem: students lack fundamental knowledge of our nation's history. According to the most recent National Assessment of Educational Progress (NAEP, 2018), only 13% of Grade 8 students were proficient on the U.S. history assessment. This randomized controlled trial (RCT) is the first large scale, national study to evaluate the impact of Mission US on students' history knowledge and skills. Purpose/Research Questions: This study investigates the impact of two Mission US games and accompanying curricular materials on Grade 810 students' U.S. history outcomes, and is guided by three confirmatory research questions: Relative to business-as-usual classrooms, does using Mission US impact students': 1) historical content knowledge; 2) ability to analyze historical documents; and 3) motivation to study history? The study also explores the potential mediating effect of historical empathy on these outcomes. We hypothesize that Mission US will improve students' historical knowledge and thinking skills, especially historical empathy, perspective-taking, and the analysis of historical documents. Setting: Researchers recruited teachers at public, private, and charter schools from 25 states across the U.S. Orientation, professional development, and assessments were virtual, but teachers implemented the intervention in their regular classroom settings. Participants: Participants included 65 Grade 8-10 social studies teachers at 60 schools and their 1,305 students, enrolled in two groups (Spring/Fall 2023). Schools were heterogeneous (e.g., traditional public schools, virtual, military leadership academy, religious schools); see Table 1 for a description of participants by treatment status. Teachers' average age was 43, and 67.7% identified as female. Teachers averaged 14 years of teaching experience; 93.8% had a social studies teaching certificate. Of the students, 42.9% were in Grade 8, 20.3% in Grade 9, and 36.8% in Grade 10. Intervention: Intervention teachers used two Mission US games and materials (12 hours total), during a three-month implementation period. Teachers were provided with curriculum guides for each era with suggested pacing. After baseline assessments and randomization, intervention teachers received 60-minutes of professional development, introducing the Mission US games, curriculum, website, and dashboard. Control teachers used an analogous website, with background documents to ensure equivalent historical content across all classrooms. Research Design: The team conducted an RCT to evaluate the impact of Mission US on students' history outcomes. Researchers randomly assigned teachers to intervention or control group after baseline assessments. Due to recruitment challenges, researchers shifted from a model with two social studies teachers per school and randomizing within school to instead randomizing at the school level, with students nested within school. All measures and procedures were pilot-tested with 11 teachers and their 234 students. To mitigate possible biases, researchers pre-registered the study design in the Registry of Educational Effectiveness Studies (REES) and collected baseline data prior to randomization; attrition was low and met What Works Clearinghouse standards. Data Collection and Analysis: All surveys and assessments were administered digitally using Qualtrics. Teachers administered baseline and post-assessments to students. Teachers completed demographic and curriculum surveys at the start of the study; weekly activity logs and virtual classroom observations monitored fidelity of implementation. Forty-three teachers provided de-identified student-level data, including demographics, prior semester GPA, attendance, enrollment, ELL, and IEP/504 plan status. Student outcomes were measured as follows: Research Question 1: Historical Content Knowledge. Using the NAEP Questions Tool, the team identified 24 multiple-choice Grade 8 questions. See Table 2 for a breakdown of question type on pre- and post-test (Cronbach's alpha=0.76-0.81; Hedges' g = 0.15). Research Question 2: Analysis and Interpretation of Historical Documents. Using the NAEP Questions Tool, the team identified 15 Grade 8 document-based multiple-choice questions and distributed these evenly across pretest (Cronbach's alpha=0.64-0.69; Hedges' g= 0.13). (See Table 2.) Research Question 3: Motivation to Study History. Students' interest and motivation to study history were measured with two instruments: the Individual Interest Questionnaire (Cronbach's alpha=0.86-0.88; Hedges' g = 0.005) and the Interest and Enjoyment Index from the 2018 NAEP U.S. History Student Questionnaire (Cronbach's alpha=0.80-0.85; Hedges' g = 0.005). Research Question 4: Historical Empathy. The research team used the approach in Hartmann and Hasselhorn's (2008) Historical Perspective Taking measure and the theoretical framework from Endacott and Brooks (2013, 2018) to develop a measure of historical empathy. The Historical Empathy Measure includes two alternate forms (Cronbach's alpha=0.78; Hedges' g = 0.029), containing 15 multiple-choice questions, 5 questions for each core component (i.e., historical contextualization, perspective-taking, affective connection). Researchers established baseline equivalence between intervention and control students for all measures. Psychometric analyses indicate acceptable reliability for all instruments. After descriptive analyses, researchers subjected each student outcome to a student-nested-within-school multilevel regression analysis where the outcome is regressed on the treatment indicator, outcome at baseline, and group-enrollment indicator (Fall/Spring). We are currently adding student and school-level covariates to the models, which will be particularly important given the heterogeneity of the schools sampled. Additional moderator and exploratory mediator analyses are underway. Findings/Results: Preliminary analyses without covariates demonstrated a positive but not significant difference in outcomes between students in the intervention and control groups for all outcome measures. These null findings may be a result of the study design changes discussed above. Using the current two-level cluster design, we plan to statistically control for differences in school characteristics to improve statistical power. Currently, our team is refining analyses and using publicly available federal data to fill in school-level variables and address missing student-level data. We will be able to fully report on our analyses by the conference. Qualitative data from teacher focus groups show Mission US impacted students' interest and, given our theory of change, we plan to explore subgroup differences in baseline interest level and differences by student characteristics. Preliminary subgroup analyses indicate that there may be heterogenous treatment effects. Conclusions Our RCT provides a case study on the tensions between sample heterogeneity and detecting impact, as well as a novel measure of historical empathy. Additional moderator and subgroup analyses will allow us to specify for which students and under what conditions Mission US might exhibit a positive impact.

Background: Students' social emotional competencies (SECs; also known as non-cognitive skills) are the skills or knowledge that allow them to successfully navigate social and academic environments such as self-management, social awareness, and relationship skills. Research has established cross-sectional associations between SECs and student outcomes (Farrington et al., 2012). Fewer studies have examined these associations longitudinally, including with econometric fixed effects methods to address unmeasured confounders. This study fills that gap. Purpose: We examine five SEC domains defined by CASEL: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making. We further divided these domains into 8 subdomains including self-management in the areas of emotions, goals, and schoolwork. Based on a small, but growing body of research (Claro & Loeb, 2019; Gordon et al., 2022), we anticipate that self-management of schoolwork will be associated with GPA. A major contribution of our study is testing these relationships in an adolescent sample. There are fewer studies on SECs and academic outcomes in high school samples compared to early childhood and elementary school (Cipriano et al., 2023). High school is relatively understudied, despite the increased academic and social demands during that period. We first describe changes in SECs and GPA between 9th and 11th grades and then examine the association between those changes. Research Questions: R1: Are SECs and GPA stable overtime? Or do they change for students from 9th to 11th grade? R2: Does a change SECs relate to a change in GPA in 9th and 11th grades? Setting: This study took place at a mid-sized urban school district in the Western United States. Participants: The study examined a longitudinal sample of high school students in the 2020 graduating cohort. We examined two time points, 9th and 11th grades (the high school years the climate survey was administered), and 4,652 students who were enrolled in school at both timepoints (see Table 1). Research Design: The current study used a quantitative secondary analysis survey design. District administrative data were linked to the annual school climate survey, where SEC data are collected. Data Collection and Analysis: Students completed the 40-item WCSD-SECA as part of the district's annual climate survey that is administered during the school day. The WCSD-SECA was designed using a rigorous mixed-method continuous measure improvement approach to align with the CASEL 5 (Davidson et al., 2018) but uses 8 subscales rather than 5, (see Table 2). Students rate how easy or difficult each of the items are for them on a 4-point scale from 1 = Very Difficult to 4 = Very Easy, and we calculated average scores for each domain. Students who answered at least one survey item for each domain were included in calculations. Sixty-one percent of students completed the survey in both 9th and 11th grades, 32% took the survey one year, and 7% never took the survey. We used multiple imputation to impute survey data for the 32% of the students who only took the survey one year. After imputing the data, we applied the first difference regression approach (Allison, 2009; Angrist & Pischke, 2009) to assess the longitudinal relationship between changes in SECs in 9th and 11th grade and GPA. Analogous to a two-panel fixed effects model, the approach uses difference scores (?) to account for time-invariant heterogeneity (Allison, 2009; Angrist & Pischke, 2009). We calculated the difference scores by subtracting 11th grade survey responses and GPA (t2) from their 9th grade survey responses and GPA (t1). Student identifiers are collected on the survey and were used to link to their school records. We used students' average Grade Point Average (GPA) on a scale of 0.0 to 4.0 as the outcome variable. We controlled for yearly academic risk, which is a composite score of risk indicators based on attendance, suspensions, and school changes in the current academic year and is predictive of GPA. Results: R1: Are GPA and SECs stable overtime? Descriptive analyses indicate that on average, GPA decreased by 0.17 points between 9th and 11th grades. Because of the large variability in the SEC differences scores (Figure 1), we examined how many students changed by half a standard deviation or more for each SEC (see Figure 2). The largest changes in the eight WCSD-SECA subscales were for self-management of schoolwork, with over a third of students (37%) reporting a decline between 9th and 11th grades and 27% reporting an increase. R2: Does a change in SECs relate to a change in GPA? To answer the second research question, we ran a first difference regression model to assess the association between change in SECs and change in GPA between 9th and 11th grade (see Table 3). Both changes in social awareness (B = -0.06, SE = 0.02, p < 0.05) and self-management of schoolwork (B = 0.11, SE = 0.02, p < 0.01) between 9th and 11th grades were associated with changes in GPA. In other words, students who reported gains in self-management of schoolwork also saw GPA gains, whereas students who reported gains in social awareness in 11th grade also experienced a decline in their GPA. Conclusion: In this study, we examined longitudinal relationships between SECs and academic outcomes in a high school sample. Consistent with past research that suggests learning strategies decline in adolescence (Bardach et al., 2023) our results show a decline in self-management of schoolwork in high school. An econometric first difference approach indicated that changes in self-management skills (as well as social awareness) are also associated with changes in GPA. The findings of this study have important implications for social emotional programming and policy. Using an econometric first difference approach, this study reveals the association between self-management skills and academic growth remains after washing away time constant effects of time constant confounders. Policies and programs that build students' self-management skills in adolescence are thus needed to better equip them to meet the increasing expectations and demands of high school. Future research should continue to examine these associations and associated programs and policies in other contexts, as this study took place in one school district.

Background: The proliferation of computer-based learning platforms (CBLPs) has caused an increased focus on understanding how to build scalable systems that optimize learning. Thus, CBLPs often rely on close response questions (e.g., multiple choice questions, "select all that apply," "arrange in the correct order," etc.) for learning tasks as they are easy to grade automatically. Furthermore, students' selection of distractors may be leveraged to provide automated feedback based on students' likely misconceptions. Yet, educators often believe that closed-response questions may lead to shallow learning (Wang et al., 2021). Alternatively, open-response questions are often considered more rigorous (Magliano et al., 2007). However, open-response questions are harder to grade automatically, and therefore, it is harder to provide timely and targeted feedback based on students' responses. Fill-in problems, in which students must provide a specific response in a text box, represent a middle ground for some content areas by requiring students to produce answers independently. Furthermore, responses to fill-in problems can be used to provide automated targeted feedback based on what the student submitted (Gurung, Baral, et al., 2023; Gurung, Lee, et al., 2023). Considering the mixed results from prior work on the learning and assessment value of closed and open-response questions (Funk & Dickson, 2011; Magliano et al., 2007; Sugrue et al., 1998), fill-in problems provide a potential alternative problem type that may combine the advantages of closed and open-response questions. Current Study: Using an experimental design, we evaluate the impact of the fill-in problems compared to multiple-choice questions (MCQs) on students' performance during a learning activity (RQ1) and its effect on their learning as measured by the post-test (RQ2). Furthermore, we must ensure that all students share any learning benefits of the features of CBLPs. Thus, following the guidance of Kizilcec & Lee (2022), we evaluate whether the feature does not widen the gap between advantaged and disadvantaged students based on students' performance before the activity (RQ3). Design: This study used mastery learning activities through ASSISTments (Heffernan & Heffernan, 2014), a math-focused CBLP. The experiment involved two middle school math mastery-based activities. Students were randomized to either receive Fill-in problems or MCQs throughout the activity. The problem order during the activity was also random. Figure 1 shows the randomization scheme, and Figure 2 presents the differences in the fill-in, MQC, and post-test. Notably, the post-test consisted of more complex transfer items. Students only took the post-test if they mastered the activity. Data: The data was collected across five school years in the United States (2017-18, 2018-19, 2019-20, 2020-21, 2021-22). Middle school teachers who used ASSISTments as an instructional tool assigned mastery learning activities to their students as part of their lessons. 192 teachers assigned the two problem sets to 383 classes. 6774 students participated in the experiment. Method: Our analysis required four steps. First, we ran a series of regression models with robust standard errors (one model for problem sequence) predicting students' accuracy (Y[subscript ij]) within the master learning activity by whether students were randomized to treatment (Fill-In[subscript i]) with fixed effects for the activities' problems (P[subscript j]) (Equation 1). This evaluates how problem type influences students' performance across the activity. Second, we assessed whether attrition differed across conditions. Third, we use a multi-level logistic regression to estimate whether problem type affects students' learning by regressing treatment assignments on students' accuracy on post-test items accounting for the student ([mu][subscript i]), class ([mu][subscript c]) and problem ([mu][subscript j]) (Equation 2). Finally, we added an interaction to Equation 2 to between Fill-In[subscript i] and students' prior performance in ASSISTments as a measure of their prior mathematical knowledge. Results: (RQ1) On the first problem sequence, students in the MCQ condition outperformed those in the Fill-In condition by an estimated five percentage points ([beta][subscript 1] = -0.05, SE = 0.012, p < 0.0001). Figure 2 displays the average performance (lines) and samples (shading) by condition across the first ten problems in the mastery learning component of the experiment. The pattern of performance differences across the activity suggests that the difficulty experienced early on by the students in the Fill-In condition potentially benefited the students later in the activity. Nevertheless, it is important to acknowledge a potential confound to this explanation as the samples differed across conditions on problem sequences due to differing mastery and attrition rates. (RQ2) Table 1 details the experiment's attrition rates and the balance test statistics. Students could attrit in multiple ways: not masting, not stating the post-test, and not completing the post-test. Overall, there weren't significant differences in attrition across conditions, but the balance between mastery rates was only marginally non-significant. Table 2 presents the model used to estimate the effect of problem type on post-test performance. Students who engaged with Fill-In problem sets were significantly more likely to provide correct responses in the post-test than those who worked through MCQs ([gamma]1 = 0.23, SE = 0.06, p > 0.001). Students in the MCQ condition had a 27% probability of getting either of the transfer problems correct. In contrast, students in the Fill-In condition had a 31% probability of getting the transfer item correct. As a robustness check, we ran this model, giving all students who attired incorrect responses, and the effect was still positive and significant. (RQ3) Table 3 displays the results of the interaction model. The interaction between the prior performance and the Fill-In problem set is significant and positive ([gamma]3 = 0.20, SE = 0.10, p = 0.042). Figure 4 plots the interaction, showing that while students with higher prior performance experienced a positive effect from fill-in problems, those with lower knowledge likely benefited from MQCs. Conclusion: Overall, this study's findings present causal evidence that problem types influence how and whether students learn. We observed that, on average, students had better learning outcomes when using mastery-based assignments with Fill-In problems compared to MCQs, but this effect may vary by students' prior knowledge. Notably, prior performance may not be a perfect measure of prior knowledge; thus, this analysis points to the need for further investigation into how problems are presented to ensure all students are learning.

Literacy skills are instrumental to unlocking a child's full potential and have a well-established link to long-term life outcomes. Children who can't read at grade level by 3rd grade are four times more likely to drop out of school, have fewer career prospects, and earn lower incomes as adults. However, despite some progress on literacy rates, almost 40% of Chicago Public Schools (CPS) students entering 9th grade still lack basic reading proficiency. Now in high school, these students are running out of time to successfully gain the literacy skills they need to graduate and thrive. Most reading interventions are focused on elementary schoolers, so there are few structured literacy opportunities available to older students, let alone interventions that can accelerate their learning enough for them to close the gap. With their pathway to literacy narrowing, it's crucial that we provide them with the urgent, intensive support they need. To fill this gap, in 2020, CPS started rolling out the Structured Literacy program to engage high schoolers with the highest need of literacy support in a robust reading intervention that focuses on decoding and encoding text. The program features small-group, individualized instruction in a separate class period. Students with similar literacy needs are grouped together to receive personalized daily instruction that incorporates evidence-based practices using the Wilson Reading System (WRS). In these sessions, the program provides opportunities for students to read culturally relevant texts at their level, augmenting a skill-based reading curriculum with age-appropriate general texts to enhance engagement. Sessions are integrated into the school day and students earn course credit as part of their high school schedule. We study the impact of the Structured Literacy program using a multi-year randomized controlled trial (RCT) starting in SY2020-21 to SY2023-24. We randomized over 2,700 students across 13 CPS high schools to either receiving the Structured Literacy program or a control condition. Preliminary findings from analysis of the first 3 cohorts suggest the program increased reading scores by 0.20 standard deviations.

Background/Context: As part of a national movement to eliminate a global tracking system that disproportionately assigned underserved students to low-level math classes, the Chicago Public Schools (CPS) mandated academic algebra for all students entering high school in 1997. The results were disappointing: low-skill students experienced elevated failure rates while high-skill students lost ground in math achievement. To rectify this failure, CPS adopted Double Dose Algebra in 2003 for students scoring below the median on a pre-test. Earlier research, using a regression discontinuity design (RDD), found that Double Dose increased algebra scores, high school graduation, college attendance, and even the college degree attainment of students attending neighborhood high schools serving predominately low-income, minority youth. However, these results are only applicable to median-skill students attending classrooms with similarly skilled peers. Median-skill students who took Double-Dose with low-skill peers did not benefit from additional instructional time. These findings pose a key dilemma. For median-skill students, the optimal policy is to increase instructional time while ensuring that their algebra classes are composed of peers having similar skills. A worry, however, is that such a policy would leave low-skill students isolated from median-skill peers. Prior studies offer competing hypotheses regarding the likely effects of such isolation. One line of reasoning suggests that concentrating low-skill students would lead to a negative stigma and such classes might also induce low expectations, perhaps low motivation among the students, and reduced outcomes. A contrasting view is that homogenous grouping would facilitate tailored instruction, thereby optimizing learning growth. This clash of perspectives generates our research questions. Purpose/Objective/Research Question: We ask whether the double-dose policy enhanced or undermined the learning of low-skill students. We assess how their classroom academic composition was affected and whether such compositional changes were related to success in algebra. Our objective is to better understand the optimal class composition when the policy goal is to help low-skill students catch up by increasing instructional time. Setting: Chicago has the third-largest school system in the country. The student population is about 50% Black, 38% Hispanic, 9% White, and 3% Asian. The FRL rates are 85%. Population/Participants/Subjects: We use 4 cohorts of first-time ninth-grade students who began high schools in 2001 through 2004. The analytic population consists of "low-skill" students, defined as those whose predicted probability of scoring below the national median is greater than .90 based on past academic performance and demographic background (N=18,140 students in 61 schools). Intervention/Program Practice: Ninth graders with 8th-grade math scores below the national median were required to take algebra plus a support course designed to help them build foundational math skills. Their teachers received curricular resources and professional development. Among low-skill students, nearly all non-special education students took Double-Dose, while only 50 percent of special education students did so, and their compliance levels varied considerably by school. Research Design: We estimate the policy impact on the treated students ("compliers") separately by special education status. Our design is a within-school version of the principal score method (Frangakis & Rubin, 2002). Specifically, we reweight pre-policy students to resemble the post-policy Double-dose takers within each school. Then, we adopt a school-specific difference-in-differences (DID) strategy to estimate the policy impact within each school. This effectively controls for fixed school effects, school-specific linear time (cohort) trends, prior academic skills, and Data Collection and Analysis: Our main outcomes are classroom compositions and algebra course failure. We analyze a two-level model after re-weighting to pre-policy students to make them resemble the population of interest (compliers or never takers). Our within-school model for student i attending school j is: Y_ij=?0_j+ ?1_j(cohort)+B_jZ +e_ij, where cohort is a linear cohort variable centered on the 2002 cohort, Z is a post-policy indicator, and e_ij is a zero-mean random error. The coefficient B_j is school specific policy effects. At the school level, we subclassified schools into three levels of policy compliance. We model the coefficients ? 1_j,B_j as a function of school compliance levels Gqj where Gq=1 is medium- and Gq=2 is high-compliance with low-complying schools as the reference group: ?(?_1j=???G_qj ?_1q ?+u_j@ ?_ATTj=???G_qj ?_q ?+b_j ) where are assumed bivariate normal in distribution with means 0, variances Tao_uu, Tao_bb and covariance, Tao_ub. Our analytic model includes the main effect of Z and the interaction between Z and compliance levels. Results: For low-skill regular education students, taking Double-Dose induced a slight decline in classroom peer skill, no change in course failures, but a substantial improvement in algebra scores. For special education Double-Dose takers, classroom peer skills and class size increased and so did their algebra course failures. In high-complying schools where most special education students took Double-Dose, peer composition remained relatively similar post-policy as they continued to take the course with other special education students. Those schools had the smallest increase in algebra course failure post-policy. In contrast, in low-complying schools, special education Double-Dose takers experienced the greatest increase in classroom peer skills and class size as they attended classes with mostly non-special education students post-policy. It is in these schools that course failure increased the most for special education Double-Dose takers. For regular education students, school compliance patterns appear to be unrelated to their classroom composition or academic outcomes. Special education non-compliers were mostly unaffected by the policy. Conclusions: This study considered how Double-Dose algebra affected low-skill students and whether concentrating these students together is potentially detrimental as suggested by some prior studies. Our results showed no such evidence. Our findings, together with previous research on Algebra reforms, guide us to sketch an optimal regime for instructional time and classroom composition for students with varying skills. High-skill students tend to benefit from Algebra instruction in high-skill classes. For median-skill and lower-skill regular education students, homogenous instruction appears to be effective with additional instructional time. Low-skill students, smaller and homogenous classes (e.g., tutoring) may be a more effective approach.

Background: The landscape of education is increasingly recognizing the impactful role of Career and Technical Education (CTE) programs in preparing students for the workforce. In Delaware, the implementation of CTE programs has been substantial, offering students opportunities to earn college credits, attain industry certifications, and gain professional experience while in high school. Previous research indicates a positive correlation between participation in CTE programs and higher rates of high school graduation, better preparation for postsecondary education, and alignment with labor market needs (Levesque et al., 2008; Bonilla, 2020; Bishop & Mane, 2004; Stevens et al., 2019). However, despite these promising findings, the causal impact of specific CTE pathways on students' educational and labor market outcomes remains less understood. This gap in the literature signifies the need for a comprehensive study that investigates the overall effectiveness of CTE programs and explores the nuances of pathway-specific effects and their implications for different student demographics. Purpose and Research Question: This research aims to elucidate the causal effects of concentration in specific CTE pathways on high school and postsecondary outcomes, including course-taking behavior, college attendance, and wages. The study seeks to answer the following questions: (1) What is the average causal effect of a school's introduction of a new CTE pathway on a student's course-taking behavior, college attendance, and wages?; (2) What is the heterogeneity in pathway-specific effects on course-taking behavior, college attendance, and wages?; and (3) Are pathway-specific effects heterogeneous by student type? Participants: The study investigates Delaware high school students who enrolled in at least one Career and Technical Education (CTE) course from 2014-15 to 2020-21 academic years. This encompasses seven complete cohorts, with a particular focus on the analysis of postsecondary outcomes for the initial four cohorts. The final sample consisted of 126,098 students distributed across 74 schools. Program: Career and Technical Education (CTE) programs are structured into several career clusters such as Health Science, Information Technology, Business Management & Administration, and Manufacturing. Each cluster has distinct programs of study, tailored to equip students with the skills and knowledge necessary for their chosen field of study and future career goals. This paper seeks to reveal the heterogeneity within the career cluster and programs of study, analyzing how differences in program specifics influence student outcomes across the diverse spectrum of career paths. Research Design: Addressing selection bias is challenging when evaluating the impact of specific CTE pathways or broader occupational clusters on student behavior and labor market outcomes. Selection bias arises as students may choose pathways aligning with their career aspirations, potentially skewing earnings associations. Similarly, schools might offer pathways based on their constraints and preferences, possibly correlating with students' labor market success. To counteract this, our approach leverages the timing of new CTE pathway introductions by schools as an exogenous factor, assuming frequent and varied offerings across school's prompt significant changes. We hypothesize that the introduction of new pathways alters the expected wages landscape, based on labor market projections, addressing school selection bias. However, student-based selection poses a challenge, particularly in Delaware, where students' ability to choose courses outside their designated zones can affect pathway selection. We mitigate this by focusing on the introduction of pathways in students assigned high schools, based on their middle school enrollments, thus aiming to neutralize biases related to both school and student selection. Data Collection and Analysis: Our primary data is derived from student-level administrative records provided by the Delaware Department of Education, encompassing details on CTE concentrator status, course records, and high school completion. Additional data on college attendance will be obtained from the National Student Clearinghouse, and wage information will be sourced from the Delaware Department of Labor. Our IV approach leverages the timing of new CTE pathways across schools in students' catchment zones defined by their middle school attendance. The structural (biased) equation is: [equation omitted](1) The IV for a pathway is the introduction of pathways in schools in year t in student i middle school-defined catchment zone. This variable takes value 1 in time t if a new pathway j is introduced in school s to which student i is assigned based on their middle school enrollment. We account for school- and year-effects with fixed effects. To address issues related to treatment effect heterogeneity, we stack the data so that each new pathway j is compared to schools for which pathway j has not yet been offered, thus providing "clean controls" for each newly treated school. Stacking the data requires us to interact the school and year effects with each "stack" or new dataset denoted g (Baker et al., 2022). Standard errors are clustered at the school-by-stack level. [equation omitted](2) The probability that a student concentrates in pathway j is determined by the introduction of new pathways in the school to which a student was assigned in middle school. The 2SLS regression is: [equation omitted]. Preliminary Findings: In our initial analysis, we provide evidence that the implementation of CTE programs yields a notable positive impact on student engagement within these specialized career clusters. However, upon closer examination, we observe variations in the degree of influence across different clusters. Specifically, certain career clusters like health sciences and STEM exhibit significant effects, whereas others such as manufacturing show relatively minimal influence. Furthermore, we delve into these heterogeneous outcomes by considering factors like gender and income status. While a positive trend persists across these demographics, the magnitudes of influence vary noticeably. For instance, females tend to gravitate more towards health sciences, whereas males show a greater inclination towards STEM fields and physical sciences. During the forthcoming paper presentation, we expect to reveal the findings from the second stage analysis, offering a more profound insight into the effects of CTE programs on both secondary and post-secondary outcomes. Conclusion: In conclusion, our findings highlight the policy relevance of this study by revealing variations in the effectiveness of program rollouts across different CTE pathways. This emphasizes the importance of tailoring strategies to enhance program uptake and rollout of CTE pathways where it can have the most significant impact.

Background: To combat stereotype threat, several school-based field trials have leveraged self-affirmation theory (Steele & Liu, 1983; Liu & Steele, 1986), wherein affirming one's important beliefs and values can buffer against identity threats. Self-affirmation theory posits that individuals are motivated to maintain a positive overall self-view. When one's self-competence is threatened (e.g., being negatively stereotyped in school), it helps to have opportunities to reflect on sources of self-worth (e.g., being a family member, enjoying sports, being creative) beyond the threatened aspect of the self, (Cohen & Sherman, 2014; Steele, 1988). Cohen et al. (2006) reported that brief writing exercises, which prompted students to reflect on and write about their core beliefs and values, reduced the Black-White grade-point-average (GPA) gap among seventh-grade students by 40%. Sherman et al. (2013) reported similar impacts for Latinx middle-school students. Longitudinal studies, also focused middle-school students, have revealed enduring impacts for students of color on middle and high school GPA (Borman et al., 2016; Borman et al., 2018; Borman et al., 2021), on-time high school graduation rates (Borman et al., 2021), and college enrollment (Goyer et al., 2017). However, Hanselman et al. (2017) caution that self-affirmation effects may be sensitive to school context, which can vary across schools. Purpose/Research Questions: This project provides a direct replication of these prior efforts at an unprecedented scale, spanning 8 districts and 16 schools, and answers three research questions (RQs): 1. Does assignment to the self-affirmation intervention impact seventh-grade students' GPAs and D and F grades? 2. Are these impacts moderated by students' race/ethnicity? 3. To what extent does engaging in self-affirming writing--as induced by the intervention--predict academic performance? Setting: The sample includes 16 schools in 8 districts across 6 states and took place in academic year 2020-21. A total of 2,398 students were randomized to experimental conditions (1,199 to treatment and 1,199 to control) and, after data attrition, the final analytical sample included 1,487 students (745 treatment and 742 control). The overall and differential attrition rates meet the What Works Clearinghouse (2022) standards for "low" attrition and a minimal risk of bias. Participants: The analytical sample of grade 7 students is diverse, with 71% classified as economically disadvantaged, 20% African American, 33% Latinx, 39% White, 3% Asian, and 5% multi-racial/ethnic. Intervention: The intervention comprises three or four brief in-class writing exercises where students reflect on personally important values. Control students engage in similar writing activities on neutral topics. Teachers distributed intervention materials without knowledge of study hypotheses or students' assignment to condition. Students took approximately 15 minutes to complete each exercise and wrote an average of 66 words when responding to the first prompt. The average completion rate across exercises was 78%. Research Design: For this multisite randomized controlled trial we randomly assigned students to experimental conditions within school and race/ethnicity blocks. Data Collection and Analysis: To estimate the intention-to-treat (ITT) impacts (RQ1), we specified a two-level random intercept model (students nested within schools) Yij=?+?(Treatmentij)+??Xij+uj+?ij In this model, ? is the coefficient representing the overall impact of self-affirmation for all student groups, and ?? Xij represents the covariates (i.e., potentially threatened status [0 = White, Asian, American Indian, and multiracial/ethnic; 1 = Black, Latinx]), prior grade 6 GPA, special education status, free or reduced-price lunch status, English language learner status). To evaluate whether students' membership in a potentially stereotype-threatened group (i.e., African American and Latinx students) moderated intervention effects (RQ2), we fit the same multilevel models, but added a treatment-by-potentially threatened status interaction term. To estimate the impact of self-affirming writing, we used an instrumental variables (IV) model. To perform this analysis, the following multilevel models were estimated using two-stage least squares (2SLS) regressions: SelfAffirm=?_0+?_1 Treatment+?_2 Treatment*Threatened+?X+uj+?ij Y_((ij))=?_0+?_1 SelfAffirm+?_2 SelfAffirm*Threatened+ ?X+uj+?ij The coefficient ?_2, is the interaction effect of the TOT by students' threatened status. Covariates were included in both stages of the instrumental variables regression, as before, to improve precision. Findings: Analysis of pre-intervention data demonstrates that the analytic treatment and control group samples are statistically equivalent with respect to demographic, GPA, and D and F grade counts (see Table 1). Impact analyses indicate that being assigned to receive the self-affirmation intervention did not have statistically significant main effects on academic performance (see Table 2) as expected. Because only Black and Latinx (potentially threatened) students tend to benefit, the statistically significant interaction effect for D and F grades shown in Table 3 is expected. As Figure 1 illustrates, potentially threatened students receiving intervention realized reductions in D and F grades, while their control-group peers did not. Though the moderating effects of treatment for potentially threatened students on the GPA outcome (see Table 3) is in the expected direction, this outcome did not reach statistical significance. The multilevel 2SLS regression analyses (see Table 4) reveal that the interaction between students' threatened status and the estimated probability of engaging in self-affirming writing is a statistically significant predictor of D and F grades. Specifically, as the predicted probability of self-affirming writing induced by treatment increases, potentially threatened students experience fewer D and F grades while non-threatened students do not (see Figure 2). Indeed, the interaction effect for engaging in self-affirming writing--the effect of the treatment on the treated--is twice as large as that found for assignment to receive the self-affirmation exercises. Conclusions: The results for the 2021-22 cohort, revealed far more promising outcomes for students vulnerable to stereotype threat than results from the prior 2019-20 and 2020-21 cohorts. Similar to the recent findings from a national study of a growth mindset intervention, which teaches middle-school students that intelligence is malleable and that intellectual ability can be developed (Yeager et al., 2019), our results suggest that subtle yet precise psychological interventions can improve student outcomes at scale. This work contributes to a growing body of evidence suggesting powerful ways to improve academic success and advance equity through social--psychological interventions (Walton et al., 2018; Yeager & Walton, 2011), which change how adolescents think or feel about themselves and their schoolwork and thereby open new avenues through which students can thrive academically.

Inequality in the shadow? Cross-national comparisons of the effects of additional instruction on math and science performance based on PISA 2018 data Background Many educational systems worldwide are experiencing a growing demand for shadow education, which often takes the form of outside-school educational activities such as private tutoring dedicated to improving student achievement (Entrich, 2021; Stevenson & Baker, 1992). Ample evidence from individual social contexts suggests that shadow education is more accessible to families with high-socioeconomic status (SES) compared to low-SES families (Atalmis et al., 2016; Liu & Bray, 2017; Smyth, 2009; Zhang, 2020). As a result, there are growing concerns that shadow education may reproduce educational inequalities (Bray, 2009; Lynch & Moran, 2006; Zhang & Bray, 2020). However, studies using data from multiple countries have yielded mixed results regarding the association between student SES and access to shadow education as well as the association between private tutoring and student achievement (Entrich, 2021; Jansen et al., 2021; Song et al., 2021; Wiseman, 2021). Therefore, more research is needed to explore the role of outside-school learning activities in the SES achievement gaps. Research Question: To contribute to the discussion on the role of shadow education in the SES achievement gaps, this study asks two questions--(1) To what extent is students' SES associated with their use of different types of additional instruction remedial, enrichment, and instruction on study skills? (2) What are the effects of different types of additional instruction remedial, enrichment, and instruction on study skills on student math and science achievement? Setting: The study uses data from six countries that participated in PISA 2018, i.e., Thailand, Kazakhstan, Greece, Korea, Hong Kong, and Ireland. These six countries were purposively selected because they have been intensively studied in the shadow education literature and they exhibit disparities in terms of geographic distribution and GDP per capita. Figure 1 shows the math and science performance as well as GDP per capita of the six countries in relation to the OECD average. Participants: Participants are 15-year-old students from educational institutions in Thailand, Kazakhstan, Greece, Korea, Hong Kong, and Ireland in PISA 2018 (N=52,807). Table 1 summarizes the number of students and students' math and science performance in each country. Intervention PISA 2018 asks whether students participate in additional instruction in different school subjects. This study looks at additional instruction in the math and science subjects. Specifically, five types of additional instruction are of interest in this study: (1) remedial instruction on math, (2) enrichment instruction on math, (3) remedial instruction on science, (3) enrichment instruction on science, and (5) additional instruction on study skills. Research Design PISA uses a two-stage sampling process: first sampling the schools and then sampling students within the participating schools. Therefore, sampling weights are associated with each student because students and schools in each country may not have the same chance of being selected. This sampling approach may increase the standard errors of population estimates. This study therefore uses a Balanced Repeated Replication (BRR) procedure to account for the cluster structure of the data and generate unbiased standard errors. Data Collection and Analysis: The study uses data from the PISA 2018 performance tests in the mathematics and science domains. The tests include an item bank and each student is only presented with a fraction of those items. To account for the variability in scores due to the different sets of items available to each student, 10 plausible values were computed for each student as estimates for their performance in each domain. PISA 2018 also collects questionnaire data from students and their schools, which provides contextual information such as SES indicators at the student and school levels. Multiple regression analyses were performed for each country to examine the relationship between additional instruction and student performance in math and science. Outcome variables are student math and science performance. The independent variables are the five different types of additional instruction listed in the Intervention section. Covariates include student gender, student learning time in math/science classes, family socio-economic status, home educational resources, percentage of fully certified teachers at school, and student-teacher ratio. Results: Table 2 presents the correlations between different types of additional instruction and student- and school-level SES indicators. Participation in different types of additional instruction was differently associated with SES indicators. In general, SES indicators are positively associated with additional instruction, especially enrichment instruction, lending support to previous findings that high-SES families tend to access more additional educational resources than low-SES families. Tables 3 and 4 present the results of the multiple regression analyses for each country. Different types of additional instruction have different effects on student math and science achievement. Remedial instruction and instruction on study skills are negatively associated with both math and science performance. Enrichment instruction is positively associated with student performance in some countries (Thailand, Greece, Korea) but not the others (Kazakhstan, Hong Kong, and Ireland). Significant results were mainly found in student-level SES indicators, while school-level SES indicators yielded few significant results. Conclusions: Despite the small number of countries included in the study, the study contributes to the growing body of cross-national comparisons on the relationship between shadow education and the SES achievement gaps. The findings of this study provide more nuance on how different types of additional instruction can be differently associated with student achievement. This demonstrates the need for further work to differentiate various types of shadow education to better understand how shadow education might reproduce or close the SES achievement gaps. Moreover, the cross-national variation in the effects of additional instruction on student achievement highlights the need to account for characteristics of different educational contexts when interpreting the role of shadow education in educational inequality.

Educational leaders through the United States have prioritized developing integrated tiered systems to assist educators in meeting students' academic, behavioral, and social and emotional well-being learning needs in an integrated fashion (McIntosh & Goodman, 2016). This is an important shift away from traditional reactive models towards proactive, systemic structures for prevention and search-and-serve mandates (IDEA, 2004). A key feature of these integrated tiered systems of support are the use of practical, reliable systematic screening tools to inform valid inference making with respect to meeting student's multiple needs: academic, behavioral, and social and emotional well-being learning needs. In most tiered systems, educators completed academic and behaviors screening tools at three times per year (fall, winter, and spring) to inform instructional practices and school policies. These data are along with other data collected as part of regular school practices to shape instruction at Tier 1, empower teachers with strategies to maximize engagement, and connect students to validated Tier 2 and 3 interventions when Tier 1 practices are insufficient. In addition to prioritizing the implementation of academic screening tools (Briesch et al., 2022), educational leaders also prioritize the need for universal behavior screening to detect students with and at risk for emotional and behavioral disorders (EBD) at the earliest signs of concern. This is a diverse group of students, including students with both major challenges of childhood and youth: externalizing (e.g., aggressive, noncompliant) and internalizing (e.g., anxious, withdrawn; Walker et al., 2005). The Student Risk Screening Scale -- Internalizing and Externalizing (SRSS-IE; Drummond, 1994; Lane & Menzies, 2009) is one of a few free-access screening tools with substantial research on a range of psychometric properties (Pelton et al., 2024). Initial inquiry indicated scores from both externalizing and internalizing scales have been predictive of critical school outcomes in both academic (e.g., oral reading fluency, course failures) and behavioral (e.g., office discipline referrals, in-school suspensions) domains (e.g., Lane, Oakes et al., 2019; Menzies et al., 2012). Yet, many studies conducted have represented samples from a single district or state. In addition, only recently have studies examined (a) measurement invariance as well as (b) more comprehensive studies examining predictivity validity. In this presentation, we report findings of three recent studies of newly adapted SRSS-IE, for use in K-12 settings. In the first study (Lane, Oakes, Buckman et al., 2023), we present new findings of the Student Risk Screening Scale -- Internalizing and Externalizing (SRSS-IE). We examined (a) SRSS-IE factor structure and (b) the degree to which measurement invariance held across sex, special education status, race, and ethnicity, as well as over time (fall, winter, spring) within each school level (elementary, middle, high). This large-scale study funded by the Institute of Education Sciences (IES) included 124 schools from four U.S. geographic regions in their first year SRSS-IE implementation collected over a ten-year period. Using confirmatory factor analysis procedures accounting for the nesting of students within teachers' classrooms, we confirmed a two-factor structure (internalizing and externalizing) and determined three items may be removed from the SRSS-IE and still result in adequate model fit. In brief, all model comparisons between configural, metric, scalar, and strict models met invariance criteria in fall, winter, and spring administrations. Furthermore, longitudinal models also met these invariance criteria. However, replication with schools in later stages of systematic screening is essential before shifting screening practices. In the second study we examined predictive validity of the newly defined Student Risk Screening Scale -- Internalizing and Externalizing (SRSS-IE 9, with 9 items) when used by teachers in school conducting SRSS-IE screening for the first time. This sample included 24,535 elementary students from 74 schools representing four geographic regions. Results revealed fall externalizing and internalizing latent factors as well as subscale scores (SRSS-E5, SRSS-I4, respectively) from the SRSS-IE 9 predicted important year-end outcomes. For example, scores predicted year-end behavioral outcomes (office discipline referrals, suspensions, and nurse visits), academic outcomes (reading and math scores and course failures), as well as referrals to special education for elementary-age students. Given many school systems currently use the SRSS-IE 12 as regular school practice, we examined these original scores as well, with result indicating externalizing (SRSS-E7) and internalizing (SRSS-I5) subscales were comparable in predictive of these outcomes. In the third study, we examined predictive validity the SRSS-IE 9 when used for the first time by middle (four geographic regions) and high (three geographic regions) school teachers from 43 schools. The sample included 11,773 middle school-aged students and 7,244 high school-aged students. Findings revealed indicated fall SRSS-IE externalizing and internalizing latent factors as well as subscale scores predicted year-end behavioral (office discipline referrals and in school suspensions) and academic (course failures) outcomes for middle and high school students as well as referrals to special education for middle school students. Internalizing scores also predicted referrals to special education for high school students. Results also indicated, externalizing and internalizing scores predicted nurse visits at the middle and high school levels with all models save for the subscale models of internalizing in middle school. Analysis of SRSS-IE 12 subscale scores for externalizing (SRSS-E7) and internalizing (SRSS-I5) also yielded comparable outcomes, with few variations. In our presentation we will discuss educational implications, illustrating how these data are used to inform instruction to meet student's multiple needs in an integrated fashion (Buckman et al., in press). Further, we discuss limitations and directions for future inquiry.

Context: Research has demonstrated the importance of teacher quality for student achievement above all other school-based factors (Aaronson et al., 2007; Goldhaber, 2002; Rivkin et al., 2005; Rockoff, 2004). Yet while states and districts have allocated significant resources toward professional learning (PL) aimed at improving instruction (Jacob & McGovern, 2015), challenges in math student achievement persist. One reason may be that the typical approaches to professional learning that are driven by district or campus priorities - rather than focusing on individual teacher needs - may fail to help teachers achieve their potential (Gates Foundation, 2014; Guskey, 2009; Kennedy, 2016; McCarthy & James, 2017; Mushayikwa & Lubben, 2009). Adult learning and motivation theories underscore the importance of allowing teachers autonomy over their own learning, as adults have "a deep psychological need to be perceived by others, and treated by others, as capable of taking responsibility for ourselves" (Knowles, 1984, p. 9). At worst, a prescriptive approach to PL may lead to resistance to the training or learning opportunities in which teachers participate (Merriam & Bierema, 2014; Knowles, 1984. Purpose: The Math Instructor Professional Learning Autonomous Network (MiPLAN) was developed to allow teachers to select, participate in, and reflect on PL opportunities of their own choice, with the goal of improving math achievement for high-need students. The PL was intended to constitute at least 80% of teachers' state-mandated PL time. This presentation will share preliminary findings from a school-level randomization study involving two cohorts spanning 2022-23 and 2023-24. These findings will provide an early look at some of the study's research questions: 1) What are the barriers to and facilitators of MiPLAN implementation? What is the impact of MiPLAN on... 2) ...teachers' attitudes and beliefs? 3) ...teachers' classroom practice? 4) ...student mathematics achievement in Grades 3-8? Program Description: The MiPLAN project is guided by the logic model shown in Exhibit 1. MiPLAN creates the conditions for teacher choice in PL by securing teachers' release from at least 80% of state mandated PL, providing resources through $8,000 voucher stipends to cover the costs of their selected PL plus additional travel stipends if needed, distributing incentives for teacher participation, and leveraging the user-friendly TEEMS platform to help teachers through the MiPLAN process. Content expertise at ESC 18-TxCEE facilitates the curation of vetted, high quality PL options, and TxCEE professional learning coaches provide support to teacher learning cycles. Teachers are encouraged to set their own learning agenda through project activities that support reflection and teacher agency. After an initial orientation, teachers select PL that meets their own learning needs with the help of a MiPLAN coach and complete at least two learning cycles over the year. Research Approach: The design is a multisite, two-cohort cluster randomized controlled trial. Schools were randomly assigned to condition within district. Half of the schools in each district were selected by lottery to participate in MiPLAN and study-related data collection activities during the 2022-23 (cohort 1) or 2023-24 (cohort 2) school year, while the remaining schools continued with their regular PL activities and only participate in study-related data collection activities. Our main impact analyses use intent-to-treat (ITT) analyses that estimate the impact of being randomly assigned to participate in MIPLAN on (a) teachers' attitudes and beliefs (RQ2, measured by survey self-report), (b) teacher classroom practice (RQ3, measured by classroom observations), and (c) student mathematics achievement (RQ4, measured by state standardized assessment scores). Findings: Recruited sample and baseline equivalence. The study sample includes 54 elementary and middle schools in 16 Texas school districts. We used rerandomization to ensure random assignment generated treatment and control groups that are similar on observable baseline characteristics (Appendix Exhibit 2). Implementation findings. Preliminary findings from Cohort 1 suggest implementation fidelity was adequate for at least half of the treatment schools (Appendix Exhibit 3). Most teachers felt they could find PL to meet their needs, and teachers largely reported that they could complete their PL and they used what they learned in their PL to improve their math instruction. In surveys and interviews, teachers reported that the support of the TxCEE coach and choosing PL aligned to teacher needs were facilitators for implementing MiPLAN. Teachers noted that a lack of coordination was a barrier to implementation, and some teachers reported having to complete MiPLAN PL in addition to school- or district-required PL as a barrier to implementation. In surveys, treatment teachers reported having more choice and being more satisfied with their level of choice in their PL. While comparison teachers attended more PL overall, treatment teachers attended slightly more math-focused PL and were more likely to "strongly agree" with positive statements about the value of the PL they attended. Impact findings. In addition to perceptions of PL choice and relevance, treatment teachers in cohort 1 had more positive attitudes than comparison teachers on constructs related to self-concept as a self-directed learner, readiness to learn, motivation to learn, and self-efficacy in math instruction. Analyses of the impact on student achievement, teachers' attitudes and beliefs, and classroom practices are underway. We will have student achievement results based on the first cohort and classroom practice and teachers' attitudes and beliefs results for both cohorts to present at the conference. Conclusions: 1. Despite the intent of the program and agreements in place before districts and schools could join, teacher choice was not as free and open as intended. 2. Nonetheless, teachers and administrators alike reported strong satisfaction with the PL that MiPLAN enabled teachers to participate in. This appears to have borne out in higher satisfaction, self-efficacy, and professional attachment when comparing MiPLAN participating teachers to teachers in the delayed implementation group. 3. Analyses will be available by the time of the conference that examine whether the teacher attitudes and beliefs correspond to positive impacts on their math instruction -- and on student math learning. 4. If the initial positive indications on outcomes are confirmed in ongoing analyses, this has implications for equitable decision making and resource allocation in school systems, even as it raises questions about sustainability.

Background/Context: Some observational studies involve multiple layers of treatment selection, specifically in the context of the extended time accommodation (ETA) for English language learners (ELLs). In ETA settings, the first selection occurs due to the eligibility rule, where students whose ELL English proficiency is below a certain threshold are eligible for ETA. This forms a regression discontinuity (RD) design. The second selection occurs by school administers who determine students' ETA receipt status under school-specific rules and constraints. Then, the third selection is made by students' decision on whether to use the ETA offered (Suk et al., 2022). Each layer of selection is influenced by different selection biases and requires different data analyses. If carefully designed, multiple analyses can allow for constructing "evidence factors." Evidence factors provide two or more independent tests of the same null hypothesis about the treatment effect, each potentially subject to different biases (Rosenbaum, 2010). For example, potential bias in the RD analysis does not affect the validity of other analyses based on other layers of selection. By combining these multiple evidence factors, researchers can strengthen causal conclusions even with a single dataset while separating independent sources of biases among them (Rosenbaum, 2023; Zhao et al., 2022). Purpose/Objective: This study investigates how to combine three evidence factors associated with ETA assignments in fuzzy regression RD designs, where ELL English proficiency serves as the running variable and noncompliance occurs through two mechanisms. Using three different treatment statues in the ETA context, we construct three evidence factors based on one treated and three different control groups. Each evidence factor aims to test the same null hypothesis--no effect of each student making use of ETA--using a stratified nonparametric test. For constructing one evidence factor, we utilize the local randomization framework in RD contexts (Cattaneo et al., 2015) that can be seamlessly applied to the randomization-based inference. We then combine p-values from these three (nearly) independent tests to reinforce causal conclusions on the ETA effect. We also evaluate the effectiveness of our approach via a simulation study and a real dataset from the National Assessment of Educational Progress (NAEP). The NAEP Assessment: Setting: The NAEP assessment used in this study is the largest continuing and nationally representative assessment in the United States (Oranje & Kolstad, 2019). Population: Our study focuses on 8th grade students with the advanced ELL proficiency who do not have disabilities. Program: The ETA program is designed to provide ELL or students with disabilities, up to three times the regular testing time during the NAEP assessment. Research Design: We employ our proposed approach on evidence factors to evaluate the effect of using ETA on student achievement. We use ELL English proficiency categories as the running variable in the RD design. Data Analysis: We analyze data from the 2017 NAEP assessment for grade 8, along with process data that contains approximately 28,000 students. Using process data, we identify students who made use of ETA during the test. Findings/Results: We present how to construct three evidence factors based on three different selection mechanisms in ETA settings. These multiple selection mechanisms form four groups: ETA users (Treatment), non-users (Control 1), non-recipients (Control 2), and ineligibles (Control 3); see Table 1. The first evidence factor comes from a comparison between ETA users (Treatment) versus non-users (Control 1). The second factor is about a comparison between ETA recipients (i.e., Treatment + Control 1) versus nonrecipients (Control 2). The third factor is based on the RD analysis that compares eligible students (i.e., Treatment + Control 1 + Control 2) versus ineligible students (Control 3), which are determined by their running variables. Importantly, the ETA effect can only occur when students make use of it; otherwise, there is no assumed ETA effect. Consequently, each of the three proposed evidence factors supports the null of no ETA effect under each of their sharp null. These three evidence factors are intentionally designed in this way to separate bias from one factor to the others, despite the potential for underpowered analyses in the second and the third factors, which include one or two control groups in the updated "treatment" group. Our posited assignment models for three treatment assignments are as follows: [equations omitted] where K[subscript k] (k =1,2,3) represents an arbitrary function of the measured covariates w[subscript ij] of individual i in stratum j, where the strata are formed using the observed covariates. [gamma][subscript k] represents the impact of the unmeasured covariate u[subscript ij,k]. "F" represents a collection of the potential outcomes and measured and unmeasured covariates, and "W" represents the window around the cutoff where local randomization is valid. Given the assignment models, we conduct a simulation study to evaluate the validity and the performance of our proposed approach. In simulations, we specifically use near-optimal generalized full matchings, so that we can match the treatment group and three different control groups simultaneously on the observed covariates. We then perform Hodges-Lehmann aligned rank tests with strata for each evidence factor and combine p-values from these three nearly independent tests via Fisher's method. We further perform the sensitivity analysis by varying the parameter value of [gamma][subscript k]'s. Lastly, we provide the demonstration of our proposed approach in the 2017 NAEP assessment and process data. Conclusions: This study will strengthen the causal evidence for evaluating the effect of making use of ETA by synthesizing three evidence factors that have been carefully tailored to the ETA context. The proposed evidence factors in fuzzy RD designs with multiple control groups from this study have the potential to be applied not only in other educational contexts but also beyond the field of education.

Background: The importance of high school completion for jobs and postsecondary opportunities is well- documented. Combined with federal laws where high school graduation rate is a core performance indicator, school systems and states face pressure to actively monitor and assess high school completion. This proposal employs machine learning techniques to identify students at-risk of exiting high school in either 9th or 10th grade. We argue that compared to students who may exit in later years of schooling, students who withdraw in the first two years are a vulnerable population and could benefit from earlier intervention, support, and services. This proposal advances the current state of knowledge in the field by (1) predicting early withdrawal of students who exit in 9th or 10th grade without eventual completion, and (2) assessing algorithmic bias for sensitive groups. Research Questions: This project investigates the following research questions to predict early exit from high school: 1: How does the prediction accuracy of supervised learning algorithms to predict high school withdrawal compare to that of traditional models (i.e., logistic regression)? 2: What are the most salient predictors of students who exit high school in 9th or 10th grade? 3: To what extent does each mode provide fair predictions across sensitive attributes such as gender, race/ethnicity, disability status, financial hardship, and English proficiency? Based on prior literature, we hypothesize that machine models will provide more predictive accuracy than an OLS regression. While there are clear hypotheses related to the importance of attendance, behavior, and coursework trajectories on high school completion, little is known about which of these aspects of middle school engagement are most predictive of exiting high school early. Setting: This project relies on existing administrative data housed at the North Carolina Education Research Data Center (NCERDC). We examine student educational records from 6th to 8th grade to predict the probability that a student will exit high school in 9th or 10th grade. The predictors include End-of- Grade test scores, attendance rates, chronic absenteeism, disciplinary infractions, school mobility, and urbanicity. Population: We examine first-time sixth-grade public school students during the 2011-2012 school year. We limit the sample to students in districts that follow a legal dropout age of 16 and those with complete graduation or exit records. We retain students who have some or all attendance and state test score data in middle grades and impute missing data with a student's unique middle school median. 94% of students in this cohort persist in the school system beyond 10th grade, compared to 6% who exit in 9th or 10th grade. Research Design: Like earlier empirical work, we compare prediction accuracy of a traditional logistic regression model to more advanced machine learning algorithms: lasso regression, ridge regression, random forests, and extreme gradient boosting (XGboost) (Mduma et al., 2019; Hung, 2017; Sansone, 2019; Coleman, 2021; Sorenson, 2018). We create binary classification models with an outcome of "1" for students who exited in 9th or 10th grade, and "0" otherwise. To evaluate model performance, we examine the area under curve (AUC), accuracy rate, sensitivity (true positive rate), specificity (true negative rate), and F-1 score. The results focus on sensitivity, or the accuracy rate for students who exit early. We follow standard machine learning practices and cross-validate models by using sixth-grade students in Fall 2010 as a training sample and sixth-grade students in Fall 2011 as a testing sample. We apply standard metrics of maximizing accuracy with parameters and hyperparameters that are standard for each algorithm. We rely on receiver-operating characteristic (ROC) curve analyses and apply a decision threshold to achieve a true positive rate of 80%. To prevent the models from being biased towards the majority class, we address class imbalance by oversampling the minority class. Specifically, we apply Synthetic Minority Oversampling Technique (SMOTE) to generate synthetic minority class observations based on the k-nearest neighbor for the minority class (Chawla et al. 2002; Fernandes et al. 2018; Anis & Ali, 2017). To examine if algorithms discriminate against certain student groups, we "blind" the algorithms to gender and race/ethnicity. We conduct an Absolute Between-ROC Area (ABROCA) slicing analysis to evaluate algorithmic fairness between the majority and minority group for student group. We focus on the following protected attributes: gender, race/ethnicity, English learner status, disability status, and economic disadvantage. Findings: We find that when models are trained with highly imbalanced data, both ensemble methods -- XGboost and random forest -- provide the highest sensitivity rate. However, we see substantial improvements in the sensitivity rates of all models that were trained with synthetic data (SMOTE). The improved accuracy in identifying the minor class comes at a penalty of a lower accuracy rate and lower specificity. Despite other models having higher sensitivity, we argue that the strongest model is with XGboost trained with SMOTE observations because it provides a higher specificity. In examining model features we find that age in 8th grade and being chronically absent in a middle school attendance are most predictive of early exit, followed by 8th grade and 7th grade absences. We detect no signs of algorithmic bias for students stratified by gender, disability status, and economic disadvantage. Conversely, we detected bias in regression-based models for students stratified by English learner status and race/ethnicity. Conclusion: Please see the Abstract PDF for the full updated abstract.

Innovation is the central driver of economic growth. Many growth models underscore the importance of actively directing more individuals to innovation (Romer, 1990; Jones, 2022). This goal can be achieved by providing early exposure to innovation to children, especially those from disadvantaged backgrounds (Bell et al., 2019; Akcigit et al., 2020; Bloom et al., 2019). But despite the consensus in the literature, there exists a noticeable absence of systematic interventions that teach innovation in school settings. The ability for innovation, i.e., generating valuable ideas for society, applies across diverse disciplines, from the sciences to the humanities, and can equip children to drive innovation in their future careers. However, a formidable challenge lies in finding education programs that can systematically nurture and harness children's ability for innovation. Especially in developing countries settings where schools often focus on memorization and tests, it is challenging to introduce, let alone lead, successful programs for innovation or other higher-order skills (Glewwe and Muralidharan, 2016; Glewwe et al., 2020). This study investigates the effects of an innovation education program for high school students, a collaboration between a state government and a non-profit organization in a resource-constrained rural Indian setting. Using a randomized evaluation, I evaluate the program's effects mainly on the ability for innovation but also on academic achievements, cognitive ability, and personality traits. This program addresses a significant disparity in innovation opportunities by targeting India's marginalized communities, particularly socially and economically isolated students. The majority of individuals who are able to pursue innovation predominantly emerge from the world's most affluent communities (Bell et al., 2019; Aghion et al., 2017). In contrast, this initiative uniquely prioritizes marginalized groups recognized in the Indian Constitution to address historical social and economic disadvantages. Scheduled Castes, often called Dalits, comprise marginalized communities that have endured discrimination and exploitation over centuries, while Scheduled Tribes, known as Adivasis, are indigenous or tribal groups with unique cultural identities. These communities still face discrimination, significantly affecting their access to education, employment opportunities, income levels, and public resources (Piketty, 2020; Munshi, 2019). Social welfare and tribal welfare schools, operating with limited resources, primarily serve students in remote and underserved areas. Policymakers aim to empower disadvantaged children attending these schools through this program, equipping them with the innovation skills that can improve their personal and societal prospects. The curriculum is designed for students to create innovations that address the problems they identify in their surroundings. A team of engineers-turned-educators with experience in teaching, especially in low-income settings, created a new curriculum for secondary school students developed after Stanford's design thinking methodology (Simon, 1980; Kelley and Kelley, 2013). This adapted program has been implemented in collaboration with five Indian state governments and the central governments of two other developing nations. The program intentionally incorporates self-directed learning pedagogy and provides adequate resources to students to ensure sustainability even in the most resource-constrained schools. Students work independently in teams, progressing through five stages of developing their frugal innovations: identifying a problem, conducting research, ideating solutions, creating prototypes, and presenting to peers and intended users for feedback. The ability for innovation is challenging to measure due to its context-dependent nature. I use two different approaches for measuring it in children. The first measures the ability for finding a valuable solution by using strategic exploration in a lab-in-the-field game, while the second assesses the quality of frugal innovations created by the students. The first measure of innovation, a lab-in-the-field game adapted from a widely cited paper in experimental economics by Ederer and Manso (2013), incentivizes exploration and measures the ability to discover optimal solutions. It is built on the concept of innovation as a result of knowledge generation through experimentation, rewarding the exploration of new untested approaches that lead to innovation rather than the exploitation of well-known ones (Arrow, 1969; Weitzman, 1979; March, 1991). The game is designed in a way that higher profits can only be achieved by being exploratory aimed at discovering new solutions, as opposed to relying solely on exploiting existing ones. The second measure of innovation, a new scale called Innovation-S, evaluates the quality of frugal innovations and was developed with input from experienced inventors, with its predictive validity established through the success of their innovations. I developed this scale with input from 58 scientists, high-impact patent recipients, and founders of both for-profit and social enterprises. I report four key results. First, the intervention improved students' ability to innovate in finding better solutions. In the innovation lab-in-the-field game, treated students demonstrated increased effort by 0.13 standard deviation. They are strategically more exploratory of discovering new solutions by 0.09 standard deviations (as opposed to being exploitative of an existing solution) and perform better in fine-tuning their exploration by 0.18 standard deviations. Second, treated students created real-world ideas that are more innovative. Over the academic year, both treatment and comparison teams submitted 2413 ideas, which were evaluated by a group of experienced innovators. I find that the treated teams scored 0.20 standard deviations higher on this new scale relative to the comparison teams. Third, I find that math scores decline, potentially indicating a trade-off of academic outcomes with the ability for innovation. Treated students performed 0.13 standard deviations worse than their peers in the comparison group, even when comparing their scores on ten identical questions that were common between the baseline and the endline assessments. This decline in scores in identical questions suggests that the students might have been less interested in answering math questions at the endline assessment. This observation is further supported by the students' self-reported decrease in enthusiasm for learning math by 0.30 standard deviations compared to the comparison group. Finally, the program had negative effective on student enthusiasm for almost all the other academic subjects, with no impact on their science performance, IQ scores, or personality traits. Students reported decreased enthusiasm, ranging from 0.11 to 0.21 standard deviations, across all the other academic subjects except science.

Background/Context: Research consistently emphasizes the importance of social-emotional learning (SEL) in education settings (Cipriano et al., 2023; Wigelsworth et al., 2023). In addition, it has become increasingly evident that educators' social-emotional competence and well-being plays a crucial role in fostering SEL among students (Braun et al., 2020; Gimbert et al., 2023). However, existing social-emotional assessment tools primarily focus on evaluating students' social-emotional skills, neglecting the essential dimension of educators' social-emotional competence and well-being. The few existing measures (Friedman & Kass, 2002; Kirk et al., 2008; Yoder, 2014) either are lengthy and time-consuming or provide a narrow perspective of educators' well-being (for a summary, see Hemi & Kasperski, 2023). Furthermore, there has been little research on the psychometric properties of existing measures of educators' social-emotional competence and well-being (Hemi & Kasperski, 2023). A lack of evidence on the measures, such as the reliability, validity, and applicability across diverse backgrounds of educators, can impact researchers' overall findings on educators' social-emotional competence. Building on an educator well-being framework, this study seeks to develop a multidimensional self-report instrument that measures educators' socialemotional competencies. Purpose: This study investigates the psychometric properties of a newly developed measure administered to a diverse group of staff members across 93 schools in Illinois. The study contributes to the research on SEL by developing a social-emotional competence survey for educators and examining the psychometric properties of the measure, thus providing the research community with a reliable and valid tool to enhance understanding of social-emotional competence and well-being across a diverse education workforce. Setting: The research--which is part of a larger impact study funded by the Institute of Education Sciences (IES) and conducted by the American Institutes for Research® (AIR®) in partnership with the Illinois State Board of Education (ISBE)--evaluates Illinois's Learning Renewal--Social-Emotional Learning (LR-SEL) programs. LR-SEL is a statewide pandemic recovery initiative to counter the critical effects of the pandemic by supporting SEL and mental well-being across the state's educational system. Information collected from this study will help ISBE, other state education agencies, and the broader education research, policy, and practice communities better understand the effects, implementation, and costs of the LR-SEL programs, with the goal of improving the ongoing social-emotional supports for districts and schools. Research Design and Participants: As part of the larger study, the research team developed an educator social-emotional competence and well-being measure comprising six scales: school working conditions (four items), learning renewal-SEL practices (four items), emotional intelligence (six items), capacity to support students (four items), general well-being (eight items), and professional well-being (four items). The conceptual framework for the survey was adapted from recent work for the Council of Chief State School Officers on educators' well-being (Yoder et al., 2023). Most of the items were adopted from prior SEL studies in Illinois (e.g., Li et al., 2023). The research team collected survey responses from 2,927 staff members in 93 public schools in Illinois between March and June 2023. The staff survey sample has a demographic makeup that closely mirrors the gender and racial diversity of educators across the state. In addition, the study team conducted follow-up cognitive interviews with nine educators to further understand their perspectives of the survey items. Data Analysis: The research team conducted Rasch analyses and multilevel multinomial logistic regression models to examine the internal reliability and construct validity of the data. The purpose of the Rasch analyses was to evaluate the reliability of person and item measurements and identify any items that exhibit misfit. Furthermore, the study aimed to identify potential biases in the measure that may affect the accuracy of the results by investigating the presence of differential item functioning (DIF) and clustering effects across gender and race. We conducted the Rasch analyses using Winsteps (version 3.91). We treated missing data using the default approach by retaining all nonmissing values for the analysis. We calculated fit statistics--including item and person reliability as well as infit and outfit mean square (MNSQ)--to evaluate item and person fit and assessed unidimensionality using principal component analysis (PCA). In addition to examining the individual level data, the research team acknowledged the clustering nature of the data with staff nested in schools. We conducted an intraclass correlation coefficient (ICC) analysis to examine the clustering effect and used multilevel multinomial logistic regression models to control for the participants' overall ability on the latent scale while predicting item response probabilities based on gender and race. Findings: The overall scale fit of the measurement instrument was good, with most items exhibiting good to acceptable model fit to the data. Analyses of differential item functioning (DIF) indicated that while the DIF for gender was negligible, a few items showed notable DIF across racial groups, including three items in the School Working Conditions domain, one item in the Emotional Intelligence domain, and two items in the Capacity to Support Students domain. The study also identified a clustering effect, indicating a tendency for participant responses to cluster with schools, which may coincide with participants' demographic characteristics. After accounting for the clustering effect with a multilevel model, we identified four items as potentially biased. Cognitive interviews about these four items revealed that staff of different racial groups may have different interpretations of some item wordings. Conclusion: The study's findings highlight the importance of conducting rigorous psychometric analyses on a novel measure for educators to ensure accurate representation of different groups and avoid interpretation biases. The findings can inform further refinement of the measure and ensure that it is culturally responsive and valid for use across different groups. Moreover, the study's findings can inform policies that promote equitable and inclusive research and practice in education settings. By identifying potential biases in measures, researchers and policy makers can work together to ensure that educators' social-emotional competence and well-being are accurately captured and supported in education settings.

Background/Context: To stimulate national and regional economic growth, policymakers and businesses have expressed great interest in growing the supply of workers with skills in science, technology, engineering, and math (STEM). A key link in this pipeline is high school, where learning experience plays a critical role in shaping student's interest in STEM careers (e.g., Bottia et al., 2015). This is also the period when females begin to lose interest in STEM (Sadler, et al., 2012). To improve STEM education, one of the national initiatives during the early 2010's focused on integrated applied STEM curricula (Gottfried & Bozick, 2016). By connecting the traditional STEM disciplines to the real-world/applied problems, the goal was to increase student participation in more meaningful and engaging STEM learning and generate greater interest in STEM or related careers. Project Lead the Way (PLTW) is one such program. PLTW began over two decades ago in a handful of high schools as career and technical education (CTE). Since then, the program has grown rapidly and is currently adopted by over 12,000 high schools nationally. In Missouri, the site of this study, the program expanded from 13 schools in 2005 to 174 schools by 2020 -- roughly 35% of all public high schools. Purpose/Objective/Research Question: Using the data on the population of Missouri public high school students, we investigate the impact of PLTW program expansion (ITT) and the impact of program participation on program participants (TOT). Our key outcomes are STEM major declaration in college, five-year STEM degree attainment, and completing a STEM degree or still enrolled in a STEM major in the fifth-year of college. Setting: Missouri has 537 traditional and charter public high schools, and 26% of them are in urban and suburban areas, and 18% and 57% are in town and rural areas, respectively. Population/Participants/Subjects: This study uses four cohorts of first-time 9th-grade students in Missouri who began public high school between 2010 and 2014. Each cohort has approximately 68,000 students attending 500 high schools. Intervention/Program/Practice: PLTW is designed to provide active, project-based learning in the largest three applied STEM fields--Engineering, Computer Science, and Biomedical Science ("Pathways"). Each pathway begins with foundational courses. They provide an overview and introduce major ideas of the field with a goal of developing an enthusiasm for further study. Advanced courses are designed to expand students' understanding of the field through deeper and specialized content. The Engineering and Biomedical Science pathways end with a capstone course which requires students to take their own idea from design through development of a product or plans to produce one. Research Design: This study exploits within-school variation in course availability across cohorts to estimate the ITT impact of PLTW expansion. We then use 2SLS to identify ToT. Our identification strategies are similar to Altonji (1995), Levine and Zimmerman (1995), Rose and Betts (2004), and Darolia et al. (2020). Data Collection and Analysis: Student data are linked to National Student Clearinghouse for post-secondary outcomes. School course-offering data contains all courses and course sections offered each semester by high school as well as class size for each course section. Following Darolia et al. (2020), we construct cohort-specific PLTW course availability, defined as the total number of PLTW course sections offered per 100 students during high school. Of all high schools, 62 schools first introduced PLTW in 2010 or earlier, 28 high schools introduced the program between 2011 and 2015, and 102 schools introduced it in or after 2016. These schools are considered "treatment" schools. Our data shows that PLTW course availability expanded over time in the treatment schools. The remaining 308 schools never offered PLTW, constituting "control schools". The following model is used to estimate the impact of program expansion (ITT impact) for student i, cohort c, and school j: Yicj = B0 + B1(cohort_c) + B2(Z_c) + B3(X_i) + B4(SFE_j) + e_icj, where cohort_c is cohort fixed effects, Zcj is course availability, Xicj is student covariates (8th-grade math and science scores, gender, race, and FRL), and SFEj is school fixed effects. The coefficient B1 captures cohort-to-cohort outcome fluctuations unrelated to Z, assumed to be common to all schools, and B2 is the effect associated with changes in PLTW availability (i.e., ITT). We use the 2-stage least squares to identify the impact on the treated. The first stage model estimates program participation using Z as an instrument. The predicted probability of program participation is entered as a predictor in the second stage model. Preliminary Results: Our ITT results show that, comparing across cohorts in the same school, PLTW participation rose when school expanded course availability. Also, within the same high school, we see improvements in STEM major declaration associated with increases in PLTW course availability. Five-year STEM degree completion is unaffected by PLTW expansion. However, when we include students who are still enrolled in STEM in their fifth-year of college as part of this outcome (i.e., this outcome equals one if students completed or are still enrolled as a STEM major), we find positive impacts of program expansion. The TOT results show that participating in PLTW increases the likelihood of STEM major declaration by nearly 10 percentage points (SE=0.02), as compared to the OLS estimate of 0.139 percentage points, SE=0.007. The impact on completing or still enrolled in STEM in the fifth-year of college is 5 percentage points (SE=0.016), vs. the OLS estimate of 0.095, SE=0.007. We find no impact on five-year STEM completion (the OLS estimate of 0.055, SE=0.005). Our next step is to investigate whether ITT and TOT impacts depend on students' academic readiness upon high school entry. We hypothesize that impacts are greater for students with higher academic readiness. We will also conduct subgroup analysis by gender. We then conduct sensitivity analysis for violations of identification assumptions (common trends assumption, exclusion restrictions, and exogeneity of instruments).

Background/Context: The transition from high school to college marks a critical juncture for students planning to pursue higher education, particularly those from historically marginalized backgrounds. Unfortunately, many of these students are left to navigate complex financial aid requirements, registration processes, and academic placement with little or no support during the summer before college (Arnold et al., 2009). As a result, they often experience "summer melt," a phenomenon wherein up to 40% of these students fail to enroll in college in the Fall immediately following high school graduation, potentially exacerbating socioeconomic disparities in postsecondary access and attainment (Castleman & Page, 2014). In recent years, behavioral "nudging" through text messaging has emerged as a potential solution to this challenge, with Castleman and Page (2015) demonstrating its effectiveness in increasing college enrollment. In their randomized controlled study (RCT), Castleman and Page used a personalized text messaging campaign to remind students of key college-related tasks while offering students the option to seek follow-up support from a counselor via text message. This intervention led to a seven percentage point increase in college enrollment among students with limited access to college-going support (Castleman & Page, 2015). However, large-scale implementations of this kind of intervention have produced mixed results, with some studies reporting minimal to null impacts on college-related outcomes (Bird et al., 2021; Oreopoulos, 2020). Purpose: The present study aims to build on the nudge literature by replicating and scaling up Castleman and Page's (2015) study to students attending lower-resourced high schools in Alabama, Minnesota, Arizona, and Kentucky. This intervention is unique in leveraging partnerships with K-12 state education agencies, state higher-education agencies, and community college partners across four states. Moreover, we extend the nudge literature by examining long-term postsecondary impacts. Research Questions: (1) What is the impact of the Text4College intervention on students' community college enrollment and persistence?; (2) What is the impact of the Text4College intervention for different student subgroups, such as gender, race, economic status, first-generation status, and locale? Data: Our primary data source is student-level administrative records from Alabama, Minnesota, Arizona, and Kentucky, offering comprehensive insights into student demographics, academic performance, and postsecondary outcomes. Additionally, we leveraged platform data provided by Signal Vine, an AI-driven messaging platform tailored to support students in their postsecondary educational journey, to gain a detailed understanding of students' interactions with text messages. Subjects/Setting: This study includes two cohorts of students. Cohort 1 (n = 937) consists of graduating high school seniors from the class of 2020 in Alabama and Minnesota. In Alabama, we surveyed 1857 students, of which 803 met our eligibility criteria. In Minnesota, we surveyed 728 students, of which 134 met our eligibility criteria. As a result of complications caused by the COVID-19 pandemic, recruitment numbers from the spring of 2020 were lower than desired. The study team included a second cohort of students recruited during the 2020-21 school year to increase the sample size. Cohort 2 (n = 1,963) included graduated high school seniors from the class of 2021 in Alabama, Minnesota, Kentucky, and Arizona. In Alabama, we surveyed 2,083 students, of whom 844 met our eligibility criteria. In Minnesota, we surveyed 1,121 students, of whom 207 met our eligibility criteria. In Kentucky, we surveyed 2,517 students, of whom 503 met our eligibility criteria. Finally, we surveyed 992 students in Arizona, of which 409 met our eligibility criteria. Intervention: Students in the treatment group received between 12-17 tailored text messages over the summer. These messages, personalized for each student, served as reminders for critical college enrollment actions such as FAFSA submission. The messages were developed in collaboration with higher education partners to ensure relevance. This intervention was facilitated by Signal Vine, enabling two-way communication. Students could respond to messages seeking additional information or support, which was promptly provided by trained college staff. Research Design: Our study employed a blocked randomized control trial (RCT) design. At participating high schools, we administered a brief survey to seniors in spring 2020 and spring 2021 to determine student eligibility for the study. We randomized students within college blocks after establishing eligibility criteria (including consent, valid cell phone number, and intent to attend a partnering college). We initially intended to perform block randomization within high schools. However, we later randomized students within college blocks due to complications arising from recruiting students during the COVID-19 pandemic. Rerandomization was implemented to ensure covariate balance. The primary goal of our analysis was to estimate the intent-to-treat (ITT) effect of receiving a text messaging intervention compared to business as usual using linear probability models of the following form: Yi=[beta]0+[beta]1Ti+X'i[beta]StudCov+SchoolFE+ei. Where Yi is the outcome for student i (coded 1 = enrollment; 0 = did not enroll); Ti is an indicator for treatment status (coded 1 = assigned to text messaging intervention; 0 = not assigned to text messaging intervention); X'i is a vector of student baseline covariates that include gender, race/ethnicity, economic disadvantage status, first-generation college student status, and the extent of support received (to what); SchoolFE is a vector of dummy indicators for the college that the students enrolled in. ß1, our coefficient of interest, is the estimate of the percentage-point difference in college enrollment between those who were assigned to the intervention and those who were not. Findings: We observed a slightly higher rate of immediate enrollment in community college among students in the treatment group, about 1.7 percentage points higher than students in the control group. However, this effect was not statistically significant (Table 1). Notably, treatment students in Alabama enrolled in community colleges at higher rates than control students, with a difference of around 4.2 percentage points. However, this effect was only marginally significant. We found no significant effects in Arizona, Kentucky, or Minnesota. While not achieving statistical significance, our sub-group analysis showed encouraging positive trends among female, Black, and first-generation students (Table 2). This suggests that the Text4College intervention may benefit these historically underrepresented groups. We are currently analyzing whether the Text4College intervention impacted students' community college persistence and plan to present those results at the SREE conference.

Background/Context: Students designated as English Learners (ELs) account for 10.4 percent of all students nationwide but have historically been provided unequal access to educational resources (Gándara et al., 2003; National Center for Education Statistics, 2023). Further, education policies are often blind to ELs' strengths and challenges, typically framing ELs' needs as only in the domain of language learning (Gándara & Rumberger, 2009; Ruiz, 1984). Though a growing body of literature examines the effect of policies targeted to ELs (i.e. reclassification, dual language programs), there is a lack of evidence on whether broad education policies are beneficial for ELs. My paper responds to this gap by estimating the effect of grade retention on ELs' education and labor market outcomes in Texas. Grade retention has long been a practice implemented across American schools. Advocates posit that an extra year will provide students with time to catch up academically and acquire foundational skills needed for subsequent grades (Goos et al., 2021). This additional year may act as a gateway for ELs to help them develop English language skills. Conversely, grade retention is a gatekeeper policy because it increases students' stress and likelihood of dropping out, which is particularly concerning for ELs who are already at a higher risk (Anderson et al., 2005; Goos et al., 2021). The causal literature on test-based retention finds positive effects for students in early grades when retention is combined with support for students. However, achievement gains typically attenuate over time and often do not translate into higher high school completion rates (Greene & Winters, 2007; Hwang & Koedel, 2023; Jacob & Lefgren, 2004, 2009; Mumma & Winters, 2023; Schwerdt et al., 2017; Winters & Greene, 2012). Conversely, 8th-grade retention typically leads to null or worse outcomes (Eren et al., 2017; Larsen & Valant, 2023). Only one consequential study has examined test-based grade retention's effects on ELs finding positive effects on students' short-term academic outcomes in Florida (Figlio & Özek, 2020). The current paper builds on this study to provide evidence of grade retention on ELs' in a different context, in multiple grades (3rd, 5th, and 8th), and on postsecondary access and earnings. Research questions: (1) What is the causal effect of grade retention on ELs' short- and long-term educational and labor market outcomes, including English proficiency, high school completion, college graduation, and earnings? (2) Does the effect of retention differ across student subgroups or school characteristics? Setting and Policy Intervention: Texas enacted grade advancement requirements for 3rd-, 5th-, and 8th-grade students tied to performance on the state standardized tests in the early 2000s. Third-grade students had to pass the reading section of the state standardized test (TAKS) while students in grades 5 and 8 had to pass both the reading and mathematics sections to be promoted to the next academic year (TEC §28.0211, 1999). Students could take each assessment up to three times. Students who failed the first administration had to receive accelerated instruction in the subject they failed before retaking the test. Students who failed the third administration were retained unless the Grade Placement Committee (GPC) unanimously decided to promote them to the next grade level. Data: I use Texas data which include information on student demographics, graduation, English proficiency, standardized test scores, postsecondary enrollment, and earnings in Texas. The data also include school and district-level characteristics and expenditures in initiatives for retained students. Approximately 700,000 ELs were enrolled in 3rd, 5th, and 8th grades between 2005 and 2011 in Texas public schools. I focus my analysis on these years because it allows for longer-term investigation of student outcomes of interest (e.g., college graduation, and wages). ELs in the analytical sample come from economically disadvantaged households, identify themselves as Latinos, and have low English Language skills. Research Design and Analysis: I use an RD analysis that exploits Texas's use of test thresholds for grade retention decisions. This approach identifies causal estimates in the local area around the threshold. Because students could retake TAKS and retention decisions were based on students' final scores, I estimate the effect of grade retention using students' scores in the third administration. To measure the impact of retention (after receiving accelerated instruction) I estimate an RD model for each outcome. The model includes a function linking students' scores to outcomes, an indicator of whether the student scored below the given threshold for retention, and an interaction term to allow the relationship between scores and outcomes to vary on each side of the cutoff (see the pdf attachment for the equation). The coefficient on the indicator for scoring below the threshold is the local Intent-to-Treat estimate. My outcome variables include English language proficiency, years to attain reclassification as English proficient, high school graduation, postsecondary entry and graduation, and earnings. I also estimate treatment-on-treated (TOT) estimates using a fuzzy RD to account for imperfect compliance. The first stage regression uses the ITT equation with the dependent variable as whether the student was retained. The second stage uses the predicted probability of retention to examine the effect on the outcomes of interests. I use a frontier RD to account for the multiple variables determining retention for 5th- and 8th-graders. For example, I limit the sample to students who failed the reading test and estimate the effect of marginally failing the math test and being retained. Analytical Strategy for RQ2: I conduct a subgroup analysis to estimate the effect of grade retention based on student and school characteristics. I estimate the models described above for each subgroup: time students spent under EL classification (1-3 years vs 4 or more years), school resources for retained students (low vs high per-pupil expenditures), and concentration of ELs. Preliminary Results: Preliminary results suggest that students who score below the threshold for retention take an additional half year to achieve reclassification as fully English proficient and are 6.6 percentage points more likely to graduate from high school. The TOT estimates show that third-grade retention increases the time of reclassification by 2.5 years and increases the probability of graduation by 31 percentage points.

Background/Context: English learners (ELs) have the potential to bring much-needed multilingual skills to the workforce, and most ELs aspire to graduate high school and earn a post-secondary credential (Gwynne, Pareja, Ehrlich, & Allensworth, 2012; Shi & Watkinson, 2019). But active ELs graduate high school at far lower rates than their English-proficient peers (U.S. Department of Education [USDOE], n.d.a), and their college enrollment and degree attainment rates are much lower than native-English speakers (Kanno & Cromley, 2013). Many schools use Early Warning Indicator Systems to identify students who are at risk of not graduating from high school, and College Readiness Indicator Systems to identify whether students are meeting important criteria to be likely to eventually obtain a college degree (see review in Balfanz & Byrnes, 2019). These indicator systems often focus on students' grades, attendance, and behavior (Allensworth & Easton, 2005, 2007; Balfanz & Byrnes 2006). When considering ELs, there may be additional indicators of academic performance that matter for educational attainment (for example, English proficiency at the beginning of high school). In addition, ELs in high school are a diverse group of students and supporting their educational attainment might require a different set of indicators for each group. Purpose/Objective/Research Question: This study examines a broad array of potential indicators of educational achievement in high school to identify what matters most for educational attainment for different groups of ELs. We include indicators of achievement used in indicator systems for all students, as well as indicators that are specific to ELs, asking: RQ1: How are high school indicators of the academic performance of different subgroups of EL students in grades nine to twelve related to their later educational attainment (high school graduation, immediate college enrollment, college degree attainment)? RQ2: Given the relationships of academic performance with educational attainment, what are the largest barriers for each of the subgroups of ELs, and what are their strengths? Our focus is on former ELs (students who demonstrated English proficiency before high school), long-term ELs without IEPs, long-term ELs with IEPs, and late-arriving (newcomer) ELs. This can help point school practitioners to the indicators that are most critical in supporting all ELs. Setting and data: The data used for this study consist of cohorts of students who started ninth grade for the first time in the fall of 2008 through 2017 in district-run schools in Chicago Public Schools. These students would have graduated high school in the spring of 2012 through 2021, if they graduated in four years, and potentially enrolled in college in fall 2012 through 2021. Data Analysis: To test which variables are the best predictors of educational attainment, we use logistic regression models predicting each educational attainment outcome (e.g., high school graduation, college enrollment, college degree attainment) for each subgroup. The models contain an increasing number of variables, and we compare the pseudo-R-squared to determine which variables have the most explanatory power by themselves, and then in combination with other predictors, to see whether a new variable improves prediction of the outcome beyond those already in the model, and what combinations are best. We also conduct a number of confirmatory analyses using different methods. Findings/Results: The educational attainment outcomes show similar relationships with high school academic indicators for all groups of ELs, except for long-term ELs with IEPs. Across all three attainment outcomes, students' GPAs tend to be the most strongly predictive, with attendance rates also strongly predictive of graduation, and test scores sometimes adding more information for college outcomes. English proficiency levels and growth on the ACCESS are not strongly predictive of educational attainment for any EL group. GPAs and attendance are also the strongest predictors for never-EL students, but ELs need to show higher achievement to have similar probabilities of graduating high school or college as never-ELs. There are some differences across the EL groups in the relationships of achievement indicators with educational attainment. For long-term ELs with IEPs, grades and attendance are less predictive than for other groups, so the combination of all ninth-grade indicators provides a much better prediction than any two alone. For other groups, two indicators tend to provide similar prediction accuracy as models with all of the potential indicators. Another difference is when the indicators become particularly predictive; the ninth-grade year is a critical year for all groups, but for late-arriving ELs, indicators become noticeably more predictive in tenth grade. This suggests a need for longer-term supports around the transition to high school for late- arriving students. Many ELs and former ELs have strong high school achievement, with former ELs and late- arriving ELs showing stronger high school performance on many indicators than students who were never-EL. However, long-term ELs need much more support in passing their classes in high school--high course failure rates are the largest barrier to educational attainment. This is especially urgent for long-term ELs without IEPs. Conclusions: Work to improve attendance and grades would have the most leverage for improving educational attainment rates for ELs and never-ELs, suggesting indicator systems can be simple and similar for all students. At the same time, ELs face additional barriers beyond other students, so the thresholds for having good probabilities of educational attainment outcomes are different, and there is a need for additional supports for college attainment beyond academic ones. English proficiency may seem like the priority for supporting ELs' educational attainment, but improving grades and attendance is actually more critical. There is a need for further research to understand why attendance and grades are less predictive for students with IEPs, and to understand how school and classroom environment factors lead many long-term ELs to disengage from school.

The implementation of Multi-Tiered System of Supports (MTSS) has been successful in elementary systems across the United States. For secondary schools, the implementation of MTSS has not been as successful. Secondary teachers often note difficulties include understanding data from different sources, what sources are available, and how it applies to interventions and student progress or growth. This study employed qualitative narrative analysis to understand better the difficulties with implantation of MTSS at the secondary level. Through semi-structured individual interviews, participants discussed their experiences with MTSS and its implementation. A two-cycle coding approach was utilized, beginning with a deductive coding approach and a second cycle was inductive analysis through in vivo coding, allowing the participants' narratives to appear. Utilizing this data, a Theory of Practice (ToP), a self-study of educators' teaching methods, pedagogy, actions, and behaviors within their teaching and reflects on their legitimization of those practices was developed. The objective of this action plan aims to have teachers develop Theory of Practice (ToP) learning objectives that drive a self-study of their teaching methods, pedagogy, actions, and behaviors within their teaching and reflect on their legitimization of those practices in reflection of data indicating student growth (Meijer et al., 2016).

Background/Context: Students classified as Long-Term English Learners (LTELs) are those who have not gained fluency in the English language, even after being enrolled in the U.S. K-12 educational system for five to seven years (Olsen, 2014). Although LTELs are not consistently tracked across states, we know they constitute a sizeable portion of the EL population in some states--for example, during the 2019-20 academic year, 200,000 students in California were LTELS, plus another 130,000 considered at risk of becoming LTELs because they are not on-track to gain fluency in English proficiency (Buenrostro & Maxwell-Jolly, 2021). In a study by the WIDA Consortium, about 13% of all ELs in grades K-2 across 15 states were projected to become LTELs based on their English proficiency scores (Sahakyan & Ryan, 2018). Even "among" English Learners, LTELs face the least access to academic opportunities, such as taking academic content courses and advanced courses (e.g., Johnson, 2019). Prior research is silent on the effects of specific policies and programs targeting the education of LTELs. Purpose/Research Question: In this study, we examine the effects of a program, known as AVID Excel, which aims to improve English proficiency and access to academic core content while also promoting college readiness for middle school LTELs. We ask: Is participation in AVID Excel associated with better English proficiency and English reading outcomes for LTEL students, compared to receiving other English Language Development (ELD) services or to no ELD at all? New evidence from this research contributes to the literature in two essential ways. First, we add urgently needed results on the reading achievement and growth patterns of LTELs over time. Reading is an important component of developing the type of academic literacy needed to access core content in key subject areas (Brooks, 2016; Goldenberg, 2013), yet the existing research base does not speak to LTEL reading growth. Second, this study is the first to examine an intervention designed specifically for LTEL students. Intervention: AVID Excel is a newly popular type of college-access AVID program designed specifically for supporting middle school LTEL success. The curriculum provides explicit instruction in ELD, academic language, and college readiness skill across six domains: (a) reading, (b) writing, (c) oral language, (d) academic vocabulary, (e) study skills, and (f) self-determination and leadership. This program differs from traditional designated ELD, which solely focuses on English proficiency. Setting: Our partner district is a large, urban district in California and enrolls about 60,000 students annually. Just under 30% of the students are classified as ELs. There are about 1,500 LTELs in grades 6-8 and 1,500 in grades 9-12 every year. Support services for LTELs in middle schools vary across sites. Some students are enrolled in dual language programs; some attend designated ELD classes (including AVID Excel); others are educated in general education classes. The AVID Excel program was first implemented in the district in 2015-16. In 2021-22, eight of the 13 middle schools in the district were offering yearlong AVID Excel for 7th and 8th grade students. Data and Analysis: We use the district's administrative records between 2012-13 and 2021-22. Data include students' (a) demographic characteristics (e.g., gender, race/ethnicity, home language); (b) annual eligibility for EL and special education (SPED) services; (c) type of ELD service received and courses taken, including AVID Excel; and (d) academic and English language proficiency assessment scores (i.e., CELDT, ELPAC, SBAC-ELA, Reading Inventory). We retained data for students who were ELs at any time in 7th or 8th grade in the analytic sample (N=8,608); 5224 were LTELs in either 7th or 8th grade, meaning they have been ELs for at least 5 years, or since 2nd or 3rd grade. Among LTELs, 19% and 17% participated in AVID Excel. Since there was no random assignment into AVID Excel, and students could self-select out of the program, we applied multivariate regressions to examine the association between AVID Excel participation and each assessment outcome, controlling for student and school characteristics. To estimate reading growth as measured by RI, we used a multilevel model (e.g., von Hippel et al., 2018). Cross-level interactions identify additional growth made by students in other programs, above and beyond growth by AVID Excel students. Models were estimated using full-information maximum likelihood estimation by calling HLM Version 8 from Stata 16 (Raudenbush et al., 2019). Our descriptive analysis informs future impact evaluations. AVID Excel is gaining prominence across the nation as an intervention targeting LTELs. Correlational findings from this study can be used to contextualize and support the design of experimental studies. Results: Overall, we found that participation in AVID Excel is associated with higher reading and ELA test scores relative to Newcomer ELD or other ELD courses and lower scores relative to No ELD. On English proficiency tests, AVID Excel students tended to score higher than the other three groups. These results generally hold after controlling for prior-year test scores. AVID Excel students grew slower or at a similar rate compared to students in other ELD services or no service at all. Thus, we can rule out positive associations between AVID Excel and reading score gains as measured by RI. (See Tables 1-4 in the PDF document.) Conclusion: Supported by within-year growth data, this study offers critical, novel evidence on the AVID Excel program and LTEL students' English language proficiency and reading achievement and growth. We leverage a unique opportunity to interrogate the efficacies of a large, coordinated program designed and delivered for this student population. The findings not only inform program development within the district but also provide critical evidence to other educational agencies serving multilingual students.

Background: Inequality along racial and economic dimensions is well-documented and widespread in educational contexts. Achievement gaps are observed among children as early as primary school and are especially notable in standardized testing (Fryer & Levitt, 2004; Fryer & Levitt, 2013; Bond & Lang 2013). In response, some observers and policymakers have called for a deeper understanding of this testing gap and the mechanisms that produce these differences. While some have suggested abandoning standardized testing entirely, others insist that measurement is essential to accountability and progress, especially regarding racial and economic equality in education. If standardized tests fail to accurately measure achievement of students across all backgrounds, understanding the exact mechanisms by which it happens is crucial. One possible pathway might be the degree to which educational content is "relatable'' to certain groups of students, whereby students learn or perform better when they encounter concepts or topics with which they are familiar or in which they hold interest. Several papers in educational psychology show that interest in a topic can impact performance on reading comprehension tests and that interests diverge by race and gender (Bray & Barron, 2004; Asher, 1979). Other existing research in this area focuses on the representation of different identity groups in educational materials, such as children's literature (Adukia et al., 2023). Objective: In this research, we measure the impact of a student's relatability to a reading passage on standardized test performance. Complementing existing research on representation of identity groups in educational content, we consider the possibility that different racial/ethnic groups have divergent experiential knowledge or interest in topics which may appear in educational content. After determining a relationship between relatability and test performance, we measure the extent to which relatability contributes to observed racial/ethnic test gaps. We hypothesize that relatability impacts test performance, which implies that relatability of test content are contributing to test score gaps between racial or ethnic groups. Setting: To conduct our research, we analyze end-of-year standardized tests for primary school students in Texas from 2013-2019. We focus on the reading comprehension section of the test for 3rd to 8th graders. For each grade and year, we use item-level responses from every test taker and the full text of the tested reading passages for our analysis. Population: Our student sample consists of more than 13 million student-exam observations. The plurality of students in our student population are Hispanic (46%), while 34% are non-Hispanic white, and 15% are non-Hispanic Black (see Table 1). Research Design: We first develop a measure of "content relatability'' for a racial/ethnic group to a piece of text through a novel measure constructed using administrative survey data on demographic differences from the American Time Use Survey and natural language processing models. We apply this measure to our sample of reading comprehension passages. This generates the relatability of each passage to each racial/ethnic group represented in our student sample. In order to identify the causal effect of this relatability measure to student performance, our estimation strategy relies only on the portion of relatability that is considered conditionally random across test takers and tests. We argue that including race-grade fixed effects and passage-level fixed effects is sufficient for identification in our context, following closely the estimation framework laid out in Borusyak et al. (2022). Results: We find that the increased relatability of a reading comprehension passage causally raises student performance on questions connected to the passage. We estimate in our preferred regression specification that a standard deviation increase in relatable topics in a passage leads to a 1.7pp increase in the probability of correct answers on that passage (see Table 2). This effect is equivalent to a 0.07 standard deviation increase in passage-level performance and a 0.05 standard deviation increase in test scores. Since test makers ultimately select passages for inclusion in exams, we consider a variety of placebo outcomes with the same model specification, including prior student performance and demographic composition, and find no effect. We next investigate the extent to which content relatability contributes to racial disparities in test scores. We first show non-parametrically that at the passage-level, there is a correlation between relatability differences across groups and performance differences across groups (see Figure 1). Using our formal regression estimates, we find that equalizing content relatability across groups would lead to a 4% smaller Black-white test gap and 4% smaller Hispanic-white test gap. We counterfactually suppose how test scores would adjust if relatability had been set to a level which most closely equalizes relatability differences across race. We show that 1% of Black students in elementary school (grades 3 to 5) could have achieved a higher state-determined, reading comprehension standard if they took a test with more racially equal relatability. Overall, we counterfactually estimate over 11,000 Black students and 15,000 Hispanic students during our sample period were designated to be at a lower reading comprehension level due to relatability. Conclusions: Our results have implications both for test writers and education policymakers. First, it highlights that in order to write balanced assessments, test makers should take into account not only the identities of characters, but also the general content of the passage or question itself as we show that this may influence performance. Second, when policymakers consider outcome differences along demographic dimensions, one additional component to examine might be the standardized tests used to calculate those differences. However, we also note that the contribution of test construction to the gaps we find are both non-negligible and modest; that is, they cannot explain a substantial portion of why Black and Hispanic students on average have lower performance on tests than white students.

Background: Since the COVID-19 pandemic, American students have demonstrated substantial losses in achievement (Amplify, 2021; Domingue et al., 2021; Dorn et al., 2020; Storey & Zhang, 2023), while chronic absenteeism doubled after the pandemic, with the largest increase among low-income students (Chang et al., 2022; Dorn et al., 2021). Among historically underserved students, the rate of academic failure is far higher than it is among other students. On the National Assessment of Educational Progress reading test (National Center for Education Statistics, 2022b), only 19% of fourth grade students qualifying for free- or reduced-price lunch (FRL) scored at or above "proficient," in comparison to 51% of students who did not qualify. While it is generally felt that higher levels of parental involvement may lead to higher academic achievement (Kim, 2022; Wilder, 2014), attendance (Epstein & Sheldon, 2010), and social-emotional development (Green et al., 2007), research on the effectiveness of family engagement on student outcomes remains mixed (Cosso et al., 2022; See et al., 2022). Schools must therefore search for solutions to address student learning, chronic absenteeism, and social-emotional development from various sources. The identification of approaches with demonstrated effectiveness through evidence appropriate to each school's unique context, population, and requirements is particularly important. This methodology has been shown to be effective in addressing student needs (Slavin et al., 2013; van Geel et al., 2016). Purpose: Given the urgent need to identify and implement effective models to improve student academic and social-emotional outcomes, researchers and practitioners need to develop a stronger understanding of the impact potential interventions may have. In this study, we will develop and provide a visual overview of the extent and distribution of rigorous, high-quality evidence on family engagement programs and interventions in support of K-12 students. Setting: The study will focus on family engagement programs and interventions implemented in elementary and secondary schools. Population: The study will focus on programs and interventions implemented with Kindergarten through 12th grade students. Intervention: The study will focus on school-based family engagement programs and interventions that could plausibly impact the following student outcomes: literacy, numeracy, attendance, and social-emotional learning. Following previous definitions of family involvement (Boonk et al., 2018), we consider school-based interventions to be those that take place in the school, either focused specifically on increasing family engagement, or addressing specific student outcomes of interest with family engagement components. Research Design: This study will utilize existing data on family engagement programs and interventions to produce an evidence gap map. Evidence gap maps (EGM) are considered to be "thematic collections of evidence on effects structured around a framework which schematically represents types of interventions and outcomes of relevance" (Snilstveit et al., 2016, p. 121). EGM illustrate both areas where there is stronger understanding, but also where there are potential areas for additional needed research or analysis (White, 2020). Data Collection and Analysis: Literature Search: A master list of family engagement programs was created, integrating lists of family engagement programs developed by the National PTA. Initial electronic searches were made of educational databases including ERIC, JSTOR, and PsycInfo, to identify studies meeting inclusion criteria for each program. To locate unpublished reports and other possible inclusions, Google Scholar, citations from identified studies and previous reviews, and each program's official websites were also searched. Inclusion Criteria: Accepted studies must: (1) Be published between 1990 and 2019; (2) Evaluate a family engagement program for preK-12 students; (3) Include a comparison group of children receiving "business as usual" services; (4) Take place in the U.S.A.; and (5) Use quantitative measures of student outcomes including reading, mathematics, attendance, or social-emotional learning. Review & Extraction: After a screening of the abstract and title for relevance, all full-text studies were read by at least two reviewers before being included in our set of included studies. For the full systematic review, a number of items were coded. For the present study, only the intervention, outcomes, and whether outcomes were statistically significant were required. The resulting data were cleaned and analyzed using the R statistical software (R Core Team, 2023). The visual evidence maps were produced using ggplot2 (Wickham, 2016) and shared via an online Shiny app. The resulting analytic code, datasets, and interactive Shiny app are publicly available (AUTHORS, 2023). Findings: A total of 48 studies of 20 Family Engagement programs were reviewed, with 34 meeting the initial inclusion criteria (Table 1). Across these included studies a total of 50 broad outcomes were collected. These outcomes were divided into 4 categories: Reading, Mathematics, Attendance, and Social-Emotional Learning. The results are summarized in screenshots of the evidence map in Figure 1. Each study is represented by one dot. A study may be represented in more than one column if it measured outcomes from different categories. The color is based on the outcome findings. A green dot means more than half of the outcomes in that study in that category were statistically significant. A red dot means the impacts in that study in that category were not statistically significant. The map illustrates that while some programs such as ParentCorps have multiple studies, others have a single study assessing their impact, while several programs have no studies of effectiveness. Additionally, outcomes were most often measures in the reading domain. Overall, there are few programs with multiple studies documenting multiple significant results across studies or domains. Conclusions: Our evidence map employs a visually intuitive system to represent the distribution and significance of outcomes offering an overview of where these programs stand in terms of empirical support. Our findings call for a concerted effort to expand the evidence base, not only by increasing the number of studies but also by ensuring a broader representation of program types and outcome measures. Future research should aim to address these gaps by focusing on under-studied programs and domains, employing rigorous methodologies to assess program effectiveness, and exploring the scalability and sustainability of successful interventions. While the evidence map provides valuable insights into the effectiveness of family engagement programs, it also underscores the critical need for a more extensive and diversified research agenda.

This investigation scrutinizes the relative effectiveness of individual versus bundled accommodations -- specifically, breaks and extended time (ET) -- on the performance, behaviors, and attitudes of eighth-grade students with disabilities during the 2017 National Assessment of Educational Progress (NAEP) digital math test. Through a detailed propensity score analysis, findings reveal that while integrating breaks with ET diminishes performance compared to the application of ET alone, the inclusion of ET with breaks substantially enhances outcomes compared to utilizing breaks in isolation. Sole reliance on breaks correlates with inferior performance metrics, reduced item completion rates, fewer test-taking actions, heightened perceptions of time pressure, and decreased interest in mathematics. These results underscore the necessity to reassess current accommodation policies to better support students with disabilities, advocating for a more nuanced approach to the provision of test accommodations. Background/Context: In the evolving landscape of educational assessment, ensuring assessments accurately reflect a student's knowledge and skills rather than the limitations imposed by disabilities remains a formidable challenge, especially within the context of digital assessments. This challenge is magnified by federal mandates, such as the Individuals with Disabilities Education Act (IDEA, 2004), which necessitate the inclusion of students with disabilities (SWDs) in large-scale assessments and allow for accommodations to mitigate the impact of disabilities (Sireci, Scarpati, and Li, 2005). Among various accommodations, extended time (ET) and breaks are two of the most commonly used accommodations designed to offset disabilities' impact, aiming for equitable assessment conditions (Wei & Zhang, 2023; Abedi et al., 2008; Lai & Berkeley, 2012). These accommodations are essential in enhancing the reliability, validity, and fairness of assessment outcomes by eliminating construct-irrelevant barriers, thereby enabling SWDs to demonstrate their competencies equitably (AERA, APA, & NCME, 2014). Purpose/Objective/Research Question: This study aims to evaluate the individual and combined effects of breaks and ET accommodations on the mathematical test performance, test-taking behavior, and attitudes towards mathematics of SWDs. It seeks to ascertain whether bundled accommodations, specifically the combination of breaks and ET, provide superior or inferior outcomes compared to each accommodation applied independently. Setting: The research utilized data from the 2017 National Assessment of Educational Progress (NAEP) digital math test, representing a diverse cross-section of eighth-grade students across the United States. Population/Participants/Subjects: The study sample comprised approximately 2,750 eighth-grade students with disabilities who participated in the 2017 NAEP math assessment. These students received accommodations in varying combinations, including breaks, ET, or both, as determined by their Individualized Education Programs (IEPs). Intervention/Program/Practice: The intervention examined in this study involves the provision of two common test accommodations (breaks and ET) to SWDs during their participation in a standardized math assessment. The study compares the efficacy of these accommodations both individually and in combination. Research Design: Employing a quasi-experimental design through propensity score analysis (Rosenbaum & Rubin, 1983), this study aimed to reduce selection bias and approximate the conditions of randomized controlled trials. This methodological approach enhances the credibility of the findings by ensuring that the comparison groups are well-matched on several covariates. Data Collection and Analysis: Data collection involved the use of NAEP's restricted-use dataset, which includes detailed response, process, and survey data from the 2017 eighth-grade math test. The analysis focused on evaluating the impact of breaks, ET, and their combination on a range of outcomes including test performance, test-taking behaviors (e.g., time spent on items, number of items completed), and attitudes towards math. Findings/Results: The analysis revealed that students receiving both breaks and ET performed significantly worse than those receiving ET alone, suggesting that the combined accommodations may introduce unintended negative effects on performance. This paradox spotlights the complex art of balancing effective support against the risk of inadvertently hampering academic performance. While breaks aim to mitigate stress and fatigue, they might also, perhaps unexpectedly, fragment a student's concentration and workflow -- a detrimental side effect in subjects demanding consistent engagement like mathematics. Intriguingly, the bundled group, in contrast to those with only breaks, exhibited heightened engagement and interaction with the assessment. This is a beacon of the psychological and behavioral advantages tied to ET, supporting Wei and Zhang's (2023) findings. The bundled group's increased time on tasks, frequent use of assistive tools, and thoroughness in attempting questions suggest that while this combination might not directly amplify academic performance, it fosters a more immersive and involved testing experience. The 'breaks only' strategy in timed assessments, however, emerges as a double-edged sword. Our data indicates that breaks alone may not suffice in supporting SWDs, potentially even detracting from their performance and overall test-taking experience. The risk lies in the unaccounted time during breaks, which may culminate in heightened stress and a sense of urgency -- outcomes diametrically opposed to the intended purpose of breaks. This insight crystallizes the notion that while breaks are indeed a valuable resource, their integration with other accommodations like ET is crucial for optimizing effectiveness in both assessment and instructional settings. Conclusions: The study's findings challenge the efficacy of bundling breaks with ET as an accommodation strategy for SWDs in digital math assessments. While ET alone appears beneficial, its combination with breaks does not confer additional advantages and may, in fact, hinder performance. These insights call for a reevaluation of accommodation policies, emphasizing the need for accommodations to be tailored to the unique needs of individual students to optimize their assessment experiences and outcomes.

Background: Magnet schools emerged during the 1960s as part of an effort to desegregate schools across the United States. Their primary purpose is to provide appealing educational settings to induce voluntary desegregation through parental choices (George & Darling-Hammond, 2021). Some of these magnet schools operate at a regional level to address interdistrict desegregation, and Hartford, Connecticut stands out as a notable example. Despite some studies demonstrating the potential of Connecticut interdistrict magnet schools (IMSs) in fostering desegregation (Quick, 2016; Orfield & Ee, 2015), there is still a gap in understanding how they impact diversity among a range of nearby school options. While the recent study by Alcaino and Jennings (2020) suggested that magnet school enrollment growth had little effect on county-level racial segregation and private school enrollment nationwide, no specific study has delved into Connecticut's unique dynamics. This study aims to provide insights into compositional enrollment dynamics between IMSs and both traditional public (TPSs) and private schools, especially Catholic schools that serve a more historically disadvantaged student population. The findings will inform policymakers concerned with offering school options to promote desegregation, not only in Connecticut but also in other states. Research Questions: Our specific research questions include: 1) To what extent does the opening of IMSs explain the change of enrollment in proximal TPS and Catholic schools? 2) To what extent does the opening of IMSs explain the change of racial and economic diversity in proximal TPSs and Catholic schools? 3) Does the impact fall differently in schools located in Sheff region and those eligible for Title-I? Setting/Intervention/Population: As a part of outcomes from 1989 Sheff v. O'Neill lawsuit, the Connecticut State Department of Education (CSDE) introduced the Comprehensive Statewide Interdistrict Magnet Plan in 2009. By the 2016-17 school year, they had established 91 IMSs, including a significant portion of Hartford's minority student enrollments (CSDE, 2016). This backdrop creates a great opportunity to learn about the impact of opening magnet schools on student composition. Our key intervention of interest is the opening of the first IMS within 2.5 miles radius from TPSs and Catholic schools. Data: The main data source in our study is the National Longitudinal School Database (NLSD), which includes public and private schools' geographic location and annual enrollment by race, and free-or-reduced lunch eligibility (Carroll et al., 2023). To distinguish IMSs from regular magnet schools, we manually checked reports published by CSDE (2016). Our main datasets span from 1998-99 through 2019-2020 school years, separately for K8 and high schools. It allows us for a more precise impact analysis by capturing the first IMS opening that share at least one overlapping grade. Our final sample for K8 captures 789 TPSs, 122 Catholic, and 58 IMSs. High school sample includes 143 TPSs and 26 IMSs. Figure 1 presents the location of schools included in our sample. Research Design: We employ staggered Difference-in-Differences and event study approaches (Callaway & Sant' Anna, 2021) to address our research questions (see Equation I). Our estimates compare schools treated by IMSs opening with those never or not-yet treated. The key outcomes of interest include 1) school-level enrollment, 2) entropy index of racial diversity, 3) racial compositions of white, Black, and Hispanic students, and 4) the proportion of low-income students (eligible for free or reduced lunch; FRPL, TPS only). We then focus on two potential sources of heterogeneity in the impact -- whether a school is in Sheff region (Hartford County), and whether a school is eligible for Title-I -- to determine whether the policy's intended target is functioning. Main Findings: Overall, the impact of IMSs openings on proximal TPSs' racial and economic diversity significantly differs by Sheff and Title I status. First, only the treated TPSs located in the Sheff region and eligible for Title I status experienced an enrollment drop following the opening of IMSs within a 2.5-mile radius, particularly high schools (Figure 2). Regarding racial composition dynamics within these schools, the white share increased, the Black share remained relatively steady, and the Hispanic share decreased (Figure 3). Second, although there is a limitation regarding the pre-trend assumption, the percentage of FRPL students in TPS high schools significantly increased following the opening of IMSs (Figure 3). Third, the total enrollment of K8 Catholic schools also decreased, but there was no significant difference in patterns based on whether they were in the Sheff region or not (Figure 4). The white share decreased, while the Black and Hispanic shares increased, showing an opposite pattern from TPSs (Figure 5). Conclusions: This study expands upon existing literature investigating the impact of free public school options on racial and economic diversity, with an emphasis on interdistrict magnet schools, thus moving beyond the predominant focus on charter schools. Our findings suggest that Hispanic students in Sheff Title I TPSs, as well as overall Catholic school students (historically serving a large body of low-income students), who are the target of IMSs policy, may opt to enroll/transfer to those schools. Moreover, our research highlights that these enrollment patterns are particularly pronounced in TPS high schools, where students may be more inclined towards longer commutes for magnet schools offering specialized programs that align with their career pathways or college goals. These findings underscore the necessity for future research employing mixed methods to delve deeper into the decision-making processes of students and parents when selecting schools. Understanding how these decisions vary across racial and economic demographics will be crucial for informing policies aimed at promoting desegregation and equity within public school choice options.

Background/Context: Math education remains a critical focus for national education improvement. As a solution, districts in the U.S. are investing in math education technologies. Research has demonstrated the potential of these technologies to close achievement gaps (e.g., Pape et al., 2012; Roschelle et al., 2016). Student math achievement is particularly important during the middle school years. 64% of fifth-grade students and 58% of sixth-grade students did not meet expectations in math (MDE, 2022). Notable losses in student motivation, engagement (Eccles et al., 1993; Eccles & Midgley, 1990), and self-concept around math (Royer & Walles, 2007) occur during the transition to middle school. As schools adopt digital programs, it is essential to attend to student affect while rigorously testing the efficacy of these platforms in improving students' achievement. Purpose/Research Question: We conducted a multi-year, large-scale randomized controlled trial of the MathSpring program during the 2020-23 school years. MathSpring is a technology-based learning environment that offers personalized content, remedial tutoring, and affective support for students, along with learning analytics reports for teachers. We investigated the extent to which the program supports math learning, growth mindset, and dispositions towards math, as well as teachers' practices. The study was guided by four research questions: Does MathSpring improve students' math achievement? Does MathSpring impact students' growth mindsets and disposition towards mathematics? What factors moderate the effects of MathSpring? What is the fidelity of implementation and the effects of implementation on learning? Setting/Participants: The study was conducted in Massachusetts in 47 public schools from 41 districts with diverse backgrounds. Sixty-four fifth- or sixth-grade math teachers participated over three cohorts. On average, 69% of the participating students were White, 7% were Black, and 15% were Hispanic. 56% of students didn't meet expectations of standards. Intervention: MathSpring is a web-based intelligent tutor [citations blinded for review] designed to support students' achievement, affect, and engagement with math. An effort-based tutoring algorithm considers a student's recent performance and level of effort to provide an adaptive selection of problems, keeping students within their zone of proximal development (Vygotsky, 1978). The platform also provides students with learning support (immediate feedback, hints, worked examples in videos) and a learning companion avatar who provides supportive statements (Fig. 1a). MathSpring features a student progress page where students can visualize their progress via a plant that blooms and grows peppers as students master more topics (Fig. 1b). Additionally, it provides student- and class-level analytics data to teachers to inform adjustments to instruction and pacing. Teachers can view effort data for individual students and the class (Fig. 1c) and use the data to determine whether students are putting in effort during problem-solving (i.e., not skipping or guessing). Research Design: The study employed a multisite, clustered randomized experimental design in which teachers were first blocked by school characteristics and then, within each block, randomly assigned to the treatment (n = 34) or control condition (n = 30). Treatment teachers were expected to use MathSpring to support math instruction at least once every other week and received professional development, ongoing support, and access to self-paced online modules. Control teachers continued using existing instructional practices and supplemental technologies and were offered training and access to the platform after the study. The study was approved by the WestEd Institutional Review Board and was pre-registered at Registry of Efficacy and Effectiveness Studies. Data Collection and Analysis: Student mathematics achievement was measured with the Massachusetts Comprehensive Assessment System (MCAS) end-of-year state standardized math assessment. The online Mathematics Diagnostic Testing Project Math Readiness Test (MRT) (Anthony, 2005) was administered as a supplemental math outcome measure. Previously validated SEL survey scales were used to measure students' growth mindset, learning strategies, and dispositions toward mathematics, including the Goal Orientations and Learning Strategies Survey (GOALS-S) (Dowson & McInerney, 2004), mindset scale (Dweck, 2006), Grit Scale (Duckworth & Quinn, 2009), and scales measuring attitudes towards mathematics (PISA 2012 Student Questionnaire). To understand the nature of implementation, we collected teacher pre- and post-surveys and instructional logs, interviews, and classroom observations. Two-level hierarchical linear models were applied to estimate MathSpring's impact on student math learning and SEL survey scales. Researchers coded the qualitative data (35 teacher interviews, 11 classroom observations, 10 observation follow-up interviews) according to four SEL sub-categories: 1) the importance of effort, 2) problem-solving strategies, 3) goals, and 4) emotions. The backend system data, which tracked student actions and teacher usage of reports, were used to analyze implementation fidelity. Findings/Results: The analytic sample included 53 teachers (30 treatment, 23 control; overall cluster-level attrition = 17.19%, differential attrition = 11.57%). Analyses of the MCAS test scores are ongoing and will be shared at the conference. Here, we report initial results from the MRT and the SEL survey scales with 1,757 students (832 treatment, 925 control; overall student-level attrition = 12.28%, differential attrition = 6.22%). Findings from the teacher interviews and observations indicated that teachers in the treatment group mentioned each coded SEL sub-category significantly more often than teachers in the control group (Table 1). Teachers used MathSpring's analytics to spark class discussions around effort and problem-solving behaviors to help students develop self-regulated learning strategies. Yet, the impact analysis results indicated that the students in the treatment group did not outperform those the control group on the MRT, and the groups did not differ on the SEL survey scales either. Conclusions: There are numerous visions for leveraging technology to enhance math outcomes. This study investigated the effectiveness of an educational technology intervention that addresses math learning and attitudes towards math. The results indicate that using MathSpring during math time influenced teaching practices; treatment teachers incorporated SEL topics more frequently with their students than control teachers. However, initial analyses did not reveal conclusive evidence of the intervention's impact on student math learning. Ongoing analyses are examining the MCAS outcome measure, fidelity of implementation, the relation between fidelity and student learning, the service contrast between conditions, and a moderation analysis of student subgroups. These findings will contribute to our understanding of the interventions that support students' math learning and SEL outcomes.

Background/Context: School districts across the United States face persistent difficulties recruiting and retaining teachers (García & Weiss, 2019; Schmitt & deCourcy, 2022). Over the last decade, every state in the south has faced a growing teacher shortage, with some facing shortages in all grades and subjects (U.S. Department of Education, 2023). The current shortage of teacher candidates is reflective of shifting perceptions of teaching as a career. Nationwide, interest in teaching has dwindled to historically low numbers among high school and college students (Bartanen & Kwok, 2023; Kraft and Lyon, 2022). The COVID-19 pandemic has only exacerbated concerns with teacher recruitment, as undergraduate students' have become less inclined to consider a career in education (Bill et al., 2022). Teacher shortage concerns, compounded with declining interest in teaching as a career, place teacher recruitment and preparation in the policy and research spotlight. Traditionally, we look to high school and college students as our pool of future teachers. States across the country have embraced early teacher recruitment efforts, such as "Grow Your Own" (GYO) initiatives, to improve interest in the profession (Edwards & Kraft, 2024). However, additional research is needed to understand how prospective teachers interact with recruitment efforts and progress through pipelines into the profession. Research Questions: This study describes how interest in a career in education has shifted in terms of high school students' aspirations to pursue a career in teaching, as well as recent trends in preparation pathways among newly hired educators. We ask: (1) What percentages of high school juniors and seniors intend to major in education?; (2) What percentages of high school juniors and seniors intend K-12 teaching as their occupation; (3) What does enrollment in "teaching as a profession" courses look like over time?; and (4) How have the certification pathways of newly certified teachers changed from 2017 to 2023? Setting: This study investigates interest in the teaching profession across eight southern states, with in-depth analysis of teacher labor markets in two states. Data & Analysis: This study combines numerous data sources. Analyses are descriptive. The first two research questions focus on analyses exploring student interest in education across eight southern states: Arkansas, Alabama, Kentucky, Louisiana, Mississippi, South Carolina, and Tennessee. We use ACT survey data to examine occupation choice trends and interest in teaching among high school students. ACT data (2013-2022) include responses to student surveys about planned college major and occupation, as well as student demographic information. Analyses primarily focus on the 525,600 Kentucky and Tennessee students who took the ACT test in 11th or 12th grade between 2013-2022, enrolled in a 2-or 4-year postsecondary institution, and provided data on their college major and occupational plans. The third and fourth research questions examine trends in the educator workforce across Kentucky and Tennessee. We draw on administrative data for Kentucky (provided via KYSTATS) and for Tennessee (provided via the Tennessee Education Research Alliance) that cover the 2016-17 to 2022-23 school years. Data include information on staffing assignments, preparation programs, and licensure. We also examine participation in "Teaching as a Profession" courses among Tennessee high school students, tracking their participation in these courses from high school into educator preparation programs (EPPs) and entry into the teacher workforce. Tennessee administrative data includes high school course-taking and student demographic data. Preliminary Findings: Across eight southern states, interest in pursuing a career in education has declined over the past decade (Figure 1). The downward trend is similarly steep among high school students regarding interest education majors in college. Over a 10-year period, interest in education occupations in Kentucky declined from 8% to 5% and in Tennessee interest dropped from 7% to 4%. Given that interest was small to start, declining interest in teaching among high school students is concerning. Although high school students' interest in education careers has declined, the racial and gender backgrounds of students indicating interest in the profession have remained unchanged. Between 2013 and 2022, students who plan to major in education and express interest in an education career in both Tennessee and Kentucky have remained predominantly white. Eighty percent of intended education majors in Tennessee and 86% in Kentucky identify as white. In 2022, students intending to major in education were 83% female in Tennessee and 77% female in Kentucky. Students interested in pursuing careers in education mirror the current racial and ethnic composition of teachers across both states (which are both majority white and female), but students signaling interest in teaching careers are not racially representative of the current student populations in Tennessee and Kentucky. Both states have increasingly diverse student populations. In Kentucky and Tennessee, students of color represent 25% and 40% respectively. Declining interest in teaching is not promising for ongoing calls and efforts to diversify the teaching workforce. However, state efforts to recruit teachers through "Grow Your Own" initiatives tell a different story. Figure 2 shows trends for high school student enrollment in Tennessee's TAP courses. Enrollment has increased as course offerings have expanded across the state, particularly among high school students of color and male students. Conclusions: While interest among high school students in education careers has declined steadily across numerous southern states, localized recruitment strategies, such as Tennessee's "Teaching as a Profession" courses, show promising results for expanding the supply of prospective teachers. It is possible that interest in becoming teachers among high school students may have declined even more over this period had these courses not been available. It is also possible that by taking an introductory teaching course, some high school students may decide that the positive aspects of becoming a teacher, such as a desire to work with young people, may outweigh the negatives, such as low salaries or student disciplinary challenges. These courses targeting high school students could be a potential avenue for revitalizing the teacher pipeline and fostering diversity within the profession. Overall, this research contributes to empirical insights into the complex landscape of teacher recruitment, emphasizing the need for targeted interventions to reverse declining interest in the profession.

Background: Despite historical efforts (Ellwood & Kane, 2000; Daun-Barnett, 2013; Perna, 2006; Kane, 1999), persisting inequities in higher education access underscore the need for innovative research approaches. Previous studies utilizing geospatial approaches often overlook critical dimensions like college readiness and financial aid while focusing on institutional locations (Sponsler & Hilman, 2016; Turley, 2009; Dache-Gerbino, 2018). This study expands the scope of the spatiality of higher education access to these aspects using innovative geospatial analysis, aligning with the conference theme of navigating future education research. It explores the interplay of academic resources, financial support, and spatial factors, particularly in metro Atlanta's diverse and segregated context. By doing so, it aims to shed new light on how these intertwined factors contribute to ongoing disparities in educational equity. Research Objective: This research seeks to understand the spatial distribution of academic readiness, financial support, and higher education institutions in Metro Atlanta, addressing three primary questions: (1) How are academic readiness, financial support, and local higher education institutions spatially distributed in metro Atlanta? (2) How, if at all, do these spatial patterns cluster in particular areas? (3) How is access to higher education academically, financially, and spatially segregated? By answering these questions, the study aims to expand the conceptualization of spatial access to higher education and analyze significant spatial clusters and patterns of educational access, providing insights into the systemic nature of these inequities. Setting: The study is set in the Atlanta-Sandy Springs-Roswell GA Metropolitan Statistical Area. As the eighth-largest metropolitan area in the U.S. and Georgia's most densely populated, this region is characterized by its diversity and notable segregation (reardon et al, 2022). The area's considerable student population, comprising 308,036 high school students across 214 schools and over 250,000 postsecondary students annually in 57 colleges and universities, offers a rich context for examining educational disparities. Additionally, the state's focused policies on college readiness and substantial financial aid support make Georgia, particularly metro Atlanta, an ideal setting for this spatial analysis of higher education access. Setting: The study is set in the Atlanta-Sandy Springs-Roswell GA Metropolitan Statistical Area. As the eighth-largest metropolitan area in the U.S. and Georgia's most densely populated, this region is characterized by its diversity and notable segregation (reardon et al, 2022). The area's considerable student population, comprising 308,036 high school students across 214 schools and over 250,000 postsecondary students annually in 57 colleges and universities, offers a rich context for examining educational disparities. Additionally, the state's focused policies on college readiness and substantial financial aid support make Georgia, particularly metro Atlanta, an ideal setting for this spatial analysis of higher education access. Population: The study focuses on 308,036 high school students in 214 schools across metro Atlanta, a diverse group in terms of socioeconomic, racial, and educational backgrounds. It also includes the broader demographic contexts of the region's college students and general population. The analysis examines these students' access to higher education, considering factors like academic readiness and financial aid availability. It emphasizes the impact of spatial distribution within the metropolitan area, exploring disparities arising from the intersection of these varied characteristics with the spatial dimensions of educational access. This comprehensive approach aims to reveal how institutional and societal structures affect educational opportunities. Intervention: While this study does not directly implement a specific intervention, it serves as a diagnostic tool, revealing prevailing inequities in higher education access within metro Atlanta. Using GIS and hot spot analysis, it identifies how access to higher education is spatially clustered, correlating with areas where specific racial and socioeconomic groups are concentrated. The findings of this study offer crucial insights into educational inequity, highlighting spatial patterns that could guide targeted interventions. Thus, the study itself acts as an indirect intervention by providing data-driven foundations for future initiatives aimed at increasing access for students from diverse backgrounds. Research Design: The study employs GIS and hot spot analysis within a critical geography framework, recognizing space as a socially constructed entity. This approach is designed to identify significant clusters of higher education access, considering the underlying social contexts for accurate interpretation. To mitigate potential biases, the study utilizes comprehensive pre-pandemic data, ensuring the validity of its findings. The interpretation of spatial data is enhanced by critical geography, which views space as a construct shaped by power dynamics and political-economic relationships. This theoretical grounding prevents oversimplified interpretations of spatial data, allowing for a more nuanced understanding of socio-spatial dynamics. Data and Analysis: This study aggregated pre-pandemic data (2018-2019) from the Georgia Department of Education and the National Center for Education Statistics, including academic readiness indicators, financial aid statistics, and geographic locations of educational institutions. The analysis employed GIS and hot spot analysis based on Getis and Ord's (1992) Gi* statistic. This method allows for examining spatial correlations between educational resources and demographic factors. Critical geography theory guided the interpretation of these spatial patterns. By analyzing these relationships, the study aims to offer a comprehensive understanding of how educational resources and demographic realities intersect to shape access to higher education. Findings: A preliminary analysis of descriptive mapping was conducted to explore the spatial distribution of higher education access in metro Atlanta. As shown in Appendix 1, the maps provide evidence of spatial patterns in the datasets. Lower scores of colleges and career ready performance index, fewer availabilities of AP courses, and fewer number of FAFSA applications completed tend to cluster in the center of metro Atlanta, and higher scores, more availabilities, and greater number of completions tend to cluster in the northern part of metro Atlanta. While these descriptive maps provide evidence of spatial patterns in the datasets and thus a rationale for approaching the datasets with geospatial approaches, it is difficult to determine the accurate spatial patterns with confidence. Extending from this preliminary analysis, hot spot analysis findings are expected to highlight clear spatial segregation in educational opportunities, suggesting that access to higher education in metro Atlanta is influenced significantly by geographical location and its associated socioeconomic and racial factors. Conclusions: The anticipated conclusion of this study is that higher education access in metro Atlanta is likely to exhibit significant spatial segregation influenced by racial and socioeconomic factors. This highlights the pivotal role of geography in educational equity, extending beyond mere institutional locations. Recognizing the study's focus on a single metropolitan area as a limitation, further research in varied settings is recommended for broader validation. The expected findings hold the potential for guiding policymakers and educators in creating targeted interventions to mitigate spatial inequities in higher education access, particularly benefiting underrepresented communities. This approach aligns with the conference theme, signaling a progressive direction for future education research.

The Cleveland Metropolitan School District (CMSD) has seen the number of English learner students in grades K-12 increase in recent years, even as overall enrollment has decreased. In addition, the English learner student population has grown more diverse in race/ethnicity, country of origin, and native language. The increasingly diverse English learner student population requires more support from the district to meet broader needs for language, cultural, and educational assistance. The Cleveland Partnership for English Learner Success is a researcher-practitioner partnership between Regional Educational Laboratory Midwest and CMSD's Multilingual Multicultural Education Office and the district's research office. The partnership has prioritized examining the characteristics of English learner students and the schools they attend to identify which of those characteristics are associated with student achievement. The partnership undertook this research to help improve district and school supports for English learner students. The authors found the number of English learner students speaking languages other than Spanish increased. From 2011/12 to 2016/17 the percentage of students whose native language was Spanish decreased from 82 percent to 71 percent. During the same period the percentage whose native language was Arabic increased from 4 percent to 9 percent, and the percentage speaking an African language doubled from 5 percent to 10 percent. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Group discussions during mathematics lessons provide important opportunities for students to deeply engage with mathematics content and to develop communication skills and strong understanding aligned with college and career readiness objectives (Walshaw & Anthony, 2008). However, such student-centered discussions are less common than teacher-led lectures in middle grades mathematics (Stigler & Hiebert, 2004). Research has found that lack of training contributes to the challenges in successfully implementing student-driven discourse (Piccolo et al., 2008). Simulated Instruction in Mathematics Professional Development--or SIM PD--is a new program being developed to meet the need for PD focused on mathematical questioning and discourse (NCTM, 2014) using the classroom simulation technology. This study used a school-level random assignment design so that groups of two to six teachers within a school were randomly assigned to either implement the program during the 2018-19 school year, or participate in data collection but wait to participate in the PD until the 2019-20 school year. The authors will share findings from a small-scale, school-level randomization study in 2018-19 to provide an early, rigorous assessment of whether the PD helps teachers develop the targeted instructional practices. Findings indicate that the program was feasible to implement, teachers received the PD positively, and classroom practice may have improved as a result of the PD. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Use of online credit recovery is a growing trend across the country, with the hope that expanding credit recovery options through online courses will help students get back on track toward graduation (e.g., Atkins, Brown, & Hammond, 2007; Gemin, Pape, Vashaw, & Watson, 2015). But expanded use of online credit recovery for high school students has outpaced the research. The authors conducted a multisite study to test a blended-learning model for credit recovery, where the main curriculum is provided through an online program and an in-class teacher provides more individualized instructional support. The authors present findings on the implementation and initial outcomes for two cohorts of students who enrolled in an Algebra 1 or English 9 credit recovery course the summer immediately after their first year of high school. The authors address three main research questions to describe how implementation of the online classes differed from the schools' business-as-usual (BAU) teacher-directed approach to credit recovery, and provide preliminary evidence about the short-term effects of the online approach: (1) How did the "instructional features" of the online classes compare to the BAU classes?; (2) How did "students' experiences" in the online classes compare to the BAU classes?; and (3) How did students' "proximal outcomes" (content knowledge and credit recovery) in the online classes compare to the BAU classes? The findings suggest that implementation of the online classes produced limited instructional contrast with the BAU condition and may have unintentionally altered grading practices. Potentially as a result, fewer students recovered the course credit in the online classes than the control classes. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Background: The number of English learner (EL) students has been steadily increasing within the United States and currently includes more than 5M students. The National Education Association (2020) estimated that by 2025, nearly 25% of all learners will be EL students. Curricula, interventions, and assessments for both diagnostic and academic purposes need further development to accommodate the growing multilingual student population. However, researchers who aim to bridge the gap in available tools often struggle to successfully engage underrepresented and marginalized communities, including the EL population, and the school districts who serve them (Bonevski et al., 2014; Erves et al., 2017). Objective: The current paper provides a descriptive review of our experience engaging with the Spanish-speaking, Hispanic/Latinx community and school districts that serve this population regarding a research study focused on development of a valid, Spanish-language social-emotional assessment. We aim to impart our approach to partnering with underrepresented communities so other researchers can more effectively engage and support them. Setting: Recruitment efforts are underway across the United States and have included virtual and in-person networking. Presently, partnerships have been established with school districts within Illinois, Texas, Michigan, and the District of Columbia. Population: Target populations include community-based organizations to help develop stakeholder-informed recruitment plans to effectively enroll Spanish-speaking Hispanic/Latinx EL students in grades 3-7; their families; and school leaders from across the United States. Practice: We engaged in several community-informed recruitment practices to help us achieve our recruitment goals (e.g., Cunningham-Erves et al., 2023). Research Design: The University Medical Center IRB approved all aspects of this research. This project began early in the pandemic, many schools shifted to remote learning. Wherein we typically identified a limited number of large school districts to collect classroom-level data and to conduct classroom-based individual recruitment efforts, this was no longer sufficient. Further, the EL students we aimed to recruit were among the most negatively impacted by COVID-19 (Long, 2022; Relyea et al., 2023; University of Cincinnati, 2021). After schools re-opened, recovery from learning loss and effectively managing their time and resources within a stressed educational system were their focus. Even with Spanish-language flyers, bilingual webpages, and paid advertisements, traditional methods of recruitment were only minimally effective. We adopted a broader approach to recruitment than previously utilized. To better engage with the community, we began presenting our research problem to community-based and inter-organizational groups. Many of these community members were the same stakeholders we needed to engage in our research. Our research team presented our past recruitment designs and current challenges and began listening. Together, we then brainstormed strategies to reach a larger audience and more effectively build trust in the community. Data Collection and Analysis: We tracked recruitment ideas, strategies, leads, partnerships, and outcomes. Our primary outcomes of interest were recruitment rates and attrition rates after engagement. Results: Here we highlight a range of strategies for reaching underrepresented communities as suggested by members within the community-based organizations. First, because the Principal Investigator was herself not a Spanish-speaking Hispanic/Latina individual, it was important to have representation within our research team and endorsement of our team or project from a trusted community, educational, or religious liaison. To this end, we enlisted the support of a English-Spanish bilingual parent who also worked within an area school district to act as a trusted liaison and to be present at in-person events. As one such example, at a Bilingual Parent Action Committee resource fair, parents would engage openly with the liaison; in one morning, eight new families signed up for information. This trusted individual can more effectively share project information with community bilingual parent groups and at area events. Next, minimize wordiness on materials and better convey the benefits of participation. If possible, utilize text messaging and other audio-based strategies. In response, we have developed Spanish informational messages that play when families are on hold while calling other offices. Through conversation with community members, to reduce attrition, it was suggested that we reduce the number of points of contact for enrollment. We modified our process to have the bilingual coordinator conduct the online enrollment process during a phone call with the parent. We also reach out to parents who provide their contact information, versus asking them to contact us. Community members suggested that enrolling participants simultaneously with other events may yield the greatest return. When families attend other school or community events, often times, children attend. If participation can occur during those events, it facilitates their completion. To engage better with school districts, ideas included providing educational sessions for educators or families on assessment, social-emotional learning, or mental health -- capitalizing on expertise within our team and providing benefit to the partner. Rather than cold-calling prospective school districts, community leaders stressed again how one-on-one engagement in person was key even for educators. Community leaders suggested hosting "lunch and learns" to share information with educators and school leaders. While we were unable to implement this strategy, we did attend key national conferences (e.g., National Association for Bilingual Education) to engage school partners with large Spanish-speaking populations. By hosting vendor tables at conferences, we made personal connections and increased visibility about our work. Finally, school leaders stressed providing materials to them for distribution to families about the partnership and research. Being able to provide schools with materials translated already into Spanish also facilitates their participation and ability to assist with recruitment efforts. Discussion: Community engagement is time-intensive. It is not enough to post a flyer or send an email. A face and a name to connect to the information is needed to establish a trusting relationship (Wallerstein & Duran, 2006). Underrepresented communities, like all communities, value transparency about what the time commitment will be and the end goal. Research opportunities that yield mutual benefit are the most effective partnerships and long-term success. Through varied outreach efforts, new opportunities arise to connect and engage with the EL community. As a result, in addition to individual participants, we developed new partnerships with four districts that have high percentages of Spanish-speaking EL students.

Context: The use of educational technology for improving K-12 math education has expanded dramatically recently. ASSISTments (Heffernan & Heffernan, 2014), a widely used digital platform, has seen its use in schools increase significantly from 800 to 20,000 teachers over the past two years. A review of evidence-based online programs by the What Works Clearinghouse named ASSISTments as one of the few digital learning programs recommended for use in response to the COVID pandemic (Sahni et al., 2021). An efficacy study conducted in Maine (Roschelle et al., 2016; Murphy et al., 2020) found that using ASSISTments significantly increased student scores on the TerraNova test (Hedges' g = 0.22, p < 0.001). From 2018-2020 we conducted a replication study to test the replicability of the findings in a heterogeneous population that more closely matches national demographics. A follow-up study was designed to measure the long-term impact of ASSISTments on student learning in 2021, one year after the intervention was over. This paper reports on the findings from the preliminary analysis of the follow-up study. Research Questions: Two research questions motivate the follow-up study: RQ1: What is the long-term impact of ASSISTments on student math outcomes at the end of Grade 8, one year after the completion of the intervention? RQ2: Do the long-term effects of ASSISTments, if exist, vary for students of different socio-economic status, race/ethnicity, or with other policy-relevant characteristics? Sample: The replication study took place in North Carolina (NC). Sixty-three schools were recruited and randomly assigned to condition (32 intervention and 31 comparison). The sample included 48 Title I schools and distributed across rural, town, suburban, and city communities. The study enrolled 102 7th grade math teachers and their classrooms. Table 1 compares demographics of participating schools to NC and US populations, and the Maine study sample. A generalizability analysis suggested the replication sample is highly representative of NC schools (generalizability index = 0.958) and schools across the nation (generalizability index = 0.899). Intervention: ASSISTments is a web-based platform that supports students' math problem-solving through immediate feedback and hints and provides teachers with reports summarizing student work to inform instruction adaptation. The intervention aligns with theory- and empirically-based instructional practices of formative assessment (Heritage & Popham, 2013) and skill development (Koedinger et al., 2013). During the study, the intervention was positioned as online support for math homework, a key opportunity for student independent practice, and for teachers to use formative data to adapt instruction (Figure 1). Research Design: The replication study used a randomized experimental design. Schools were randomly assigned to either intervention or business-as-usual comparison condition. ASSISTments was implemented by all Grade 7 teachers in intervention schools over two consecutive years--teachers learned to use ASSISTments for a year (2018-19) and then we measured the immediate outcome for students at the end of teachers' second year of experience (2019-20). Students who were in Grade 7 in 2019-20 comprised the analytic sample for the research questions. The students maintained their conditions and were followed longitudinally for another year when their Grade 8 performance (long-term outcome) was measured in spring 2021 (see the study overview in Figure 2). During the follow-up year in 2020-21, no interventions were provided to 8th grade teachers or students. Data: We obtained demographic, enrollment, and state assessment performance data from the state-wide database for all 63 schools. Student's long-term learning outcome was measured by Grade 8 state standardized End of Grade (EoG) math test. Grade 6 EoG math test scale scores from spring 2019 served as the baseline measure. Individual demographic covariates included gender, race/ethnicity, and economically disadvantaged status (EDS). School-level covariates included average 6th grade EoG score, 7th grade enrollment size, Title 1 eligibility, percentage of students with EDS, and percentage of ethnic groups. There were 9,073 students who (a) have 6th grade EoG score, and (b) enrolled in a participating school either as a 6th-grader in 2018-19, or as a 7th-grader in 2019-20 if the school didn't have 6th grade in 2018-19. Among these students, 5,991 have 8th-grade EoG test scores (2,961 intervention, 3,030 comparison) and were included in the analysis. 34% students (n=3,082) took an accelerated or alternative math course and did not take the Grade 8 EoG test. Their performance was analyzed separately and were not included in the paper. Analysis and Findings: There was no school-level attrition. Overall student-level attrition on Grade 8 EoG test was 33.97% (mostly due to the accelerated alternative math course enrollment) and the differential attrition was 0.31%. Counting all students with non-missing outcome data, the overall and differential attritions were 12.83% and 1.81%, respectively. We examined baseline equivalence on students' 6th grade EoG scores, gender, and ethnicity and found no significant differences between conditions. To address RQ 1, we conduct an intent-to-treat (ITT) analysis using two-level hierarchical linear regression models (students nested within schools), controlling for student's Grade 6 scores and other student- and school-level covariates. To address RQ 2, we added a cross-level interaction term of the school-level treatment variable and student subgroup indicator and conducted a series of moderator analysis to examine whether ASSISTments had differential impacts on the students with policy-relevant background characteristics. Results showed that students at intervention schools performed significantly better than those in the comparison group on 8th grade EoG (p = 0.011, g = 0.10) (Table 2). The moderator analyses showed that the intervention benefited minority (non-White) students significantly more than majority (White) students (p = 0.003, g = 0.14) and that the impact was more profound for Hispanic students than non-Hispanic students (p = 0.014, g = 0.13) (Table 3 and Figure 3). Conclusions: The follow-up study was motivated by the need to explore the long-term impact of ASSISTments use with diverse populations. The results suggested that it had a sustained long-term impact on students' math learning, after one year of intervention. The program also helped close the achievement gap among students of different ethnicities. Future works include additional moderator or mediator analyses on ASSISTments usage in 8th grade, benchmarking the effects, and estimating the cost-effectiveness.

Background: Public schools are among the first civic institution with which many individuals have prolonged, meaningful social interaction. Lessons about the authority, power, and fairness of civic institutions, conveyed through disciplinary and social control practices, may be part of the "hidden curriculum," that teaches students about racially inequitable unspoken norms, beliefs, and values held by society. We summarize three pathways by which the discipline gap in schools may contribute to civic disempowerment. First, and most directly, schools are key providers of civic education. They provide students with knowledge and skills on topics like politics, history, government, and current events (1,2). Students who are excessively removed from the classroom because of exclusionary discipline are more likely to miss out on this education. Second, schools shape students' attitudes about civic engagement. Schools influence students' concern for the common good, their understanding of whether their school, as a representative of a civic institution, is fair and just, and their belief in whether it is possible to change systems for the better (3). The discipline gap been found to damage its victims' sense of fairness in school and to lead to disaffection and alienation (4-6). Third, education is associated with the financial security and job characteristics that make civic engagement more possible (7,8). The discipline gap has been strongly linked to risk of drop out and therefore to diminished economic well-being (7,9). Moreover, exclusionary discipline has been tied to increased risk of incarceration, which can lead to outright disenfranchisement (10). Purpose: Using Critical Race Theory, Social Cognitive Theory, and Social Control Theory, we propose that disparities in civic engagement may be among the long-term consequences of racially disproportionate discipline and social control in schools. We tested this hypothesis using voting behavior. This study builds on previous work in this area (11) first by strengthening the theoretical basis for a connection between exclusionary disciplinary experience and social control in school and civic engagement in adulthood and second by extending the empirical test of those theoretical bases. We predict that disproportionate discipline and control in schools are among the insults experienced by Black students that inform their social learning about the fairness of civic institutions and that therefore erode their sense of political efficacy thereby making them less inclined to vote. Our study tested three hypotheses: first, disciplined students will be less likely to vote as adults, with the effect size increasing with the number of suspensions received. Second, the number of social control measures in school will be continuously and negatively associated with voting as an adult. Third, the effects of both suspension and social control on voting behavior will be modified by race, with Black, non-Hispanic students experiencing greater reductions in voting than non-Black students. Methods: With national data from approximately 15,369 students who participated in the Education Longitudinal Study between 2002 and 2012, this retrospective cohort study examined the relationships between discipline (i.e., how often a 10th grade student was suspended), social control (i.e., the number of social control measures in place in their school), and voting behavior into adulthood and how those relationships varied by race after considering several individual-, household-, and school-level variables, including student perceptions of school climate. Following descriptive and bivariate analyses, we used multilevel logistic regression with household- and school-level fixed effects to test our hypotheses. Our first model looked at only main effects. In the second model we added the interaction between Black non-Hispanic race and discipline history. In the third model we included a second interaction term between Black non-Hispanic race and social control. Findings: As hypothesized, we found that the relationship between exclusionary discipline and voting behavior was negative and demonstrated a dose-response pattern. In model one, students who were suspended one or two times in their sophomore year had approximately 16.5% lower odds of being a regular voter relative to those who were never disciplined (p=0.003). Students suspended three or more times had 25.4% lower odds (p<0.001). The interaction of race and suspension history in Model Two was significant for students who had been suspended three or more times (p=0.021), confirming part of our interaction hypothesis. Taking the main and interaction effects into account, Black non-Hispanic students suspended one or two times had an odds of voting regularly that was 26.2% lower than non-Black students who had never been suspended (OR=0.738, p=0.059). Black students suspended three or more times had an odds of voting regularly that were 33.5% lower than non-Black students who had never been suspended (p=0.021). The relationship between number of social control measures and voting behavior was positive and significant. Every additional social control measure was associated with a 1.3% increase in the odds of regular voting (p=0.002); the interaction by race was not significant. This contradicted our hypotheses. Conclusion: Our discipline-related findings suggest that schools are testing grounds where students develop a lived experience of fairness and justice, which they may then apply to other civic activities. Future studies should use the paths enumerated above and richer, mixed methods data to examine the mechanisms at play. This understanding will be necessary to advise school administrators on how and why to address the discipline gap (e.g., through disciplinary policy reform and implementation of restorative alternatives). This study also encourages education researchers, practitioners, policymakers, and advocates to think about the long-term effects of the "hidden curriculum" from a racial equity perspective. At a moment in our history when American Democracy seems gravely at risk, we must think about how our institutions communicate racialized messages of privilege, power, and precarity.

Background/Context: Recently there has been increased emphasis on disaggregating students' experiences collected through surveys based on student identities (Arredondo, 2016). Yet demographic data collection is problematic by nature, often reducing the complexity of an individual's racial, gender, or sexual identity to a single category. Racial identity is much more complex than what is captured by commonly accepted racial categories. It includes three concepts: self-identity (the meaning individuals attribute to their race/ethnicity), category (which category individuals selected from the discrete survey options), and identification (the category observers would place them in) (Rockquemore et al., 2009). Like race, gender is not unidimensional and is a combination of biological sex at birth, gender identity (how an individual perceives their own gender), gender expression (how gender is presented), and gender label (how an individual labels or, in some cases, does not label the intersection of biology, identity, and expression) (Temkin et al., 2017). Similarly, sexual orientation is multidimensional and is defined in terms of sexual identity (how an individual labels their sexual orientation), sexual attraction (who an individual is and is not attracted to), and sexual behavior (whom an individual engages in sexual acts) (Temkin et al., 2017. Despite the need for data collection that includes racial, gender, and sexual orientation demographics, the discrete nature of survey questions diminishes the complexity of these identities into discrete and often mutually exclusive categories. Even with the most robust and comprehensive demographic questions, creating survey questions that fully capture the complexity of individual identities remains challenging. However, best practices suggest including an open-ended other question allowing individuals to self-report their identity. Purpose: This study examines how students in grades 3-12 self-identify when given the option to express their racial, gender, and sexual orientation through a write-in other choice on a survey. The questions guiding our study are: 1. How do students respond when they can self-identify their race, gender, or sexual identity? 2. In what ways do students' understanding of racial, gender, and sexual orientation differ from the discrete identity categories provided? Setting and Sample: The data analyzed in this study were drawn from the 2022 student restorative culture survey (SRCS) administered to students in grades 3-12 in a school district in New England. Across the district's eleven schools, 1,760 students in grades 3-12 completed the 2022 survey. Based on discrete demographic questions, Table 1. provides details of the survey respondents. Of the students who responded to the survey, 11% responded to the open-ended race option, 3% to gender, and 5% to sexual orientation. Research Design and Analysis: The 2022 survey was administered by the school district using GoogleForms during class time to allow teachers to answer clarifying questions that might arise. At the end of the survey, students were asked a series of demographic questions (see Appendix A) where they could check all that apply and/or fill in the opened 'Other' response. Our study employed a questionnaire variance design (Creswell & Clark, 2018) to analyze the open and closed-ended demographic questions. To analyze the responses, researchers one and two reviewed the responses independently and identified emerging themes. We then came together to discuss and refine the themes and identified cases where we did not converge and, through the discussion, further refined the themes into their finalized versions. The last review of the responses by each researcher was assessed for interrater reliability using Cohen's kappa (1968), which revealed near-perfect agreement (race = 0.82, gender = 0.88, sexual orientation = 0.84). Results: Racial Identity: Six themes emerged in the racial demographic category (see Table 2). Roughly 25% of students with an open-ended response also checked a racial category. Of these students, 8.7% selected white, 4.6% African, 3.5% Asian, Asian American, 3.6% Black/African American, 2.6% Multiracial, 1.5% American Indian/Alaskan Native, and <1% Hispanic or Latinx. Gender Identity: Five themes emerged in the gender demographic category (See Table 3). Roughly 26% of students with an open-ended response also checked a gender category. Of these students, five students selected 'boy' (33%), five students selected 'neither boy nor girl' (33%), three students selected 'girl' (20%), one student selected 'I am not sure yet' (6.6%), and one student selected 'Decline to answer' (6.6%). Sexual Orientation: Four themes emerged in the sexual orientation demographic category (See Table 4). Roughly 17% of students with an open-ended response also checked a sexual orientation category. Of these students, six selected 'Bisexual or Pansexual' (43%), three selected 'I am not sure yet' (21%), two selected 'Asexual' (14%), two selected 'I don't know what this question means' (14%), and finally one student selected 'Decline to answer' (7%). Conclusion: Study results highlight the importance of collecting robust student demographic questions within K-12 surveys. Even though the original demographic categories were more extensive than more commonly used survey demographic questions, many students still felt the need to self-identify regarding race, gender, and sexual orientation. These results reinforce the need for those collecting data on K-12 students to include categorical measures of identity and allow students to express their understanding of these complex constructs.

Background and Purpose: Sleep is an important determinant of health in adolescence, but many adolescents around the world are sleep-deprived. Compared to children and adults, getting enough sleep is especially challenging for adolescents due to the natural alteration of circadian rhythms that govern sleep-wake cycles. The delayed circadian rhythm does not necessarily lead to sleep deprivation unless late morning sleep is disturbed. Unfortunately, early school start times require adolescents to wake up early in the morning when their body "clocks" point to sleep time. As the negative impacts of inadequate sleep of adolescents have been increasingly recognized over the past decade, professional organizations, including American Academy of Pediatrics, have issued policy statements advocating secondary schools begin at 8:30 or later. With this increased attention to later school start times, many studies have investigated relationships between school start times and various students' outcomes. With respect to mood and substance use outcomes, while some correlational studies show a relationship between later start times and better mood and lower substance use, recent reviews suggest that strong evidence of the causal impact of school start time policies is missing. To fill this gap in knowledge, the present study exploiting a natural experiment in Korea that delayed school start times to between 8:30 and 9:00 am and leveraged a nationally-representative longitudinal survey to examine the impact of the policy on adolescents' sleep and substance use. Data and Participation: This study used the first to sixth waves of the Korean Children and Youth Panel Survey 2010 (KCYPS), which is longitudinal, nationally representative data of 7th graders in 2010. Administered by the National Youth Policy Institute, KCYPS annually tracked 2351 7th graders from 2010 to 2016, a year after they graduated from high school. KCYPS collected information on demographics, sleep, socio-emotional well-being, health, and developmental environment including family background. Importantly, this survey includes data on students' outcomes before and after a later school start time policy. Our outcomes of interest were sleep and substance use. The analytic sample was limited to students in either treatment regions or comparison regions, and students who moved across regions across waves were excluded. Since the final survey (i.e., 7th wave) was conducted after high school graduation when students were no longer exposed to the policy, it is excluded from the analysis. The final analytic sample sizes varied across models because some outcomes were surveyed only at some waves. To be specific, the final sample sizes were 6,324 observations from 1,133 students for models with sleep outcomes and 5,279 observations from 1,120 students for substance use outcomes. Policy Intervention: In September 2014, Gyeonggi province in Korea implemented a 9:00 am school start policy, which requires all elementary and secondary schools in the region to begin at 9:00 am. Following Gyeonggi province, some other regions implemented similar policies: Jeollabuk-do delayed school start time by 30 minutes in 2014, and Gwangju and Incheon delayed school start times to 8:30-9:00 and 8:40-9:00 am in 2015, respectively. Given that more than half of schools started before 8:00 am prior to policy adoption, school start times have been delayed approximately 30 minutes to one hour as a result of the policy. Research Design and Analysis: We employed a difference-in-differences (DD) design to estimate the impacts of later school start times on students' sleep and substance use, leveraging variation in a later school start time policy adoption across regions and years. Specifically, using a two-way fixed effect (TWFE) approach, which is an extension of the canonical 2X2 DD design, we compared changes in outcomes of adolescents before and after a later school start time policy adoption in treatment regions with changes in comparison regions. Our main model includes individual and year fixed effects to adjust time-invariant factors and general trends in outcomes. In addition to the main model, we estimated a model with time-varying covariates to account for time-varying individual-level factors. We clustered the standard errors at the region level to account for the nested characteristics of data and the level of treatment (i.e., students within regions) and account for heteroscedasticity. Also, we used sampling weights to make the analysis nationally representative. We conducted several supplemental analyses to better understand dynamic effects of later school start times and to check the robustness of our findings. First, we conducted event-study analyses of sleep and substance outcomes to identify the dynamic effect of later school start times and to examine the pre-treatment parallel trend assumption--that in the absence of a later school start time policy, trends in treatment and comparison regions would have been similar. In this model specification, we included pre-policy years (i.e., leads) and distinct post-policy years (i.e., lags) instead of one treatment indicator. Next, we incorporated recent advances in the econometrics literature to address potential bias in the TWFE approach. Findings: Adoption of a later school start time policy was initially associated with 20 minutes increase in sleep duration driven by delayed wake time and consistent bedtime. In addition, the policy was associated with statistically significant reductions in monthly smoking and drinking frequencies. However, approximately a year after implementation, the observed increase in sleep duration shrank to 6 minutes and became statistically nonsignificant. Similarly, the observed reduction in smoking and drinking was attenuated a year after. Conclusion: We demonstrated that policies that delay school start times can have health benefits beyond increases in sleep duration--getting more sleep can reduce unhealthy behaviors for adolescents. Our study also provides the first evidence of the benefits of delayed school start time policies in East Asia, and has insights on the sleep-health link for policymakers across Global regions. However, our results also serve as a warning to policymakers that, without measures to ensure the persistence of increased sleep, the benefits of delayed start times may be relatively short-lived.

Background/Context: In order to stimulate national and regional economic growth, policymakers and businesses have expressed great interest in growing the supply of workers with skills in science, technology, engineering, and math (STEM). At the same time, considerable efforts are being devoted to narrowing race and gender gaps in the STEM training pipeline. A key link in this pipeline is high school, where learning experience plays a critical role in shaping student's interest in STEM careers (e.g., Bottia et al, 2015; Corin, et al 2020). This is also the period when females begin to lose interest in STEM (Sadler, et al., 2012). To improve STEM education, one of the national initiatives during the early 2010's focused on integrated applied STEM curricula (Gottfried & Bozick, 2016). By connecting the traditional STEM disciplines to the real-world/applied problems, the goal was to increase student participation in more meaningful and engaging STEM learning and generate greater interest in STEM or related careers. Project Lead the Way (PLTW) is one such program. PLTW began over two decades ago in a handful of high schools as career and technical education (CTE). Since then program has grown rapidly and is currently adopted by over 12,000 high schools nationally. In Missouri, the site of this study, the program expanded from 13 schools in 2005 to 174 schools by 2020--roughly 35% of all public high schools. Purpose/Objective/Research Question: Rigorous empirical evidence on applied STEM program is limited. This study investigates the impact of PLTW program offering (ITT) and the participation impact on participants (TOT). The outcomes include college enrollment, STEM major declaration, overall degree attainment, and STEM degree attainment. Setting: Missouri has 537 traditional and charter public high schools, and 26% of them are in urban and suburban areas, and 18% and 57% are in town and rural areas, respectively. Population/Participants/Subjects: We use three cohorts of first-time 9th-grade students who began high school in AY2010, AY2013, and AY2014 (N? 68,000 per cohort). Racial compositions are 75.8% White, 16.5% Black, 4.2% Hispanic, and 3.6% others. The FRL rate is 42.4%. The analytic population consists of students in two types of schools: schools that began offering PLTW for the 2013 and/or 2014 cohorts, but not for the 2010 cohort (N=92 schools and 43,578 students) and schools that never offered PLTW for the three cohorts (N=273 schools and 49,788 students). The two types of schools are academically similar, but they differ considerably in demographic characteristics: PLTW schools are larger in size; are located in urban and suburban areas; have larger Black or Hispanic student populations; and have lower FRL rates than non-PLTW schools (Table 1). Intervention/Program/Practice: PLTW is designed to provide active, project-based learning in the largest three applied STEM fields--Engineering, Computer Science, and Biomedical Science ("Pathways"). Each pathway begins with foundational courses. They provide an overview and introduce major ideas of the field with a goal of developing an enthusiasm for further study. Advanced courses are designed to expand students' understanding of the field through deeper and specialized content. The Engineering and Biomedical Science pathways end with a capstone course which requires students to take their own idea from design through development of a product or plans to produce one. Research Design: First, we apply a Difference-in-Differences analysis (DiD) to estimate the impact of program offering for PLTW schools. This is estimated by deciles of "prognostic scores"--predicted probability of completing a STEM degree given pre-treatment predictors (math and science test scores and GPAs) . Students in non-PLTW schools serve as a comparison group, providing an estimate of the outcome difference between the 2010 and later cohorts in the absence of PLTW. Next, the propensity-score adjusted DiD is used to estimate the impact of program participation on participants (TOT). Propensity scores--the probability of program participation given pre-treatment covariates-- are used to construct the "would be treated" and "would be untreated" populations from the 2010 cohort who are made similar to, respectively, the 2013/2014 program participants and non-participants in the same PLTW school. The would-be-untreated and observed-untreated students and students in non-PLTW schools are two comparison groups for the DiD analysis. In additional DiD analysis, these two comparison groups are weighted/matched to resemble the program participants. Data Collection and Analysis: Student data are provided by Missouri Department of Elementary and Secondary Education. The K-12 data are linked to National Student Clearinghouse data for post-secondary outcomes. The ITT impact for student i, cohort c, and school j is estimated as; Y_icj=d_ij+ [beta]eta_cd(d_ij*PLTW_j*Coh_c)+G(X_ij)+ Coh_c*d_ij+ S_j+ e_ij, where dij are indicators for prognostic-score deciles; PLTW distinguishes PLTW and non-PLTW schools; X is student covariates; Coh_c is cohort indicators; s_j is school fixed effects. The coefficients, [beta]_c_d, indicate the cohort- and decile-specific ITT impact. The ITT analysis avoids selection bias around program participation, but estimates may be small due to low participation rates, especially for the 2013 cohort (Table 2). The model for TOT impact is; Y_icj=a0+ a1(Untreat_coh10,ij) + a2 (Treat_coh10,ij))+ a3(Coh_c )+ a4(Coh_c*Untreat_ij ) + delta(Coh_c*Treat_ij)+a5(X_ij)+ e_ij,, Where a0 is the average outcome for the 2010 students in non-PLTW schools; Untreated_coh10 and Treated_coh10 are, respectively, the PS-matched/weighted "would be untreated" and "would be treated" 2010 students in the PLTW school. The program impact is given by delta. Note that a_3=a_4 under the paralell trends assumption. Results: First, Table3 presents base-line outcomes. Table 4 shows positive ITT effects on STEM major declaration and STEM degree completion for the top deciles. Program participants also had higher STEM enrollment and degree completion (Tables 5 and 6). Conclusions: Our evidence suggests that an applied STEM program like PLTW has a potential to increase post-secondary TEM outcomes among students with higher STEM readiness. The study's limitations include bias due to unobserved confounders. We will conduct sensitivity analyses to address this issue.

Study after study has demonstrated that postsecondary success involves a range of factors from neighborhood context to family history to individual aspiration (Kim & Rifelj, 2021). Kauffman Scholars Incorporated (KSI), a post-secondary scholarship program based in Kansas City, understands these factors and has tailored its programming accordingly. Throughout its 20-year history, KSI has focused on supporting low-income students in attaining postsecondary credentials. Whereas the program has changed over time, it has consistently offered college scholarships, academic tutoring, and mentoring to students starting in 7 grade. Over 800 Scholars have earned college degrees since KSI's inception. KSI's goals extend beyond participating individuals. They also aim to create a ripple effect--having impacts on people in Scholars' lives, community organizations, and the local career-professional pipeline. These goals are reflected in wrap-around services such as one-on-one coaches and community partners who guide KSI scholars and their families through barriers that often derail students from their postsecondary track. th NORC and Bowman Performance Consulting (the NORC team) have joined together to evaluate KSI's impact over the past two decades. The traditional linear cause-effect evaluation methodology is insufficient to fully understand and describe the complex nature of transformational change at this scale. Understanding system-wide impacts requires holistic thinking, innovative and flexible approaches, and a radical commitment to partnership with all stakeholders. Our research approach, which draws upon the Culturally Responsive Indigenous Framework and Equitable Evaluation, meets these criteria. We think of this as telling the story of the ripple effect of KSI over twenty years. We will examine impacts from the direct source (the Scholar), rippling out to their families, their other connections, and ultimately, to the broader community of Kansas City. In our evaluation study, we: (1) investigate to what extent KSI achieved its goals, (2) describe what factors enabled or constrained KSI scholars and alumni in achieving KSI goals, discuss what ways KSI impacted Kauffman Foundation strategies, investments, and current work, and (3) delineate the ways KSI Scholars and alumni have impacted the local community. This study is ongoing and will be completed by September 2023. This proposal describes our methodological approach and the purposeful ways we embed equity throughout the research process. We will be ready to present findings in the Fall. Data: Nobody understands a person's story better than the person who has lived it. As such, we purposefully embed participant-centered ideals that honor the tenets and conceptual framing for culturally responsive practices (Ladson-Billings, 2008; Bowman, 2018) with a focus on interpersonal, experiential, and consequential validity (Kirkhart, 2010) throughout our approach. The NORC team will conduct Storytelling interviews with 20 past and present Scholars. Each Scholar will identify a person in their lives to interview, gathering their story of KSI's impacts beyond the individual Scholars. We empower the Scholar in multiple ways throughout the process. First, we co-construct the Storytelling protocol with each Scholar, and practice the skill of interviewing another person, allowing their story to be told in their own words. Part of this practice will include participating in an interview (as the interviewee), where a member of the NORC team will interview the Scholar about their experiences with KSI. Second, we put Storytelling agency in the Scholars' hands, asking them to select a connection in their lives to interview, helping us understand the ripple effect of KSI across people. This approach employs Storytelling to build relationships across researchers, Scholars, and the community. It also grounds the evaluation work in the nuanced contextual elements that shaped KSI's impact over time, surfacing and infusing meaning throughout the data collection and analysis process. Our project also includes a survey of the 2,000+ current and former Scholars, a survey of household members of these Scholars, a survey of community organizations impacted by KSI, interviews with KSI staff, and secondary data collected by Kauffman on their Scholars over time. The Storytelling interviews are the heart of the study--they provide the story of KSI's impact from those who lived and continue to live through the experience. The other data contextualize the Storytelling data, providing important information about the demographic composition of the Scholars, programmatic goals and reasons for programmatic design decisions, and data on KSI's ripple effects on the community and Scholars' family units. Methods: Storytelling Analysis: The NORC Team will utilize a combination of inductive and deductive coding strategies to synthesize themes and describes KSI's range of impacts over time. We will utilize KSI's vision, mission, and value statements as normative guideposts through which to analyze the storytelling data (e.g., how do the values of the KSI program express themselves throughout KSI Scholar and Scholar connection stories?). All primary data collection tools include questions that mirror the Storytelling Interview questions. We will first examine responses to these questions, honing in on what parts of KSI were most impactful for Scholars as well as for the local community. We will create a cross-data matrix, comparing responses across participant type, surfacing themes of commonality and divergence. By analyzing data in this way, we gain the benefit of extra nuance and detail by drawing on Scholars' lived experiences, the experiences of chosen ones in their lives, the stories of the KSI staff who supported them, and information from the communities they live within. Landscape Analysis: Analyzing KSI's secondary data will provide a landscape of KSI's impact contextualize the stories we collect. We will also draw on the Scholars, family member, and community stakeholders survey data. The survey data allows us to identify specific program features that were associated with program satisfaction and other outcome measures. We will also compare across Scholar groups to hone in on how programmatic elements shaped Scholars' lives. Conclusion: Evaluation is, at its heart, telling the stories of people's lives in ways that are authentic, meaningful, and participant-centered. Our project employs an intentionally responsive evaluation design, centering participants while purposefully placing agency for telling their stories in their hands. We strongly believe that embedding equity and repositioning the roles of researchers, participants, and community in these ways deeply strengthen the research process from beginning to end.

Background/Context: The proposed paper will share findings from a dissertation study examining mindfulness as a tool to reduce the impacts of educator bias on school discipline disparities. Teachers are the single most important factor in determining student achievement compared to any other aspect of schooling, yet studies are finding that the majority of teachers hold implicit biases much like the general population (Starck et al., 2020). Despite often good intentions, these biases can result in the differential treatment of students of color and students with disabilities, leading to disparate behavioral and academic outcomes (Losen, 2018). For instance, Black students diagnosed with an emotional/behavioral disability are punished more frequently and severely for the same behaviors compared to their White peers with and without disabilities (Losen, 2018). Mindfulness offers a novel approach to addressing the mental conditions in educators that can lead to biased disciplinary decisions in the classroom (Chang et al., 2022). Purpose/Objective/Research Question: In the proposed paper presentation, I will share findings from a pre-registered, double-blind randomized field experiment with teachers testing the hypothesis that a brief mindfulness training may reduce the impacts of bias in educators' discipline decisions. The presentation will answer the following research questions: (1) Does mindfulness training positively affect participants' interpretations of student behaviors and recommended use of punitive discipline? (2) Is a reduction in punitive discipline mediated by a reduction in the likelihood to interpret the students' behavior as problematic overall? Within Black and White student subgroups? (3) Does mindfulness training reduce disparities between Black and White student groups with and without disabilities as measured by participants' interpretations of student behaviors and their recommended use of punitive discipline? Setting: The study was conducted online through the Qualtrics platform. Teachers were invited to participate in the study via email. Interested teachers were first invited to complete a brief screener survey to determine their eligibility and then were taken to the Qualtrics platform to participate in the intervention or control activity. Population/Participants/Subjects: Secondary teachers (n=412) from public middle and high schools in four urban districts were recruited to participate in the study. The districts were selected intentionally given their publicly reported disparities in suspensions between Black and White male students with and without disabilities (identified through the Civil Rights Data Collection). Intervention/Program/Practice: Teachers who were eligible to participate in the study first answered pretest questions on their state of mindfulness and discipline philosophy (5 minutes). Teachers were then randomized at the individual level by treatment (Mindfulness vs. Control) and vignette race (Black vs. White) (20 minutes). The intervention included a brief introduction to mindfulness practice and included a guided mindfulness meditation exercise adapted from prior mindfulness-based interventions (Hanley et al., 2020). The control exercise entailed learning about the benefits of technology in the classroom, informed by prior intervention studies of a similar paradigm (Okonofua et al., 2022). After completing the exercise, all participants then responded to two vignettes depicting a student with and without a disability and the order was randomly counterbalanced. Participants then responded to post-test questions assessing their state of mindfulness, bias and basic demographic details (5 minutes) and were debriefed. Research Design: The research design entailed a double blind randomized experiment and followed a 2 (Condition: Treatment x Control) x 2 (Student Race: White x Black) x 2 (Student Disability Status: With x Without) mixed between and within subjects experimental design. Participants were randomly assigned at the individual level to receive either the treatment or control condition (between subjects) and to review two student vignettes depicting either Black or White students (between subjects). All participants viewed one vignette in which the student has a learning/behavioral disability, as referenced in the vignette materials, and one vignette in which the student did not have a disability (within subjects). The order of viewing a student with or without a disability was counterbalanced. Data Collection and Analysis: In response to the first research question, independent samples t-tests and regression were used to assess differences in interpretations of student behavior and recommendations for punitive discipline between the treatment and control groups. For regression, I created a dummy variable for the treatment and control group, which served as the independent variable predicting the primary outcomes (interpretations and recommendations). I included pre-test covariates assessing participants' state of mindfulness and discipline philosophy in the model. In response to the second research question, mediation analyses were performed in all student groups combined and within subgroups by student race. In response to the third research question, independent sample t-tests were conducted to compare group means between Black and White students with and without disabilities in the treatment and control conditions. The data was then reorganized from wide into long format, enabling disability and race to be treated as separate variables. A regression analysis was then performed to examine whether there was a main effect of race or disability status on the interpretation and recommendation outcomes as well as whether there was a significant interaction between group assignment and race and disability status. Findings/Results: In response to the first research question, regression indicated that the treatment/control dummy variable significantly predicted participants' interpretations of student behavior and recommendations for punitive discipline. In response to the second research question, a partial mediation was found overall and for the White student subgroup. A full mediation was found for the Black student subgroup. In response to the third research question, independent sample t-tests revealed that differences between the Black and White student groups in the treatment and control conditions were not significant, with the exception of the difference between the Black and White student without a disability in the treatment condition, which was significant. Interestingly, the White student without a disability received the harshest punishment across the treatment and control groups, followed by the Black student without a disability, White student with a disability and Black student with a disability. Additionally, regression also revealed that disability status was a significant predictor for the interpretation composite variable and race was not. The interaction of race and disability status as well as the three-way interaction between group assignment, race and disability status were also not significant for the interpretation of behavior. For the recommendation composite variable, regression revealed that both race and disability status were significant predictors. The interaction terms, however, were not significant. Conclusions: There are several limitations in this study. First, the online delivery mechanism may not approximate real-life classroom scenarios adequately. Responding to vignettes, for example, presents challenges to the transferability of the intervention effects to classrooms. Nevertheless, the findings of this study are encouraging, suggesting that mindfulness interventions may be a promising approach to shifting teachers' negative interpretations of the behaviors of Black and White students with and without disabilities and their subsequent use of punitive discipline. While the results did not indicate a reduction in disparities, the intervention group did interpret the student behaviors within each student subgroup significantly less negatively and responded with significantly less punitive discipline, which suggests mindfulness training may positively impact disciplinary outcomes for all students. Going forward, there is an opportunity to study the effects of the intervention in school contexts and to examine discipline outcomes in school settings following the intervention to assess its transferability to classrooms.

Advocates for the use of a multi-measure system of performance evaluation argue that multiple measures may better capture meaningful differences in employee effectiveness and help align effort with organizational objectives. This may be particularly important in organizations such as schools that produce multiple outcomes. In this paper I use a leave-year-out estimation approach, similar to that outlined by Chetty, Friedman and Rockoff (2014), and estimate the relative contributions of the subjective (supervisor observations and student surveys) versus objective (value-added) evaluation measures in explaining teacher effectiveness at increasing cognitive and non-cognitive skills. I use data from a large urban public school district where teacher compensation is tied to the evaluation measures. Estimates reveal that the subjective measures provide information on teacher effects on short-run achievement and behaviors and long-run achievement in ways that value-added does not. Under the multi-measure system, test score value-added has highly significant relationships with achievement and absences in the current and subsequent year. Furthermore, teachers with high supervisor observation scores increase contemporaneous and subsequent year achievement. Not only are the supervisor observations related to achievement, but they also explain variation on top of test score value-added for both the contemporaneous test scores and subsequent year reading. Interestingly, student survey scores also capture current and subsequent year achievement, and they remain significantly related to contemporaneous absences even after controlling for the other evaluation measures. In fact, of the three high-stakes evaluation measures, teachers with high student survey scores are the most effective in reducing contemporaneous absences. The student survey effects on achievement and absences are more pronounced for students with low prior achievement and high prior absences. This finding is consistent with the idea that a caring teacher may particularly matter more for the low performing students and has implications for equity. The pattern of results is similar between subsequent year and high school outcomes, with the exception that supervisor observations are not significantly related to any of the high school outcomes when conditioned on test score value-added and survey score, while student surveys provide some additional information for Algebra I test scores. High survey score teachers also increase the probability of students attending a public high school within the district. Findings in this paper establish a foundation for the larger investigation of the effects of the evaluation and compensation reform on student outcomes.

Background/Context: Taking even one Advanced Placement (AP), International Baccalaureate (IB), dual enrollment, or other college-preparatory course is positively associated with student outcomes including test scores; high school graduation; and college matriculation, major choice, and completion (Adelman, 2006; An & Taylor, 2019; Author, 2017, 2020; Avery et al., 2018; Conger et al., 2021; Jackson, 2014). Benefits are especially large for students from historically underrepresented backgrounds (An, 2013a, 2013b; Lee et al., 2022; Long et al., 2012). However, large inequities in advanced course participation persist, despite high postsecondary aspirations among students of color and students from low-income backgrounds (Schneider & Saw, 2016). These inequities are substantially driven by within-school inequities in advanced course enrollments (Kolluri, 2018; Patrick et al., 2020; Xu et al., 2021). Purpose/Research Question: We examine whether an automatic enrollment policy in Washington state increases access to advanced courses for students who have historically been underrepresented in these courses. Setting: Academic Acceleration was first implemented in 2010-11 by an urban district serving about 20,000 students south of Seattle. Two years later, the Washington state legislature passed a law encouraging all districts to adopt the policy. Between 2013-14 and 2015-16, 72 of Washington's 263 districts with at least one high school adopted the policy. In 2019, the legislature updated the policy and mandated that all districts adopt Academic Acceleration by 2021-22. Intervention/Program/Practice: Academic Acceleration attempts to increase equity in advanced course enrollment in two ways. First, it requires high schools to automatically identify students as eligible for advanced courses if they met the state standard on statewide assessments, rather than relying on teacher, counselor, student, or parent input. Second, it makes advanced courses opt out rather than opt in for qualified students by requiring schools to automatically enroll all qualified students into the next highest level of coursework in the same or related subject(s). Research Design: We employed a difference-in-differences model to examine changes in course enrollment in policy-implementing districts compared to changes in comparison districts. We define advanced courses as AP, IB, Cambridge International, Running Start (a dual enrollment program at community colleges), or College in the High School (a dual credit program in high schools). We also examined GPA, enrollment in a subsequent advanced course in grade 12 (for grade 11 students), and high school graduation. We estimated separate models by advanced ELA or social studies courses, advanced mathematics courses, and any advanced course regardless of subject area for students in grades 11 and 12. We estimated differential outcomes for qualified students and students in the URM and FRPL groups using two- and three-way interactions. We used linear probability models, implemented two-way fixed effects, and clustered standard errors at the district level (the level of policy variation). Data Collection and Analysis: We used administrative, student-level data from the Washington Office of Superintendent of Public Instruction for all 11th- and 12th-grade students in Washington public schools from 2010-11 through 2018-19. 263 Washington district have at least one high school. We excluded two very early adopting districts and 15 late adopting districts for insufficient pre- or post-policy data. We also excluded six districts with no pre-period high school enrollment. We generated a matched analytic sample of 72 districts in our three implementation cohorts and 168 comparison districts that did not adopt Academic Acceleration before the 2019 legislative update. The prepolicy implementation trends fail the parallel assumptions test for the 2015 and 2016 cohorts. There is an upward trend in the rate of advanced course taking in treatment districts before implementation but not comparison districts, and the formal test (De Chaisemartin & d'Haultfoeuille, 2020) rejects the null hypothesis of parallel trends (p = 0.03). Propensity score matching did not resolve the violations As a result, our findings should be viewed as descriptive, not causal. Findings/Results: Academic Acceleration was associated with a 5.3 percentage point increase in the probability of enrollment in any advanced course for all students, regardless of qualified status, and a 4.8 percentage point increase in the probability of advanced mathematics courses enrollment for qualified students. It also was associated with a 5.1 percentage point increase in qualified students' chances of taking any advanced course in the 2017 cohort. In most cases, Academic Acceleration was associated with a similar change in the odds of advanced course enrollment for qualified URM or FRPL-eligible students and qualified non-URM or non-FRPL-eligible students. In the 2017 cohort, however, qualified non-URM and non-FRPL-eligible students in policy-implementing districts experienced 7.4 and 6.5 percentage point larger increases in the probability of taking advanced courses, respectively, than qualified URM and FRPL-eligible students. Across cohorts, policy implementation was associated with a 2 percentage point larger increase in ELA/social studies enrollment, and with a 3.1 and 3.8 percentage point larger increase in mathematics enrollment, for URM and FRPL-eligible students, respectively, compared with non-URM and non-FRPL-eligible students. Taken together, these changes translate into narrowed enrollment differences by URM and FRPL status. Notably, treated districts experienced increases in advanced course enrollment for URM groups relative to non-URM groups regardless of qualified status. Students' performance in courses targeted by the policy and other courses remained similar or showed small increases from pre- to post-policy. The policy also was associated with a 4.6 percentage-point higher probability of enrolling in another advanced math course in 12th grade (for 11th graders), and no difference in the probability of on-time high school graduation. Conclusions: Increases in the probability of advanced mathematics course enrollment among qualified students in treatment districts suggest that Academic Acceleration may have identified and enrolled qualified students who would not have enrolled before policy implementation. Interestingly, the increases in any advanced course enrollment for all students (qualified and not qualified) suggest that districts with Academic Acceleration policies also may have broadened participation by expanding their eligibility criteria beyond standardized test scores or encouraging students to try an advanced course outside the core subjects. Findings beyond enrollment indicate the policy increased access to advanced courses without decreasing student performance. However, demographic inequities remained, especially in advanced mathematics.

Background: This paper considers a new policy and statistical framework for evaluating K-12 schools and policies that aligns with diversity, equity, and inclusion values. The new approach broadens the standard approach to accountability and evaluation by combining features of evaluation and multi-level growth models with approaches used in systems of early warning, college/career/workforce readiness, and predictive analytics. A hallmark of these latter systems is that they focus on student outcomes beyond student test scores, including student attendance and suspensions, enrollment in challenging courses, grades, and high school and college graduation. We expand the focus to outcomes such as participation in English language learner (ELL) and other programs. The expanded focus on a broader set of student outcomes supports a DEI focus by providing a more complete portrait of the malleable factors that affect students with very different demographic backgrounds. An important feature of the new framework is that by focusing on post high school college and career outcomes, the framework connects K-12 outcomes to the long-term outcomes that students, parents, and educators value. Research Questions: (RQ1) How is it possible to connect K-12 outcomes at the elementary, middle, and high school grades with medium and long-term outcomes that range from reading proficiency in 3rd grade to post high school outcomes such as college graduation? (RQ2) How is it possible to incorporate extensive K-12 data in each grade into models of high school and post high school outcomes to promote statistical precision, transparency, and feasibility? (RQ3) What are the appropriate statistical tools to highlight the DEI dimensions of K-12 student and school-level outcomes? Setting and Participants: The paper addresses key policy and statistical issues and progress in developing and implementing a DEI-aligned evaluation and accountability system in collaboration with four large, but diverse, school districts. Two of the districts are especially interested in extending these systems to incorporate Pre-K-3 data, which has less commonly been included in these systems, given the bias toward federally required assessments in grades 3-8. The other two districts are especially interested connecting middle and high school outcomes and experiences to post high school outcomes. Research Design: A key objective of the research is to calibrate model parameters that connect K-12 outcomes with long-term outcomes. These calibrated coefficients can then be used to evaluate college and career readiness of middle and high school students while the students are still enrolled in school, rather than waiting until these long-term outcomes are observed. To address RQ1, it is necessary to connect K-12 outcomes with long-term outcomes, time spans that in some cases are exceptionally long. The simplest strategy is to use long-span longitudinal panels to connect, for example, 8th grade outcomes with college graduation, a period that covers 11 years of data. This is generally not the preferred approach, however, because longitudinal data is often not available for that length of time and calibrated parameter estimates based on long-span data reflect school policies and contextual factors (such as the COVID pandemic) that likely fail to capture the up-to-date relationships between the outcomes. Our preferred approach is to estimate separate short and medium-span models and chain link them together to produce calibrated coefficients that reflect more up-to-date relationships between the variables. Our research addresses the econometric conditions that must be satisfied to validly implement this approach. A key condition is that the short and medium-span models must be connected using very strong variables. The connecting variables must be strong outcomes in the first model and strong predictors in the second model. We have demonstrated that data measured in the first year after nominal high school graduation satisfies this requirement. To address RQ2, we consider options for creating variables that efficiently and transparently incorporate: (a) potentially large amounts of information and (b) information that may not be available for all students. The first task is especially necessary for models based on high school transcript data where, in principle, student differences in measures and outcomes in all four grades could be salient. We consider alternative data reduction approaches to create parsimonious measures that focus on four major dimensions: attendance and suspensions, course taking, test scores, and grades. To address the second task, we employ multiple imputation to address missing data. This approach is especially valuable in K-3 models where different schools and types of students vary in their use and coverage of different data elements. To address RQ3, we expand the standard multilevel modelling approach to estimate nonlinear models that allow for flexible modelling of within and between school effects of different demographic variables. This approach allows us to assess the degree to which outcomes such as enrollment in Algebra differ due to between-school differences in poverty, within-school differences in poverty, or both. We find, for example, that differences in Algebra enrollment in one of our districts is strongly related to both student poverty and school-level poverty. This framework allows for a more detailed and nuanced measures of the efficacy of schools and policies. Data sources: The study uses student and school-level data on all students from all four districts included in the research. As discussed above, the study utilizes an extensive battery of data, including K-12 and post high school data. The latter data were obtained from the National Student Clearinghouse. In all districts, data is available from school years 2016-17 to 2022-23 (the latter available this Summer) and thus reflects student growth during three periods: pre-COVID, peak-COVID, and "post" COVID recovery. Findings and Conclusions: We have developed and demonstrated options for building a new policy and statistical framework for evaluating K-12 schools and policies that aligns with diversity, equity, and inclusion (DEI) values. To do so, we have addressed the three research questions considered above. The new framework incorporates a much richer set of variables than typically considered in evaluations and accountability systems and provides tools for considering the DEI dimensions of these systems.

Background/Context: Observational studies often employ regression discontinuity (RD) designs and multiple control-group designs to explore the causal quantities of interest. RD designs assess policy and program effectiveness by assigning subjects to treatment based on whether they exceed a pre-defined cutoff. RD designs are classified into two types depending on subjects' compliance with the assigned treatment status. In particular, when compliance is not complete, fuzzy RD designs are used to estimate the local average treatment effect (LATE) at the cutoff (Imbens & Lemieux, 2008). Multiple control groups arise when there are various reasons for students' non-participation in treatment, and their value depends on the availability of supplementary information about unobserved biases. For example, when one control group produces positive unobserved bias and the other produces negative bias, the two control groups can yield consistent and unbiased estimates of bounds on the treatment effect (Rosenbaum, 1987). However, despite the potential benefits of using multiple control groups, there has been limited research on their utilization in RD designs. Purpose/Objective: This paper aims to combine fuzzy RD designs and multiple control-group designs to evaluate the effect of extended time accommodations (ETA) on student performance. Two control groups are formed in ETA settings, one for students who did not receive ETA and another for students who received ETA but did not use it. Our approach enables causal identification and estimation of the average treatment effect on the treated (ATT) at the cutoff from the fuzzy RD design, along with its bounds formed by utilizing the two control groups, which serves as a sensitivity analysis. We also propose flexible estimators based on machine learning (ML) to construct the bounds on the ATT at the cutoff. The NAEP Assessment/Setting/Population/Program/Research Design/Data Analysis: Setting: The NAEP assessment used in this paper is the largest continuing and nationally representative assessment in the United States (Oranje & Kolstad, 2019). Population: Our study focuses on 8-th grade students with the advanced English language learner (ELL) proficiency who do not have disabilities. Program: The ETA program is designed to provide ELL or students with disabilities, up to three times the regular testing time during the NAEP assessment. Research Design: We employ our proposed RD design with multiple control groups to evaluate the effect of ETA on student achievement. We use ELL English proficiency categories as the running variable in the RD design. Data Analysis: We analyze data from the 2017 NAEP assessment for grade 8, along with process data containing approximately 28,000 students. Using process data, we identify students who made use of ETA during the test. Findings/Results: We present the key identifying assumptions for a fuzzy RD design and outline our estimation strategy for constructing causal bounds using multiple control groups. We define a binary treatment variable A[subscript ij] [element of] {0, 1} for student "I" from school "j," which is determined by a discrete running variable X[subscript ij]; specifically, A[subscript ij] = 1 if X[subscript ij] [less than or equal to] x[subscript c] and A[subscript ij] = 0 if X[subscript ij] > x[subscript c] where x[subscript c] is a cutoff score. We also define T[subscript ij] = 1 if student "ij" used ETA and T[subscript ij] = 0 otherwise, and define the observed outcome Y[subscript ij]. In addition to the four identifying assumptions for a fuzzy RD design with a discrete running variable (i.e., outcome regression function, treatment regression function, local monotonicity, and local exclusion restriction) (Lee & Card, 2008; Suk et al., 2022), we require another identifying assumption as: (A5) Local Absence of Always-Takers: Pr(T[subscript ij](1) = T[subscript ij](0) = 1 | X[subscript ij] = x[subscript c]) = 0. Assumption (A5) ensures that the LATE at the cutoff is equivalent to the ATT at the cutoff (denoted as [tau subscript ATT](x[subscript c])) because all the treated cases are compliers, rather than a mixture of compliers and always-takers. This indicates one-sided noncompliance. In our ETA context, students ineligible for ETA are highly unlikely to use ETA, and it renders the existence of always-takers impossible in our setting. To construct the bounds on the ATT at the cutoff, we employ two control groups in the ETA setting. We use T[subscript ij] [element of]{C[subscript 0], C[subscript 1], C[subscript 2]} to indicate the combined ETA receipt and use statuses where ETA users are denoted as C[subscript 0], ETA nonusers (though offered) denoted as C[subscript 1], and ETA non-receivers denoted as C[subscript 2]. We assume that in our ETA context, the first control group (C[subscript 1]) yields positive unobserved bias, whereas the second control group (C[subscript 2]) yields negative unobserved bias; see Figure 1 for a graphical representation of multiple control groups in the ETA setting. Using this information, we derive lower and upper bounds on the ATT at the cutoff as: [equation omitted]. Moreover, we propose utilizing flexible ML-based estimators to construct bounds on [tau subscript ATT](x[subscript c]) to prevent bias from model misspecification. To achieve this, we employ "off-the-shelf" ML-based causal inference methods, notably Bayesian additive regression trees (Hill, 2011), the targeted maximum likelihood estimator (van der Laan & Rubin, 2006) with ensemble learning, and causal forest (Wager & Athey, 2018), to obtain the conditional average treatment effects on the treated. With our causal assumptions in place, we estimate the ATT of ETA on math scores at the cutoff using the two-stage least square estimator and its bounds using the ML-based bound estimators based on the 2017 NAEP assessment and process data. Conclusions: This paper proposed a novel approach that utilizes a fuzzy RD design with multiple control groups to evaluate the ATT at the cutoff along with its bounds. In particular, the use of multiple control groups, identified through process data, enables the construction of the bounds that provide a useful sensitivity check. The proposed approach offers theoretical and practical insights into evaluating the effects of testing accommodations in education research.

This paper estimates the effects of earning industry-recognized certifications (IRCs) in high school on downstream outcomes. IRCs are a form of alternative education credential issued by industry groups or corporations to individuals seeking to acquire knowledge or skills in a particular sector. This credential type has grown rapidly among high school students as school systems incorporate them into accountability systems and students respond to labor market demand for skills. Since IRCs are awarded on the basis of an objective passing score, students who just barely fail or barely pass are likely to differ only in their likelihood of earning a certification. I thus use regression discontinuity design to estimate the effect of earning a certification on post-high-school intentions, postsecondary enrollment, and earnings. Conditional on first exam attempts, IRC earners are more likely to express their intent to attend a four-year-college and are just as likely to actually enroll--effects that operate through college readiness indicators and not career and technical education. However, within seven years following the first IRC attempt, students on the margin of passing earn no more than their counterparts. The results suggest that for marginal examinees, earning an IRC may represent a form of skill accumulation or signal that boosts interest in and enrollment in four-year college, but these effects do not translate into higher short-term earnings.

Background/Context: Access to rigorous courses in high school is essential for college success (ACT, 2005), but low-income students and students of color take fewer demanding courses than their white and Asian counterparts (Cowan Pitre, 2014; Theokas & Saaris, 2013). The issue of inequitable access to more advanced courses requires a deeper dive into course prerequisites, policies, and other requirements. This paper presents early results from a statewide longitudinal mixed methods study of high school students' course trajectories and the supports that schools provide to enhance college-readiness. Purpose: This study examines potential inequities in access to rigorous high school curricula based on school locale, demographics, race. The specific research questions are: 1. According to district course catalogs, what course trajectories do high schools offer, and what prerequisites does each school have that limits access to those courses? and 2. According to students' individual transcript data, what are students' actual course trajectories in high school mathematics, and how does the prevalence of trajectories into more rigorous courses vary across race groups? Setting: All data for this study come from the state of Delaware. Population: This study includes all Delaware high schools and more than 8,000 high school students from the graduating class of 2020. Practice: This paper focuses on practices and policies related to course offerings in English language arts (ELA) and mathematics in high school grades, including contextual differences across schools as well as differences in prerequisites that may support or hinder equitable access to rigorous curricula within some schools. Research Design: This is an observational study. The design is intended to identify and quantify current inequities in policy and practices through descriptive statistics, inferential tests of subgroup comparisons, and data visualizations. Data Collection and Analysis: For the course catalog analyses under research question 1, we consolidated data gathered from course catalogs of high schools across the state, and we then used qualitative coding to categorize each course (based on course name and course description) into five categories of rigor (college prep, honors, Advanced Placement, Dual-Enrollment, and International Baccalaureate). We then created interactive course trajectory models in RStudio using the "collapsibleTree" package to visualize all the available ELA and mathematics course trajectories at each of the 42 schools (see Figure 1 for an example). For the analyses of student transcript data under research question 2, we classified course titles and course codes in transcript data for more than 8,000 students using individual-level data from Delaware's statewide longitudinal data system (SLDS). To identify the most common trajectories and examine the flow of individual students' mathematical trajectories over their four years of high school, we produced Sankey diagrams of math courses from 9th through 12th grades (see Figure 2 for an example) across all students and schools, for individual schools, and for subgroups of students by race. Findings/Results: Results from the course-sequence models and advanced-course-offering profiles confirm that high schools have varying practices, rules, and prerequisites regarding ELA and mathematics course enrollment. Some schools have policies clearly restrict access to advanced courses, while other schools have few if any restrictions. Fisher's exact test revealed that differences in enrollment restrictions were found to be unrelated to racial composition or to average SAT scores. However, Title I schools were less likely (p<0.001) to offer unrestricted access. Results from the student trajectory analyses reveal that (a) there are many transitions where students jump up or down in course rigor, especially in their junior and senior years, (b) transitions are more prevalent in Math than in ELA, and (c) non-white students are less likely to transition into AP, Dual-Enrollment, or IB courses. Conclusions: The results presented in this paper suggest that access to challenging curriculum may not be equitably distributed across schools and school districts, and may not be equitably distributed across subgroups of students within some schools. However, the school-level differences identified here appear to be unrelated to three key demographic factors (e.g., percent non-white, percent passing the SAT college-ready benchmark, school size). The factors that explain why different high schools in Delaware provide more or less opportunity to enroll in advanced courses are one focus of the next stages of this five year research project.

Background: Over the past decade, schools across the United States increasingly turned to online learning (Gemin, Pape, Vashaw, & Watson, 2015; Queen & Lewis, 2011). While many espouse the promise of online credit recovery (OCR) courses, the abrupt shift to online learning during the COVID-19 pandemic renewed concerns about theses courses' effectiveness as tools for equitable outcomes (Tate & Warschauer, 2022). These concerns are exacerbated by a dearth of rigorous and mixed evidence about the effectiveness of OCR courses (Viano, 2018). A correlational study of OCR in Florida found that students in online courses were more likely to earn a C or better than students in face-to-face courses (Hughes, Zhou & Petscher, 2015). A study of North Carolina high school students who failed a course found that students who enrolled in OCR were more likely to graduate but had lower test scores than students who repeated the face-to-face version of the course (Viano & Henry, 2020). In addition, Heinrich et al. (2019) found that in a large urban midwestern district "online course-taking is not benefiting students or reflecting real learning" (p. 2,174). To date, there have been two randomized controlled trials of OCR. The first study found that students who took an OCR course in Chicago were less likely to earn credit and learned less than students who took a face-to-face course (Heppen et al., 2017), but found no statistically significant differences in longer-term outcomes, including high school graduation (Rickles et al., 2018). The second study found that students who took an OCR course in a large, urban school district were, on average, less likely to earn credit for ninth grade English than students who took a teacher-directed course but found no statistically significant differences in credit recovery rates in Algebra 1 or in content knowledge across both subject areas (Rickles et al., in press). Purpose: This paper builds off the second randomized study to examine how OCR affects longer-term high school outcomes relative to the business-as-usual (BAU) teacher-directed credit recovery classes. We address the following research questions: (1) What was the impact of taking OCR on students' credit accumulation over four years of high school?; (2) What was the impact of taking OCR on students' probability of on-time high school graduation?; (3) How did the longer-term effects of OCR vary across sites?; and (4) How did students' prior academic performance and engagement moderate the longer-term effects of OCR? Intervention: The intervention for this study was an Algebra 1 or English 9 online curriculum for the credit recovery course, where an online provider supplied the main course content and curriculum, and the school provided the credentialed in-class teacher who could supplement the digital instruction. The BAU classes primarily relied on traditional teacher-directed instruction, where teachers had latitude in the curriculum and instructional materials for the class. For both the intervention and BAU conditions, students took the class within a standard classroom during the district's 5-week summer session. Key characteristics of the online learning model tested in this study are presented in Exhibit A.1. The theory of action for the study is presented in Exhibit A.2. Population and Setting: The analyses in this paper are based on 613 students in 28 Algebra 1 classes across 13 high schools and 1,124 students in 70 English 9 classes across 19 high schools, all within a large urban district. All students entered 9th grade in the 2017-18 school year (Cohort 1) or the 2018-19 school year (Cohort 2) and failed their Algebra 1 and/or at least one semester of their English 9 course and enrolled in a credit recovery course the summer between their 9th and 10th grade year. For both cohorts, the credit recovery classes took place before the COVID-19 pandemic. Exhibit A.3 presents descriptive statistics and baseline equivalence for the Algebra 1 and English 9 student samples. Research Design: Students were randomly assigned to take their credit recovery course in an online class (treatment) or a BAU class (control). Random assignment took place within blocks defined by school, course, and cohort. In some schools, blocks were further defined by which semesters of the course the students failed during their 9th grade year. Data Collection and Analysis: We utilized extant district student background and course data for all analyses in this paper. Measures are described in Exhibit A.4. All analyses were conducted separately for Algebra and English. We estimated average treatment effects for the intent-to-treat student sample using regression models that control for student characteristics and randomization blocks. For the analysis of between-site heterogeneity, we estimated site-specific treatment effects by adding site-by-treatment interaction terms (fixed effects) to the main impact model specification. For the student-level moderator analysis we estimated separate moderation models for two indicators of prior academic performance (8th grade assessment scores and 9th grade GPA) and an indicator of school engagement (9th grade attendance). Findings: We are conducting analyses for this paper and therefore do not currently have findings to present. We expect to have full results by August 2023. Conclusions: The findings will provide needed evidence about the relative effectiveness of OCR to keep students on track toward graduation over time and obtain a high school degree. While earlier results from this study indicate that the online classes were more difficult to pass but may have improved short-term content knowledge for certain students in certain subject areas, it is not clear whether those proximal effects had implications for students' progress toward graduation. To help interpret the findings, the paper will use data from a larger sample of the district's students, focus group data from participating teachers, and interview data from students who took credit recovery during the pandemic to help situate the impact findings within the broader context of teaching, learning, and high school progression during the pandemic. Within this context, findings about the longer-term effects of OCR should inform future efforts to address the needs of students who fall behind early in their high school career.

Background/Context: There were over 500,000 open computing jobs across the country in 2018, but only 35% of U.S. high schools offer computer science classes, and only 8-10% of STEM graduates study computer science (Code.org, 2018). In 2021, only 5.1% of all Bachelor's degrees conferred were in Computer and Information Sciences (IPEDS, 2018). STEM is the future, but the present pipeline is insufficient to meet demand. K-12 Computer Science course participation is the leading indicator of what happens in university. The College Board found that students who try AP Computer Science courses in high school are 6 times more likely to study Computer Science in college, and participation is 7 to 10 times higher for minority and female students, respectively (Wyatt, Feng, & Ewing, 2020). Purpose/Objective Research Questions: This study evaluates the impact of an intervention that integrates increased access to Computer Science Principles (CSP) with curriculum and professional development support for CSP teachers. Specifically, we examine how the design of a computer science-integrated math curriculum, the Beauty and Joy of Computing (BJC), affects students' participation and performance in computer science in high school and in college. Sharing this knowledge will contribute significantly to the field, helping other secondary schools improve STEM preparation for underrepresented students nationwide. We aim to address three questions: what is the impact of enrollment in BJC Advanced Placement (AP) CSP curriculum on: 1) student AP CSP test scores, 2) student computer science attitudes and engagement, and 3) AP CSP enrollment at the school? Setting: This intervention was implemented at IDEA Public Schools, a network of charter schools based in Texas. Since 2000, IDEA has grown from a single campus serving 150 students in the Rio Grande Valley to its current enrollment of 44,653 students across 79 schools in two states and is the fastest- growing network of tuition-free, Pre-K-12 public charter schools in the United States. In spring 2021, IDEA high schools were randomly assigned to one of two AP CSP curricula in the 2021-2022 academic year, 15 schools to receive Code.org, the usual curriculum, and 15 schools to receive the Beauty and Joy of Computing (BJC). Population/Participants/Subjects: IDEA's current student population across all schools is 89.2% Hispanic, 5.5% African American, 88.6% economically disadvantaged, and 32.9% English-language learners. Additionally, 45.9% are considered at-risk for dropping out of school due to multiple academic, social, and economic factors. Our target population is comprised of students, between the 9th and 12th grade, who requested to take the AP CSP course in the spring before randomization, students who did not make a spring course request, even if they enrolled into the course later, are not included in the evaluation. Intervention/Program/Practice: After randomization, teachers in schools that were assigned the BJC curriculum participated in a one- week training at the beginning of the school year. Throughout the year, teachers attended bi-weekly webinars facilitated by the CS curriculum manager at each school. Teachers were also able to collaborate and communicate through online communication channels. Teachers primarily taught with their school's adopted materials in fall 2021, but frequently used other supplementary materials in spring of 2021. Research Design: This study utilizes a Randomized Controlled Trail (RCT) design to estimate the impacts of the BJC AP CSP curriculum. Specifically, we first conduct school-level randomization blocks and then conduct a constrained randomization to balance covariates. This is to ensure that the treated and comparison schools are statistically similar at baseline. The school blocks are created based on schools' prior history of offering an AP CSP curriculum. We then implement procedures described by Morgan and Rubin (2012) to further constrain the set of possible randomizations and achieve optimal balance for two key continuous covariates: (a) prior mathematics standardized achievement and (b) prior overall grade point average (GPA). Based on the district's historical data, these two covariates emerged as the strongest predictors of our key outcome (i.e., AP CSP scores). Our main impact model is a two-level mixed-effects model estimating treatment-control differences for continuous AP CSP scores, controlling for baseline covariates and fixed effects of randomization blocks. We also conduct exploratory subgroup analyses based on student-level characteristics, such as student grade level, gender, or prior mathematics achievement. Data Collection in Analysis: We collect administrative records from IDEA public schools that include information on student enrollment, demographic characteristics, student test scores, as well as school-level demographics and prior achievement. We also conduct a survey in both the fall of 2021 and spring of 2022. In the survey, we collect data from students on their computer science engagement, computer science confidence and interests, and computer science postsecondary and career intentions. Findings/Results: About half of students initially enrolled in spring 2021 did not remain in the course or otherwise did not take the AP CSP test in spring 2022. Our baseline equivalence analyses focus on student who did remain in the AP CSP course (i.e., those with AP CSP test scores as outcomes). The main concern with attrition is that different types of students might have left the BJC and Code.org conditions, however, findings from our baseline equivalence tests provide evidence that the two groups overall remain comparable. Also, where there are some slight differences (e.g., limited English proficiency rates), we statistically control for them by including those variables as covariates in our regression model. The average AP CSP scores were low in both conditions (compared to the national average of 2.91). About one-quarter of test takers (25%) earned a passing score of 3 or higher (compared to national average of 64%). The results of our main model show that there were no statistically significant differences between the BJC and Code.org groups, nor do we find statistically significant differences for subgroups. Although the study survey response rates were low, the results of analyzing the survey data show that students in the treatment group responded more favorably on all student engagement constructs than comparison students, the size of the differences between the treatment and comparison condition were small and none were statistically significant. There were also no statistically significant differences for CS interests, confidence, or postsecondary and career intentions.

Background/Context: Conditional cash transfers (CCTs) have become one of the most prevalent and rapidly growing social assistance strategies in developing countries. Typically, education-focused CCT programs provide monthly or bimonthly cash payments to students who meet specific attendance requirements. There is a wealth of literature documenting the positive effects of CCTs on outcomes such as enrollment, attendance, and dropout rates across various countries and contexts (see Garcia & Saavedra, 2017 for a comprehensive review). CCT interventions are often implemented across multiple school sites (e.g., Barrera-Osorio et al., 2011; 2019). While understanding the overall impact of a CCT program is essential for informing policy recommendations, it is also crucial to investigate questions related to cross-site variation in CCT effects in order to gain a deeper understanding of how CCTs function and under which circumstances they are most or least effective. In the present study, we employ rigorous analytical techniques to quantify and characterize the heterogeneity of CCT program impacts across different sites. As a case example, we utilize data from the Conditional Subsidies for School Attendance (Subsidios Condicionados a la Asistencia Escolar) program implemented in Bogota, Colombia. This experimental evaluation of a CCT intervention aimed to improve student retention, reduce dropout rates, and decrease child labor among low-income secondary school students (Barrera-Osorio et al., 2019). Purpose/Objective/Research Question: This study addresses three research questions: 1. What is the overall impact of CCT programs on secondary education outcomes, including on-time enrollment, dropout rates, and secondary graduation, as well as medium-term tertiary education enrollment in universities, vocational institutions, and unclassified establishments? 2. To what extent do the average treatment effects (ATEs) of CCT programs vary across school sites in terms of both secondary and tertiary education outcomes? 3. How much of the variation in CCT program effects across school sites can be attributed to "compositional" differences in student populations, such as family income and parents' education levels, as opposed to "contextual" differences in school-level characteristics? If all sites had the same population of students, what would be the ATEs for each site? Setting/Population/Intervention/Design: In 2005, the Secretary of Education in Bogota, Colombia's capital city, implemented the CCT program in two of the poorest districts among the 20: San Cristobal and Suba. In the San Cristobal district, students in grades 6 to 11 were randomly assigned to one of three CCT experimental groups: "basic" treatment, "savings" treatment, or control (refer to Barrera-Osorio et al., 2019 for details). In the Suba district, participants in grades 9 to 11 were randomly assigned to one of two groups: "tertiary" treatment or control. These two multisite experiments involved a total of 13,491 participants across 99 sites within the two districts (see Figures 1 and 2). Barrera-Osorio and colleagues combined the original program registration and randomization records with administrative data sources to track educational outcomes over time. As primary outcomes, we analyze on-time secondary school enrollment, secondary school exit exam participation, and tertiary enrollment by institution type. Data Analysis: To address Research Questions 1 and 2, the main challenge involves utilizing the observed heterogeneity in estimated effects (see Figures 3 and 4) as an indicator of heterogeneity in effects across a broader range of sites, while accounting for the sampling variation that contributes to some of the observed heterogeneity (Rubin, 1981; Meager, 2019). We employ the Bayesian hierarchical framework to distinguish genuine heterogeneity from sampling variation (refer to the Appendix for details). To quantify the heterogeneity of site-specific treatment effects of CCT programs, we calculate and report the shrinkage factors for each outcome, which measure the proportion of total variation attributable to between-site variation relative to sampling variation. To address Research Question 3, we apply the reweighting method developed by Lu et al. (2021) to decompose treatment effect variation resulting from "compositional" differences in the distributions of student-level covariates from variation due to "contextual" differences in school-level features. Their method re-weights (or "transports") each school to have a common distribution of student-level covariates, allowing the remaining variation in impact to capture contextual differences between sites. We compute the approximate balancing weights using the balancer R package. Findings/Conclusions: "The general impact of CCTs on educational outcomes." Our findings show that the ATEs on secondary and tertiary education outcomes generally align with the patterns reported by Barrera-Osorio et al. (2019). However, when considering cross-site impact variations, the estimated ATEs reveal an increased posterior probability of a null intervention effect due to uncertainty at both within-site and between-site levels (see Figures 5-8). The fixed-effects approach, which does not account for variation between sites, may lead to an underestimation of the predicted CCT effect's uncertainty in new or future contexts. "Treatment effect heterogeneity of CCT programs." The estimated shrinkage factor metrics range from 0.02 to 0.29, with the majority of values falling between 0.05 and 0.12. This indicates that approximately 88% to 95% of the initially observed variation in CCT treatment effects is due to within-site sampling variation (see Table 1). The cross-site impact variations, measured in effect size units, are estimated to range from 0.04 to 0.22, with the 25th and 75th percentiles being 0.07 and 0.14, respectively. According to the definition provided by Weiss et al. (2017), the cross-site impact variations observed in the CCT program are considered modest or moderate. "Decomposition of compositional and contextual effects." The reweighting did not significantly alter the distribution of site-specific ATEs; however, some site-specific ATE estimates were modified. We estimate that less than 8.2% of the variance in site-specific ATEs can be explained by observed differences in student-level covariates. Thus, we conclude that variations in impact across sites in these CCT experiments are primarily driven by contextual differences in site-level characteristics.

Background: Early Colleges (ECs) are a type of rapidly growing dual enrollment program proven to be successful in helping underrepresented students get a jumpstart on their postsecondary education. To date, the most rigorous evidence on the impact of ECs has come from two lottery-based natural experiments conducted by the SERVE Center (Edmunds et al., 2020) and American Institutes for Research (AIR) (Song et al., 2021), respectively. Both studies found a positive impact of ECs on a variety of student outcomes both during and after high school. Song and colleagues (2021), for example, found that students who won EC admission lotteries had significantly higher rates of college enrollment and degree attainment within 6 years of expected high school graduation than control students. Building on the original AIR EC impact study, this follow-up study assesses the longer-term impact of ECs on participants' workforce and life outcomes with data collected up to 14 years after expected high school graduation. Purpose and Research Questions: This study is intended to expand the existing evidence base on ECs by addressing the following research questions (RQs) about the longer-term impact of ECs--(1) What is the impact of ECs on workforce and other life and financial outcomes 12-14 years after expected high school graduation? and (2) Does the impact of ECs vary by participant background characteristics? Setting: The original AIR EC impact study included 10 ECs located across five states (North Carolina, Ohio, South Carolina, Texas, and Utah) that met certain selection criteria. This follow-up study focuses on seven ECs in four states, excluding three ECs from North Carolina because we did not have student identifying information needed to administer a follow-up survey to participants from those schools. All seven schools were relatively small high schools (i.e., fewer than 150 students per grade), with four schools in urban areas, two in mid-sized cities, and one in a small town. Participants: In total, 2,115 students (875 treatment and 1,240 control) participated in 19 admissions lotteries conducted between 2005-06 and 2007-08 by the seven ECs included in this study. Of those students, about 77% were non-White, and 59% were low-income. Analyses for workforce and other life outcomes will be based on 969 study participants (473 treatment and 496 control) who provided data on these outcomes through a follow-up survey. The EC Program: Led by the Bill & Melinda Gates foundation, the EC High School Initiative began in 2002 with the goal of increasing the opportunity to earn a postsecondary credential for students who are traditionally underrepresented in higher education (see Figure 1). To achieve this goal, ECs partner with colleges and universities to offer students an opportunity to earn an associate degree or up to 2 years of college credits during high school at no or low cost to their families. ECs also provide a rigorous and supportive high school environment to help students navigate and succeed in college coursework. These supports are essential for achieving ECs' mission of improving the educational outcomes of students who are traditionally underrepresented, who may struggle academically, or whose parents did not attend college. Research Design: This study is a multisite natural experiment with student-level random assignment based on admission lotteries conducted by ECs. Data Collection and Analysis: To assess the longer-term impact of ECs, we administered a follow-up survey to study participants between September 2022 and March 2023 that captured data on workforce and other life outcomes such as college debt, employment status, income, job satisfaction, and home ownership. The survey instrument was reviewed by an independent technical advisory group comprised of experts in ECs; survey methodology; and diversity, equity, and inclusion. The survey instrument was also piloted by volunteers with diverse background characteristics. The response rate of the follow-up survey was 48.2% (55.4% for treatment students and 40.2% for control students), which is noteworthy given the unusually long timeline for the follow-up survey (15-17 years after the initial admission lotteries). For all our statistical analyses, we will apply survey weights to address missing data and minimize potential bias due to nonresponse (McCaffrey et. al., 2013). To answer RQ1, we will conduct intent-to-treat (ITT) analyses to estimate the impact of winning an EC admissions lottery on each given outcomes, regardless of whether the student actually enrolled in the EC. To estimate the overall ITT effects across lotteries, we will construct a two-level model that accounts for the clustering of students within lotteries. The treatment indicator will be group-mean centered at the student level to make sure the comparisons of EC students and control students are made within rather than across lotteries and thus produce unbiased estimates. To answer RQ2, we will estimate the different impact of ECs by incorporating an interaction between treatment status and a given student characteristic into the student-level equation of the two-level ITT model. We will explore whether the EC impact differed significantly by students' gender, minority status, low-income status, and eighth-grade mathematics and English language arts achievement. Findings: We will begin to analyze follow-up survey data in May 2023, and findings will be ready before the 2023 SREE conference. Conclusions: Few studies have followed study participants from high school through young adulthood to assess the longer-term impact of education programs on students' workforce and other life/financial outcomes. Despite its limitations (e.g., limited generalizability), this study will fill in a critical gap in the evidence base about the longer-term impact of ECs by following the participants in a natural experiment for as many as 14 years after expected high school graduation. To strengthen the evidence base on this promising program, further research on ECs is needed that assesses the impact of ECs for more recent cohorts in more diverse settings and explores conditions that facilitate or hinders the success of ECs.