Search

Inequality in the shadow? Cross-national comparisons of the effects of additional instruction on math and science performance based on PISA 2018 data Background Many educational systems worldwide are experiencing a growing demand for shadow education, which often takes the form of outside-school educational activities such as private tutoring dedicated to improving student achievement (Entrich, 2021; Stevenson & Baker, 1992). Ample evidence from individual social contexts suggests that shadow education is more accessible to families with high-socioeconomic status (SES) compared to low-SES families (Atalmis et al., 2016; Liu & Bray, 2017; Smyth, 2009; Zhang, 2020). As a result, there are growing concerns that shadow education may reproduce educational inequalities (Bray, 2009; Lynch & Moran, 2006; Zhang & Bray, 2020). However, studies using data from multiple countries have yielded mixed results regarding the association between student SES and access to shadow education as well as the association between private tutoring and student achievement (Entrich, 2021; Jansen et al., 2021; Song et al., 2021; Wiseman, 2021). Therefore, more research is needed to explore the role of outside-school learning activities in the SES achievement gaps. Research Question: To contribute to the discussion on the role of shadow education in the SES achievement gaps, this study asks two questions--(1) To what extent is students' SES associated with their use of different types of additional instruction remedial, enrichment, and instruction on study skills? (2) What are the effects of different types of additional instruction remedial, enrichment, and instruction on study skills on student math and science achievement? Setting: The study uses data from six countries that participated in PISA 2018, i.e., Thailand, Kazakhstan, Greece, Korea, Hong Kong, and Ireland. These six countries were purposively selected because they have been intensively studied in the shadow education literature and they exhibit disparities in terms of geographic distribution and GDP per capita. Figure 1 shows the math and science performance as well as GDP per capita of the six countries in relation to the OECD average. Participants: Participants are 15-year-old students from educational institutions in Thailand, Kazakhstan, Greece, Korea, Hong Kong, and Ireland in PISA 2018 (N=52,807). Table 1 summarizes the number of students and students' math and science performance in each country. Intervention PISA 2018 asks whether students participate in additional instruction in different school subjects. This study looks at additional instruction in the math and science subjects. Specifically, five types of additional instruction are of interest in this study: (1) remedial instruction on math, (2) enrichment instruction on math, (3) remedial instruction on science, (3) enrichment instruction on science, and (5) additional instruction on study skills. Research Design PISA uses a two-stage sampling process: first sampling the schools and then sampling students within the participating schools. Therefore, sampling weights are associated with each student because students and schools in each country may not have the same chance of being selected. This sampling approach may increase the standard errors of population estimates. This study therefore uses a Balanced Repeated Replication (BRR) procedure to account for the cluster structure of the data and generate unbiased standard errors. Data Collection and Analysis: The study uses data from the PISA 2018 performance tests in the mathematics and science domains. The tests include an item bank and each student is only presented with a fraction of those items. To account for the variability in scores due to the different sets of items available to each student, 10 plausible values were computed for each student as estimates for their performance in each domain. PISA 2018 also collects questionnaire data from students and their schools, which provides contextual information such as SES indicators at the student and school levels. Multiple regression analyses were performed for each country to examine the relationship between additional instruction and student performance in math and science. Outcome variables are student math and science performance. The independent variables are the five different types of additional instruction listed in the Intervention section. Covariates include student gender, student learning time in math/science classes, family socio-economic status, home educational resources, percentage of fully certified teachers at school, and student-teacher ratio. Results: Table 2 presents the correlations between different types of additional instruction and student- and school-level SES indicators. Participation in different types of additional instruction was differently associated with SES indicators. In general, SES indicators are positively associated with additional instruction, especially enrichment instruction, lending support to previous findings that high-SES families tend to access more additional educational resources than low-SES families. Tables 3 and 4 present the results of the multiple regression analyses for each country. Different types of additional instruction have different effects on student math and science achievement. Remedial instruction and instruction on study skills are negatively associated with both math and science performance. Enrichment instruction is positively associated with student performance in some countries (Thailand, Greece, Korea) but not the others (Kazakhstan, Hong Kong, and Ireland). Significant results were mainly found in student-level SES indicators, while school-level SES indicators yielded few significant results. Conclusions: Despite the small number of countries included in the study, the study contributes to the growing body of cross-national comparisons on the relationship between shadow education and the SES achievement gaps. The findings of this study provide more nuance on how different types of additional instruction can be differently associated with student achievement. This demonstrates the need for further work to differentiate various types of shadow education to better understand how shadow education might reproduce or close the SES achievement gaps. Moreover, the cross-national variation in the effects of additional instruction on student achievement highlights the need to account for characteristics of different educational contexts when interpreting the role of shadow education in educational inequality.

The Cleveland Metropolitan School District (CMSD) has seen the number of English learner students in grades K-12 increase in recent years, even as overall enrollment has decreased. In addition, the English learner student population has grown more diverse in race/ethnicity, country of origin, and native language. The increasingly diverse English learner student population requires more support from the district to meet broader needs for language, cultural, and educational assistance. The Cleveland Partnership for English Learner Success is a researcher-practitioner partnership between Regional Educational Laboratory Midwest and CMSD's Multilingual Multicultural Education Office and the district's research office. The partnership has prioritized examining the characteristics of English learner students and the schools they attend to identify which of those characteristics are associated with student achievement. The partnership undertook this research to help improve district and school supports for English learner students. The authors found the number of English learner students speaking languages other than Spanish increased. From 2011/12 to 2016/17 the percentage of students whose native language was Spanish decreased from 82 percent to 71 percent. During the same period the percentage whose native language was Arabic increased from 4 percent to 9 percent, and the percentage speaking an African language doubled from 5 percent to 10 percent. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Many public and private initiatives have focused on improving prekindergarten (PK) through Grade 12 mathematics education, given that the mathematics content learned before college is foundational for later learning and college success across STEM subjects. Although decades of educational research have studied PK-12 mathematics interventions, the field lacks a comprehensive understanding of which interventions work, for whom, and under what conditions. This meta-analytic project aimed to examine the heterogeneity in mathematics intervention effects by synthesizing 25 years of randomized experiments of interventions designed to improve mathematics achievement. It describes intersections of interventions, intervention components, and outcomes that appear especially promising for future intervention research. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Math education has been in the forefront of education policy discussions for over two decades. Interest in math by politicians, policymakers, researchers, and practitioners stems, in part, from the understanding that the economy will depend on jobs in science, technology, engineering, and mathematics (Clements, 2004). While math is increasingly taught in formal early childhood education settings, young children's exposure to math is less prevalent at home. The authors analyze the effects of 32-week, parent-facing text-message curricula on pre-kindergarten math achievement. The text-messaging program provides parents with small, easy-to-implement activities that leverage everyday routines and household materials. The program is designed to combat common barriers to behavior change identified in behavioral economics. This study demonstrates that a text-messaging program for parents based on behavioral economics principals can improve the mathematics development of pre-kindergarteners, with a combination program more effective than a pure math program. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Despite a history that can be tracked to 1867, the US Department of Education is a relative newcomer to applying evidence standards to education research. The What Works Clearinghouse (WWC) reviews intervention research to support evidence-based decision making. This is done by applying standards of research quality to studies and giving those studies ratings that indicate the extent to which the findings are trustworthy. In addition, nonprofit entities have also invested in vetting research toward more informed decision-making in education and other policy areas. There is much overlap in the interventions of interest to the WWC and other research vetting entities. The authors model the comparative approach using a case study of the "Career Academies" program, reviewed under WWC Group Design Standards, Blueprints for Healthy Youth Development (Univeristy of Colorado Boulder, 2019), and Social Programs that Work (Arnold Ventures', 2019) standards. The goal of this comparison is to examine these and other standards frameworks, extending the case study of "Career Academies" toward the goal of illuminating differences in framework-specific assessments of interventions that could confound decision making for practitioners. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Group discussions during mathematics lessons provide important opportunities for students to deeply engage with mathematics content and to develop communication skills and strong understanding aligned with college and career readiness objectives (Walshaw & Anthony, 2008). However, such student-centered discussions are less common than teacher-led lectures in middle grades mathematics (Stigler & Hiebert, 2004). Research has found that lack of training contributes to the challenges in successfully implementing student-driven discourse (Piccolo et al., 2008). Simulated Instruction in Mathematics Professional Development--or SIM PD--is a new program being developed to meet the need for PD focused on mathematical questioning and discourse (NCTM, 2014) using the classroom simulation technology. This study used a school-level random assignment design so that groups of two to six teachers within a school were randomly assigned to either implement the program during the 2018-19 school year, or participate in data collection but wait to participate in the PD until the 2019-20 school year. The authors will share findings from a small-scale, school-level randomization study in 2018-19 to provide an early, rigorous assessment of whether the PD helps teachers develop the targeted instructional practices. Findings indicate that the program was feasible to implement, teachers received the PD positively, and classroom practice may have improved as a result of the PD. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Use of online credit recovery is a growing trend across the country, with the hope that expanding credit recovery options through online courses will help students get back on track toward graduation (e.g., Atkins, Brown, & Hammond, 2007; Gemin, Pape, Vashaw, & Watson, 2015). But expanded use of online credit recovery for high school students has outpaced the research. The authors conducted a multisite study to test a blended-learning model for credit recovery, where the main curriculum is provided through an online program and an in-class teacher provides more individualized instructional support. The authors present findings on the implementation and initial outcomes for two cohorts of students who enrolled in an Algebra 1 or English 9 credit recovery course the summer immediately after their first year of high school. The authors address three main research questions to describe how implementation of the online classes differed from the schools' business-as-usual (BAU) teacher-directed approach to credit recovery, and provide preliminary evidence about the short-term effects of the online approach: (1) How did the "instructional features" of the online classes compare to the BAU classes?; (2) How did "students' experiences" in the online classes compare to the BAU classes?; and (3) How did students' "proximal outcomes" (content knowledge and credit recovery) in the online classes compare to the BAU classes? The findings suggest that implementation of the online classes produced limited instructional contrast with the BAU condition and may have unintentionally altered grading practices. Potentially as a result, fewer students recovered the course credit in the online classes than the control classes. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Context: A perennial problem in applied statistics is the inability to justify strong claims about cause-and-effect relationships without full knowledge of the mechanism determining selection into treatment. Few research designs other than the well-implemented random assignment study meet this requirement. Researchers have proposed partial solutions that rely on non-randomized comparison groups and baseline covariates. Despite their limitations in providing unbiased causal inference in practice, observational studies are needed to compare outcomes between non-randomly formed treatment and comparison groups. To measure potential bias, researchers conduct within study comparisons (WSC) that compare causal estimates from observational studies and experiments that test the same hypothesis using the same treatment group, outcome, estimand, and other design particulars (LaLonde 1986; Cook et al. 2008). Objective: This paper conducts a meta-analysis of 46 WSCs that compare both mean bias and its variance over eight feasible observational designs. These designs result when three design elements are factorially combined: (1) the presence or not of a pretest measure of the outcome, (2) whether the non-equivalent comparison group is local to the treatment group or not, and (3) whether there are or are not at least four other covariates. Thus, one design includes all three elements; another has none; and the other six include any one or any two. Research Design: The sample consists 46 WSC studies that focus on educational interventions and outcomes. We evaluate standardized bias estimates at three levels: contrast, hypothesis, and study. Contrast-level bias refers to the bias estimate for each outcome, population, and analytical choice. In this study, we analyzed 603 contrasts from 46 WSCs, where both standardized bias estimates and approximate standard errors were available. For the remaining contrasts, we could only obtain standardized bias estimates, as the papers lacked the necessary information to calculate the standard error associated with the bias estimate. Hypothesis-level bias is the average of contrast-level bias estimates in a specific outcome domain. A study can contain multiple hypotheses. This analysis aims to answer the question of how the intervention affected the full study sample's math performance, rather than its effects on different locations or target groups (Friedlander and Robins 1995). In our sample, we have 247 hypothesis-level bias estimates. Study-level bias is the average bias across all hypothesis-level estimates within a single study. This approach has been commonly used in previous syntheses of WSC results, as demonstrated by studies such as Chaplin et al. (2018) and Coopersmith et al. (2022). This paper uses the frequentist multi-level model proposed by Pustejovsky and Tipton (2021), which employs a random-effects meta-regression method implemented through the R package metafor, and cluster-robust variance estimation (CRVE) implemented through the R package clubSandwich. This approach is most appropriate for this paper as this model allows each design subgroup to have its own estimate of average bias and variation in bias at the contrast, hypothesis and study levels. Using this within-subgroup variation, random effects are computed for each of the eight design subgroups. The bias calculated for a design subgroup is independent of the other subgroups. Standard errors are clustered at the study-level. Results: Our subgroup correlated and hierarchical effects analysis reveals that the average bias of the standardized bias estimates lies within 0.1 standard deviations of the experimental estimate for all the eight design subgroups (see Figure 1). The average standardized bias is at or close to 0 for several of these subgroups. The variance around the average bias reflects how consistently studies produce treatment effect estimates that are comparable to those from randomized trials. At the top, the design subgroup D does not include any of the three key design elements and has the largest standard deviation. In contrast, design subgroup L_P_C, which contains all three key design elements, namely local comparison group, pretest measure of the outcome and a set of rich covariates, has the smallest standard deviation and consistently performs well across different specifications and levels of analysis. Figure 2 compares the predicted percentages of effect sizes that are close to unbiased for the three levels of analysis. We find that two design subgroups perform consistently well across all three levels of analysis. For subgroups L_P_C and D_P_C, 90% or more studies deliver effect sizes within 0.1 standard deviations of the experimental estimate. Both these subgroups contain a pretest as well as rich covariates and one of them also utilizes a local comparison group. Two more subgroups show promising results - the subgroup of studies that use a pretest and rich covariates (D_P_C) and that use a local comparison group in conjunction with a pretest (L_P). Over 75% of studies using these design combinations deliver effect sizes within 0.1 standard deviations of the experimental estimate at all three levels of analysis. Conclusions: In summary, our analysis reveals that the average bias falls within 0.1 standard deviations of the experimental estimate for all the eight observational study designs. Moreover, as more design elements are incorporated, the standard deviation of bias tends to decrease, and when all three elements are utilized, there is a high likelihood (at least 90 percent) that bias will fall within 0.1 standard deviations of the experimental estimate. Notably, quasi-experiments that include a pretest measure of the outcome generally outperform other designs in reducing bias, and a pretest plus four other covariates is as effective as a quasi-experiment with all three elements. Based on a meta-analysis of bias estimates from 46 studies, our findings indicate (1) which quasi-experimental designs are preferable and (2) which ones consistently yield causal estimates that closely align with those from a comparable randomized experiment.

Background: The number of English learner (EL) students has been steadily increasing within the United States and currently includes more than 5M students. The National Education Association (2020) estimated that by 2025, nearly 25% of all learners will be EL students. Curricula, interventions, and assessments for both diagnostic and academic purposes need further development to accommodate the growing multilingual student population. However, researchers who aim to bridge the gap in available tools often struggle to successfully engage underrepresented and marginalized communities, including the EL population, and the school districts who serve them (Bonevski et al., 2014; Erves et al., 2017). Objective: The current paper provides a descriptive review of our experience engaging with the Spanish-speaking, Hispanic/Latinx community and school districts that serve this population regarding a research study focused on development of a valid, Spanish-language social-emotional assessment. We aim to impart our approach to partnering with underrepresented communities so other researchers can more effectively engage and support them. Setting: Recruitment efforts are underway across the United States and have included virtual and in-person networking. Presently, partnerships have been established with school districts within Illinois, Texas, Michigan, and the District of Columbia. Population: Target populations include community-based organizations to help develop stakeholder-informed recruitment plans to effectively enroll Spanish-speaking Hispanic/Latinx EL students in grades 3-7; their families; and school leaders from across the United States. Practice: We engaged in several community-informed recruitment practices to help us achieve our recruitment goals (e.g., Cunningham-Erves et al., 2023). Research Design: The University Medical Center IRB approved all aspects of this research. This project began early in the pandemic, many schools shifted to remote learning. Wherein we typically identified a limited number of large school districts to collect classroom-level data and to conduct classroom-based individual recruitment efforts, this was no longer sufficient. Further, the EL students we aimed to recruit were among the most negatively impacted by COVID-19 (Long, 2022; Relyea et al., 2023; University of Cincinnati, 2021). After schools re-opened, recovery from learning loss and effectively managing their time and resources within a stressed educational system were their focus. Even with Spanish-language flyers, bilingual webpages, and paid advertisements, traditional methods of recruitment were only minimally effective. We adopted a broader approach to recruitment than previously utilized. To better engage with the community, we began presenting our research problem to community-based and inter-organizational groups. Many of these community members were the same stakeholders we needed to engage in our research. Our research team presented our past recruitment designs and current challenges and began listening. Together, we then brainstormed strategies to reach a larger audience and more effectively build trust in the community. Data Collection and Analysis: We tracked recruitment ideas, strategies, leads, partnerships, and outcomes. Our primary outcomes of interest were recruitment rates and attrition rates after engagement. Results: Here we highlight a range of strategies for reaching underrepresented communities as suggested by members within the community-based organizations. First, because the Principal Investigator was herself not a Spanish-speaking Hispanic/Latina individual, it was important to have representation within our research team and endorsement of our team or project from a trusted community, educational, or religious liaison. To this end, we enlisted the support of a English-Spanish bilingual parent who also worked within an area school district to act as a trusted liaison and to be present at in-person events. As one such example, at a Bilingual Parent Action Committee resource fair, parents would engage openly with the liaison; in one morning, eight new families signed up for information. This trusted individual can more effectively share project information with community bilingual parent groups and at area events. Next, minimize wordiness on materials and better convey the benefits of participation. If possible, utilize text messaging and other audio-based strategies. In response, we have developed Spanish informational messages that play when families are on hold while calling other offices. Through conversation with community members, to reduce attrition, it was suggested that we reduce the number of points of contact for enrollment. We modified our process to have the bilingual coordinator conduct the online enrollment process during a phone call with the parent. We also reach out to parents who provide their contact information, versus asking them to contact us. Community members suggested that enrolling participants simultaneously with other events may yield the greatest return. When families attend other school or community events, often times, children attend. If participation can occur during those events, it facilitates their completion. To engage better with school districts, ideas included providing educational sessions for educators or families on assessment, social-emotional learning, or mental health -- capitalizing on expertise within our team and providing benefit to the partner. Rather than cold-calling prospective school districts, community leaders stressed again how one-on-one engagement in person was key even for educators. Community leaders suggested hosting "lunch and learns" to share information with educators and school leaders. While we were unable to implement this strategy, we did attend key national conferences (e.g., National Association for Bilingual Education) to engage school partners with large Spanish-speaking populations. By hosting vendor tables at conferences, we made personal connections and increased visibility about our work. Finally, school leaders stressed providing materials to them for distribution to families about the partnership and research. Being able to provide schools with materials translated already into Spanish also facilitates their participation and ability to assist with recruitment efforts. Discussion: Community engagement is time-intensive. It is not enough to post a flyer or send an email. A face and a name to connect to the information is needed to establish a trusting relationship (Wallerstein & Duran, 2006). Underrepresented communities, like all communities, value transparency about what the time commitment will be and the end goal. Research opportunities that yield mutual benefit are the most effective partnerships and long-term success. Through varied outreach efforts, new opportunities arise to connect and engage with the EL community. As a result, in addition to individual participants, we developed new partnerships with four districts that have high percentages of Spanish-speaking EL students.

Background: What is the role of state legislatures in improving public education? Is it to provide funding and standards for student outcomes and then step aside? Or should legislatures and state education agencies also govern how districts educate students? In practice, states hold school districts accountable for academic achievement while also requiring them to prove compliance with a variety of regulations on educational inputs (Cohen et al., 2017). In 2015, in Texas, this traditional approach to regulation was upended. Today, any Texas district with an acceptable academic and financial rating can declare itself a District of Innovation (DOI) and opt out of any regulation that does not apply to the state's charter schools. These districts are still held accountable to student outcomes but may meet those academic standards however they choose. Deregulation advocates argue that regulations restrict schools from tailoring inputs to their unique circumstances and from serving their student populations as those circumstances require (Elmore & Fuhrman, 1995; Hanushek, 2003). On the other hand, many fear that without input regulations, some district leaders would make inappropriate decisions and thereby harm students and increase inequality (Elmore & Fuhrman, 1995). These fears were evident following the passage of the DOI statute as the state's teachers' unions issued warnings to their members and argued that the statute would life "legal protections that…protect students and educators" (Texas AFT, 2021). Purpose: To provide empirical evidence on the role of state education regulation in determining school decisions and student outcomes, this paper uses the Texas DOI statute to shed light on 1) which regulations districts are most enthusiastic to remove (RQ 1); 2) the extent to which deregulation changes local decisions on educational inputs (RQ 2); and 3) the extent to which deregulation impacts student outcomes (RQ 3). Recognizing that equity is of particular concern with deregulation, this paper focuses on gaps in educational inputs and academic outcomes for rural students, economically disadvantaged students, and students of color. Setting and Population: The Texas DOI statute passed in 2015 and was followed by a steady but enthusiastic response from districts. By the 2016-17 school year, 178 districts had claimed District of Innovation status. An additional 509 districts followed suit in the 2017-18 school year, along with 119 districts in 2018-19, and another 102 districts between 2020 and 2022. As of June 2022, 90% percent of school boards have voted to become a District of Innovation. Research Design: One typical approach to assessing impacts with variation in treatment timing is to employ an event study design. However, recent research demonstrates that weighting schemes embedded in these approaches produce biased and unintuitive impact estimates (Goodman-Bacon, 2021). I therefore follow the advice of Callaway and Sant'Anna and estimate the impact of District of Innovation status using a series of difference-in-difference (DID) estimates (2020). This strategy has the advantage of producing well-understood impact estimates while capitalizing on treatment variation and controlling for time-invariant confounders (by controlling for pre-treatment levels) and history effects (by controlling for comparison group trends). The key assumption in this design is that each cohort would have experienced the same change in outcomes as the comparison group, if not for changes resulting from District of Innovation status itself. I probe and address this assumption by 1) estimating the "impact" of District of Innovation status on school-level inputs and outcomes in the years before the innovation plan was implemented; and 2) controlling for observable differences in cohort characteristics. Data Collection and Analysis: My data analysis requires information contained in District Innovation Plans which are posted on district websites. Using web-scraping and natural language processing, I extract from these documents each district's exempted regulations and the date when they become of District of Innovation. I then link my scraped data to an administrative dataset of school-level data from the 2011-12 to 2019-2020 school years. Findings: I find that Texas school districts have responded enthusiastically to the opportunity to opt out of input regulations. Nearly 90% of districts have claimed District of Innovation status and, of these, 87% have exempted teacher certification requirements, 44% have exempted elementary class size maximums, and 42% have exempted the minimum class time requirement. Regulatory preferences are also highly correlated with district geography; rural districts are particularly eager to exempt teacher certification requirements and restrictions on teacher contracts, while urban districts are particularly eager to exempt restrictions on student attendance and responses to student behavior. These distinct responses suggest that district leaders feel that input regulations do not address contextual needs. Yet, despite enthusiastic adoption, I find that regulatory freedom under the District of Innovation statute caused limited changes in observable school inputs. I estimate the impact of regulatory freedom on twelve key educational inputs including the percent of uncertified teachers, out-of-field teachers, average elementary class sizes, and student-teacher ratios. Each of these impact estimates is a relatively precisely estimated zero within the first four years of implementation. Further, though there are substantial gaps in educational inputs among average schools and those serving high proportions of Hispanic, Black, rural, and economically disadvantaged students, the District of Innovation statute did little to either exacerbate or lessen these inequities in the medium-term. Finally, I find a statistically insignificant and insubstantial 3-year effect of the District of Innovation statute on student achievement: less than 0.01 school level standard deviations (SD) in mathematics and reading. Conclusions: Taken together, these results indicate that though school boards are generally in favor of increased regulatory flexibility, this flexibility did not dramatically alter school inputs within four years or school outcomes within three years. On the other hand, there are still stark inequalities in Texas schools. Thus, if policy-makers and activists wish to increase the standard of educational inputs or academic achievement for rural, Black, and Hispanic students, these results suggest that it would be better to invest in alternative levers like accountability, incentives, resources, and improved compliance, rather than deregulation.

Background/Context: Since the No Child Left Behind Act of 2002, disaggregated data reporting requirements have illuminated differences in school outcomes between demographic subgroups. These results are publicized in part for parents to identify "effective" schools for their children (Fusarelli, 2004; Husband & Hunt, 2015). Households now have disaggregated data on standardized test scores, graduation rates, attendance, and discipline. How do households update their beliefs about schools when considering these data? This study demonstrates the link between prior history, school outcome data, and parent expectations. Fitting with the SREE conference theme, Embedding Equity for Better Methods, Evidence, and Practice, I use a pre-post survey experiment to identify how race/racism and prior experience shape parents' school expectations. Recognizing that race and racism are permanent features in society (Ladson-Billings & Tate, 1995), I foreground two QuantCrit principles: questioning the supposed neutrality/objectivity of numbers, and valuing the experiential knowledge of marginalized people (Gillborn et al., 2018). The type of information that families seek when identifying a school for their child and the interpretation of that information are likely colored by the decisionmaker's identities and their experiences in a world that treats people differently based on their identity (Cooper, 2005). Knowing how these expectations are racialized and hearing how minoritized parents, specifically, perceive schools will help illuminate problem areas for school leaders to address and continuously monitor. Additionally, this study helps identify what and how data can empower parents to navigate decision environments. Purpose/Objective/Research Question: Hypothesizing that parent expectations for their children's school experiences are important drivers of school choice/accountability and that these expectations are influenced by their past experience with a school as well as the performance of their child's same-race peers, in this study, I answer--(1) to what extent are parent expectations for their children driven by their past experience with a school, and (2) how does having knowledge of both subgroup and school-level performance as opposed to only having aggregate data affect parent expectations? Setting: To answer this study's research questions, I conduct an original, pre-post online survey experiment using the CloudResearch platform. Population/Participants/Subjects: Participants in this study are a national sample of parents of children in grades 3-8 who take state standardized tests. I recruited a total of 655 parents, including more than 200 parents with children of color. Intervention/Program/Practice: For this pre-post survey experiment, respondents first estimate their child's peers' school performance, both for the school as a whole and for children of the same racial and gender (intersectional) background as their child. Subsequently, they indicate their baseline expectations for their child along several measures. For this baseline test, parents rate the likelihood (on a continuous scale from 0-10) that their child scores proficient on the state standardized test, for example. After completing the pretest measures, I ask participants to consider a new school for their child. Participants are told this new school is like their current one in demographics and size, and then they are presented data about the school's performance. The data presented in the posttest is of two general types: aggregate, school-level data or demographic subgroup data with the aggregate data. While considering the data, parents are again asked the likelihood their child would score proficient on the state standardized tests, have on-track attendance, etc. Research Design: I randomize participants to be in one of two broad experimental conditions. In Group 1, participants see school-level data for their first posttest measure. Group 2 evaluates the new school based on subgroup data in addition to the aggregate data. More specifically, the respondents in Group 2 are randomized to see one of three types of subgroup data: race, gender, or intersectional. Regardless of the experimental group, the statistics presented in the data table are the same as what the participants provided earlier about the school's performance and their child's demographic peers' performance. Data Collection and Analysis: To analyze the data collected via Qualtrics, I first determine group equivalence of the experimental conditions by regressing the pretest means on each experimental group. Subsequently, I begin the pre-post analysis by computing the difference between the posttest expectations and the initial baseline expectations for each parent. I conduct t-tests to compare the baseline expectations to the posttest expectations. When compared to the baseline, the posttest expectations show how expectations are driven by one's history with a school, as the school presented is otherwise the same as their child's current school. To estimate the effect of subgroup data on parent expectations as opposed to only having school-level data, I compare the average pre-post difference between the experimental conditions using multivariate OLS regressions with controls. Because I have randomized assignment to the experimental groups, the average difference between the groups is attributable to the type of data presented. With each of these analyses, I probe to see if there are racial differences in the response to the data. Findings/Results: Across the board, parents had significantly lower expectations in the imagined school than when thinking about their child in their current school. The difference in expectations tended to be greatest for parents of students of color. The initial results also indicate that there are significant differences in expectations when provided subgroup data in addition to school-level data. I show that the type of subgroup data presented -- race, gender, or intersectional -- also matters and matters differently between and within races. Conclusions: Among other things, these findings speak to the familiarity that parents have with their current school, their relationships, their level of confidence, as well as their agency in their school. When parents, especially parents of students of color, see school-level data, they interpret the data by making adjustments based on who their child is, their experience with and prior knowledge of a particular school, and, more generally, how schools treat children like theirs. Recognizing the limitations of a survey experiment about a hypothetical school, future work of mine uses a field experiment in conjunction with administrative data to further investigate the relationship between prior history and knowledge, data, and parental expectations.

Background: Teachers in developing countries face a complex challenge at work--teaching students who are first-generation learners, in classrooms with wide heterogeneity in learning levels, and in frequently resource-poor settings (Muralidharan et al., 2019; World Development Report, 2018). The presence of many of these potentially binding external constraints might suggest to teachers that their effort matters little in influencing student learning, leading to a perception of low returns to effort. These beliefs could be reinforcing if teachers invest less effort, observe low returns and then update their beliefs. This hypothesis is supported by cross-country studies that document a widespread consensus among teachers that 'there is little they can do to help a student learn if parents are uneducated, or have personal or financial problems' and that 'a student's home environment directly limits the influence of their teaching' (Sabarwal et al., 2022; Young Lives, 2016-17). From a policy perspective, it is unknown whether teachers' beliefs about returns to effort are malleable and can be influenced through targeted interventions. This idea has been extensively studied in the psychology and education literatures under the theoretical frames of teacher locus of control and teacher efficacy (Rose and Medway, 1981; Gibson and Dembo, 1984, Tschannen-Moran et al., 1998), however, it is unclear whether these constructs can be used as a policy level to impact student learning at scale, especially in the context of developing countries. The question is even more relevant in a post-covid context where learning lags are likely to adversely impact classroom environments and strain psychological reserves of teachers, in the absence of adequate support and resources. Research Question: In this project, I examine the extent to which beliefs about perceived control over the parameters of education production function influence the choice of teacher effort and impact student learning. The research aims to answer the following questions: (a) To what extent are teachers' beliefs about perceived control malleable through exposure to a psychosocial intervention? (b) What is the causal impact of perceived control beliefs on teachers' effort levels and students' academic outcomes? The study's main hypothesis is that using a psychosocial intervention grounded in positive psychology and behavioral science, shifts teachers' perceived control over education production, increases teachers' effort in the classroom and improves student learning in Math. Research Design: To study these questions, I conduct a randomized controlled trial with 83 schools in a large school chain in north India. An experimental design allows me to exogenously manipulate teacher beliefs to examine malleability and enables me to isolate the effect of beliefs on other outcomes. Randomization is done at the school-grade level. Each teacher is mapped to a unique school-grade. I conduct pairwise randomization across all possible grade pairings within schools. With this design, the study is well-powered to detect effects at the student- and teacher-level, in line with related work. Setting: The study is conducted in a large rural school chain of 129 schools in northern India. The private school chain is run by a charitable trust and provides high-quality and subsidized education to rural populations. Participants: The study sample consists of teachers and students in grades 2, 4, 6, and 8 across 83 schools, comprising 292 teachers and around 5,600 students. I work with non-adjacent grades within each school to minimize the possibility of spillovers across adjacent grades. Intervention: I use an evidence-backed psycho-social intervention that has been implemented by McKelway (2021) in a similar context outside education. The intervention targets perceived control through two approaches -- a skill-building approach that enhances self-efficacy, and a control-enhancing approach that develops an internal locus of control. The modules are crafted to use cognitive restructuring strategies and use contextualized stories about role-models, motivational quotes and affirmations that highlight primary and secondary forms of control. Teachers undergo five weeks of online training for two hours each week, with a main session, a debrief session, and a weekly assignment to reinforce concepts. The training is conducted after school hours to minimize disruption to the school timetable. To minimize the scope for demand effects, the curriculum is purged of any references to teaching skills or classroom practices. Teachers in the treatment arm are exposed to psychological content covering five topical areas -- character strengths, self-efficacy, emotion-management, goal-setting, and problem-solving. Teachers in the placebo arm received training of a similar duration and format but with psychologically inactive content. Data: I collect data through 4 rounds of primary data collection -- at baseline, immediately after, three and six months after the intervention. Multiple endline rounds enable a rich description of the persistence of impact. I collect primary data on teachers' beliefs and efforts through interviews and classroom observations. I access data on school- and student-level characteristics from school administrative records. To measure teachers' beliefs, I use a novel experimental task using a multiple price list approach to credibly elicit teachers' perceived control beliefs about improving performance of students in the bottom tail of ability distribution. The study's secondary analyses investigate mediating variables and alternative channels through which the intervention might impact teacher and student outcomes. Findings: The study is currently in field with an ongoing last round of endline data collection. Preliminary results indicate balance on observable characteristics at baseline and a positive impact on perceived control beliefs as measured by the multiple price list task immediately after the intervention. Analysis on teacher effort and student learning outcomes, as well as on later measures is ongoing, and will be available by the time of the conference. Conclusion: The need to raise learning outcomes among students in the developing world has spurred a range of education initiatives. Governments spend millions of dollars in providing in-service teacher training even though evidence suggests limited productivity of these investments. This raises a need to find training models that address core teaching challenges and are effective at scale. This project holds promise to show important evidence about efficacy of a novel psychosocial intervention that can be scaled up in relevant settings.

A college degree typically helps graduates earn more when they enter the labor force and is a key driver of economic mobility. However, many students who enroll in college do not finish, reflecting numerous academic, financial, and social emotional barriers students may face to succeeding in college and completing a degree, from being underprepared for college coursework to struggling to afford tuition or other expenses like books and housing. The COVID-19 pandemic brought additional challenges and amplified existing ones (U.S. Department of Education 2021; Hotez et al. 2022). Funded by The Boston Foundation, Success Boston brings together Boston Public Schools (BPS), local colleges and universities, the City of Boston, and various nonprofit organizations. In 2022, the initiative recommitted to improving BPS graduates' postsecondary completion rates, with a target rate of 70 percent across all groups of students (The Boston Foundation 2022). A core strategy of Success Boston, transition coaching offers support to students in addressing challenges they face in college. The Success Boston Coaching (SBC) model matches recent graduates of BPS with a coach, who meets with them regularly during their first two years of college and supports them as they navigate academic, financial, and social barriers. Our quasi-experimental study looks at SBC's effects on students' four-, five-, and six-year postsecondary completion rates, across more than 9,000 students across 50 colleges who initially enrolled in college in 2015, 2016, or 2017. To do this, we conducted propensity score matching to construct a comparison group with similar demographic and baseline academic achievement to SBC students, and who enrolled in the same college in the same year. To measure SBC's effects, for each outcome, we estimate a linear regression model that includes indicators for matching blocks and matching characteristics, among the matched comparison and SBC students. We find that students who received SBC graduated in four years at rates that are 21 percent higher than those of noncoached students and in five years at rates that are 15 percent higher. These effects are meaningful because completing college in four years or five years, rather than in six or more years, can help students avoid the costs of additional years of college and allow them to enter the labor market sooner. At six years after entering college, differences in graduation rates between coached and noncoached students were no longer statistically significant, suggesting that some noncoached students had caught up to the coached students at this point. Findings suggest that the coaching program could be an important component in helping move citywide postsecondary completion toward its of 70 percent target. That said, only about 49 percent of coached students, and a similar proportion of all BPS students, complete college in six years, well short of the 70 percent target. Students will continue to need support, perhaps through extending coaching beyond students' first two years in college, for the city to achieve the initiative's goal. To focus its efforts on students who could most benefit from the support, SBC serves primarily students from groups traditionally underrepresented in postsecondary education, such as members of racial/ethnic minority groups and students from economically disadvantaged backgrounds.

Background: For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require each document first be manually coded for constructs of interest by trained human raters. These hand-coded scores are then used as a measured outcome for an impact analysis, with the average scores of the treatment group compared to the control (possibly adjusting for other observed covariates, etc.). This process, the current standard, is time-consuming and limiting; even the largest human coding efforts are typically constrained to measure only a small set of dimensions across a subsample of available texts. We propose using modern machine learning tools to assist with such human coding efforts, allowing more researchers to use text as an outcome. Achieving this end is important: while difficult to use, text is a critical outcome to consider. In K-12 settings, for example, students' academic success is partially dependent upon their writing proficiency--a cognitive, linguistic, and social task that requires writers to communicate with a non-present audience (Gee, 2015). Despite its importance for students' learning, however, writing research continues to lag behind reading research (Juzwik et al., 2006). One key impediment to text-based research may be the difficulty associated with analyzing text. Purpose: We aim to increase power of large-scale randomized evaluations with text outcomes in a manner that preserves the validity of a human-coded construct. In particular, we propose augmenting standard human coding efforts by taking advantage of "untapped" observations -- those documents not manually scored due to time or resource constraints -- as a supplementary resource. This can increase the power of an impact assessment, given a fixed human-coding budget, by leveraging the full collection of documents. Methods: Our core idea is to, first, hand-code a random sample of the original documents; next, we train a machine learning model on this subset that can then be used to predict scores for the entire corpus. We then use methods from survey sampling to adjust our impact estimate. As we prove in our paper, this ensures any biases from the machine learner are adjusted for, so we maintain focus on the human-coded construct. Through a simulation study based on data from a recent field trial in education, we show that our approach indeed reduces the scope of the human-coding effort needed to achieve a desired level of power. In particular, consider a collection of N documents where Y is the score for document i with respect to a human-coded construct and Z is an indicator for treatment. Our approach, a methodological combination of causal inference, survey sampling methods, and machine learning, has four steps: (1) select and code a sample of n

Background: Traditionally, SREE has traditionally focused on examining the effects of educational interventions with rigorous causal analysis. However, studies of racial/ethnic inequities in education do not always lend themselves to experimental or even quasi-experimental designs. Further, well formulated descriptive studies can provide crucial information about the current state of education and can lead to the generation of causal questions and designs. This study documents racial/ethnic, class, and English learner (EL) disparities in gifted identification through the lens of intersectionality using administrative data from seven school districts. Race, ethnicity, SES, and EL are not independent and unitary concepts. Instead, they intersect and interact in ways that must be considered in educational research (Collins, 2015). Substantial work has documented disproportional representation in school discipline, special education placement, and gifted education placement (Hamilton et al., 2018; Hamilton et al., 2020; Long et al., in press; Shores et al., 2020). A subset of studies examined the same educational outcomes after controlling for student poverty, prior achievement, or special education eligibility (e.g., Backes et al., 2021; Grissom & Redding, 2019; Morgan et al., 2015; Long et al., in press). The results of studies comparing students of different racial/ethnic groups are very different, depending on whether or not studies control for academic achievement or socio-demographic factors such as poverty. For example, although Latinx students are less likely than their White peers to be identified as gifted, after controlling for background factors, Grissom and Redding found the identification gap between White and Latinx students disappeared. As more educational interventions, particularly post-pandemic, focus on improving equity, it is essential to understand the various factors that relate to disproportional outcomes. Otherwise, efforts to mitigate disproportional outcomes may emphasize or implement ineffective solutions. Intersectionality, as an analytic strategy, may provide an important perspective the complex interplay between various sociodemographic factors (Collins, 2015). Purpose: The purpose of this study is to understand how race/ethnicity and free- or reduced-price lunch (FRL) eligibility and EL intersect and interact to produce disparities in gifted identification rates across intersectional groups. Setting: Our sample includes student level data on academic achievement, ability, and student demographics across 6 districts. The combined dataset includes 19463 students nested within 1119 classrooms in 285 schools (see table 1). These districts have an average of 19% black, 46% Latino, 26% White, and 66% FRL. They also have notable variation in racial composition and FRL composition. For example, among the largest four districts, one is predominantly Latino, two are predominantly White, and one has nearly equal proportions of Black, Latino, and White students. We also complement this analysis with a replication in a 7th district that provided two years of 2nd grade data collected under a separate data agreement. Across the two cohorts in district 7, there are a total of 25,114 students in 937 classrooms. Research Design: We conducted a series of mixed-effect logistic regression analyses to determine the intersectional effects of race/ethnicity, FRL and EL status on gifted identification. We tested three models: Model 1 included race/ethnicity, FRL and the race/ethnicity by FRL interaction; Model 2 added EL to Model 1 plus the race/ethnic by EL interaction, and the FRL by EL interactions; and Model 3 added achievement and ability to the Model 2 variables. (see table 2). Our analysis allowed us to directly evaluate the relative contribution of each variable in explaining the odds of a student being identified as gifted. We first estimated these models with a students within classrooms within schools within district model that included the pooled data from districts 1-6. Next we estimated a subset of multilevel models of students within classrooms within schools for the largest districts (districts 1, 2, 3 plus the supplemental district 7). Findings Models 1 and 2 show a Black, Latino, FRL, and EL disadvantage in gifted identification (see Table 3 and Figure 1). When ability, and achievement are added to the models the lower levels of identification for Black students, Latino Students, low-income students, and EL students all diminish and are no longer statistically significant (see Tables 3-6 and Figures 1-4). To better understand the differences in rates of gifted identification in models with and without ability and achievement we examined the differences in ability, achievement, and teacher rating scales by subgroup (see Table 6). There is about a 0.5 standard deviation difference in achievement, ability, and TRS between Non-FRL and FRL students. And there is a 0.6, 0.5, and 0.3 difference in achievement, ability, and TRS, respectively, for Black versus White students. In the joint models for districts 1-6 there are few statistically significant interaction effects, however there is a positive interaction effect by FRL and race/ethnicity in district 7's unconditional model. Conclusion: First, this study shows that Black and Latinx students in poverty are at a double disadvantage in gifted identification. There is a persistent effect of both race and poverty when both are included in the model and these disadvantages have a similar magnitude. However, in most districts (District 7 is the exception) there are no statistically significant interaction. Second, this study shows that identification gaps largely disappear after controlling for ability and achievement. The different results in the conditional and unconditional model appears to be driven by very large sub-group differences in ability and achievement. These findings suggest that the most effective policy to address under-representation would be to address early achievement and ability gaps instead of addressing potential "bias" in the gifted identification process. Third, our analysis revealed some notable between district differences. For example, there is a positive interaction effect for Black and FRL and Latinx and FRL in district 7 only. Also, in district 7, holding ability and achievement constant, Black students are actually more likely to be identified as gifted. Although non-causal, we believe that well-designed descriptive analysis of unconditional and conditional disparities can help identify the relative influence of race versus class on educational inequalities and can help to inform education policy.

Background and Context: Every student is a powerful learner capable of succeeding in mathematics. The EF+Math Program aims to dramatically improve math outcomes for Black and Latinx students and students of all races experiencing poverty (hereafter, our Priority Students). We are a community of researchers, developers, educators, and students who are developing research-informed mathematics learning approaches for grades 3-8 that combine executive function (EF) skills, conceptual understanding and multi-step problem solving, and equity. Our Inclusive Research and Development (R&D) model (Angevine et al., 2019; EF+Math Program, 2023) places an intentional focus on equity throughout the R&D process and centers educators and students as crucial partners from the start. In the Inclusive R&D process, we encourage cycles of design, iteration, and implementation in both research and development processes. Through these cycles of iteration, our teams have adjusted and refined their instruments to center and affirm our Priority Students. The portfolio of research studies take place in several elementary and middle school classrooms across the country. Objective and Analytic Approach: Our R&D teams are testing the math learning approaches that affirm the brilliance of our Priority Students, and that provide our students with rich and rigorous math learning opportunities. As such, they have designed innovative studies to examine the feasibility and efficacy of these approaches in districts, schools, and classrooms. As part of designing these studies, our teams critically examined the instruments traditionally employed to measure EF skills and psychosocial constructs such as belonging, math identity, and self-efficacy. The examination of these instruments catalyzed our teams to modify, adjust, and redesign the existing instruments to reflect our commitments to equity. As R&D teams have engaged in studies to examine the feasibility and efficacy of their math learning prototypes, we have examined the cycles of study design, implementation, and iteration across the portfolio. We analyzed each cycle based on its purpose and process for iteration, as well as the instrument that was revised. Three themes regarding embedding equity into measurement tools arose across the portfolio. We present these three themes below. Findings: Approach 1: Accessibility for All Students: Instruments are often designed with a specific population in mind, and this population can often exclude individuals for whom we wish to improve outcomes (Matsumoto, D. & Yoo, S. H., 2006). Such instruments often undergo validation studies with samples that follow this same exclusionary pattern. Many instruments that measure psychosocial constructs are designed for students of higher grades or have only been validated with White student populations. Through intentional design and thoughtful adjustments, our teams have modified these instruments to both accommodate and center our priority students. Specifically, educators have provided feedback regarding item language, length, and formats for student responses. Our teams are also engaged in validating adapted instruments for appropriateness of use with our priority students. Approach 2: Affirming the Brilliance of Students through Asset Framing: When examining psychosocial factors, items can sometimes signal that students are struggling, failing, or otherwise performing at a level that is not desirable (Russell, 2013; Stinson, D., 2013). These signals can be even stronger for students of marginalized identities (Schweinle & Mims, 2009; Stinson D., 2013)--like our priority students. Across the portfolio, several instruments included language that could signal to students that they are struggling math learners. Through co-design and iteration with educators, teams rewrote such items to reflect not only the brilliance of our students, but also to reflect that their psychosocial attributes do not make them "challenged" or "inferior". This included broadening response options as well as embedding additional framing for students before they engaged with the instruments. With these adaptations, students are able to be introspective about how they show up as math learners, rather than believing that they are underperforming. Approach 3: Leveraging multiple measures: By nature, latent constructs are difficult to observe, and instruments designed to measure such constructs are often unable to paint a holistic picture of the construct being measured (Russell, 2013). This might lead to inchoate understandings of these constructs, or interpretations of data can be taken out of context. Additionally, regarding psychosocial and/or affective states of students can often (and detrimentally) be situated as static, permanent, and immutable. For example, a student may be deemed "lazy" or "unfocused" if they exhibit low affect on a particular measure or seem easily distracted one day. Such interpretations can potentially reify deficit assumptions of students from marginalized backgrounds. To disrupt these biased and deficit applications of construct interpretation, R&D teams--in conjunction with educators and students--have developed approaches to not only measure constructs in multiple ways, but to also enact such measurement often and in real time. Some of our teams use a suite of instruments to measure executive function states, as well as other psychosocial constructs, like math identity, self-efficacy, belonging, and math identity. This allows researchers to honor the nuanced nature of constructs, and provides a more accurate and robust understanding of students. Such frequent and just-in-time information around EF skills and psychosocial states honor the fluid and dynamic nature of these constructs, thereby reinforcing an asset-framing about our Priority Students as dynamic, complex, and multifaceted human beings. Conclusion: Through critical examinations of instruments used in feasibility and efficacy studies of math learning prototypes, and subsequent cycles of design, implementation, and iteration, our teams of researchers and developers found opportunities to improve the instruments in ways that affirm our priority students and more accurately measure psychosocial constructs such as belonging, math identity, and self-efficacy. The practices that the R&D teams engaged in reflect our commitments to equity and our commitments to affirming and centering Black and Latinx students and students of all races experiencing poverty.

Background: Students face many challenges completing a postsecondary education program, including both administrative hurdles (e.g., course registration) and course management challenges (e.g., scheduling study time, finding academic support). Text-based nudging in the postsecondary context has been a particularly effective tool when targeting administrative tasks (Castleman & Page, 2015, 2016, 2017; Castleman, Page & Schooley, 2014; Page, Castleman & Meyer, 2019; Page, Meyer, Lee, & Gehlbach, 2022) or encouraging use of specific academic services (Pugatch & Wilson, 2018). Personalized outreach from a trusted source, such as one's school or a peer, has been more effective at moving student behavior (Avery et al, 2020; Debnam, 2017). However, in other settings outreach around college academic performance and encouraging specific study skills has not yielded significant effects (Oreopoulos & Petronijevic, 2019). Research Question and Motivation: This study examines whether text-based nudging can support students' performance in an introductory college course. Course performance matters for college success. Both marginal (e.g., passing the class) and inframarginal (earning a B vs. a C) course performance can affect students' choice of major, progression to upper-level courses, eligibility for financial aid, and college persistence. We recognize the limited reach of an intervention in one course on affecting ultimate college completion and view this work as part of a broader ecosystem of student support. Setting & Method: We conducted a randomized controlled trial of a course chatbot designed to (1) provide timely reminders of assignments; (2) provide customized feedback on students' progress; (3) connect students to academic supports; and (4) serve as an additional channel of communication between students and their instructors. We pre-registered the study with the Registry of Efficacy and Effectiveness Studies (REES). We randomized 1,568 students enrolled in an asynchronous, virtual introductory government class to the treatment or control condition across three semesters. We targeted this course given its high historical failure/withdrawal rate and given it was a large-enrollment course where students might struggle to connect with the instructional team. All students received standard course communications from the professor (primarily via email). Treatment students additionally received 2-3 weekly text messages from the chatbot providing information and inviting interactions. The weekly messaging schedule included a digest of upcoming assignments, customized information about whether students had overdue assignments, and encouragement from the course TA. Around each of the asynchronous course exams, students received reminders about a "quiz me" bot function to try sample exam questions, reminders about when the exam opened, and nudges to start the exam if students had not done so by the exam closing date. Students could text the bot at any time and receive automatic replies from the bot knowledge base of answers or, when there was not a clear bot reply available, a notice that the course teaching assistant would address their question shortly. Population: We implemented the intervention at Georgia State University in Atlanta, GA. As highlighted in Table 1, about 43% of students were Black and 15% were Hispanic, about a quarter of students were first-generation students. The share of freshmen each term ranged from 43% to 78% of the course. We hypothesized the intervention may be especially useful for first-generation students who may not have mentors available to provide guidance on effective college course management (Jack, 2016; Lareau, 2003; Walton & Cohen, 2007) and for students who struggled academically (either based on lower prior grades or those re-taking the course after an unsatisfactory grade their first attempt). We viewed as an open question whether the chatbot might differentially benefit freshmen (who are new to college course management) or upperclassmen (particularly the type of upperclassmen enrolling in a course typically completed during freshman year). Research Design: To estimate the treatment effect, we use a regression model of the following general form: Y[subscript ir] = [alpha subscript r] + [beta]T[subscript i] + X[gamma] + [epsilon subscript ir]. Where Y[subscript ir] represents the outcome for study participant i, T[subscript i] is the indicator for assignment to treatment and is equal to one if the study participant i is randomized to the academic chatbot group, X represents a vector of individual-level baseline characteristics (included primarily to explain residual variation in outcomes and to improve precision of estimation as a result), and [epsilon subscript ir] is a random error term. We include implementation semester fixed effects ([alpha subscript r]). We modify this analytic model to include an interaction between treatment and student characteristics for subgroup analyses. Findings: The academic chatbot significantly shifted students' final grades, increasing the likelihood students received a B or higher in the course by five percentage points (Table 2). We find the chatbot affected grades at different points in the distribution for different groups -- for example, increasing the likelihood that freshmen earned an A while for students re-taking the course, the treatment increased the likelihood of passing the course. On end-of-course surveys, students reported enthusiasm for the chatbot, with 92 percent of respondents recommending its continued use in the course and expansion to other courses. About half of treated students wrote into the chatbot, with 22% of all treated students using the "quiz me" tool. In the main paper, we supplement our impact analysis with insights about efforts to scale the chatbot across classes, sharing data on faculty and staff time required during the development, evaluation, and scale phases. We further leverage open-ended responses on end-of-course surveys to provide a portrait of students' experiences during the intervention and to provide insights into the mechanisms through which the intervention affected final grades. Conclusions and Implications: This intervention builds on an ongoing research-practice partnership with Georgia State University, and these insights have informed adaptation of the chatbot for an introductory economics and introductory chemistry course. Our analysis highlights the efficacy of a low-cost tool to supplement student learning and help students develop course management skills. This work further provides an analytic framework for studying interventions implemented in a shifting context. We offer recommendations on how to systematically document variations in sample, intervention context, and implementation to provide a comprehensive picture of whether, for whom, and under which circumstances an intervention is effective.

Background: In fourth grade, students must solve single- and multi-step word problems during classroom instruction and on standardized tests. The majority of mathematics items on high-stakes assessments are word problems; thus, word-problem proficiency is necessary to demonstrate successful mathematics performance. However, many students are inadequately prepared to solve word problems. Although word-problem solving is demanding for many students, those experiencing or at-risk for mathematics difficulty (MD) often demonstrate lower word-problem performance and make significantly more errors relative to peers without MD (Kingsdorf & Krawec, 2014). Moreover, students with MD often have less access to high-quality mathematics opportunities compared to students without MD (McLeskey & Waldron, 2006). A beneficial approach for improving word-problem skills of students with MD is through schema instruction. In word-problem intervention using schemas, students first identify a word problem as belonging to a problem type and then use a specific solution strategy associated with that schema to solve the problem. Developing schemas for categorizing word problems proves valuable by helping students understand novel problems as belonging to familiar categories (e.g., Cooper & Sweller, 1987; Ng & Lee, 2009). Recognizing schemas for categorizing word problems greatly influences whether students answer problems correctly (Fuchs et al., 2014) and has been found to be more effective than other techniques for teaching word-problem solving to students with MD (Zhang & Xin, 2012). For our intervention, we also included a pre-algebraic reasoning component based on work by Powell et al. (2010, 2021) related to understanding the equal sign. Pre-algebraic reasoning skills include the ability to interpret the equal sign as relational and success with solving equations with variables, multiple operator symbols, and parentheses. Prior research suggests when students learned a relational definition of the equal sign, equation-solving performance improved, which, in turn, improved word-problem solving. Purpose/Research Question: The purpose of this study was to determine the efficacy of a 12-week, word-problem intervention for Grade 4 students experiencing MD, and to determine whether one of two approaches to multi-step problem solving was more efficacious. Students with MD were randomly assigned to (a) word-problem intervention with multiple equations, (b) word-problem intervention with a single equation, or (c) the business-as-usual control. We identified MD as students with mathematics performance at or below the 25th percentile on a word-problem measure. We asked the following research questions: What is the efficacy of a word-problem intervention focused on improving word-problem outcomes for fourth graders with MD? Is there a difference as to whether students use multiple equations or a single equation to represent multi-step word problems? Setting: We conducted this study in a large, urban school district in a southwestern state. Participants: We implemented a randomized-controlled trial to examine the effects of an intensive word-problem intervention for Grade 4 students identified with MD. After receiving approval from our university Institutional Review Board, we recruited and screened a total of 714 fourth graders. Students who performed <25th percentile on a word-problem measure (Jordan & Hanich, 2000) during screening were identified as at-risk for MD and deemed eligible for the study. Eligible fourth graders (n = 132) were randomly assigned to the word-problem intervention with multiple equations (n = 43), word-problem intervention with single equations (n = 47), or business-as-usual (n = 42). Intervention: The word-problem interventions included 36 individual lessons with the interventionist, implemented three times per week, with each session lasting 30 min. Each lesson included five activities: (1) math fact flash cards, (2) interventionist-led activities about the equal sign, (3) interventionist-led lesson featuring schema instruction, (4) schema sorting practice, and (5) cumulative review. In Activity #1, interventionists showed addition, subtraction, multiplication, and division math fact flash cards to students during two, 1-min timings. At the end of the second 1min timing, students graphed the highest score from the two trials. During Activity #2, students received instruction on solving equations and the meaning of the equal sign. To understand the equal sign as a relational symbol, students solved standard and nonstandard equations with parentheses and multiple operator symbols using concrete manipulatives, hand-drawn pictures, or equations presented with numbers and symbols. For Activity #3, interventionists provided scaffolded instruction to students to set up and solve three word-problem schemas. The intervention included single- and multi-step word problems featuring the combinations of the following schemas: total, difference, change, equal groups, and equal shares. Students in the multiple-equations intervention learned to use a separate equation for each schema within multi-schema word problems. Students in the single-equation intervention learned to use a single equation that represented multiple schemas within a multi-schema equation (e.g., (9 + 4) x 3). In Activity #4, interventionists read aloud word-problem flash cards for 1 min as students identified the problem type. In Activity #5, students participated in a paper-and-pencil review, which consisted of completing 9 fluency problems in 1 min and one word problem in 2 min. Interventionist fidelity data for the 12 interventionists was collected on >20% of sessions through in-person supervisory observations and analysis of audio-recorded sessions. After intervention, all eligible students were post-tested to determine if differences existed across the intervention and control conditions. Research Design: This study is a randomized-controlled trial with student participants assigned to conditions. Data Analysis: The data collected for this presentation is from Year 1 of a third-year study in which we collect data from three cohorts of students. We will present data collected from the 2021-2022 school year. Mulit-level modeling will be used to analyze the data once the three cohorts of data collection has occurred. Preliminary Findings: Preliminary findings indicates differences conditions on a word-problem outcome measure, F(2, 115), 22.022, p < 0.001. Contrasts indicated students receiving word-problem intervention with either multiple equations (p < 0.001, CI [4.21, 8.19]) or single equations (p < 0.001, CI [3.51, 7.71]) outperformed students in the BaU. There was no difference between active word-problem conditions, p = 0.562, CI[-1.42, 2.59].

Background/Context: The purpose of this paper is to highlight the importance of supporting educator, and particularly teacher, mental health and well-being when working to improve educational systems. Although education, and teaching in particular, has long been characterized as a stressful field of work (Kyriacou, 2001), outside of occupational burnout, there is a paucity of research on the prevalence of stress-related symptoms among educators (Hirshberg et al., 2023). Understanding the prevalence of symptom clusters such as stress, anxiety and depression is important because high levels of these symptoms are prospectively associated with occupational outcomes across diverse occupations. Since the beginning of the COVID-19 pandemic, several studies have reported very high rates of teacher and/or educator occupational burnout, stress, and anxiety and depressive symptoms (Hirshberg et al., 2023; Kush et al., 2022; Pressley, 2021). These studies suggest that greater attention and resources need to be allocated to supporting educator mental health. However, these data were collected during the enormous uncertainty of the first two years of a global pandemic, raising the possibility that educator mental health has normalized as pre-pandemic life routines have returned. Using baseline data from a randomized controlled trial conducted in the 2022-2023 school year, we tested the preregistered hypothesis that COVID-related fears would explain little variance in educator symptoms of stress, anxiety, depression and occupational burnout, that symptoms will remain high even though life has returned to pre-pandemic routines, and that features of the school environment will moderate symptoms levels. We will also report on preregistered replication hypotheses that, just as during the initial phases of the pandemic, assignment to a four-week smartphone-based well-being training would significantly improve educator mental health relative to assignment to an attention control condition. Setting: Public school employees in a large Southern pK-12 school district that serves a high proportion of students from low resources environments. The intervention occurred via an app on participants phones. Population/Participants/Subjects: Participants were 830 employees of a public school district who worked in that district in any capacity. Around 54% of participants were lead classroom or special education teachers. Eighty-five percent self-reported as female, 71% as White, 13% as Black, 3% as Hispanic or Latino, with the remaining 11% reporting as more than one race or Asian, American Indian or Alaskan Native, or Native Hawaiian (<1% each). Intervention/Program/Practice: The Healthy Minds Program (HMP) is a smartphone-based well-being training constructed around a new, four pillar model of well-being (Dahl et al., 2020). The awareness pillar includes mindfulness and other attention-related practices. The connection pillar consists of loving-kindness, compassion, and additional techniques to strengthen prosocial dispositions. Insight practices aim to clarify the function of the self and the processes by which experience, memory, and identity inform perception. Purpose practices are intended to clarify and help one embody deeply held values throughout daily activities. The full HMP is one-year long, with three months of content for each ACIP pillar. However, the app is highly modifiable, allowing researchers to customize program features, including duration. Content in the app takes two primary forms: "learns" and "practices." Learns provide a didactic overview of the program's theoretical framework along with summaries of relevant research supporting these theories. Practices provide instruction in ACIP techniques intended to strengthen specific skills. In this study, a conceptual replication of a prior randomized controlled trial of the app in educators (Hirshberg et al., 2022), we used a four-week version of the app that had one week of content for each pillar. Research Design: This study was a two-arm, randomized attention controlled trial of the Healthy Minds Program app. Primary outcomes were assessed at baseline (T1), weekly during the intervention (T2-T4), post-intervention (T5), and at five month follow-up (T6). Secondary outcomes were assessed at T1, T5, and T6. Process measures hypothesized to be mechanisms of improved mental health were assessed at T1-T5. Data Collection and Analysis: Most of the data were self-reported. Ecological momentary assessment was used around T1, T5 and T6 but is not reported on here. Administrative records of 2023-2024 school year employment will be provided. Administrative records of students of teacher participants will be provided by the district in the summer of 2023 and 2024. Baseline data was analyzed with Pearson product moment correlations, linear regressions, and logistic regressions. Trial data will be analyzed using linear mixed effects models with time nested within participants. The time (T1-T5 and T1-T6) by GROUP (treatment, control) interactions are of primary interest. The primary preregistered outcome is an aggregated of stress, anxiety, and depressive symptoms (psychological distress). Secondary outcomes include well-being, occupational burnout, loneliness, and perceptions of the work environment. Findings/Results: Analyses of baseline data demonstrate that educator mental health remains at crisis levels even as life has returned to normal. On average, this sample of educators' anxiety and depressive symptoms were around one standard deviation and 0.6 standard deviations above the population average, respectively. More than 85% of participants reported moderate or high stress, with 44% reporting clinically significant levels of anxiety and 34% clinically meaningful levels of depressive symptoms. Although COVID-related anxiety explains significant variance in both anxiety and depression (around 7%), most of the variance is not associated with COVID fears and school contextual features explain relatively more variance mental health (e.g., administrator trust 8%, satisfaction with co-workers 9%). Trial data will be analyzed over the summer of 2024. Conclusions: Stress, anxiety and depression remain at alarmingly high levels among educators, with significant implications for school systems nationwide. Addressing the mental health crisis among educators demands an urgent and coordinated public health campaign that shows no substantive signs of emerging. Should the replication hypotheses be supported by the data, mobile health well-being training like the Healthy Minds Program app, which is free to use, may be an accessible and effective approach to supporting educator mental health and wellbeing, and potentially supporting improved educator effectiveness.

Context: The use of educational technology for improving K-12 math education has expanded dramatically recently. ASSISTments (Heffernan & Heffernan, 2014), a widely used digital platform, has seen its use in schools increase significantly from 800 to 20,000 teachers over the past two years. A review of evidence-based online programs by the What Works Clearinghouse named ASSISTments as one of the few digital learning programs recommended for use in response to the COVID pandemic (Sahni et al., 2021). An efficacy study conducted in Maine (Roschelle et al., 2016; Murphy et al., 2020) found that using ASSISTments significantly increased student scores on the TerraNova test (Hedges' g = 0.22, p < 0.001). From 2018-2020 we conducted a replication study to test the replicability of the findings in a heterogeneous population that more closely matches national demographics. A follow-up study was designed to measure the long-term impact of ASSISTments on student learning in 2021, one year after the intervention was over. This paper reports on the findings from the preliminary analysis of the follow-up study. Research Questions: Two research questions motivate the follow-up study: RQ1: What is the long-term impact of ASSISTments on student math outcomes at the end of Grade 8, one year after the completion of the intervention? RQ2: Do the long-term effects of ASSISTments, if exist, vary for students of different socio-economic status, race/ethnicity, or with other policy-relevant characteristics? Sample: The replication study took place in North Carolina (NC). Sixty-three schools were recruited and randomly assigned to condition (32 intervention and 31 comparison). The sample included 48 Title I schools and distributed across rural, town, suburban, and city communities. The study enrolled 102 7th grade math teachers and their classrooms. Table 1 compares demographics of participating schools to NC and US populations, and the Maine study sample. A generalizability analysis suggested the replication sample is highly representative of NC schools (generalizability index = 0.958) and schools across the nation (generalizability index = 0.899). Intervention: ASSISTments is a web-based platform that supports students' math problem-solving through immediate feedback and hints and provides teachers with reports summarizing student work to inform instruction adaptation. The intervention aligns with theory- and empirically-based instructional practices of formative assessment (Heritage & Popham, 2013) and skill development (Koedinger et al., 2013). During the study, the intervention was positioned as online support for math homework, a key opportunity for student independent practice, and for teachers to use formative data to adapt instruction (Figure 1). Research Design: The replication study used a randomized experimental design. Schools were randomly assigned to either intervention or business-as-usual comparison condition. ASSISTments was implemented by all Grade 7 teachers in intervention schools over two consecutive years--teachers learned to use ASSISTments for a year (2018-19) and then we measured the immediate outcome for students at the end of teachers' second year of experience (2019-20). Students who were in Grade 7 in 2019-20 comprised the analytic sample for the research questions. The students maintained their conditions and were followed longitudinally for another year when their Grade 8 performance (long-term outcome) was measured in spring 2021 (see the study overview in Figure 2). During the follow-up year in 2020-21, no interventions were provided to 8th grade teachers or students. Data: We obtained demographic, enrollment, and state assessment performance data from the state-wide database for all 63 schools. Student's long-term learning outcome was measured by Grade 8 state standardized End of Grade (EoG) math test. Grade 6 EoG math test scale scores from spring 2019 served as the baseline measure. Individual demographic covariates included gender, race/ethnicity, and economically disadvantaged status (EDS). School-level covariates included average 6th grade EoG score, 7th grade enrollment size, Title 1 eligibility, percentage of students with EDS, and percentage of ethnic groups. There were 9,073 students who (a) have 6th grade EoG score, and (b) enrolled in a participating school either as a 6th-grader in 2018-19, or as a 7th-grader in 2019-20 if the school didn't have 6th grade in 2018-19. Among these students, 5,991 have 8th-grade EoG test scores (2,961 intervention, 3,030 comparison) and were included in the analysis. 34% students (n=3,082) took an accelerated or alternative math course and did not take the Grade 8 EoG test. Their performance was analyzed separately and were not included in the paper. Analysis and Findings: There was no school-level attrition. Overall student-level attrition on Grade 8 EoG test was 33.97% (mostly due to the accelerated alternative math course enrollment) and the differential attrition was 0.31%. Counting all students with non-missing outcome data, the overall and differential attritions were 12.83% and 1.81%, respectively. We examined baseline equivalence on students' 6th grade EoG scores, gender, and ethnicity and found no significant differences between conditions. To address RQ 1, we conduct an intent-to-treat (ITT) analysis using two-level hierarchical linear regression models (students nested within schools), controlling for student's Grade 6 scores and other student- and school-level covariates. To address RQ 2, we added a cross-level interaction term of the school-level treatment variable and student subgroup indicator and conducted a series of moderator analysis to examine whether ASSISTments had differential impacts on the students with policy-relevant background characteristics. Results showed that students at intervention schools performed significantly better than those in the comparison group on 8th grade EoG (p = 0.011, g = 0.10) (Table 2). The moderator analyses showed that the intervention benefited minority (non-White) students significantly more than majority (White) students (p = 0.003, g = 0.14) and that the impact was more profound for Hispanic students than non-Hispanic students (p = 0.014, g = 0.13) (Table 3 and Figure 3). Conclusions: The follow-up study was motivated by the need to explore the long-term impact of ASSISTments use with diverse populations. The results suggested that it had a sustained long-term impact on students' math learning, after one year of intervention. The program also helped close the achievement gap among students of different ethnicities. Future works include additional moderator or mediator analyses on ASSISTments usage in 8th grade, benchmarking the effects, and estimating the cost-effectiveness.

Background: The need for evidence-based, effective interventions is especially great in literacy -- particularly for young students, mastering early literacy skills is predictive of later school success (Slavin et al., 2011). Understanding the relationship between letters and sounds, also known as the alphabetic principle, is essential for children to develop the solid foundation required to build phonics and phonemic awareness skills (Boyer & Ehri, 2011; Byrne & Fielding-Barnsiey, 1989). These skills are particularly difficult to learn for students who are deaf or hard of hearing (D/HH). Limited access to sounds makes literacy development challenging -- so that evidence-based explicit interventions are necessary (Easterbrooks & Huston, 2008; Kyle & Harris, 2011). While a broad literature exists, and has been synthesized, regarding reading and writing interventions for struggling or at-risk readers, this same effort has not been conducted for students who are D/HH (Easterbrooks & Dostal, 2020; Strassman & Schirmer, 2013). Objective: This study aims to produce a comprehensive and rigorous up-to-date review on literacy programs specifically developed for students who are D/HH. Through its work, the study expects to promote effective use of proven literacy programs for these students. Setting: The review included findings from studies that examined the efficacy of programs and practices designed to improve literacy outcomes for students who are deaf or hard of hearing in schools serving students from grades preK-12. Population: The review included studies of students who are deaf or hard of hearing in grades preK to 12. Intervention: Interventions examined included reading and writing programs and practices designed to improve literacy outcomes for students who are deaf or hard of hearing. Research Design: This study uses a best-evidence synthesis approach (Slavin, 1986), which combines traditional techniques of systematic review with narrative descriptions of individual programs and studies. Data Collection and Analysis: Inclusion Criteria The review used the following inclusion criteria: 1. Published/publicly available in 2010 or later. 2. Evaluated the effects of a reading or writing intervention on literacy outcomes of children who are D/HH. 3. Study population are students in PreK through grade 12. Literature Search Procedure: A broad literature search was carried out in an attempt to locate every study that could possibly meet the inclusion requirements. Electronic searches were made of educational databases, including ERIC, EBSCO, and PsycInfo. Other internet search engines, educational publisher websites, and third-party evaluator websites were also searched. Citations from identified studies and previous reviews of interventions were examined for possible inclusion. Study Review and Coding Procedures: Once potential studies were identified, we screened studies for relevance, and reviewed the study full texts against the inclusion criteria using the Covidence online platform. Finally, we extracted details of the included studies by coding them into a standardized database. Findings: A total of 85 studies were identified that met the inclusion criteria (Figure 1). Overall, these studies exhibited an assorted set of methodological approaches, with 69% of studies using single subject designs and 31% of studies using comparison group designs including quasi-experimental and randomized designs. While many studies were methodologically sound, some studies exhibited flaws such as issues with interrater reliability (Beal-Alvarez et al., 2012). The specific approaches of the interventions in the included studies fell into three main categories: programs targeting vocabulary (e.g., Teach Your Child to Read in 100 Easy Lessons paired with Visual Phonics, Bedrock Literacy), programs targeting writing skills (e.g., Strategic and Interactive Writing Intervention, Self-Regulated Strategy Development), and programs targeting phonics and decoding skills (e.g., Foundations for Literacy, Corrective Reading-Decoding with Visual Phonics). Within each category, there were effective programs identified. There was substantial heterogeneity across these studies. The settings for these studies were also quite varied, ranging from residential schools for the deaf, to classrooms within public schools. Additionally, the populations were quite diverse, with varying levels of hearing loss and access to amplification via hearing aids and cochlear implants, as well as students with languages other than sign language or English spoken at home. This diversity highlights that this student population cannot be treated as a homogenous block, and that focusing on means may ignore that the effectiveness of these interventions is likely population- and context-dependent. Finding a match between program and user is likely key. Conclusions: Many programs exist to support literacy instruction for students who are deaf or hard of hearing. While the effectiveness of these programs is variable, effective models do exist. Schools should be encouraged to choose from among these proven programs to ensure all students have access to effective literacy instruction.

Background: In the immediate aftermath of school closures related to COVID-19, efforts to unpack the impact of the pandemic on student outcomes surged. As schools have returned to fully in-person learning, that focus has shifted towards the implementation and evaluation of interventions to remediate or accelerate impacted learning. However, measuring the ongoing effects of the pandemic on students is crucial for districts and states to be able to monitor when recovery is taking place, and for which students--not only for education agency leaders, who rely on such data to inform the administration of interventions and application of ESSER funds, but also for the broader education field, who need these data to paint an accurate picture of the current state of student achievement and growth. Research Questions: This paper addresses the following six research questions: 1. What was the magnitude of cumulative average learning change, by grade and by state, over each phase of the COVID pandemic: (a) Fall 2019 (or Winter 2019/20) to Fall 2020, (b) Fall 2019 to Winter 2020/21, (c) Fall 2019 to Spring 2021, (d) Fall 2019 to Fall 2021, and (e) Fall 2019 to Winter 2021/22 (with data for this last period available in Winter 2022) 2. Did magnitude of learning change differ for different types of students, particularly along African American students, economically disadvantage students and students with lower prior achievement? 3. What was the magnitude of recovery, if any, from learning losses experienced during the early phases of the pandemic? Are there any heterogeneous effects of recovery along the dimension of race or Data Collection. We utilize a rich historical set of data that includes spring state assessment data and interim assessment data measured in the fall, winter, and spring over multiple years before and after onset of the pandemic. A major strength of the research is that it utilizes assessment data from multiple state assessments and from multiple interim assessments, including NWEA MAP, Renaissance Star, Curriculum Associates i-Ready, and HMH Reading Inventory and Math Inventory. Hence, as mentioned above, the results are expected to provide evidence on whether learning loss is sensitive to the type of assessment used to measure student achievement and to assess the robustness of the results. Methods: This paper uses multi-level achievement growth models (Chetty, Friedman, and Rockoff (2014) and Kane and Staiger (2008)) designed to identify the effects of the covid pandemic on growth in student achievement over the period Fall 2019 to Winter 2021/22, using prior data from school years 2017-18 and 2018-19 as baseline, pre-covid data. The models embody features of evaluation models and standard growth and value-added models. The evaluation lens prompts us to view the COVID pandemic as a type of treatment (or intervention) and the pre-COVID period as a control. Moreover, given that some districts and schools provided instruction at home while other provided instruction in school (or a hybrid blend of the two), it is potentially important to consider these types of instruction as essentially different interventions. Consistent with this view, we address the need to control for possible differences in the treatment interventions and control periods and samples with respect to the composition of students and schools. These considerations are potentially important given that participation rates in interim assessments during the COVID-affected years have differed substantially from participation rates in prior years and the types of schools providing in-home versus in-school instruction likely have differed. Our models allow slope coefficients to vary across years. The model thus allows for the likely possibility that the effects of demographic characteristics and prior achievement may have changed because of the pandemic, with at-risk students experiencing lower growth than in prior years. This model addresses the concern that the expected decline in growth during covid-affected years may have been especially acute for some schools and for some students within schools. Findings: Although we intend to report findings based on the most up-to-date data at the time of the fall conference, our findings based on data through Fall 2021 confirm that learning loss has been substantial for students. However, the evidence suggests that the magnitude of learning loss varied across schools and was highly correlated with factors associated with school and family learning supports. Larger learning losses occurred for students who were enrolled in remote or hybrid learning compared to in-person learners. Conclusions: We develop a model for measuring learning change that estimates separate value-added growth models for comparable periods of time before and after the COVID-19 pandemic, treating the pre-COVID period as a type of control group and the post-COVID period as a type of treatment group. This approach explicitly models different impacts of COVID-19 on students with different characteristics. It can be used to measure learning change both on average and for students in particular demographic groups in a way that is robust to differences in school assignments, demographic makeup, and student achievement between available pre- and post-COVID student samples. Using data from multiple interim assessments in one state, we find evidence of persistent learning lag to Fall 2021, both overall and across students by economic disadvantage and by English language learner status, as shown in figure 1 through figure 3. We are planning on updating results using the latest data which will be available in summer 2023.

Background: Public schools are among the first civic institution with which many individuals have prolonged, meaningful social interaction. Lessons about the authority, power, and fairness of civic institutions, conveyed through disciplinary and social control practices, may be part of the "hidden curriculum," that teaches students about racially inequitable unspoken norms, beliefs, and values held by society. We summarize three pathways by which the discipline gap in schools may contribute to civic disempowerment. First, and most directly, schools are key providers of civic education. They provide students with knowledge and skills on topics like politics, history, government, and current events (1,2). Students who are excessively removed from the classroom because of exclusionary discipline are more likely to miss out on this education. Second, schools shape students' attitudes about civic engagement. Schools influence students' concern for the common good, their understanding of whether their school, as a representative of a civic institution, is fair and just, and their belief in whether it is possible to change systems for the better (3). The discipline gap been found to damage its victims' sense of fairness in school and to lead to disaffection and alienation (4-6). Third, education is associated with the financial security and job characteristics that make civic engagement more possible (7,8). The discipline gap has been strongly linked to risk of drop out and therefore to diminished economic well-being (7,9). Moreover, exclusionary discipline has been tied to increased risk of incarceration, which can lead to outright disenfranchisement (10). Purpose: Using Critical Race Theory, Social Cognitive Theory, and Social Control Theory, we propose that disparities in civic engagement may be among the long-term consequences of racially disproportionate discipline and social control in schools. We tested this hypothesis using voting behavior. This study builds on previous work in this area (11) first by strengthening the theoretical basis for a connection between exclusionary disciplinary experience and social control in school and civic engagement in adulthood and second by extending the empirical test of those theoretical bases. We predict that disproportionate discipline and control in schools are among the insults experienced by Black students that inform their social learning about the fairness of civic institutions and that therefore erode their sense of political efficacy thereby making them less inclined to vote. Our study tested three hypotheses: first, disciplined students will be less likely to vote as adults, with the effect size increasing with the number of suspensions received. Second, the number of social control measures in school will be continuously and negatively associated with voting as an adult. Third, the effects of both suspension and social control on voting behavior will be modified by race, with Black, non-Hispanic students experiencing greater reductions in voting than non-Black students. Methods: With national data from approximately 15,369 students who participated in the Education Longitudinal Study between 2002 and 2012, this retrospective cohort study examined the relationships between discipline (i.e., how often a 10th grade student was suspended), social control (i.e., the number of social control measures in place in their school), and voting behavior into adulthood and how those relationships varied by race after considering several individual-, household-, and school-level variables, including student perceptions of school climate. Following descriptive and bivariate analyses, we used multilevel logistic regression with household- and school-level fixed effects to test our hypotheses. Our first model looked at only main effects. In the second model we added the interaction between Black non-Hispanic race and discipline history. In the third model we included a second interaction term between Black non-Hispanic race and social control. Findings: As hypothesized, we found that the relationship between exclusionary discipline and voting behavior was negative and demonstrated a dose-response pattern. In model one, students who were suspended one or two times in their sophomore year had approximately 16.5% lower odds of being a regular voter relative to those who were never disciplined (p=0.003). Students suspended three or more times had 25.4% lower odds (p<0.001). The interaction of race and suspension history in Model Two was significant for students who had been suspended three or more times (p=0.021), confirming part of our interaction hypothesis. Taking the main and interaction effects into account, Black non-Hispanic students suspended one or two times had an odds of voting regularly that was 26.2% lower than non-Black students who had never been suspended (OR=0.738, p=0.059). Black students suspended three or more times had an odds of voting regularly that were 33.5% lower than non-Black students who had never been suspended (p=0.021). The relationship between number of social control measures and voting behavior was positive and significant. Every additional social control measure was associated with a 1.3% increase in the odds of regular voting (p=0.002); the interaction by race was not significant. This contradicted our hypotheses. Conclusion: Our discipline-related findings suggest that schools are testing grounds where students develop a lived experience of fairness and justice, which they may then apply to other civic activities. Future studies should use the paths enumerated above and richer, mixed methods data to examine the mechanisms at play. This understanding will be necessary to advise school administrators on how and why to address the discipline gap (e.g., through disciplinary policy reform and implementation of restorative alternatives). This study also encourages education researchers, practitioners, policymakers, and advocates to think about the long-term effects of the "hidden curriculum" from a racial equity perspective. At a moment in our history when American Democracy seems gravely at risk, we must think about how our institutions communicate racialized messages of privilege, power, and precarity.

Background/Context: Recently there has been increased emphasis on disaggregating students' experiences collected through surveys based on student identities (Arredondo, 2016). Yet demographic data collection is problematic by nature, often reducing the complexity of an individual's racial, gender, or sexual identity to a single category. Racial identity is much more complex than what is captured by commonly accepted racial categories. It includes three concepts: self-identity (the meaning individuals attribute to their race/ethnicity), category (which category individuals selected from the discrete survey options), and identification (the category observers would place them in) (Rockquemore et al., 2009). Like race, gender is not unidimensional and is a combination of biological sex at birth, gender identity (how an individual perceives their own gender), gender expression (how gender is presented), and gender label (how an individual labels or, in some cases, does not label the intersection of biology, identity, and expression) (Temkin et al., 2017). Similarly, sexual orientation is multidimensional and is defined in terms of sexual identity (how an individual labels their sexual orientation), sexual attraction (who an individual is and is not attracted to), and sexual behavior (whom an individual engages in sexual acts) (Temkin et al., 2017. Despite the need for data collection that includes racial, gender, and sexual orientation demographics, the discrete nature of survey questions diminishes the complexity of these identities into discrete and often mutually exclusive categories. Even with the most robust and comprehensive demographic questions, creating survey questions that fully capture the complexity of individual identities remains challenging. However, best practices suggest including an open-ended other question allowing individuals to self-report their identity. Purpose: This study examines how students in grades 3-12 self-identify when given the option to express their racial, gender, and sexual orientation through a write-in other choice on a survey. The questions guiding our study are: 1. How do students respond when they can self-identify their race, gender, or sexual identity? 2. In what ways do students' understanding of racial, gender, and sexual orientation differ from the discrete identity categories provided? Setting and Sample: The data analyzed in this study were drawn from the 2022 student restorative culture survey (SRCS) administered to students in grades 3-12 in a school district in New England. Across the district's eleven schools, 1,760 students in grades 3-12 completed the 2022 survey. Based on discrete demographic questions, Table 1. provides details of the survey respondents. Of the students who responded to the survey, 11% responded to the open-ended race option, 3% to gender, and 5% to sexual orientation. Research Design and Analysis: The 2022 survey was administered by the school district using GoogleForms during class time to allow teachers to answer clarifying questions that might arise. At the end of the survey, students were asked a series of demographic questions (see Appendix A) where they could check all that apply and/or fill in the opened 'Other' response. Our study employed a questionnaire variance design (Creswell & Clark, 2018) to analyze the open and closed-ended demographic questions. To analyze the responses, researchers one and two reviewed the responses independently and identified emerging themes. We then came together to discuss and refine the themes and identified cases where we did not converge and, through the discussion, further refined the themes into their finalized versions. The last review of the responses by each researcher was assessed for interrater reliability using Cohen's kappa (1968), which revealed near-perfect agreement (race = 0.82, gender = 0.88, sexual orientation = 0.84). Results: Racial Identity: Six themes emerged in the racial demographic category (see Table 2). Roughly 25% of students with an open-ended response also checked a racial category. Of these students, 8.7% selected white, 4.6% African, 3.5% Asian, Asian American, 3.6% Black/African American, 2.6% Multiracial, 1.5% American Indian/Alaskan Native, and <1% Hispanic or Latinx. Gender Identity: Five themes emerged in the gender demographic category (See Table 3). Roughly 26% of students with an open-ended response also checked a gender category. Of these students, five students selected 'boy' (33%), five students selected 'neither boy nor girl' (33%), three students selected 'girl' (20%), one student selected 'I am not sure yet' (6.6%), and one student selected 'Decline to answer' (6.6%). Sexual Orientation: Four themes emerged in the sexual orientation demographic category (See Table 4). Roughly 17% of students with an open-ended response also checked a sexual orientation category. Of these students, six selected 'Bisexual or Pansexual' (43%), three selected 'I am not sure yet' (21%), two selected 'Asexual' (14%), two selected 'I don't know what this question means' (14%), and finally one student selected 'Decline to answer' (7%). Conclusion: Study results highlight the importance of collecting robust student demographic questions within K-12 surveys. Even though the original demographic categories were more extensive than more commonly used survey demographic questions, many students still felt the need to self-identify regarding race, gender, and sexual orientation. These results reinforce the need for those collecting data on K-12 students to include categorical measures of identity and allow students to express their understanding of these complex constructs.

Background: Community colleges are the primary door to higher education for students of color and those from low-income backgrounds (Bailey et al., 2015; Calcagno et al., 2008). To raise attainment and reduce inequalities in completion, colleges have adopted many student success initiatives--from first-year experience programming and basic needs supports to early alert systems and curricular reforms (Crisp & Taggart, 2013; Goldrick-Rab, 2010). While many strategies are backed by causal evidence (WWC, 2016a, 2021a, 2021b), there is a dearth of information on their costs, including what resources are required to implement, scale, and sustain these strategies. Our work represents the most comprehensive set of cost studies around student success initiatives to date. Nested within a series of eight community college case across three states, we intensively (1) identify the resources required to operate a diverse array of student success programs, (2) pair these resources with national prices, and (3) identify an average cost of each initiative, including what costs are borne by various stakeholders. We then pair these costs with available causal evidence on impacts to identify which initiatives appear most cost-efficient. Purpose/Research Questions: This work makes a substantial contribution to the literature on student success programs, where existing cost studies rest scattered across few works (Belfield et al., 2016; Levin & Garcia, 2013). This work also serves as a guide for administrators and policymakers when considering which programs to implement and what resources are required to achieve equitable outcomes. By pairing causal impacts with costs, we provide immediately useful information for stakeholders seeking to efficiently and effectively raise attainment and equity. To achieve these aims, our work is guided by the following questions: 1. What resources are required to operate a given student success intervention? 2. What is the average cost of the intervention? What share of costs are borne by various stakeholders? 3. Which interventions are most cost-effective at raising outcomes? Setting/Interventions: Our work builds upon case studies that seek to understand how high-performing colleges achieve equitable student outcomes. Within each institution, they draw evidence from interviews with 20 or more internal and external stakeholders and identify a suite of interventions that (1) each college [individually and consistently] noted as essential to their student success strategy and (2) are backed by empirical evidence or internal institutional research. Our cost studies focus on these eight, diverse success interventions. Table 1 provides an overview of our partner colleges, including information on the size of each and their share of Pell-eligible and racially-minoritized students. Table 1 also identifies the interventions of interest, including: first- and second-year student experience (FYE/SYE), prison education/follow-up re-enrollment support; Black/men of color initiative; early alert with complementary supports; basic needs center; and college/career-readiness courses. Our final two colleges/interventions are currently being identified. Research Design, Data, and Analysis: We rely upon the ingredients method to holistically identify all resources (and quantities) required to operate each intervention (Levin et al., 2018). This includes a series of interviews with program leaders/staff, as well as iterative surveys and resource questionnaires. We use empirically-guided cost assumptions to price each ingredient and aggregate costs within functional categories (Shand & Bowden, 2021). Our prices are drawn from national sources (Amazon, BLS, CUPA-HR salary survey, and E$timator/CostOut). When necessary, each is adjusted for inflation, includes fringe, and amortized over time. Aggregating costs (sum of quantity x price) provides the total cost of each initiative, which we disaggregate to identify costs borne by stakeholders (informed by interviews/questionnaires; Levin et al., 2018). Finally, when causal evidence is available, we pair costs with impact estimates in a cost-effectiveness framework to identify which initiatives appear more (or most) efficient at raising a common student outcome (retention or completion; Levin et al., 2018). Findings/Results: Tables 2 and 4 present resource inventories and average costs for a basic needs center; Tables 3 and 5 for FYE/SYE. Cost studies for prison education/re-enrollment, Black/men of color, early alert, and college/career-readiness coursework are underway (to be completed by the end of spring). Cost studies for the remaining colleges and interventions will be completed by summer (in advance of the conference). The basic needs center provides students with food, clothing, and supplies, as well as grants for childcare, transportation, and emergency issues. It also connects students to community-based programs and services. These strategies are supported by causal evidence (WWC, 2020). A basic needs center of this scope uses approximately $1 million in resources/year. This particular center serves roughly 1,500 students/year, resulting in a resource use of $680/student. However, only 64% of costs are borne by the institution, while 36% are covered by donations/activities of external organizations. This FYE/SYE program provides priority registration, personal coaching, customized advising, and access to a Student Success Center. This is also supported by causal evidence (WWC, 2016b). A FYE/SYE program of this scope uses approximately $750,000 in resources/year. Given that it serves roughly 560 students/year, its resource use is $1,340/student. The vast majority of costs are in personnel who provide advising and coaching to each student for at least 4 hours per year, though students also face requirements to participate, including time commitments beyond that of coursework, resulting in induced costs of roughly $16,000/year. Upon the completion of all cost studies, we will leverage a cost-effectiveness framework to pair each cost with available evidence on the intervention's ability to raise a common retention or completion outcome, allowing us to identify which intervention(s) are most cost-efficient. Conclusion: In all, our work represents the most comprehensive set of cost studies around student success initiatives to date and not only makes a substantial contribution to existing literature on the costs of student success programs but also compiles actionable information for college administrators and policymakers when considering which programs to implement and what resources are required to achieve equitable student outcomes.

Background/Prior Research: Research shows that teacher quality is the single most powerful influence on student learning (e.g., Nye et al., 2004; Rivkin et al., 2005), yet teachers report that they are not well prepared to implement standards- based mathematics education with heterogeneous groups of students. Banilower et al. (2018) found in a national survey of science and mathematics teachers that only 41% of elementary school math teachers felt well prepared to differentiate mathematics instruction to meet the needs of diverse learners, and 15% felt well prepared to incorporate students' cultural backgrounds into mathematics instruction. Given teachers' self-reported lack of preparation to teach high-quality mathematics to diverse learners, it is not surprising that mathematics performance in the United States is low, especially for high-need student populations. Clarke and Hollingsworth (2002) propose that teacher quality, along with school context, classroom practice, and student mathematics achievement, are the key components to consider when conceptualizing teachers' professional growth, which is an ongoing, dynamic, and interactive enterprise (Figure 1). Our study seeks to add additional evidence to the literature that shows how teacher beliefs (dispositions, attitudes) about teaching and learning "serve as filters, frames, and guides for teacher practice"(Fives et al., 2014, p.249). We focus on teachers' self-efficacy beliefs related to their comfort and preparedness for teaching students with diverse needs and abilities. Teacher efficacy is an outcome that has been empirically linked to improved student achievement and teacher psychological well-being, adaptability, and adjustment (e.g., Goddard et al., 2007), and job satisfaction (Skaalvik & Skaalvik, 2014). We propose that PD programs that help teachers understand and respond to individual students' strengths and needs can support equity in the classroom by building teachers' preparedness and comfort, which enhances the likelihood of teachers' engaging in differentiated instruction to support all students. Intervention: Math for All (MFA) is a 40-hour PD program designed to equip Grades K-5 teachers with skills in using a neurodevelopmental (ND) framework (Barringer et al., 2010) to (a) analyze mathematical tasks, (b) develop student learning profiles, and (c) adapt mathematics lessons in ways that make them accessible to all students while maintaining the original goals of the lesson. Teachers participate in five workshops in which they learn about different aspects of the ND framework and use it to adapt and reflect on their own lessons. MFA is designed for both general and special education teachers, and an integral part of the professional learning is the collaboration between the two, as they plan adaptations for common lessons, implement them with students they share, and reflect on their work. Purpose: The goal of this paper is to present data from two randomized controlled trials (RCTs) and one quasi-experimental study (QED) designed to assess the impact of MFA. The six research questions we addressed through analyses presented in this paper are as follows: 1. What is the impact of MFA on K-5 teachers' reports of preparedness for teaching diverse students? 2. What is the impact of MFA on K-5 teachers' reports of comfort with teaching diverse students? 3. Does the impact of MFA on comfort and preparedness vary by mode of PD delivery (Math for All developers vs. local staff developers)? 4. Does the impact of MFA on comfort and preparedness vary by grade level (early elementary K-2 vs. upper elementary grades 3-5)? 5. Does the impact of MFA vary on comfort and preparedness by locale? and 6. How are preparedness and comfort related to teachers' reports of instructional practices? Data Collection & Analysis: In all three studies, we administered a teacher survey at the beginning and end of the school year that included measures of teacher preparedness and comfort. For the two RCTs, we also administered instructional logs that asked teachers to report on their practices at the end of several targeted weeks. Internal consistency of all three scales is high, as measured by Cronbach alphas. We tested multilevel models (MLMs) that included the baseline scores and an indicator for study condition (MFA PD or business as usual [BAU]). Findings and Conclusions: This body of work provides strong causal evidence based on two RCTs that MFA had an impact (i.e., statistically significant, medium-to-large effect sizes) on teachers' self-reported sense of comfort and preparedness for teaching diverse students. This is critical, given repeated survey findings that most teachers report feeling underprepared to work with diverse students (Banilower et al., 2013; 2018). Although this study has several key limitations (e.g., large baseline differences in RCT #1; lower teacher participation rates in the QED), the consistency of results across both RCTs and the QED indicates stability in the patterns of findings, which point to positive impacts of MFA on elementary teachers' comfort and preparedness for teaching diverse students. The fact that the results were similar across the three studies, even though the modes of implementation and PD dosage differed, also speaks to the scalability of the MFA program. The RCT data demonstrate consistent, modest, positive correlations among preparedness, comfort, and instructional practices, which complement prior reports of the positive impact of MFA on instructional practices, as measured via classroom observations of mathematics lessons and scores using the CLASS rubric (Duncan et al., 2018). As we seek to understand the impacts of PD on teacher and student outcomes, it is important to "open the mystery box" and flesh out the mechanisms by which PD can affect teacher practice (which in turn, affects student performance). The data presented here suggest that teacher preparedness and comfort may be key mediators to consider in our models of teacher quality.

Background/Context: Accelerated Study in Associate Programs (ASAP) is a comprehensive program developed by the City University of New York (CUNY) that has demonstrated remarkable success for associate degree-seeking students, nearly doubling the three-year graduation rate in a randomized controlled trial (Scrivener, et al., 2015). The model has been replicated in seven states, with similar impacts (Miller, et al., 2020), and has been noted as an evidence-based model by researchers, policymakers and funders. Many public institutions serving bachelor's degree-seeking students also struggle with low on-time graduation rates -- nationally, the most recent four-year graduation rate for first-time full-time students was 42.4% and rates were lower for students who were Black (24.4%) and Hispanic (32.6%) than White (46.9%) and Asian or Pacific Islander (50.6%) (USDOE, 2021). At CUNY, the most recent four-year graduation rate was 36.2%, with similar differences by race and ethnicity (CUNY OAREDA, 2023). To address these low graduation rates, CUNY implemented a version of the ASAP model, called Accelerate, Complete, Engage (ACE), for bachelor's degree-seeking students. This study seeks to test the effectiveness of ACE to understand whether the ASAP|ACE model works for bachelor's degree students as well. Research Questions: (1) ("Confirmatory") Do students who participate in the CUNY ACE program have a higher "on-time" (four-year) bachelor's degree graduation rate than they would have had if they did not participate in the program? and (2) ("Exploratory") For all outcomes, do the results vary for students across different subgroups of sex, race/ethnicity, and socioeconomic background? Four-year graduation is defined as having earned a bachelor's degree prior to the beginning of fall 2022 (yes vs. no). ACE is hypothesized to increase "on-time" graduation rates and help students graduate more quickly. Five-year graduation will also be analyzed for the final report. Setting: The study was launched in fall 2018 at John Jay College of Criminal Justice, the first CUNY senior college to implement the ACE model. Participants: ACE is designed to serve a broad population of first-time students. Students were eligible for the study if they met criteria including: applying for financial aid, meeting residency requirements and enrolling in approved majors. A total of 570 students enrolled in the study, with 285 assigned to each condition. Close to half of participants identified only as Hispanic or Latino/a (48.3%), with smaller percentages identifying only as White (14.2%), Black (10.2%) and Asian or Pacific Islander (9.7%). The majority of participants were Female (70.0%) and were eligible for Pell grants (71.8%). Students were on average 18 years old at the start and half (50.4%) were the first generation in their family to go to college. Program Model: Students were initially eligible to receive support for up to four years, although funding was later secured for a fifth year. ACE is a comprehensive program that includes the following components: (1) Early Engagement and Community Building: An ACE orientation in the summer prior to matriculation and community building activities offered each semester; (2) Academic Momentum: Expectations included attempting at least 15 credits each semester and enrolling in summer or winter sessions as needed to stay on track; (3) Advising and Career Development: Advisors had relatively small caseloads (maximum of 150 students) and were assigned to each student, staying with them from entry through graduation, which allowed them to build strong relationships and provide personalized support. Students were required to meet their advisor one-on-one each month and attend a monthly group seminar. Data and technology tools were used to help advisors effectively serve students. Students received career development support through workshops and activities; (4) Academic Support: Students were strongly encouraged to utilize tutoring or other services; and (5) Financial Supports: Tuition and fee gap coverage, funds for textbooks and course materials, and a MetroCard for public transportation on New York City subways and buses. Research Design: The study used an experimental design in which students were randomly assigned to the program or control condition with a 50:50 ratio. Baseline equivalence analysis confirmed that the two groups were equivalent across all key variables. Data Collection and Analysis: Data were merged from the CUNY Institutional Research Database and the National Student Clearinghouse (NSC), capturing data on enrollment and degrees at postsecondary institutions nationwide. Intention-to-treat (ITT) analyses were conducted using logistic regressions with pre-program control variables and effect sizes were calculated to measure the practical importance of findings in addition to statistical significance. Findings: The ACE program had a statistically significant positive impact on the likelihood of graduating with a bachelor's degree within four years. ACE students had a higher graduation rate (58.8%) than the control students (46.4%), a difference of 12.4 percentage points with a notable effect size, considered substantively important (Cox index = 0.304). Exploratory analyses found positive effects for nearly all subgroups, although only those for the largest subgroups were statistically significant. Effects for Hispanic or Latino/a, Black, White and Asian or Pacific Islander students were close in size (ranging from 16.7 to 20.3 percentage points) and effects were also similar in size for male and female students (13.0 and 11.4 percentage points). The program had a larger effect on students who were not eligible for Pell grants (21.3 percentage points) than those who were eligible (11.1 percentage points). Conclusions: This study found that ACE has a statistically significant and substantively important positive impact on four-year graduation, with similar estimated effects across lines of race/ethnicity and sex. CUNY has expanded ACE with a focus on serving transfer students with an associate degree, providing a future opportunity to analyze impacts across colleges and to better understand the generalizability of the model. Study limitations include: the sample size was not large enough to rigorously investigate potential differential effects and estimate whether ACE narrows equity gaps; implementation data were not collected to assess service contrast; the study relied only on administrative data and did not include qualitative data or reflect student voices. The results from the study to date suggest that the ACE program has provided critical student support and that it may be a promising model for institutions to adopt as they invest in student success initiatives.

Background and Purpose: Sleep is an important determinant of health in adolescence, but many adolescents around the world are sleep-deprived. Compared to children and adults, getting enough sleep is especially challenging for adolescents due to the natural alteration of circadian rhythms that govern sleep-wake cycles. The delayed circadian rhythm does not necessarily lead to sleep deprivation unless late morning sleep is disturbed. Unfortunately, early school start times require adolescents to wake up early in the morning when their body "clocks" point to sleep time. As the negative impacts of inadequate sleep of adolescents have been increasingly recognized over the past decade, professional organizations, including American Academy of Pediatrics, have issued policy statements advocating secondary schools begin at 8:30 or later. With this increased attention to later school start times, many studies have investigated relationships between school start times and various students' outcomes. With respect to mood and substance use outcomes, while some correlational studies show a relationship between later start times and better mood and lower substance use, recent reviews suggest that strong evidence of the causal impact of school start time policies is missing. To fill this gap in knowledge, the present study exploiting a natural experiment in Korea that delayed school start times to between 8:30 and 9:00 am and leveraged a nationally-representative longitudinal survey to examine the impact of the policy on adolescents' sleep and substance use. Data and Participation: This study used the first to sixth waves of the Korean Children and Youth Panel Survey 2010 (KCYPS), which is longitudinal, nationally representative data of 7th graders in 2010. Administered by the National Youth Policy Institute, KCYPS annually tracked 2351 7th graders from 2010 to 2016, a year after they graduated from high school. KCYPS collected information on demographics, sleep, socio-emotional well-being, health, and developmental environment including family background. Importantly, this survey includes data on students' outcomes before and after a later school start time policy. Our outcomes of interest were sleep and substance use. The analytic sample was limited to students in either treatment regions or comparison regions, and students who moved across regions across waves were excluded. Since the final survey (i.e., 7th wave) was conducted after high school graduation when students were no longer exposed to the policy, it is excluded from the analysis. The final analytic sample sizes varied across models because some outcomes were surveyed only at some waves. To be specific, the final sample sizes were 6,324 observations from 1,133 students for models with sleep outcomes and 5,279 observations from 1,120 students for substance use outcomes. Policy Intervention: In September 2014, Gyeonggi province in Korea implemented a 9:00 am school start policy, which requires all elementary and secondary schools in the region to begin at 9:00 am. Following Gyeonggi province, some other regions implemented similar policies: Jeollabuk-do delayed school start time by 30 minutes in 2014, and Gwangju and Incheon delayed school start times to 8:30-9:00 and 8:40-9:00 am in 2015, respectively. Given that more than half of schools started before 8:00 am prior to policy adoption, school start times have been delayed approximately 30 minutes to one hour as a result of the policy. Research Design and Analysis: We employed a difference-in-differences (DD) design to estimate the impacts of later school start times on students' sleep and substance use, leveraging variation in a later school start time policy adoption across regions and years. Specifically, using a two-way fixed effect (TWFE) approach, which is an extension of the canonical 2X2 DD design, we compared changes in outcomes of adolescents before and after a later school start time policy adoption in treatment regions with changes in comparison regions. Our main model includes individual and year fixed effects to adjust time-invariant factors and general trends in outcomes. In addition to the main model, we estimated a model with time-varying covariates to account for time-varying individual-level factors. We clustered the standard errors at the region level to account for the nested characteristics of data and the level of treatment (i.e., students within regions) and account for heteroscedasticity. Also, we used sampling weights to make the analysis nationally representative. We conducted several supplemental analyses to better understand dynamic effects of later school start times and to check the robustness of our findings. First, we conducted event-study analyses of sleep and substance outcomes to identify the dynamic effect of later school start times and to examine the pre-treatment parallel trend assumption--that in the absence of a later school start time policy, trends in treatment and comparison regions would have been similar. In this model specification, we included pre-policy years (i.e., leads) and distinct post-policy years (i.e., lags) instead of one treatment indicator. Next, we incorporated recent advances in the econometrics literature to address potential bias in the TWFE approach. Findings: Adoption of a later school start time policy was initially associated with 20 minutes increase in sleep duration driven by delayed wake time and consistent bedtime. In addition, the policy was associated with statistically significant reductions in monthly smoking and drinking frequencies. However, approximately a year after implementation, the observed increase in sleep duration shrank to 6 minutes and became statistically nonsignificant. Similarly, the observed reduction in smoking and drinking was attenuated a year after. Conclusion: We demonstrated that policies that delay school start times can have health benefits beyond increases in sleep duration--getting more sleep can reduce unhealthy behaviors for adolescents. Our study also provides the first evidence of the benefits of delayed school start time policies in East Asia, and has insights on the sleep-health link for policymakers across Global regions. However, our results also serve as a warning to policymakers that, without measures to ensure the persistence of increased sleep, the benefits of delayed start times may be relatively short-lived.

Background: In educational settings, individuals are often best served by an intervention that is adapted over sequential stages to suit their initial and changing needs. The salience of an adaptive intervention is, perhaps, most clear in the classroom. Learning itself is a sequential process: mastering a given concept or technique frequently necessitates a thorough understanding of the preceding concepts. Following an initial lesson or assignment, a classroom teacher may begin monitoring each student to identify those meeting or failing to meet criteria for early signs of success, and then offer each student targeted support based on their needs (Dawson et al., 2014; Arendale, 1994; Rowan et al., 2019). Outside the classroom, as well, there are myriad scenarios where it may be necessary to adapt and re-adapt intervention, both at the school and school-district level. Increasingly, there is interest by educators in determining how best to make sequences of intervention decisions (Raudenbush, 2008). Such "adaptive interventions" (AI) determine which treatment should be offered to a student or participant at any given stage of the intervention. Also referred to as dynamic treatment regimens or dynamic instructional regimes (Raudenbush, 2008), these adaptive interventions tailor the treatment to best-serve the needs of the participants. To optimize an AI, researchers may utilize sequential, multiple assignment randomized trials, or SMARTs (Lavori and Dawson, 2004; Murphy, 2005). SMARTs are a type of factorial design (Murphy and Bingham, 2009) where some or all participants are randomized multiple times to one or more treatment options at critical decision points in the AI (Almirall et al., 2018). SMARTs, frequently utilized in the medical and behavioral intervention sciences, are growing in popularity in education sciences as well. For example, Kim et al. (2019) and Fleury and Towson (2021) use SMARTs to inform development of AIs aimed, respectively, at personalizing print and digital content for early elementary students and at improving reading in preschool children with autism. Purpose: We introduce statisticians and methodologists in education sciences to adaptive interventions and SMART designs. Next, we provide education scientists with a suite of easy-to-implement techniques that may lead to increased statistical efficiency when analyzing data from a SMART. We demonstrate the benefits of the efficiency techniques through a comprehensive simulation study. Research Design: An adaptive intervention (AI) is a pre-specified set of decision rules that guides how best to serve the baseline and ongoing needs of individuals (Seewald et al., 2020a). These rules tailor the provision of treatment at critical points throughout the intervention. There are four aspects of AIs: decision points, intervention options, decision rules, and tailoring variables (Seewald et al., 2020a). Decision points are the times at which an intervention decision is made; the set of interventions available at a decision point are the intervention options. These may include, among others, different interventions or different intensities of interventions. The decision rule determines which option is selected for an individual at a given decision point. The decision rule makes this determination based on the value of one or more tailoring variables, which consist of known information collected prior to or at the current decision point. In SMARTs, which are used to optimize AIs, participants take part in multiple stages of the intervention, corresponding to each decision point where individuals may be randomized to two or more intervention options. There are many different SMART designs, but we focus on the prototypical SMART, seen in Figure 1. In the prototypical SMART, all participants are randomized during the first stage of the intervention. At subsequent stages, only non-responders are re-randomized to an adjusted intervention. Techniques to Increase Efficiency: In this paper, we present four techniques that may be used to improve statistical efficiency when analyzing data from a SMART. Technique 1: It is well-known that incorporating baseline covariates may increase efficiency in outcome analysis (Bloom et al., 2007). For example, controlling for a pre-test score will often substantially improve precision. The gains in efficiency should typically remain when analyzing SMART data as well. Technique 2: Rather than using known inverse probability of assignment weights, it may be possible to realize gains in efficiency by estimating the weights, either through the sample proportions assigned to each treatment or even through modeling (Hernan et al., 2002; Hirano et al., 2003; Brumback, 2009; Almirall et al., 2014). Technique 3: As is standard in many education studies, often we are able to collect longitudinal outcomes when conducting SMARTs. Obtaining longitudinal data (e.g. an intermediate outcome immediately preceding the second-stage randomization) should permit the researcher to realize substantial gains in efficiency. Technique 4: Rather than assuming a constant variance across time, model the variance as a function of the time point. Simulations: We present a comprehensive simulation study using modifications of the data generative models presented in Seewald et al. (2020b) in order to better understand whether, and under what conditions, the four techniques lead to improvements in statistical efficiency. We compare each technique to an analytical approach that adopts none of these techniques, estimating efficiency in terms of ratios of root mean-squared-errors (rMSE). Results: Results are presented in Table 1. When both [rho], the within-person correlation, and [nu], the correlation between outcome Y and covariate X, are small, the techniques provide marginal gains to efficiency. Increasing [rho] leads to greater gains in efficiency for each technique, but particularly for Techniques 3 and 4 which directly incorporate the longitudinal outcomes. Increasing [nu] generally corresponds to greater relative efficiency for each technique in comparison with the baseline scenario. This benefit is larger for Techniques 1 and 2, the two methods that rely on incorporating information from X into their estimation procedures. Conclusion: In this paper, we introduced adaptive interventions and SMARTs to education scientists. We presented four techniques that may increase statistical efficiency when analyzing SMART data. We showed through a comprehensive simulation study that all four techniques have the ability to improve efficiency.

Background: School district implementation of a four-day school week (4DSW) calendar has skyrocketed in recent years, despite little evidence regarding the impact on student and teacher outcomes. Between 1999 and 2019, the number of 4DSW districts increased by over 500% (Thompson et al., 2021). Since 2019, the rate of 4DSW adoption has increased even more. In Missouri, the setting for the study herein, the number of 4DSW districts -- mostly small and rural -- increased by over 300%, with over 30% of the state's 500+ districts operating a 4DSW schedule. There is still much to learn about the impacts of a 4DSW on students, teachers, and communities, however a growing body of literature finds mixed, yet mostly negative, effects on student achievement (Thompson & Ward, 2020; Morton et al., 2022; Thompson et al., 2022; Anderson & Walker, 2015; Morton, 2023; Morton, 2021). Furthermore, districts that adopt the 4DSW 1) serve significantly higher proportions of FRPL-eligible students than 5DSW districts (Thompson et al., 2021) and 2) are primarily small, rural districts that face unique staffing challenges (Thompson et al., 2021, Nguyen, 2020). It is therefore critical to advance our knowledge of the effects of a 4DSW to better inform policymakers, practitioners, families, and scholars. Prior to the COVID-19 pandemic, the most cited reason for adopting a 4DSW was financial pressure with teacher retention and other challenges particularly salient to rural districts as the second most cited reason (Thompson et al., 2021). Indeed, rural districts struggle to match salary offerings and lack other non-pecuniary benefits associated with non-rural areas (e.g., access to healthcare, amenities, etc) (Nguyen, 2020). Therefore, the 4DSW may be appealing to teachers and aid rural districts in keeping high-performing teachers, especially early-career teachers. More recent, post-pandemic, data suggests that teacher recruitment and retention has become the leading rationale district leaders cite for implementing a 4DSW, particularly in small, rural districts (Office of Education Policy, 2023). Despite this rationale I am aware of no study that examines the causal effects of the 4DSW on teacher turnover or retention. This study seeks to fill this crucial gap. Purpose: As mentioned above, rural districts may struggle to match salary offerings, lack other non-pecuniary benefits associated with non-rural areas, and face unique staffing challenges in comparison to their urban counterparts (Nguyen, 2020; Goldhaber et al., 2020); therefore, rural districts have been implementing 4DSWs as a teacher retention tactic. Empirical evidence finds that teachers are less likely to turnover in schools with better working conditions (Nguyen et al., 2020); to the extent that teachers perceive a 4DSW as an improved working condition, it would be expected that turnover decreases because of policy adoption. However, if a 4DSW is not enough to retain or attract teachers, then turnover would not be affected. This study answers the following research questions: (1) What is the effect of the 4-day school week on teacher retention? (a) To what extent does this effect vary across teacher demographics, including teacher race, gender, and years of experience? Setting and Population: This study examines 4DSW adoption in Missouri. As of the 2022-23 academic year, 144 school districts in Missouri -- nearly 30% of the state's districts - are using the 4DSW calendar, with even more slated to adopt the policy in the 2023-24 academic year. These districts are predominantly rural, though at least one urban district in the state has adopted the 4DSW. Panel data from 2005-06 through 2021-22 will come from the Missouri Department of Secondary and Elementary Education (DESE) and include 1) teacher-level administrative data and 2) student-level achievement and demographic data. Methods and Data: I will use difference-in-difference (DID) and time series models to answer the research questions. The primary DID model is as follows: y_isdt=??FDSW?_dt+A_isdt ?+B_st ?+C_dt ?+?_d+?_t+?_ist where y_isdt is an indicator of teacher turnover equal to 1 if teacher i in school s in district d in year t is not in the same district in year t+1. ? is the coefficient of interest and ?FDSW?_dt is an indicator variable equal to 1 in the year district d implements2 the FDSW. I will control for: teacher-level characteristics (A_isdt) including race, gender, education level, subject, and years of experience; school-level characteristics (B_st) including prior year teacher turnover, proportions of student race, gender, FRPL, ELL, gifted, and special education status; and comparable district-level characteristics (C_dt). ?_d are district fixed effects and ?_t are year fixed effects; ?_ist is the error term. An event study analysis will also be conducted, including the same terms but will estimate yearrelative-to-treatment point estimates rather than pooling post-treatment effects. Finally, due to the staggered timing of treatment, I will also test robustness of results to alternative DID estimators per de Chaisemartin & D'Haultfeuille (2020), Callaway & Sant'Anna (2021) and others.

Background: Evidence of educational improvement is essential for ensuring that we are providing students with meaningful and optimal learning experiences. But what counts as equitable evidence of educational improvement? Our presentation will discuss one organization's attempt to create an environment in which evidence is contextualized in ways that align with commitments to equity. Educational improvement is a complex construct that must be attended to in holistic ways. Such evidence must consider all of the ways a learning approach is at play in sociocultural interactions between students, teachers, and their environments, including institutional and cultural contexts (Cohen, et al., 2003; Kyriakides, et al., 2023). Impact is often measured as outcomes on standardized tests. However, the full impact of a learning approach cannot be captured in isolated data on outcomes or treated solely as cause and effect (Coburn & Turner, 2011; Firestone & Gonzalez, 2007). In alignment with an equity orientation in educational research, studies which aim to understand a learning approach's promise for impact should consider multiple perspectives and data sources. Quantitative and qualitative data should be combined - and well contextualized - to understand complex narratives and unpack nuances of teaching and learning. Further, educator and student voices should be leveraged when considering what evidence of impact is important. Finally, research in partnership with educators and students requires flexibility and agency to ensure studies are designed and implemented in ways that align with their partners' needs. Context and Objectives: We are an organization that works to actively manage teams of researchers, developers, and educators who are engaged in Inclusive Research and Development (R&D). Our model of Inclusive R&D (Angevine, et al., 2019; EF+Math Program, 2023) intentionally centers educators and students as critical partners in the design and study of learning approaches. Our community of R&D teams is working to explore the hypothesis that executive functioning, developed within the context of mathematics learning and combined with an equity-oriented approach to teaching and learning, can narrow opportunity gaps in mathematics outcomes for Black and Latinx students and students of all races experiencing poverty (EF+Math Program, 2019). The R&D teams are iteratively designing and researching learning approaches across grades 3-8 to understand the potential for supporting students' mathematics learning. In the current phase of our work, teams are engaging in preliminary impact studies while continuing to iterate upon their current approaches. As with any R&D work that is exploring a new hypothesis, there is a clear need for evidence of educational improvement to justify continued exploration of these learning approaches. We posit that what comprises that evidence must be responsive to the current state of education research and center the voices of educators and students who are being impacted by the learning approaches themselves. As an Inclusive R&D organization, we are positioned to take a stance on what counts as evidence of educational improvement in alignment with our equity orientation. In this session, we will provide insight to our organizational learnings about conducting impact evaluations of iteratively designed learning approaches. Specifically, we will discuss how our organization's approach promotes equity through flexible, yet explicit goal setting and an inclusive approach to data. Findings: First, our Inclusive R&D organization moves together towards shared goals, even though different approaches or processes may be unique in how these goals are achieved. The organization's goals are set in a way that supports contextualized progress; we are committed to providing guidance to R&D teams and partners that both hold an expectation of rigor and provide areas for adaptation within contexts. For example, pilot study guidance included expectations that data for preliminary studies should be collected, analyzed, and interpreted using methodologies and instruments that are culturally informed and developed with community input. This guidance is explicit in terms of expectations, but flexible for teams' enactment. As teams have had to adapt to the constraints and capacities of their partner districts and educators, their study designs and implementation features have shifted to serve their partners' needs. Teams have found ways to gather data regarding student and educator experiences that create opportunities for comparisons between implementations, while minimizing burden on participants. Educator and student data is leveraged to triangulate with other data sources across studies. Mixed methods approaches not only provide nuanced detail regarding particular aspects of an implementation, but also help to further contextualize data collected via quantitative approaches. Further, our organization is conducting preliminary impact studies, while still engaging in Inclusive R&D cycles. We are both aiming to gather rich data of the approaches' impact on student math learning and understanding how these approaches can continue to be improved. Our organization provides opportunities for educators to be embedded throughout both the design and research of the learning approaches. Partner teachers and students who are participating in the preliminary studies provide valuable feedback through surveys and interviews, sharing how an approach is useful to them and how it can be improved. Educators and students also have opportunities to provide insight regarding the ways growth is assessed throughout the study. Iteration is possible not only for the learning approaches themselves, but also in the ways our community selects measures and interprets data. Our Inclusive R&D approach allows for repeated opportunities to revise what is considered as evidence of impact, so that we can more closely connect research and practice, towards the goal of transforming teaching and learning. Conclusion: Our community is navigating tensions around needing to gather evidence of educational impact while also wanting to explore the deep nuances of practice that can lead to transforming learning experiences for priority students. Disrupting practices around educational effectiveness can be challenging when acting as an individual or small team; our organization has the opportunity to create change in terms of institutional practices and culture. We aim to share our current learnings and strategies with the community to consider ways that we can come together to push for more equitable notions of evidence of impact that are responsive to current educational contexts.

Background: Special education services are individualized supports provided to students with disabilities (SWD) that should help them make appropriate educational progress given their abilities, educational needs, and goals (Endrew F. v. Douglas County School District, 2017). Concerns about the benefits of these services have emerged given the increasing numbers of SWD and costs of providing services (Chambers et al., 2014; NCES, 2022), the widening opportunity gap between students with and without disabilities (NCES, 2021; Gilmour et al., 2013), the potential for stigmatization that may result from being labeled as needing services (Shifrer, 2013), and observations that special education services are often insufficiently intensive to result in appropriate educational progress (Gilmour et al., 2013; Vaughn & Wanzek, 2014). Yet, conclusions about the effects of being assigned to special education that are derived from correlational research or from studies that do not adjust for potential confounds (including adjusting for baseline "prima facie" differences between students with and without disabilities) can lead to mischaracterizations of their effectiveness. Objectives: To our knowledge, no review has assessed the state of causal evidence for the effectiveness of special education services. We help address this gap by examining special education treatment effects, both overall and across several related outcome domains, in a moderated, multivariate, multilevel meta-analysis framework. Results offer the field an evidence-based baseline of the extent to which receiving U.S. special education services may affect common academic and non-academic learning outcomes. We also evaluated whether these effects may be moderated by two different adjustments for baseline differences between students: whether the study used a quasi-experimental design [QED] or descriptive design, and whether effects were compared among SWD who vary in the amount and timing of their special education services, or relative to students who receive general education. Research Design: We searched several scholarly search engines and databases for studies meeting the following criteria: (1) Examined the relation between receiving business-as-usual special education services (i.e., not specific interventions or programs) and academic, behavioral, and/or social-emotional outcomes; (2) Article was peer-reviewed or grey literature (e.g., dissertation, working paper); (3) Occurred in pre-K through 12th grade settings in the United States; (4) Reported at least some numerical results corresponding to the effects of receiving special education; and (5) Utilized a QED or a descriptive research design (controlled for at least one covariate). This led to a final sample size of K = 40 studies using data from 23 datasets and N = 1,721 effect sizes for inclusion in our meta-analysis. Our full search process is displayed in Figure 1. Population: Sample sizes ranged from n = 68 to n = 14.4 million students (median treatment n = 690; median control n = 1,175). The mean age at special education service receipt was 9.09 years, and the mean age at outcome measurement was 10.62 years. See Table 1 for additional study information. Analysis: We analyzed multivariate effect sizes in a random-effects meta-analysis to deal with the potential for both nonindependence and correlated outcomes (Assink & Wibelink, 2016; Cheung, 2015; Konstantopoulos, 2011). We evaluated random rather than fixed effects because the underlying "true" effect sizes may differ across studies (Harrar et al., 2021). Because we often extracted multiple effect sizes from the same study (e.g., both achievement and behavior outcomes), and because some studies used the same datasets, we evaluated a threelevel meta-analysis in which participants were nested within studies, who were nested within dataset (Colich et al., 2020). We also used meta-regression to explore potential moderators of effects including type of outcome and study design/comparison group. All analyses were conducted in R version 4.1.1. Findings/Results: Results are presented in Table 2. As study designs more rigorously adjust for baseline differences (e.g., from left to right; columns indicate all included studies, QEDs, studies that only included SWD, and QEDs that only included SWD), more positive effects of receiving special education services are noted, suggesting that descriptive studies contrasting students with and without disabilities may be biased by baseline differences even with covariate adjustment. There were larger, positive effects for graduation (z = 0.16-0.24), postsecondary outcomes (z = 0.15-0.23), and other outcomes (z = 0.26-0.38) across study design. The average effect of special education on achievement reversed in directionality across study design, increasing from negative but non-significant (z = -0.07) to both positive and significant among QEDs that only included contrasts among SWD (z = 0.12). Similarly, the negative effect noted for behavior across all studies diminished in magnitude to an effect size that was not meaningfully or significantly different from zero among these more rigorous study designs (z = -0.07 to -0.01). We found little evidence for other moderation effects (Table 3) and no evidence of publication bias. Conclusions: We examined the effects of assignment to business-as-usual special education services rather than specific interventions or programs. We were unable to examine for the exact special education services or evidence-based practices delivered to SWD. The impact of special education services on academic achievement was positive but small in effect size across study designs that made a better attempt to adjust for baseline differences, ranging from z = 0.04 to 0.12. This result suggests that special education services may not be sufficiently intensive to improve academic achievement. Special education services may have a negative effect at worst (z = -0.13) and a neutral effect at best (z = -0.01) on student behavior. This finding is consistent with the idea that special education services might have both positive and negative effects on SWD. However, and optimistically, our results also imply that despite any immediate negative repercussions, referral for special education services may confer long-run benefits to SWD. The effects of special education may be especially positively associated with non-academic outcomes (e.g., truancy, grade repetition, physical health, college preparation, or family/home outcomes), with effect sizes ranging from z = 0.21 to 0.38. The relative lack of studies that utilize QEDs comparing SWD to other SWDs suggests that further causal research is needed to understand the mechanisms by which special education may bestow this longitudinal advantage.

Background/Context: Mathematics education, especially at early grade levels, continues to be an important focus for national education improvement efforts, and more districts are introducing technology as part of their solution. Along with increasing consistent delivery of personalized and individualized learning to students, technology- and game-based programs with high-quality, engaging content offer a scalable, efficient, and effective way to help learners master targeted skills, thus supporting teachers in providing tailored instruction. Furthermore, technology- and game-based programs, with their capacity to assess and generate data on student learning, can provide information that helps teachers better understand their students' learning and plan instruction. Highly engaging educational technology innovations targeting early elementary students, such as My Math Academy, can close the learning gap and prepare students for success in mathematics. As a supplemental curriculum, My Math Academy consists of game-based activities with adaptive learning trajectories, performance dashboards that help teachers support students' learning, and offline activities that extend in-game learning experiences. Purpose/Objective/Research Question: We are conducting a large-scale randomized controlled trial to examine the efficacy of My Math Academy, a widely used, game-based, supplemental online curriculum that enables early elementary school students' personalized learning through embedded assessments, adaptive learning trajectories, and a performance dashboard that helps teachers monitor student progress and personalize learning. The study will use school-level random assignment and address the following research questions: (1) Does My Math Academy improve students' achievement in mathematics as measured at the end of kindergarten? (2) Do students who use My Math Academy during both kindergarten and first grade score higher on math achievement assessment at the end of first grade than those in the control group? (3) What are the relationships between students' My Math Academy usage, their game-based performance, and their achievement in mathematics as measured at the end of kindergarten, and then at the end of first grade? (4) What factors moderate the effects of My Math Academy? Do the effects of My Math Academy vary for students with different demographic characteristics (e.g., gender, SES, race/ethnicity, or different prior achievement levels? Setting: The study will take place in kindergarten and first grade classrooms in public elementary schools. We seek to recruit a sample of schools that includes low-performing schools and schools serving students from diverse socioeconomic backgrounds. Population/Participants/Subjects: The study sample will include 66 public elementary schools in California and Illinois, involving approximately 132 teachers and 1,980 students in kindergarten and first grade, respectively. Intervention/Program/Practice: My Math Academy is an adaptive learning system designed to help elementary age children build a strong understanding of fundamental number sense and operations. It includes 31 games with over 300 activities, covering 130 number sense and operations concepts and skills for pre-kindergarten through second grade. My Math Academy leverages games as a vehicle for playful engagement, learning in context, and formative assessments (Dohring et al., 2019). The program advocates for a mastery-based personalized learning approach (see Bingham et al., 2018; Plass & Pawar, 2020). The program integrates evidence-centered design -- an assessment framework that provides a principled alignment of the concepts, skills, and abilities a game is designed to teach with evidence of learning and task design (Clarke-Midura et al., 2012; DiCerbo et al., 2015; Shute, 2011), which enables the estimation of students' competency levels via in-game learning evidence (Mislevy, Almond, & Lukas, 2003; Shute, 2011). My Math Academy has been used by hundreds of thousands of children and several thousand teachers since 2018. Self-funded pilot studies have demonstrated the promise of the program (Bang & Li, 2020; Thai, Li, & Schachner, 2018). Using My Math Academy, students will work through personalized learning pathways with interactive, story-based learning activities that are customized to individual needs. They will receive instruction on math concepts and skills, scaffolding, and feedback to help solve difficult problems, and embedded assessments to gauge their understanding of individual concepts and skills. Teachers will receive real-time reports to facilitate their review of students' usage and progress and adapt instruction accordingly. The theory of change (Figure 1) shows how the theoretically grounded components lead to changes in math achievement. Research Design: The study will use a clustered randomized experimental design. Schools will be randomly assigned to either the treatment or control condition and will be asked to participate for two full school years (kindergarten: 2023-2024 and first grade: 2024-2025). Treatment teachers and students in kindergarten and first grade will use My Math Academy as a supplemental curriculum, three time per week. Teachers will receive professional development before they start implementing My Math Academy and use it along with the accompanying student data as a tool to personalize classroom practices. In the control condition, teachers will continue their existing instructional practices and supplemental technologies. Data Collection and Analysis: Data on implementation fidelity and contrast between conditions will be collected during both years. Cost data will be gathered systematically throughout the study. The Test of Early Mathematics Ability, third edition (TEMA) will be the primary measure of student early mathematics skills. Efficacy will be examined using a three-level hierarchical linear regression model of mean differences in TEMA scores between students in two conditions, controlling for prior achievement and other covariates. Moderator analyses will examine the impact of the intervention on the learning of students with low-baseline mathematics achievement and different demographic backgrounds. Mediator analyses will examine the link between student learning outcomes and mediating variables such as engagement in mathematics learning. Exploratory analyses will address the accumulated impact of multi-year My Math Academy exposure on student outcomes. Analysis will also be conducted to examine cost and cost-effectiveness of My Math Academy.

Research in education, economics, and psychology suggests that teachers contribute in a variety of ways to the learning environment. They form relationships with students and help inculcate effective learning habits (Battistich et al., 1997; Davis, 2003; Thomas & Oldfather, 1997). Teaching practices mediate students' feelings of belonging and engagement (Liu & Loeb, 2021; Rowley et al., 2019). They manage classroom behavior and establish expectations for students' work (Ferguson & Danielson, 2014; Kane et al., 2011). These teaching skills contribute to student academic outcomes in the short run and have lasting effects on the educational trajectories of students (Blazar & Kraft, 2017; Jackson, 2018; Rose et al., 2022). But understanding which teaching skills contribute to these outcomes is an important empirical question given that teaching effectiveness is multidimensional. Several scholars have also identified school climate as a potential component of culturally responsive teaching (Banks, 2004; Brookover et al., 1978; Ferguson, 2016; Thapa et al., 2013). This is consistent with research documenting disparities in students' experiences with the learning environment. For instance, researchers have found that teachers and administrators disproportionately subject students of color to discipline (Barrett et al., 2021; Liu et al., 2022; Shi & Zhu, 2021b). Others have found that teachers have lower expectations for Black students, on average, and that Black students consequently have less access to advanced coursework (Ferguson, 2003; Hart, 2020; Papageorge et al., 2021; Shi & Zhu, 2021a). These findings suggest that improvements in the school climate may have benefits for students of color. In this paper, we use student surveys conducted by the Massachusetts Department of Elementary and Secondary Education (DESE) beginning in 2018. The Views of Climate and Learning (VOCAL) survey yields scores on three dimensions (engagement, safety, and environment) and nine topics (cultural and linguistic competency, relationships, participation, emotional safety, physical safety, bullying, instruction, mental health, and discipline) of school climate. Although the survey is intended to measure school climate, the survey asks questions specifically about classroom activities and students' relationships with teachers. The survey includes one topic explicitly aligned with the instructional environment, which asks about how teachers support student learning and students' interest in the curriculum as well as whether teachers have high expectations for all students. Other questions, particularly in the participation topic, ask about how well teachers encourage engagement in school. And the relationships and emotional safety topics both include questions about the strength of student-teacher relationships. Finally, one topic specifically intends to measure cultural competence, with questions about demographic representation in instructional materials; the extent to which school personnel respect students regardless of their backgrounds; expectations for advanced coursework; general respect from school personnel; and inclusive social environments. We first find that perceptions of school climate are not uniform for all students. Students of color have significantly lower perceptions of the cultural competency of teachers and other school faculty. These differences are most pronounced for Black students, particularly in middle and high school, and appear even when we compare students within the same classroom. Black students additionally report worse experiences on topics aligned to student-teacher relationships, bullying, and mental health outcomes. Similar findings have been reported in the extensive literature on school climate (Thapa et al., 2013). We then use standard statistical approaches to isolate teacher effects, which we refer to as climate value added (VA). We investigate how climate VA contributes to other student outcomes and explore how the impacts of teachers vary by student race and ethnicity. We estimate that the variation in teacher effects on school climate is about 0.1 standard deviations, which is comparable in magnitude to teacher effects on tests and behavioral measures (Hanushek & Rivkin, 2010; Jackson, 2018). Climate VA is also predictive of student survey responses in out-of-sample analyses that examine differences in teacher assignments within schools. It is also positively correlated with teacher contributions to test scores, and to a lesser extent, an index of several nontest measures. We further document that students' assessments of climate, as shaped by their teachers, differ by student race/ethnicity. Teachers who improve school climate tend to do so for all students, but the perceptions of white students and students of color do differ for individual teachers. We estimate climate VA separately by student racial/ethnic groups and find the correlation between climate VA obtained from white students and students of color is about 0.7. The perceptions of students of color differ most significantly on topics aligned with cultural competency, participation in school, and comfort approaching teachers about emotional or personal problems. We also find evidence of student-teacher racial-matching effects on school climate. Matches improve school climate perceptions by about 0.06 SDs for Black students, which is about 50% of the discrepancy in climate reports between white and Black students. Teachers who contribute positively to school climate also improve academic achievement. We estimate that a one SD improvement in climate VA increases student test scores by about 0.02 SDs for both white students and students of color. When we estimate climate VA separately by student racial/ethnic groups, we find that an inclusive climate--as measured by the portion of a teacher's contribution to school climate that accrues specifically to students of color--has similar effects on student achievement, but only for students of color. A one SD increase in the inclusive climate measure is also associated with an improvement of about 0.02 SDs in test scores for students of color. Our findings provide additional evidence for culturally responsive teaching as a distinct dimension of effective teaching and provide some new evidence on specific mechanisms. Taken together, our findings suggest that teachers contribute to the school climate, that students respond to these contributions, and that they have consequences for student achievement. We also find that these skills overlap with culturally responsive instruction. Teachers' ability to engage students in coursework, hold high expectations for all students, and form supportive relationships contribute to academic achievement for students of color.

Background/Context: Access to education is seen as a critical aspect of personal success, with substantial implications for education systems and social well-being. In this context, the effectiveness of free college tuition is a hot topic in current higher education research (Bell, 2020), as there is no clarity on whether the benefits outweigh the large costs that many countries are incurring. In recent years, important results have provided evidence of the causal effects of free-college policies on student success. Molina and Rivadeneyra (2021) evaluate the 2008 free-college tuition granted for public universities in Ecuador, finding that the tuition fee elimination significantly increased college participation and shifted people into higher-skilled jobs. Similarly, Bettinger et al. 2019 found evidence that California financial aid has increased undergraduate and graduate degree completion. Over the past decade, many countries have adopted targeted free college tuition initiatives in recent years in order to increase access to higher education for underprivileged people. In Indonesia, the "Kartu Indonesia Pintar - Kuliah" (Smart Indonesia Card - College) targeted relief college tuition initiative was launched in 2020 with the goal of encouraging human capital development among students from low-income households. This initiative has piqued the interest of policymakers, educators, and academics alike since it provides a unique chance to assess the impact of such programs on student success. Purpose/Objective/Research Question: In this article, we identify the causal effect of targeted relief college tuition in several dimensions of student success, such as high-school retention, college enrollment, holding part-time jobs, and socioeconomic inequality gaps. Our findings expand the current literature because, as far as we know, this is the first evaluation of a largescale, nationwide-level, targeted tuition-relief policy, conditional on higher education institutions' performance (through a three-level college accreditation scale), in a developing country. Setting: The "Kartu Indonesia Pintar - Kuliah" (Smart Indonesia Card - College) program is taking place throughout Indonesia, a country consisting of thousands of islands with a population of over 270 million people. Indonesia has a diverse population with various cultural and linguistic backgrounds, and the country is divided into 34 provinces and numerous regencies and cities. The "Kartu Indonesia Pintar - Kuliah" program is implemented in all of these regions, with a particular focus on underprivileged areas, such as rural and conflict-prone regions, where students face significant barriers to accessing higher education. Population/Participants/Subjects: The population for evaluating this program would be school and college-aged people in Indonesia who are eligible for the "Kartu Indonesia Pintar - Kuliah" program. The total number of eligible students would vary from year to year, depending on the number of students who meet the program's criteria. We base our analysis on various features and outcomes, such as: (1) High school retention rates; (2) Family socioeconomic status; (3) College enrollment; and (4) Holding part-time jobs. Intervention/Program/Practice: The "Kartu Indonesia Pintar - Kuliah" (Smart Indonesia Card - College) program is a targeted relief college tuition initiative that was launched by the Indonesian government in 2020. The program's primary objective is to increase access to higher education for students from low-income households, who might otherwise face financial barriers to pursuing a college education. Under the program, eligible students receive a smart card, known as the "Kartu Indonesia Pintar- Kuliah," which is loaded with a certain amount of funds that can be used to cover college tuition fees and living costs. The amount of funding provided to each student is based on the university's program accreditation they attend and the cost of living index in that region. To be eligible for the program, students must meet certain criteria, including being enrolled in an accredited college or university program in Indonesia, coming from a low-income household, being enrolled in several government social assistance programs, and having good academic potential. Additionally, the program prioritizes students who come from underprivileged regions, such as rural areas or conflict zones, and those who belong to marginalized groups, such as women, people with disabilities, and indigenous peoples. The "Kartu Indonesia Pintar - Kuliah" program is implemented through a partnership between the Ministry of Education and Culture and the Ministry of Social Affairs. The program is funded through the government budget, as well as donations from private corporations and individuals. Research Design: (1) We start with linear regression with fixed effects at the district level. (2) Then we implement a Difference in Difference model that exploits variation across cohorts differentially exposed to the policy, and geographic variation in access to college. Data Collection and Analysis: We use the Indonesian National Socio-Economic Survey (SUSENAS). This survey is representative of the district level. We have full access to data from the years 2016, 2017, 2018, 2020, 2021, and 2022. Approximately 1.2 million individuals are surveyed per year. Preliminary findings: Our preliminary results indicate that receiving the program has a statistically significant effect on college enrollment. Within the region, students who receive the college tuition relief are 0.021 points more enrolled in college, on average, in the population compared to those who do not receive the college tuition relief.

Purpose: Decades of scholarship have drawn on Weiss's four categories of research use (Weiss, 1980; 2005; Weiss & Bucuvalas, 1980) -- instrumental use, symbolic use, conceptual use, and imposed use -- to examine how research use unfolds in school districts (e.g., Amara et al., 2004; Penuel et al., 2017). In this paper, we theorize a new form of research use: latent use. We do so by drawing on sociocultural learning theory (Wenger, 1998) and using empirical data from our comparative case study of research use across four districts. We define latent use as when educators participate with research-embedded tools in ways that guide their work practice. Latent use occurs both when educators describe themselves as using research but also when they do not explicitly acknowledge their use of research. Research Design: We surveyed 12-15 district leaders from the thirty largest district offices. We then purposively sampled four districts -- districts with high and low connections to outside sources of research, and districts that possessed greater and fewer routines. These districts serve from approximately 70,000 to 180,000 students, with over 50% of students eligible for free or reduced lunch. Over 18 months, we conducted a comparative case study on these districts' research use in literacy professional development (PD), collecting 140 interviews, 50 hours of observations, and 127 artifacts. For data analysis, coded and analyzed all references to tools in interviews and observations. We also coded for references to research embedded in all 127 tools. Finally, we conceptualized latent use by writing longitudinal memos of how district leaders used these research-embedded tools in all four districts. Findings: Latent use was pervasive across all four districts. Here, we draw on sociocultural theory (Wenger et al., 1998) to illustrate an example of latent use. In Juniper District, district leaders provided feedback on literacy PD using a research-embedded slide deck in Figure A. Uva says: so, let's do a whip around, and get the positives... Sharon says: I liked that the deck came back to the standards. Danika adds: I liked that it connected between reading and writing, with the standards. That's an important shift. Teagan adds: I want to second what Danika said on the reciprocal nature of reading and writing. I did a session last week with teachers, and that's something that they're missing. As illustrated in this excerpt, latent use has four defining characteristics. The first defining characteristic is that it involves tools that are not research products (e.g., journal articles) but nonetheless have research embedded in them (e.g., research citations), such as slide decks. Figure A is an example of a research-embedded tool. The second defining characteristic is that latent use requires district leaders to participate with a research-embedded tool. Through this participation, district leaders can make this tool meaningful in the context of their own work. We see this when Teagan made connections between the content of the slide deck on the reciprocal connections between reading and writing and her own experiences providing PD. The third defining characteristic is that latent use requires a research-embedded tool to guide how district leaders do their work. Here, the research-embedded slide deck, and specifically, Fordham and colleagues' research (see Figure A), focused and framed district leaders' discussions around literacy PD. The fourth defining characteristic is that latent use can occur regardless of whether district leaders acknowledge themselves as using research or not. When Danika and Teagan noted the "reciprocal nature of reading and writing," they were referencing Fordham's research idea without explicitly acknowledging its research base. Now that we have conceptualized latent use, we now detail how it differs from Weiss's existing categories of research use. Instrumental use. Instrumental use is when educators draw on research to fill a gap in knowledge when making a decision. During instrumental use, educators must intentionally seek out research to fill a specific gap in their knowledge. In contrast, educators who engage in latent use may not see themselves as using research. Additionally, whereas instrumental use informs formal decisions, latent use can guide a wider range of work practices. Symbolic use. Symbolic use is defined as the use of research to influence or persuade others about a decision or legitimize a decision that has already been made. When educators engage in symbolic use, they use research to justify what they were already doing, where research does not shift educators' work practices. In contrast, latent use involves educators using the research-embedded tools in ways that do inform their work practice. Imposed use. Imposed use is defined as when educators are required to use research when selecting an intervention curriculum or practice so they can receive federal or state funding. In contrast, latent use is not required by any entity. Conceptual use. Conceptual use is defined as the use of research ideas to inform how district leaders think about a problem or a solution to that problem. Whereas conceptual use involves changes to district leaders' thinking, latent use involves changes to leaders' work practices. Conclusion: Our discovery of latent use highlights a new pathway -- educators' use of research-embedded tools -- through which research informs practice. What was especially striking about latent use is that district leaders who engaged with latent use did not necessarily see it as a form of research use. For this reason, latent use may be hidden from view from both those engaging in latent use and researchers studying research use. Without attention to latent use, existing scholarship may be under-representing how frequently educators use research. Latent use raises important questions for embedding equity in research. Consider when the research embedded in tools is implicit or invisible (Cain & Allen, 2017). When educators do not see the full range of ways that they are using research during latent use, what are the implications for how they participate with research and who (and who does not) gets to meaningfully shape research agendas?

Preschool children are developing social and emotional skills, like recognizing and expressing emotions, building relationships with peers and adults, following directions, and solving problems. To develop these skills, young children rely on support from adults, including teachers, who model and teach children social and emotional skills in the context of consistent and positive interactions and relationships (Cantor et al., 2019; Silkenbeumer et al., 2016). As they learn these skills, preschool children engage in behaviors that adults find challenging, such as having tantrums, not following directions, and various forms of aggression (e.g., biting, hitting; Snell et al., 2012). Though these behaviors are often developmentally appropriate, they can cause frustration for early childhood educators, who identify child behavior as a central challenge of their profession (Haslip & Donaldson, 2021). Much of child behavior is in the "eye of the beholder", and not all behavior is interpreted equally. Perception of child behavior varies based on teacher characteristics (Jeon et al., 2014; Yoder & Williford, 2019), child characteristics (Arbeau & Coplan, 2007; Brady & Woolfson, 2008), and characteristics of the behavior itself (Coplan et al., 2015; Nungesser & Watkins, 2005). Teacher attributions about behavior are the beliefs that teachers have about why children behave in certain ways (Carter et al., 2014; Nemer et al., 2019). Teachers may hold attributions that are characterized by beliefs that a child behaved the way they did on purpose, was motivated to get something for themselves, was intentionally trying to annoy the teacher, and is deserving of blame (i.e., high level of child responsibility), as well as beliefs that the reason the child's behavior comes up often, the behavior is not likely to change, and has to do with something internal to the child (i.e., high level of child causality). When teachers hold attributions that place high levels of responsibility on the child for the behavior, they may be more likely to blame children and families (Martin et al., 2017), more likely to use exclusionary teaching practices (e.g., time out, suspension, expulsion), and less likely to use practices that support positive behavior (Carter et al., 2014; Miller et al., 2017). Despite the importance of teacher attributions, few studies have examined attributions in early childhood contexts. To address this gap, we use structural equation modeling to explore the relationship between attributions and teaching practices. We used data from 183 preschool teachers (see Table 1 for descriptives) collected as part of a relationship-based dyadic intervention (i.e., Banking Time; for full procedures, see Williford et al., 2017). At the beginning of the preschool year, teachers reported on their attributions using the Preschool Teacher Attributions measure (PTA; Carter et al., 2014). The PTA is a vignette measure that is designed to capture teacher's general attributions about various types of challenging behavior, adapted from the Attributional Style Measure for Parents (ASMP; O'Brien & Peyton, 2002). This measure included five types of child behavior: noncompliance to teacher requests, aggression towards peers, aggression or disrespect towards the teacher, interruption, and noncompliance with the routine. The teacher rated each behavior across eight dimensions (purposefulness, globality, stability, motivation, internal-external locus, blame, negative intent, and controllability), using a 6-point Likert scale ranging from strongly disagree (1) to strongly agree (6). Teachers also reported on their use of proactive and inappropriate teaching strategies, using 26 items from the Teacher Strategies Questionnaire (TSQ; Webster-Stratton et al., 2003). Teachers reported on how often they used each strategy within their classroom on a scale from rarely/never (1) to very often (5). We conducted a series of confirmatory factor analyses and used the resulting latent factors as the teacher attribution predictor and teaching practices outcome. This resulted in two latent factors for the PTA measure (Causal and Responsibility, r = 0.852; RMSEA = 0.066, SRMSR = 0.044, CFI = 0.962), and two latent factors for the TSQ measure (Proactive practices: e.g., prepare children for transitions, give clear positive directions; Exclusionary practices: warn or threaten to send child out of classroom if they don't behave, send child home for misbehavior). Structural equation models explored the relationship between teacher attributions and teaching practices, controlling for teacher covariates (see Table 2). In partial support of our hypothesis, responsibility attributions predicted exclusionary practices (§ = 0.678, SE = 0.250, p = 0.007). Teachers who had higher responsibility attributions tended to also report more frequent use of exclusionary practices. However, causal attributions were not related to exclusionary practices (§ = -0.231, SE = 0.261, p = 0.376), and neither causal nor responsibility attributions predicted proactive teaching strategies (§ = 0.193, SE = 0.270, p = 0.476 and § = -0.290, SE = 0.259, p = 0.263, respectively). In terms of covariates, teacher race and center type predicted use of exclusionary practices; White teachers reported slightly more frequent use of exclusionary practices (§ = -0.209, SE = 0.126, p = 0.09), and teachers at private centers reported more frequent use of exclusionary practices, compared to Head Start teachers (§ = -0.346, SE = 0. 138, p = 0.012). No covariates predicted teachers' use of proactive practices. The finding that responsibility attributions relate to exclusionary practices is in line with qualitative research that found that teachers who request expulsion have different beliefs about children and families (Martin et al., 2017; Zulauf & Zinsser, 2019). Given the high rates of soft exclusion, suspension, and expulsion in early childhood settings generally, and in particular for boys, children of color, and children with disabilities (Meek & Gilliam, 2016; Zeng et al., 2020; Zinsser et al., 2022), understanding how attributions may work to push teachers towards using exclusionary practices is an important area of future research. Changing teacher attributions to be more supportive may be a promising target for reducing and ultimately eliminating the use of exclusion (Davis et al., 2020). More research is needed to understand how teacher attributions and other psychological factors are related to teaching practices to inform future interventions and policies to better support educators and children.

Background: Research clearinghouses (CHs) seek to play an important role in identifying successful social programs and practices. The evidence-based policy movement has grown tremendously over the last few decades and has gotten institutionalized in the form of CHs in the US. CHs (1) specify standards for evaluating the quality of evidence from single studies, (2) judge how well each study meets these quality criteria to determine the weight its results should be given; and when more than one acceptable study exists, (3) they synthesize causal estimates to reach a conclusion equivalent to whether an intervention is clearly effective, or merely promising, or not yet demonstrated to be even promising, or it is ineffective or even harmful. Purpose: This paper explores the construct validity of "evidence-based" in the social sciences, with a focus on education and socio-behavioral development of youth. It asks -- how different are the effectiveness ratings of a single program, and whether the size of the difference might reflect disparate understandings of the meaning of "evidence-based". Specifically, this paper: (1) compares the scientific criteria 24 CHs use to determine intervention effectiveness, (2) estimates how comparable CH ratings of the same program are in the fields of education and socio-behavioral development, and (3) probes why CHs differ in how evidence-based a program is. Data and Methods: We found a total of 43 CHs in the US and UK. Out of these, we kept the 24 that conducted effectiveness rating of programs/policies, that did not borrow their standards from other CHs and that could be found on the internet. CHs vary greatly in terms of the policy domains they focus on; some CHs deal with multiple policy domains. We identified 12 CHs that rated educational interventions and 13 that explicitly deal with sociobehavioral development. We first compared the methodological criteria used by these 24 CHs to rate the quality of individual studies as well as the standards of evidence they use to declare a program evidence-based. To estimate the consistency in effectiveness ratings, we collected the names and ratings of individual programs from CH websites. The sample consists of 1,359 programs in the field of education and 2,525 in socio-behavioral development. Since each CH uses its own rating scheme, we rescaled the ratings of all CHs to a three-point scale -- 1 (Recommended), 2 (Promising) and 3 (Not Recommended). Program ratings are fully consistent when both CHs give a program a score of 1 or 2 or 3. In contrast, when a program receives ratings of 1 (recommended) and 3 (not recommended), it is considered fully inconsistent. Ratings of 1 and 2 indicate moderate consistency as CHs differ in the confidence with which they endorse it. Moderate inconsistency happens when a program is rated 2 and 3. Thereafter, to understand the reasons for why ratings of the same program differ, we conducted case studies of 5 programs in each policy field and probed four hypotheses for differences in ratings. These hypotheses are: variation in studies included, different versions of the program were examined, effects were analyzed on different outcomes and differences in evidence standards used to draw judgements about effectiveness. Results: We first compared the standards of 24 CHs and find that all CHs prefer RCT studies over other studies. Even though CHs accept quasi-experimental studies, they are almost always rated lower than RCTs (though some exceptions exist). There is some variation in the scientific criteria CHs use, especially about methodological attributes like replication, temporal post-intervention follow-up, evaluator independence etc. We find that only approx. 17% of the education programs and 18% of socio-behavioral development are rated by 2 or more CHs. When two CHs rate a program, approximately 18% of education programs are found effective, i.e., either both CHs rated them 1, or both rated them 2 or a combination of 1and 2. Full consistency in ratings pairs stands at around 30% for most education programs. However, CHs agree more on programs that they do not recommend, i.e., receive two ratings of 3. There are comparatively fewer programs that both CHs endorse at the highest levels. Moderate consistency (ratings of 1 and 2) is not as common and ranges from 8.5% to 20% but its probability increases with the increase in number of clearinghouses ratings per program. Moreover, at least a quarter programs show maximal disagreement (ratings of 1 and 3). In the field of socio-behavior development, we find a relatively higher level of full consistency though other patterns of results replicate. 55% of the programs rated by 2 CHs show full consistency. This is nearly 48% for programs rated by 3 CHs, 38% and 37% for programs rated by 4 and 5 CHs. Again, agreement is most common for programs that are not endorsed and least for programs that are endorsed. Among programs rated by 2 CHs, nearly 42% received ratings of 3 from both CHs and only 4% were endorsed by both CHs with ratings of 1. However, unlike in education, maximal disagreement in ratings is not as common and ranges from 9.8 to 15.6% across the entire sample. From the case studies, we concluded that the most likely explanation is that CHs differ in the design features CHs require to give programs a high rating. The bar for attaining the highest rating differs. In particular, the requirements of replication and temporal persistence of effects can play an important role. Discussion: Evidence-based language is commonly used in the public rhetoric though we find that this might be more of an aspiration right now than a matter of scientific consensus. The levels of consistency we found in program ratings are perhaps lower than what one might expect. Complete consensus on methodological issues is neither possible nor desirable but it may be beneficial to find partial consensus on which methodological features of the evidence are necessary to recommend a program's effectiveness.

Background/Context: In order to stimulate national and regional economic growth, policymakers and businesses have expressed great interest in growing the supply of workers with skills in science, technology, engineering, and math (STEM). At the same time, considerable efforts are being devoted to narrowing race and gender gaps in the STEM training pipeline. A key link in this pipeline is high school, where learning experience plays a critical role in shaping student's interest in STEM careers (e.g., Bottia et al, 2015; Corin, et al 2020). This is also the period when females begin to lose interest in STEM (Sadler, et al., 2012). To improve STEM education, one of the national initiatives during the early 2010's focused on integrated applied STEM curricula (Gottfried & Bozick, 2016). By connecting the traditional STEM disciplines to the real-world/applied problems, the goal was to increase student participation in more meaningful and engaging STEM learning and generate greater interest in STEM or related careers. Project Lead the Way (PLTW) is one such program. PLTW began over two decades ago in a handful of high schools as career and technical education (CTE). Since then program has grown rapidly and is currently adopted by over 12,000 high schools nationally. In Missouri, the site of this study, the program expanded from 13 schools in 2005 to 174 schools by 2020--roughly 35% of all public high schools. Purpose/Objective/Research Question: Rigorous empirical evidence on applied STEM program is limited. This study investigates the impact of PLTW program offering (ITT) and the participation impact on participants (TOT). The outcomes include college enrollment, STEM major declaration, overall degree attainment, and STEM degree attainment. Setting: Missouri has 537 traditional and charter public high schools, and 26% of them are in urban and suburban areas, and 18% and 57% are in town and rural areas, respectively. Population/Participants/Subjects: We use three cohorts of first-time 9th-grade students who began high school in AY2010, AY2013, and AY2014 (N? 68,000 per cohort). Racial compositions are 75.8% White, 16.5% Black, 4.2% Hispanic, and 3.6% others. The FRL rate is 42.4%. The analytic population consists of students in two types of schools: schools that began offering PLTW for the 2013 and/or 2014 cohorts, but not for the 2010 cohort (N=92 schools and 43,578 students) and schools that never offered PLTW for the three cohorts (N=273 schools and 49,788 students). The two types of schools are academically similar, but they differ considerably in demographic characteristics: PLTW schools are larger in size; are located in urban and suburban areas; have larger Black or Hispanic student populations; and have lower FRL rates than non-PLTW schools (Table 1). Intervention/Program/Practice: PLTW is designed to provide active, project-based learning in the largest three applied STEM fields--Engineering, Computer Science, and Biomedical Science ("Pathways"). Each pathway begins with foundational courses. They provide an overview and introduce major ideas of the field with a goal of developing an enthusiasm for further study. Advanced courses are designed to expand students' understanding of the field through deeper and specialized content. The Engineering and Biomedical Science pathways end with a capstone course which requires students to take their own idea from design through development of a product or plans to produce one. Research Design: First, we apply a Difference-in-Differences analysis (DiD) to estimate the impact of program offering for PLTW schools. This is estimated by deciles of "prognostic scores"--predicted probability of completing a STEM degree given pre-treatment predictors (math and science test scores and GPAs) . Students in non-PLTW schools serve as a comparison group, providing an estimate of the outcome difference between the 2010 and later cohorts in the absence of PLTW. Next, the propensity-score adjusted DiD is used to estimate the impact of program participation on participants (TOT). Propensity scores--the probability of program participation given pre-treatment covariates-- are used to construct the "would be treated" and "would be untreated" populations from the 2010 cohort who are made similar to, respectively, the 2013/2014 program participants and non-participants in the same PLTW school. The would-be-untreated and observed-untreated students and students in non-PLTW schools are two comparison groups for the DiD analysis. In additional DiD analysis, these two comparison groups are weighted/matched to resemble the program participants. Data Collection and Analysis: Student data are provided by Missouri Department of Elementary and Secondary Education. The K-12 data are linked to National Student Clearinghouse data for post-secondary outcomes. The ITT impact for student i, cohort c, and school j is estimated as; Y_icj=d_ij+ [beta]eta_cd(d_ij*PLTW_j*Coh_c)+G(X_ij)+ Coh_c*d_ij+ S_j+ e_ij, where dij are indicators for prognostic-score deciles; PLTW distinguishes PLTW and non-PLTW schools; X is student covariates; Coh_c is cohort indicators; s_j is school fixed effects. The coefficients, [beta]_c_d, indicate the cohort- and decile-specific ITT impact. The ITT analysis avoids selection bias around program participation, but estimates may be small due to low participation rates, especially for the 2013 cohort (Table 2). The model for TOT impact is; Y_icj=a0+ a1(Untreat_coh10,ij) + a2 (Treat_coh10,ij))+ a3(Coh_c )+ a4(Coh_c*Untreat_ij ) + delta(Coh_c*Treat_ij)+a5(X_ij)+ e_ij,, Where a0 is the average outcome for the 2010 students in non-PLTW schools; Untreated_coh10 and Treated_coh10 are, respectively, the PS-matched/weighted "would be untreated" and "would be treated" 2010 students in the PLTW school. The program impact is given by delta. Note that a_3=a_4 under the paralell trends assumption. Results: First, Table3 presents base-line outcomes. Table 4 shows positive ITT effects on STEM major declaration and STEM degree completion for the top deciles. Program participants also had higher STEM enrollment and degree completion (Tables 5 and 6). Conclusions: Our evidence suggests that an applied STEM program like PLTW has a potential to increase post-secondary TEM outcomes among students with higher STEM readiness. The study's limitations include bias due to unobserved confounders. We will conduct sensitivity analyses to address this issue.

Overview: This study examines the complex effects on families when mothers reenroll in school at any level. The population of mothers in school--especially in college--is large and predominantly composed of women of color (Reichlin Cruse et al. 2019; Anderson 2022). Though there is a mounting emphasis on increased education and skills in the workforce, a persistent call for single mothers being self-sufficient, and a pressing need to break intergenerational cycles of poverty, little is known about how mothers and their children fare in the short and long term after school reentry. In this research, I characterize how school reentry and degree completion affect families over two decades, but I also observe large differences in the impacts of school reenrollment between Black, Latina, and White mothers and their children. The overall results, without disaggregation by race, are published in an Urban Institute brief (see Anderson 2022). This not-yet-published follow-up study examines the differential results by mothers' race accompanied by racial equity-informed policy analysis. Context: Millions of parents pursue education or training after starting a family. National data show that students who are parents make up nearly a quarter (22 percent) of the postsecondary student population overall, and a larger share of students at two-year colleges (26 percent) and among certain demographic groups (NPSAS:16). For example, 40 percent of all Black female college students have minor children, and 50 percent of Black mothers go back to school after they have children (Anderson 2022). Enrolled students with children are disproportionately students of color. Thus, there are important racial and ethnic equity implications of promoting student-parent success. Student parents have strong assets--they are motivated to succeed so they can set a good example for their children and better support their families. And they earn higher average grades than nonparenting students in the first semester of college (DeMario 2019). But student parents often struggle to complete degrees when they are not visible or accommodated in their classes and in college life. They also often do not receive appropriate advising, struggle to afford tuition and other costs considering their other financial obligations, and lack social supports while in school (Generation Hope 2020; Green 2013; Anderson 2020). Those with younger children may struggle to find and afford child care and be challenged in balancing school, parenting, and often work responsibilities (Goldrick-Rab, Welton, and Coca 2020; Hess et al. 2014). And structural barriers related to race and gender likely discourage college enrollment and limit success. Research and Methods: This is the first quantitative research that reveals the complex experiences of families when mothers pursue their educational goals. I explore the trajectory of wellbeing indicators for mothers and children for up to two decades after school reentry using data from the National Longitudinal Survey of Youth 1979 (NLSY79) and the Child and Young Adult Supplement, which tracks biological children of women in the NLSY79. I estimate impacts of school reentry on mothers' short- and long-term educational, economic, and social outcomes using individual fixed effects with propensity-score-determined comparison groups. The individual fixed effects are only applicable to outcomes that can be measured both before and after a mother's school reentry (e.g., labor market effort and marital status). I compare distal outcomes that cannot be benchmarked before the mother's school reentry (e.g., depression at age 50) using propensity score matching (PSM) between mothers who returned to school and otherwise similar mothers who did not continue their schooling. I also complete an analysis of child educational, economic, and social outcomes through young adulthood. The child analysis only lends itself to the PSM approach because the individual fixed effects approach would limit the analysis to a much smaller subset of children than is available in a less restrictive analysis. When I compare the individual fixed effects results to the PSM results for the subset of children and outcomes that support both, there are few substantive differences. In this mixed-methods study, additional and contemporary insights come from in-depth qualitative interviews with mothers who were pursuing additional education in fall 2018 in the national capital region. Findings: The overall study shows that mothers who reenrolled in school attained more educational credentials, worked more, and earned more in the long run than similar mothers who did not reenroll. However, mothers who reenrolled were less likely than comparison mothers to be married; family income were lower long term (because of lower marriage rates); and they experienced some negative physical and mental health effects later in life. Also, mothers' reenrollment related to small, short-term gains in children's vocabulary and reading scores but also more behavioral problems that persisted into early adulthood. In the long term, children of mothers who went back to school had better academic attainment, but this did not result in earnings gains in their early careers. The positive effects of school reenrollment were much larger and almost all negative effects were mitigated when student mothers completed college degrees: they did not have income declines or negative health effects, and their children saw earnings gains even before age 25. But outcomes differ notably for Black, Latina, and white mothers. Black mothers who reenrolled school did not see earnings gains, even though they completed more education and worked more than the matched comparison group. Latina mothers and mothers who completed a college degree fared better than other subgroups in educational and economic outcomes relative to their respective comparison groups, though Latina mothers who reenrolled had higher rates of depression at age 50 on average than the comparison mothers. The children of these mothers saw similar patterns to their parents. Qualitative interviews with student mothers and racial equity-aware policy analysis point to explanations for these differentials. Importance: This research addresses the intergenerational effects of education, which is how inequities reproduce over time. It is important to bring light to heterogeneous effects of educational engagement by race to inform policy and practice solutions that promote equity in the medium term and justice in the longer term.

Study after study has demonstrated that postsecondary success involves a range of factors from neighborhood context to family history to individual aspiration (Kim & Rifelj, 2021). Kauffman Scholars Incorporated (KSI), a post-secondary scholarship program based in Kansas City, understands these factors and has tailored its programming accordingly. Throughout its 20-year history, KSI has focused on supporting low-income students in attaining postsecondary credentials. Whereas the program has changed over time, it has consistently offered college scholarships, academic tutoring, and mentoring to students starting in 7 grade. Over 800 Scholars have earned college degrees since KSI's inception. KSI's goals extend beyond participating individuals. They also aim to create a ripple effect--having impacts on people in Scholars' lives, community organizations, and the local career-professional pipeline. These goals are reflected in wrap-around services such as one-on-one coaches and community partners who guide KSI scholars and their families through barriers that often derail students from their postsecondary track. th NORC and Bowman Performance Consulting (the NORC team) have joined together to evaluate KSI's impact over the past two decades. The traditional linear cause-effect evaluation methodology is insufficient to fully understand and describe the complex nature of transformational change at this scale. Understanding system-wide impacts requires holistic thinking, innovative and flexible approaches, and a radical commitment to partnership with all stakeholders. Our research approach, which draws upon the Culturally Responsive Indigenous Framework and Equitable Evaluation, meets these criteria. We think of this as telling the story of the ripple effect of KSI over twenty years. We will examine impacts from the direct source (the Scholar), rippling out to their families, their other connections, and ultimately, to the broader community of Kansas City. In our evaluation study, we: (1) investigate to what extent KSI achieved its goals, (2) describe what factors enabled or constrained KSI scholars and alumni in achieving KSI goals, discuss what ways KSI impacted Kauffman Foundation strategies, investments, and current work, and (3) delineate the ways KSI Scholars and alumni have impacted the local community. This study is ongoing and will be completed by September 2023. This proposal describes our methodological approach and the purposeful ways we embed equity throughout the research process. We will be ready to present findings in the Fall. Data: Nobody understands a person's story better than the person who has lived it. As such, we purposefully embed participant-centered ideals that honor the tenets and conceptual framing for culturally responsive practices (Ladson-Billings, 2008; Bowman, 2018) with a focus on interpersonal, experiential, and consequential validity (Kirkhart, 2010) throughout our approach. The NORC team will conduct Storytelling interviews with 20 past and present Scholars. Each Scholar will identify a person in their lives to interview, gathering their story of KSI's impacts beyond the individual Scholars. We empower the Scholar in multiple ways throughout the process. First, we co-construct the Storytelling protocol with each Scholar, and practice the skill of interviewing another person, allowing their story to be told in their own words. Part of this practice will include participating in an interview (as the interviewee), where a member of the NORC team will interview the Scholar about their experiences with KSI. Second, we put Storytelling agency in the Scholars' hands, asking them to select a connection in their lives to interview, helping us understand the ripple effect of KSI across people. This approach employs Storytelling to build relationships across researchers, Scholars, and the community. It also grounds the evaluation work in the nuanced contextual elements that shaped KSI's impact over time, surfacing and infusing meaning throughout the data collection and analysis process. Our project also includes a survey of the 2,000+ current and former Scholars, a survey of household members of these Scholars, a survey of community organizations impacted by KSI, interviews with KSI staff, and secondary data collected by Kauffman on their Scholars over time. The Storytelling interviews are the heart of the study--they provide the story of KSI's impact from those who lived and continue to live through the experience. The other data contextualize the Storytelling data, providing important information about the demographic composition of the Scholars, programmatic goals and reasons for programmatic design decisions, and data on KSI's ripple effects on the community and Scholars' family units. Methods: Storytelling Analysis: The NORC Team will utilize a combination of inductive and deductive coding strategies to synthesize themes and describes KSI's range of impacts over time. We will utilize KSI's vision, mission, and value statements as normative guideposts through which to analyze the storytelling data (e.g., how do the values of the KSI program express themselves throughout KSI Scholar and Scholar connection stories?). All primary data collection tools include questions that mirror the Storytelling Interview questions. We will first examine responses to these questions, honing in on what parts of KSI were most impactful for Scholars as well as for the local community. We will create a cross-data matrix, comparing responses across participant type, surfacing themes of commonality and divergence. By analyzing data in this way, we gain the benefit of extra nuance and detail by drawing on Scholars' lived experiences, the experiences of chosen ones in their lives, the stories of the KSI staff who supported them, and information from the communities they live within. Landscape Analysis: Analyzing KSI's secondary data will provide a landscape of KSI's impact contextualize the stories we collect. We will also draw on the Scholars, family member, and community stakeholders survey data. The survey data allows us to identify specific program features that were associated with program satisfaction and other outcome measures. We will also compare across Scholar groups to hone in on how programmatic elements shaped Scholars' lives. Conclusion: Evaluation is, at its heart, telling the stories of people's lives in ways that are authentic, meaningful, and participant-centered. Our project employs an intentionally responsive evaluation design, centering participants while purposefully placing agency for telling their stories in their hands. We strongly believe that embedding equity and repositioning the roles of researchers, participants, and community in these ways deeply strengthen the research process from beginning to end.

Introduction: Missing data are bound to occur in education intervention research. Reasons vary but always have the consequence of reducing sample sizes and can, under certain conditions, seriously bias estimated intervention effects. A wide array of methods have been developed for handling missing data (Enders, 2023). Whereas older approaches largely involve simply ignoring missing cases (e.g,. complete case analysis), with newer approaches, missing data can actually be recovered (e.g., multiple imputation; Enders, 2023; Schafer & Graham, 2002). As with any statistical method, though, certain assumptions must be met. Indeed, the appropriateness of any handling method depends on the pattern of missingness and the mechanism thought to underlie it (e.g., missing completely at random; Enders, 2023; Little & Rubin, 2020). Accordingly, information about the pattern, mechanism, and handling method should be clearly reported (Toste et al., 2023). Although missing data reporting and handling have not traditionally been emphasized within special education intervention research (Gersten et al., 2005), recent quality indicator standards have placed an increased focus on these matters (Toste et al., 2023). Though clear recommendations have emerged (Enders et al., 2006), it is unclear what practices are currently used. Previous reviews of the broader education literature indicate that reporting practices are minimal and that researchers largely rely on traditional handling approaches (e.g., complete case analysis; Cheema, 2014). As in other fields, this may reflect gaps in methodological training (Murtonen & Lethtien, 2003). To our knowledge, no review has focused specifically on special education researchers. In this study, we therefore aimed to take stock of missing data reporting and handling practices in special education group design research. Purpose and Research Questions: Our purpose for this review was to summarize missing data reporting and handling practices in special education group design intervention research. We had three primary questions. First, what are the locations (e.g., item, case), amount, and reasons (e.g., attrition) for missing data? Although design and reporting standards in special education specify that authors should clearly describe this information (Gersten et al., 2005), based on previous reviews, we hypothesized that reporting would on average be minimal but vary considerably across studies. Second, what missing data mechanisms (e.g., missing completely at random) are reported and how are they tested? We again hypothesized that reporting would be minimal but vary widely across studies. Third, how are missing data handled? What methods are used and are sensitivity analyses performed? Based on previous reviews, we hypothesized that traditional handling methods (e.g., complete case analysis, pairwise deletion) would be used more often than modern methods (e.g., multiple imputation). Methods: Research Design: Our study is a systematic review. We followed the What Works Clearinghouse process for systematic reviews, which involves defining the review protocol, systematically identifying relevant literature, screening studies for eligibility, reviewing studies deemed eligible, and reporting those findings (Institute of Education Sciences, 2020). Screening and Inclusion Criteria: Our sample of studies was drawn from a larger project aimed at examining how null effects are reported in special education intervention research (Cook et al., 2023). To identify studies for the project, special education journals reported in the 2019 Journal Citation Reports were electronically searched. Studies were included if they (a) used a quantitative group design (randomized controlled trial, quasi-experimental designs with a comparison group without random assignment to groups, regression discontinuity designs) and within-group designs (e.g., single-group, pre/post-test design), (b) empirically evaluated the effect of an intervention on at least one dependent variable, and (c) used inferential statistics to evaluate the effect of the intervention for one or more groups of participants. Studies were excluded if they (a) used a research design other than the specified group designs (e.g., single-case design studies, purely descriptive studies, qualitative studies, secondary data analyses), (b) only examined differences or relations between groups on non-manipulative variables (e.g., examining differences in behavior between males and females, or between students with learning disabilities and at-risk learners: examining the relation of age to reading achievement), or (c) did not report statistical significance of inferential statistics evaluating the effect of the intervention. Using these criteria, over 1,428 relevant studies were screened for eligibility. Out of the relevant literature screened, 130 articles were deemed eligible for inclusion in our review. Data Collection: We imported articles that met the inclusion criteria into Covidence, an online tool for screening and extracting data in systematic reviews and meta-analyses. To analyze the studies, we adapted a codebook from Eekhout and colleagues (2012). We began by randomly selecting five articles not included in our sample, coding two of them together, and then revising the codebook to improve its alignment with our research questions and usability. We then coded the remaining articles independently and discussed discrepancies. We repeated this training process with additional articles not from our sample until we met an a priori criterion of 90% total agreement. Analysis: We will report frequencies of missing data reporting and handling methods in special education group intervention research. We will compare these results to past and current design and reporting standards for the special education field (Gersten et al., 2006; Toste et al., 2023). Results: Results are presently not available. Study coding and analyses are scheduled to be completed by June 2023.

I. Purpose of the Study: Detecting whether interventions work or not (through main effect analysis) can provide empirical evidence regarding the causal linkage between malleable factors (e.g., interventions) and learner outcomes. In complement, moderation analyses help delineate for whom and under what conditions intervention effects are most salient. Such analyses provide empirical evidence that is the source material for effective policy development (Raudenbush & Liu, 2000; Spybrook, Kelcey, Dong, 2016; Dong et al., 2018, 2021; IES & NSF, 2013). The purpose of this study is to improve the design of experiments by developing optimal design strategies for experiments probing moderation and main effects. II. Background and Relevant Literature: Differential Sampling Costs Across Treatment Conditions and Levels: In real-world experimental designs, researchers must consider sampling costs, fixed budgets, and complex differences in sampling costs across treatment conditions and levels/units. For example, in a multisite-randomized trial to evaluate the effects of the Reading Recovery program on students' reading achievement, the costs of adding a treated student may additionally include the resources for delivering 60- to 100-days of 30-minute individual lessons, the professional development of teachers, and the supplemental work time of teachers as intervention providers (May et al., 2015). There are a broad range of documented experiments with differential costs across treatment conditions and levels (Springer et al., 2011; Greenleaf et al., 2011; Jacob et al., 2015; Jayanthi et al., 2017; Clements et al., 2011; Lewis et al., 2013; Hiscock et al., 2008). Sample Allocation and Optimal Design: III. Research Design We use the following two-step macro structures to develop the proposed framework. Optimal Design of Experiments Detecting Moderation Effects: We use the (multivariate) first-order derivative method to develop the optimal design framework for moderation effects. This approach has been used to develop the optimal design for main effects (Shen & Kelcey, 2020, 2022a, 2022b). Optimal Design of Experiments Investigating Moderation and Main Effects: We use a meta-heuristic optimization method, ant colony optimization (ACO; Dorigo & Stützle, 2004; Socha & Dorigo, 2008), to identify the optimal sample allocation in experiments investigating both moderation and main effects. The procedures of implementing the ACO algorithm to identify the jointly optimal sample allocation, including validation methods, are presented in Appendix A. IV. Preliminary Results: Next, we present the results for two-level multisite randomized trial with a site-level binary moderator as an example to demonstrate the feasibility of the research design and the utility of the proposed work. We do not include covariates in this example to ease the presentation. We omit the statistical models and other equations because of the space limitations. Optimal Sample Allocation for Moderation Effects: Unconstrained Optimal Sample Allocation: The resulting optimal design parameters (individual-level sample size per cluster) and (proportion of units in the treatment) must satisfy the two equations in Appendix A. Although there are no algebraic solutions for the optimal and from the two equations, we can solve for the roots numerically through iteration. This iteration method has been outlined in Shen & Kelcey (2020) and implemented in the R package odr. Power Analysis in Optimal Design: Once optimal design parameters are identified, we can perform power analysis using these parameters. The power analysis results in optimal design will have these features: Approximately achieving the maximum statistical power under the same budget and using the least resources to produce the fixed power. The same features are valid under a constraint if used in obtaining a constrained optimal sample allocation. Optimal Sample Allocation for Moderation and Main Effects: To identify the optimal sample allocation detecting both moderation and main effects in the same two-level multisite trial, we can use the ACO algorithm presented in Appendix B. Illustration: Let the cost structures of sampling be , , (e.g., May et al., 2015), intraclass correlation coefficient be 0.20 ( ; e.g., Hedges & Hedberg, 2007), treatment-by-site variance be 0.01 ( ), and the proportion of treatment-by-site variance explained by the moderator be 0.30 ( ). Optimal Sample Allocation for Moderation Effects: The identified optimal design parameters for moderation effects are and . The statistical power is maximized at the optimal allocation under the same budget (Figure 1). When designs are sub-optimal, they will require additional budget percentages (ABP) than the optimal one to maintain the same level of statistical power (Figure 2). Jointly Optimal Sample Allocation for the Moderation and Main Effects: In terms of the jointly optimal allocation for both moderation and main effects in the two-level multisite trials, the ACO algorithm can efficiently identify jointly optimal allocation. It returns results within one second for 300 iterations that can provide precise enough results. The results show that when a design departs from the jointly optimal sample allocation, power decreases for at least one of the effects under the same budget (see Figures 3 to 5). V. Contribution to the Field: This study will enrich the knowledge base for future design literature as the proposed framework can be extended to other types of designs. This study and its application in the field have great potential to shift the next-generation studies to more effective and efficient ones. The updated package odr not only will facilitate the computation, but its power analysis functions explicitly accommodating costs and budget will also encourage thoughtful consideration of efficient design strategies.

Introduction: The What Works Clearinghouse (WWC) reviews rigorous research on educational interventions with a goal of identifying "what works" and making that information accessible to educators and policymakers. In rating the quality of causal research, the WWC has historically prioritized internal validity over external validity. One critique of the WWC, and quantitative research in general, is that there has been too little attention to external validity (Hedges, 2018; Joyce, 2019; Ming & Goldenberg, 2021). The larger concern is that paying too little attention to external validity could lead to inaccurate conclusions about the potential effectiveness of interventions in different contexts (Briggs, 2008; William, 2019). This white paper uses publicly available WWC study data to explore the settings, student populations, and outcome domains included in the WWC evidence base. The purpose of this paper is to map the available evidence to highlight where there is an abundance of evidence as well as where more high-quality research is needed. This white paper addresses two questions: (1) For which study settings, outcome domains, and student populations is there high-quality research (or a lack thereof) according to the WWC? (2) Which study settings, outcome domains, and student populations are represented in the WWC's evidence base about 'what works' in education? In other words, for which settings, outcome domains, and populations did studies find positive and statistically significant effects according to the WWC? Data: The advantage of using WWC study data is that studies must demonstrate internal validity and outcome measures must demonstrate reliability to be included (WWC, 2022d). Therefore, WWC study data are one source for high-quality research on educational effectiveness in education. The final study sample includes 4,048 findings in 1,064 studies. Each study may include more than one finding. Method: The evidence base is presented visually in the form evidence gap maps (EGMs). EGMs are designed to show where relatively little research exists and more research is needed, as well as where research may be ripe for new syntheses of evidence (Saran & White, 2018). The target audience of this paper is funders of research, researchers, and educators looking for the availability of evidence. Findings: The WWC database offers the most evidence of educational intervention effectiveness on student outcomes in math- and literacy-related outcome domains. In contrast, the WWC offers relatively little evidence in other academic subjects, such as science and social studies, and on educator outcomes. This limits the WWC's capacity to inform evidence-based practices in those domains. The available evidence generally appears to align with the WWC's systematic review efforts. Therefore, to increase the number of findings on outcomes beyond general math and literacy, the WWC may need to commission systematic reviews in other topic areas, and funders of research might also consider funding additional research focused in these areas. Educators hoping to identify interventions that are effective for specific student populations, or better understand the types of students who participated in the studies reviewed by the WWC, may encounter difficulties doing so. Students with individualized education plans (IEPs) appear to be underrepresented in the WWC evidence base. Excluding studies with missing data, 61% of studies had 10% or fewer students with IEPs, while 15% of public school students nationwide received special education services in 2020-21 (NCES, 2022a). Moreover, only 6% of studies included more than 20% students with IEPs, and of these studies, very few or none included outcomes on college readiness, postsecondary, science, social studies, functional skills (i.e., life skills), or English language development. Similarly, only 14% of studies included more than 20% English learners, and of these studies, very few to none of these examined outcomes relating to college readiness, postsecondary education, school progress, behavior, school climate, science, social studies, or educators. However, missing data in the WWC database complicate our understanding of its true representativeness: IEP data were missing in 67% of studies and English learner data were missing in 50% of studies. Given the importance of understanding "what works for whom," it is critical that researchers adequately report--and WWC reviewers consistently record--the characteristics of student samples in impact studies. Evidence maps depicting the extent to which the WWC assigned evidence tiers to studies--indicating that the studies found positive and statistically significant effects--demonstrated considerable variation across outcome domains. Some outcome domains were assigned evidence tiers at higher rates than others. For example, the K-12 school progress, postsecondary progress, and school climate domains were assigned evidence tiers at higher rates than other domains. Domains that are more narrow in scope, such as numbers and operations and writing, were also assigned evidence tiers at higher rates than other domains. One potential explanation is that effect sizes are typically larger--and therefore the studies are more likely to receive an evidence tier--when the outcome domains are "narrow" in scope, when there is lower variation in scores, and when the measures themselves are "narrow" and more tightly aligned with an intervention as compared to when the outcome is a broad measure of interest, like a state test (Boyd et al., 2008; Kraft, 2020; Kraft, 2023; Ruiz-Primo, Shavelson, Hamilton, & Klein, 2002; Somers et al., 2022; Wolf & Harbatkin, 2022). Consumers of research should be careful to review the outcome measures and domains in studies to ensure overlap and generalizability with their constructs of interest. Moreover, comparing effect sizes across studies is problematic because it may lead to greater implementation of interventions whose impacts have only been studied on narrow measures and whose impacts on standardized measures of student achievement are unknown. While prior work has focused on why effect sizes might differ across outcome domains and measures (Kraft, 2020; Wolf & Harbatkin, 2022), there has been less research on how to make sense of new research findings given differences in effect size distributions (Newcomer, Hall, Pandey, Reginal, & White, 2023). More methodological work is needed. In conclusion, researchers should contextualize the findings in program evaluations and provide appropriate nuance and caveats. Avoiding overly simplistic conclusions about complex educational interventions can only help to build trust in research findings.

Background/Context: The proposed paper will share findings from a dissertation study examining mindfulness as a tool to reduce the impacts of educator bias on school discipline disparities. Teachers are the single most important factor in determining student achievement compared to any other aspect of schooling, yet studies are finding that the majority of teachers hold implicit biases much like the general population (Starck et al., 2020). Despite often good intentions, these biases can result in the differential treatment of students of color and students with disabilities, leading to disparate behavioral and academic outcomes (Losen, 2018). For instance, Black students diagnosed with an emotional/behavioral disability are punished more frequently and severely for the same behaviors compared to their White peers with and without disabilities (Losen, 2018). Mindfulness offers a novel approach to addressing the mental conditions in educators that can lead to biased disciplinary decisions in the classroom (Chang et al., 2022). Purpose/Objective/Research Question: In the proposed paper presentation, I will share findings from a pre-registered, double-blind randomized field experiment with teachers testing the hypothesis that a brief mindfulness training may reduce the impacts of bias in educators' discipline decisions. The presentation will answer the following research questions: (1) Does mindfulness training positively affect participants' interpretations of student behaviors and recommended use of punitive discipline? (2) Is a reduction in punitive discipline mediated by a reduction in the likelihood to interpret the students' behavior as problematic overall? Within Black and White student subgroups? (3) Does mindfulness training reduce disparities between Black and White student groups with and without disabilities as measured by participants' interpretations of student behaviors and their recommended use of punitive discipline? Setting: The study was conducted online through the Qualtrics platform. Teachers were invited to participate in the study via email. Interested teachers were first invited to complete a brief screener survey to determine their eligibility and then were taken to the Qualtrics platform to participate in the intervention or control activity. Population/Participants/Subjects: Secondary teachers (n=412) from public middle and high schools in four urban districts were recruited to participate in the study. The districts were selected intentionally given their publicly reported disparities in suspensions between Black and White male students with and without disabilities (identified through the Civil Rights Data Collection). Intervention/Program/Practice: Teachers who were eligible to participate in the study first answered pretest questions on their state of mindfulness and discipline philosophy (5 minutes). Teachers were then randomized at the individual level by treatment (Mindfulness vs. Control) and vignette race (Black vs. White) (20 minutes). The intervention included a brief introduction to mindfulness practice and included a guided mindfulness meditation exercise adapted from prior mindfulness-based interventions (Hanley et al., 2020). The control exercise entailed learning about the benefits of technology in the classroom, informed by prior intervention studies of a similar paradigm (Okonofua et al., 2022). After completing the exercise, all participants then responded to two vignettes depicting a student with and without a disability and the order was randomly counterbalanced. Participants then responded to post-test questions assessing their state of mindfulness, bias and basic demographic details (5 minutes) and were debriefed. Research Design: The research design entailed a double blind randomized experiment and followed a 2 (Condition: Treatment x Control) x 2 (Student Race: White x Black) x 2 (Student Disability Status: With x Without) mixed between and within subjects experimental design. Participants were randomly assigned at the individual level to receive either the treatment or control condition (between subjects) and to review two student vignettes depicting either Black or White students (between subjects). All participants viewed one vignette in which the student has a learning/behavioral disability, as referenced in the vignette materials, and one vignette in which the student did not have a disability (within subjects). The order of viewing a student with or without a disability was counterbalanced. Data Collection and Analysis: In response to the first research question, independent samples t-tests and regression were used to assess differences in interpretations of student behavior and recommendations for punitive discipline between the treatment and control groups. For regression, I created a dummy variable for the treatment and control group, which served as the independent variable predicting the primary outcomes (interpretations and recommendations). I included pre-test covariates assessing participants' state of mindfulness and discipline philosophy in the model. In response to the second research question, mediation analyses were performed in all student groups combined and within subgroups by student race. In response to the third research question, independent sample t-tests were conducted to compare group means between Black and White students with and without disabilities in the treatment and control conditions. The data was then reorganized from wide into long format, enabling disability and race to be treated as separate variables. A regression analysis was then performed to examine whether there was a main effect of race or disability status on the interpretation and recommendation outcomes as well as whether there was a significant interaction between group assignment and race and disability status. Findings/Results: In response to the first research question, regression indicated that the treatment/control dummy variable significantly predicted participants' interpretations of student behavior and recommendations for punitive discipline. In response to the second research question, a partial mediation was found overall and for the White student subgroup. A full mediation was found for the Black student subgroup. In response to the third research question, independent sample t-tests revealed that differences between the Black and White student groups in the treatment and control conditions were not significant, with the exception of the difference between the Black and White student without a disability in the treatment condition, which was significant. Interestingly, the White student without a disability received the harshest punishment across the treatment and control groups, followed by the Black student without a disability, White student with a disability and Black student with a disability. Additionally, regression also revealed that disability status was a significant predictor for the interpretation composite variable and race was not. The interaction of race and disability status as well as the three-way interaction between group assignment, race and disability status were also not significant for the interpretation of behavior. For the recommendation composite variable, regression revealed that both race and disability status were significant predictors. The interaction terms, however, were not significant. Conclusions: There are several limitations in this study. First, the online delivery mechanism may not approximate real-life classroom scenarios adequately. Responding to vignettes, for example, presents challenges to the transferability of the intervention effects to classrooms. Nevertheless, the findings of this study are encouraging, suggesting that mindfulness interventions may be a promising approach to shifting teachers' negative interpretations of the behaviors of Black and White students with and without disabilities and their subsequent use of punitive discipline. While the results did not indicate a reduction in disparities, the intervention group did interpret the student behaviors within each student subgroup significantly less negatively and responded with significantly less punitive discipline, which suggests mindfulness training may positively impact disciplinary outcomes for all students. Going forward, there is an opportunity to study the effects of the intervention in school contexts and to examine discipline outcomes in school settings following the intervention to assess its transferability to classrooms.

Nearly one million undergraduates take Introductory Statistics each year, and both the importance of statistics and viewpoints about how it should be taught have changed markedly over the last several decades (Guidelines for Assessment and Instruction in Statistics Education College Report, ASA Revision Committee, 2016). Modern introductory statistics courses seek to teach statistics in the context of real-world investigations, and there is growing consensus that understanding statistics ethics is a key learning outcome for the course. Yet, course instructors have few resources to help them teach or assess ethics. Our review of over 150 items used in five popular statistics concept inventories identified only a single item dealing with ethics. Digital Promise is leading the development of the Benchmark Assessment for Statistics Introductory Concepts (BASIC), a test designed to measure a student's ability to apply the key concepts taught in a modern, postsecondary introductory statistics course. Because the assessment will be used as the common learning outcome measure in a large-scale study, all BASIC items have a multiple-choice format. Development of pilot items on ethics has been particularly challenging in the absence of previously developed items for this learning outcome or even a consensus around what behaviors would indicate competency in understanding statistical ethics. Items can easily become no different from those used to assess understanding of other learning outcomes, such as sampling or hypothesis testing, or become as bland as "don't be evil." Creating an item context around a relevant social issue can easily bring preconceptions associated with different identities into play. This poster will summarize the assessment item development process, review the initial ethics items developed for the BASIC, and share the results of a differential item function analysis on key demographic variables, including Race/Ethnicity, Gender, and Age. Data is currently being collected in Spring 2023, and this In-The-Pipeline poster will be the initial results of the pilot test of this assessment.

Background: Implementation quality is a significant factor in determining whether a program has its intended impact (Durlak and DuPre, 2008; List et al., 2021). Therefore, when studying a program's efficacy, researchers must ensure that the program is implemented as intended to estimate its impact. To evaluate whether a program is being implemented properly in the classroom during a study, researchers generally conduct classroom observations or rely on teacher self-reports (Dart et al., 2012; Schoenwald et. al., 2011). However, live observations are time-consuming and costly because they require multiple observations for each teacher to reliably estimate fidelity metrics (Cross et al., 2015; Gresham, et. al, 2017). Although self-reposts require fewer resources, they are less reliable (Gresham et al., 2017; Lee et al., 2008). In this study, we explore how data automatically collected by online learning platforms can be leveraged to evaluate implementations and how these data can help us understand who has access to high-fidelity environments. Purpose/Objective/Research Question: (1) Which variables should be used as metrics of students' dosage and engagement across conditions? (2) How much of the variance in student-level dosage and engagement metrics are associated with the students' implementation contexts? (3) Can we classify high and low-fidelity implementation contexts using the dosage and engagement metrics? (4) Do students of different demographics receive different levels of fidelity? Setting: The analyses for the current study used data from a randomized control trial described in Section 3. This study employed student-level randomization into the following programs: two gamified conditions (OLP1 and OLP2), OLP3 of traditional problems with immediate feedback, and OLP3 without immediate feedback (active control). Population: Initially, 52 seventh-grade mathematics teachers from 11 schools were recruited to participate in the study from a large, suburban district in the Southeastern United States during the summer of 2020. The district provided demographic data for the 3,434 participating students. Of the analytic sample, 48.88% of the students were identified as White, 27.14% as Asian, 14.04% as Hispanic, 4.86% as Black, 0.69% as Native American, 0.04% as Pacific Islander, and 2.98% as two or more races. The district did not provide data for 1.37% of the students. About half the students were identified by the district as female (48.14%) and male (51.81%), with less than 1% of the population not reporting gender. The district did not provide any data on non-binary students. About one-tenth (9.10%) of students had an Individualized Education Plan. Programs: During the school year, teachers were expected to provide students with 13 half-hour minute sessions to complete 30 assignments within their respective OLP. Each OLP used a different method of teaching algebraic knowledge. The first, sixth, twelfth, and thirteenth sessions were 40-45 minute assessments of their algebra knowledge and attitudes toward math. Sessions 2-5 and 7-11 were activities in their assigned technology. In both the assessments and the activity sessions, students' usage and progress for each program were captured by their respective technologies. As students solved problems in the systems, the platforms also logged information about students' actions and performance. Design: To create metrics of student dosage and engagement (RQ1), we first calculated variables from the log-file data of each program, conducted exploratory data analysis, and standardized fidelity metrics within programs. Variables identified as suitable measures of dosage and engagement across programs in RQ1 are used to produce fidelity Setting: The analyses for the current study used data from a randomized control trial described in Section 3. This study employed student-level randomization into the following programs: two gamified conditions (OLP1 and OLP2), OLP3 of traditional problems with immediate feedback, and OLP3 without immediate feedback (active control). Population: Initially, 52 seventh-grade mathematics teachers from 11 schools were recruited to participate in the study from a large, suburban district in the Southeastern United States during the summer of 2020. The district provided demographic data for the 3,434 participating students. Of the analytic sample, 48.88% of the students were identified as White, 27.14% as Asian, 14.04% as Hispanic, 4.86% as Black, 0.69% as Native American, 0.04% as Pacific Islander, and 2.98% as two or more races. The district did not provide data for 1.37% of the students. About half the students were identified by the district as female (48.14%) and male (51.81%), with less than 1% of the population not reporting gender. The district did not provide any data on non-binary students. About one-tenth (9.10%) of students had an Individualized Education Plan. Programs: During the school year, teachers were expected to provide students with 13 half-hour minute sessions to complete 30 assignments within their respective OLP. Each OLP used a different method of teaching algebraic knowledge. The first, sixth, twelfth, and thirteenth sessions were 40-45 minute assessments of their algebra knowledge and attitudes toward math. Sessions 2-5 and 7-11 were activities in their assigned technology. In both the assessments and the activity sessions, students' usage and progress for each program were captured by their respective technologies. As students solved problems in the systems, the platforms also logged information about students' actions and performance. Design: To create metrics of student dosage and engagement (RQ1), we first calculated variables from the log-file data of each program, conducted exploratory data analysis, and standardized fidelity metrics within programs. Variables identified as suitable measures of dosage and engagement across programs in RQ1 are used to produce fidelity metrics which are then used in subsequent analysis to address RQ2 and RQ3. Next, to evaluate how much of the dosage and engagement metrics variance is associated with components of the implementation context (RQ2), we estimated a series of multilevel models. Each model included random intercepts for school, teacher, and class regressed on the fidelity metrics identified and standardized for RQ1, and computed ICCs to evaluate the importance of the students' contexts in predicting their dosage and engagement. Next, we conducted a latent profile analysis to identify different fidelity profiles. We aggregated the fidelity metrics identified in RQ1 at the context level (school, teacher, or class) with the highest ICCs from the analysis for RQ2. Finally, to understand whether their students had different opportunities to be in high-fidelity contexts, we estimated logistic regression models regressing whether or not the students were in a high-fidelity context on their demographic data. Data Collection: Students' log-file data were collected by their respective programs. The researchers did not have access to the OLP2 database; however, before and after students solved problems in OLP2, they logged into OLP3 and answered a series of self-reported questions about their starting and ending progress in the program. All data from this study are available through the Open Science Foundation (https://osf.io/r3nf2/). Variables included in the analyses of fidelity are presented in Table 1. Results: Three metrics emerged as best suited to evaluate the students' dosage and engagement in their respective programs: the percentage of assignments students started, the total number of problems they attempted during the study, and their average problems completed per assignment (Table 2). Teachers had the highest intraclass correlations on each fidelity metric, suggesting that in this study, the teachers were the most important contextual factor in determining implementation fidelity (Table 3). Next, we identified two latent profiles of teachers: high and low fidelity (Figure 1). Finally, we found that students who were not White or Asian and those with Individualized Education Plans (IEPs) were significantly less likely to be taught by high-fidelity teachers compared with their peers (Table 4), yet these effects went away after accounting for students pretest algebraic knowledge scores. These findings reveal bias in access to high-fidelity implementations, where lower-performing students are less likely to have access, and these students are more likely to be students of color. These findings highlight the importance of implementation fidelity monitoring, and future work should how to ensure equitable access to high-quality programs in high-fidelity environments.

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton, 2013). Propensity scores model the sample selection process as a function of observable characteristics, or covariates, that affect both sample selection and moderate treatment effects (Kern et al., 2016). Once estimated, propensity scores are used to match or reweight the study so that the resulting sample is compositionally similar (based on the covariates) to individuals in the inference population. When certain core assumptions hold, propensity score-based methods can provide bias reduced estimates of population average treatment effects (PATEs; Chan, 2017). A core assumption needed for propensity scores is that the sample and population share common support (Tipton, 2014). This assumption requires that the propensity scores include all treatment effect moderators that also affect sample selection. Additionally, every school in the sample must have a comparable school(s) in the inference population, where comparability is based on the covariates used in the propensity scores. While the notion of common support centers on treatment effect moderators, these variables are generally difficult to identify in practice, primarily due to the small study samples seen in practice (Tipton, 2014; Tipton et al., 2017). Although the true treatment effect moderators may be unknown, there may be covariates that are correlated with the true moderators, leaving open the question of whether including variables that are proxy moderators affects estimation and inferences in generalization studies. Focus of Study: The purpose of the current study is to assess the effects of including proxy moderators, specifically weak proxies (variables that are weakly correlated with the true moderators), on inferences for generalization. The motivation for this focus is two-fold. First, many publicly available data sources include aggregate variables (at the school or district level), but the true moderators may be at the classroom or student level. As a result, aggregate variables may only be weakly correlated with the true moderators. Second, when researchers have access to a potentially large set of weak proxies, an important question is whether including them in the propensity score model and analysis yields any benefit for estimation. This can have implications for researchers when various types of covariates on a population are accessible and decisions are made about which ones to include in a generalization analysis. Research Design: We conducted a simulation study to examine the effects of propensity score augmentation with weak proxies. The simulation varied two main parameters: (i) covariate type based on their correlation with the true moderators of treatment impact and (ii) the number of weak proxy moderators included in the propensity score model. For each iteration, we generated nine covariates X_i, i=1,...,9, of which {X_1,X_2,X_3} were the true treatment moderators, {X_4,X_5,X_6} were strongly correlated with the moderators with an average correlation of ? ?=0.8, and {X_7,X_8,X_9} were moderately correlated with the moderators with ? ?=0.4. All covariates were simulated as Normal variables. In addition to the nine covariates, we simulated sets of covariates that had average correlations |(?|) ?<0.10 with the true moderators. Together, we examined three scenarios in which the propensity score model contained p=11, 41, and 81 weak proxies in addition to the strongly and moderately strong moderators {X_4,...,X_9}. Results and Conclusion: We applied four estimators of the population average treatment effect (PATE) to assess the effects of propensity score augmentation with weak proxies on their performance. Table 1 provides a description of each estimator. Table 2 provides summary statistics of the propensity score distributions in the simulated samples and populations. Note that the results are based on the realistic scenario when the true moderators {X_1,X_2,X_3} are unknown and not part of the estimated propensity scores. Table 3 provides the average absolute bias and root mean squared error (RMSE) of each estimator where both measures are computed in relation to the true simulated PATE. Table 2 illustrates that as more weak proxies are added to the propensity score model, the median and maximum estimated propensity score increases and the overall skewness of the distribution decreases in the sample. In contrast, the median propensity score in the population decreases with additional weak proxies, dropping to a value of 0.009 in the model with 81 weak proxies. This is accompanied by an overall increase in skewness. This implies that with more weak proxies added, there were some population schools with estimated propensity scores that were much smaller than average. The changes in skewness appear to be more pronounced in the sample (decrease of 40%) compared to the population (increase of 7%) between the first row (no weak proxies) and last row (81 total weak proxies). Table 3 shows that as the propensity score model was augmented with weak proxies, the average bias and RMSE consistently increased for IPW and this was followed by TMLE. In the final row with 81 (total) weak proxies, the average RMSE increased by 23% and 21% for IPW and TMLE, respectively. The results for subclassification were smaller where the average RMSE increased by 15% when the number of weak proxies increased from 41 to 81. For BART, the results were inconsistent though the average bias and RMSE were higher when the model included weak proxies compared to the model without them. This is potentially due to the increased variability from including weak proxies in the analysis. These results highlight two main implications. First, adding weak proxies affects the propensity scores for both the sample and population by affecting the median and skewness of the distributions. Second, augmenting the propensity score model with weak proxies affects the bias of estimators like IPW and TMLE the most. However, the pace at which the bias changed was not necessarily aligned with the number of weak proxies included. Additionally, estimators that do not depend on the propensity score, like BART, are also potentially impacted when weak proxies are incorporated in the analysis.

Advocates for the use of a multi-measure system of performance evaluation argue that multiple measures may better capture meaningful differences in employee effectiveness and help align effort with organizational objectives. This may be particularly important in organizations such as schools that produce multiple outcomes. In this paper I use a leave-year-out estimation approach, similar to that outlined by Chetty, Friedman and Rockoff (2014), and estimate the relative contributions of the subjective (supervisor observations and student surveys) versus objective (value-added) evaluation measures in explaining teacher effectiveness at increasing cognitive and non-cognitive skills. I use data from a large urban public school district where teacher compensation is tied to the evaluation measures. Estimates reveal that the subjective measures provide information on teacher effects on short-run achievement and behaviors and long-run achievement in ways that value-added does not. Under the multi-measure system, test score value-added has highly significant relationships with achievement and absences in the current and subsequent year. Furthermore, teachers with high supervisor observation scores increase contemporaneous and subsequent year achievement. Not only are the supervisor observations related to achievement, but they also explain variation on top of test score value-added for both the contemporaneous test scores and subsequent year reading. Interestingly, student survey scores also capture current and subsequent year achievement, and they remain significantly related to contemporaneous absences even after controlling for the other evaluation measures. In fact, of the three high-stakes evaluation measures, teachers with high student survey scores are the most effective in reducing contemporaneous absences. The student survey effects on achievement and absences are more pronounced for students with low prior achievement and high prior absences. This finding is consistent with the idea that a caring teacher may particularly matter more for the low performing students and has implications for equity. The pattern of results is similar between subsequent year and high school outcomes, with the exception that supervisor observations are not significantly related to any of the high school outcomes when conditioned on test score value-added and survey score, while student surveys provide some additional information for Algebra I test scores. High survey score teachers also increase the probability of students attending a public high school within the district. Findings in this paper establish a foundation for the larger investigation of the effects of the evaluation and compensation reform on student outcomes.

Background: It has become increasingly important to shift education research and programming from a deficit focused perspective to a positive youth development perspective in order to identify and promote factors that support students' positive functioning and outcomes. One outgrowth of positive youth development is Positive Mental Health (PMH), which describes students' positive social, emotional, and cognitive well-being and is associated with numerous short- (e.g., academic achievement, academic engagement) and long-term (e.g., improved mental and physical health) outcomes. The study of school-related PMH is particularly important for student success, as numerous facets of PMH (e.g., thriving, flourishing) have been shown to improve students' trajectories of positive functioning. Recent research has begun to identify factors that promote school-related PMH for students, such as social support and self-esteem, yet further research is needed to identify factors that promote PMH among a diverse group of students. Previous studies have been limited by the sociodemographic make-up of their sample, which has restricted researchers' ability to compare and contrast factors that contribute to changes in PMH in students over time. Furthermore, previous approaches to measuring PMH have often failed to account for potential measurement non-invariance across sociodemographic factors. Failure to account for measurement non-invariance when exploring students' PMH can lead to biased conclusions, as measurement non-invariance and true differences in mean levels of the latent construct may be conflated. Objectives: This study focuses on measures of school-related PMH. Specifically, we leverage data from students randomized to the control condition across 8, tier-2 RCTs to explore measurement non-invariance and latent mean differences in reports of students' school-related PMH. We hypothesize that students' gender identity, race/ethnicity, geographic region, and socioeconomic status will be associated with the way in which teachers and students report on students' PMH. After accounting for partial invariance, we hypothesize that these predictors will also be associated with differences in school-related PMH latent means at baseline. Setting: The 8 preventive intervention RCTs (i.e., Coping Power; Lochman & Wells, 2004) took place across 3 states over the last 20+ years, and the current analyses were conducted solely among students randomized to the control (i.e., school-as-usual [SAU]) condition. The interventions were predominantly delivered in elementary schools, whereas two of the interventions were delivered in secondary schools. Participants: Students in the RCTs were eligible if they demonstrated elevated levels of externalizing behaviors, as rated by teachers or parents. For the current analyses, only students in the control condition (SAU) were included in order to avoid influences of any potential intervention effects, given that evidence suggests the potential for an interaction between school-related PMH and the active components of the preventive intervention. Research Design: The current study uses an Integrative Data Analysis (IDA) approach to harmonize data across several studies. IDA provides a framework to help account for semantic differences among similar measures collected across studies (e.g., differing versions/language of the same measures), provides an approach to harmonize questions and responses across different measures (e.g., combining two conceptually similar subscales from two distinct measures), and provides a methodological framework to account for other cross-study variations in measurement (e.g., distinct populations of students). A benefit of the IDA approach is it leverages a larger sample size of underserved and marginalized populations, thus creating the opportunity to explore more equity-specific questions among students that are generally underrepresented in individual studies. Data Collection and Analysis: We will semantically harmonize questions across multiple measures in which students and/or teachers reported on the students' school-related PMH. From the overall set of measures, a series of steps were taken to identify a subset of subscales or specific items relevant for the current IDA. First, we identified all measures with school-related PMH items that were collected in at least 1 of the 8 studies. Second, we identified a set of constructs that previous literature has identified that comprise different aspects of school-related PMH (e.g., student connectedness, school connectedness, teacher connectedness, school-related self-concept). Third, multiple rater semantic harmonization techniques were employed to map similar items across constructs from these measures. Student-reported PMH items were semantically harmonized across multiple measures, including the BASC, School Attitudes and Bonding, and Perceived Self Competence Scale. Additional multiple rater logical harmonization of response options was conducted due to the varied response options across measures. Teacher-reported PMH was solely collected on the BASC, and thus only required analytic harmonization across versions. Once the item-set is finalized and response options and questions across measures are harmonized, we will employ Moderated Non-Linear Factor Analysis (MNLFA) to explore measurement non-invariance and estimate factor scores that take into account differential item functioning (DIF). We will test for DIF based on the students' gender identity, race/ethnicity, geographic region, and SES for school-related PMH. Results: We will discuss the numerous decision points required when harmonizing student- and teacher-reported PMH across multiple measures. We will report on the significant intercept and loading DIF from our MNLFA focused specifically on teacher-reported PMH, as cross-measure harmonization was not required. Conclusions: Given the association between student PMH and both short- and long-term academic outcomes, it is necessary to explore how the measurement of this construct may differ across a diverse range of students, especially when considering multiple reporters. By accounting for DIF when estimating factor scores and exploring predictors of DIF, we will be able to ascertain which predictors are associated with differences in measurement versus differences in levels of the latent construct. Subsequent research can then be conducted to explore trajectories of students' PMH over time that more adequately reflect the students' underlying PMH by accounting for potential biases in measurement. These trajectories will then enable us to identify factors that promote PMH across all students, as well as factors that are unique to students from underserved and/or marginalized backgrounds. In addition to highlighting methodological innovations of IDA, implications for intervention will be discussed.