Search

Regional Educational Laboratory Southwest partnered with the Arkansas Department of Education (ADE) to examine Arkansas's middle and high school indicators of postsecondary readiness and success, building on an earlier study of these indicators (Hester et al., 2021). Academic indicators include attaining proficiency on state achievement tests, grade point average, enrollment in advanced courses, and community service learning. Behavioral indicators include attendance, suspension, and expulsion. Using data on statewide grade 6 cohorts from 2008/09 and 2009/10, the study examined the percentages of students who attained the readiness and success indicators and the percentages of students who attained postsecondary readiness and success outcomes by gender, race/ethnicity, eligibility for the National School Lunch Program, English learner student status, disability status, age, and district locale. The study also examined whether the predictive accuracy, specificity, and strength of the indicators varied by these student groups. Three key findings emerged. First, the attainment of indicators of postsecondary readiness and success differed substantially for nearly all student groups, with the number of substantial differences on academic indicators exceeding those on behavioral indicators. The largest number of substantial differences in the attainment of academic indicators were between Black and White students, between students eligible and ineligible for the National School Lunch Program (an indicator of economic disadvantage), and between students who entered grade 6 before and after age 13. Second, attainment of postsecondary readiness and success outcomes varied substantially across student groups, with the largest differences between students with and without a disability. Third, predictive accuracy (the percentage of students with the same predicted and actual outcomes) and strength (the relative importance of a single indicator) were similar across student groups in most cases. Leaders at ADE and in Arkansas districts can use these findings to identify appropriate indicators of postsecondary readiness and success and to target supports toward student groups who most need them. These findings can help leaders identify and address disparities such as inequitable access to resources and supportive learning environments. [For the appendixes, see ED624015.]

The Texas Education Agency launched the Grow Your Own (GYO) grant program in 2018 to encourage districts to develop or expand existing high-quality education and training courses for high school students and to support district-employed paraprofessionals (including instructional aides and long-term substitute teachers) to pursue certifications that would allow them to enter full-time teaching roles. This study aimed to help state education leaders in Texas understand the progress of districts in implementing the GYO program and the early outcomes of participants. This study analyzed data from 2015/16 through 2020/21 for districts that received GYO funding in the first two grant cycles and districts in the same geographic locales within the same regions that did not receive GYO funding (comparison districts). The study found that the majority of districts awarded a GYO grant were in rural areas and small towns. GYO districts were more likely to have a smaller enrollment and had a higher average percentage of Hispanic students than comparison districts. The findings suggest that the program appeared to meet the Texas Education Agency's goal of providing opportunities to students and paraprofessionals in rural and small school settings and students of color to participate in GYO activities. The study also found that the percentage of students completing education and training courses in GYO districts was low (about 10 percent) during the grant years, and the percentage was similar in comparison districts before and after the grant awards. A disproportionate share of students who completed education and training courses in GYO districts were female. Although it is too soon to tell whether the GYO program will, over time, increase the size and diversity of the state's teacher pool, leaders at the Texas Education Agency can use these early findings to both understand the progress of districts in achieving the GYO grant program aims and help identify aspects of the program that might need further investigation. [For the appendices to this report, see ED623385.]

Louisiana's Believe and Prepare pilot program, supported by grants from the Louisiana Department of Education, aimed to prepare teacher candidates or in-service teachers through a residency with a mentor and a competency-based curriculum. To improve teacher preparation and teacher residencies, state and teacher education leaders in Louisiana sought to better understand the early career outcomes for participants in the pilot program. This study analyzed data for the three cohorts that participated in the program between 2014/15 and 2016/17. A majority (76 percent) of pilot participants were enrolled in a university-based teacher preparation program. The study examined certification, employment, and retention outcomes for a subset of pilot participants who were teacher candidates or early career teachers (together referred to as early career Believe and Prepare pilot participants). About 30 percent of early career Believe and Prepare pilot participants who attained a Level 1 professional certificate in 2015/16-2017/18 were certified in a high-need subject, as defined by the Louisiana Department of Education (middle grades math and science, secondary math and science, or special education), and 28 percent of participants who entered teaching in 2015/16-2018/19 taught in a high-need subject in their first year of teaching. Early career pilot program participants who completed a residency in a primary school were more likely than those who completed a residency in a nonprimary school to attain a Level 1 professional certificate. Participants who completed a residency in a charter school were less likely than those who completed a residency in a noncharter school to attain a Level 1 professional certificate. (Louisiana does not require Level 1 certifications for charter schools.) Of early career Believe and Prepare teachers who entered teaching in 2015/16-2017/18, 89 percent were retained in the state for a second year, 76 percent were retained in the same district, and 71 percent were retained in the same school. Among these teachers the within-state retention rate was lowest for teachers in high-need subjects, and the within-school retention rate was lowest for secondary and middle grades math and science teachers. [For the Study Snapshot, see ED612483. For the appendixes, see ED612484.]

The Regional Educational Laboratory Southwest Educator Effectiveness Research Alliance expressed interest in investigating annual teacher mobility in Texas. This resulting study, using data from the 2011/12-2015/16 school years, first asked how large teacher mobility was and how much of that movement was between schools in the same district, how much was between districts in Texas, and how much was out of public school teaching in Texas altogether. The study also addressed the relationships between teacher mobility and teachers' personal and professional characteristics, school-level student characteristics, and schools' average teacher ratings (under a new system piloted in 2014/15). The study used 2011/12-2015/16 data collected by the Texas Education Agency on all Texas public schools. It also used data collected by the Texas Education Agency during the 2014/15 pilot of the Texas Teacher Evaluation and Support System (T-TESS) in 57 school districts-about 5 percent of districts in Texas. This report provides state and district policymakers in Texas with updated information on trends in teacher mobility and on correlates of mobility in the teaching workforce, offering a systematic baseline for monitoring and planning. The findings will enable policymakers to formulate a strategic, targeted approach for recruiting and retaining teachers rather than relying on generic approaches for increasing the overall supply of teachers or improving recruitment. For example, informed efforts might target attracting and retaining teachers in specific fields (such as special education), at certain stages of their career (such as novice teachers), or in certain geographic areas. Moreover, the analysis enriches the knowledge base about schools' teacher retention and mobility in relation to the quality of the teaching force and may inform policy discussions about the importance of a stable teaching force for teaching effectiveness. [This document was produced in collaboration with the Regional Educational Laboratory Southwest Educator Effectiveness Research Alliance.]

Federal initiatives and other research have led states across the nation to sharpen their focus on teacher evaluation in recent years. In 2009 a seminal report, "The Widget Effect," from The New Teacher Project revealed that in districts using a binary rating system to evaluate teachers, less than 1 percent of teachers received an unsatisfactory rating. The remaining 99 percent were, in effect, like widgets, undifferentiated as individual professionals. Since then, a growing body of research on teacher evaluation systems has indicated that classroom observation ratings often cluster around the middle or high end of evaluation scales. The research has also found that observation ratings are susceptible to several biases, such as incoming student achievement and school, classroom, and rater characteristics. Thus many education decision-makers lack sufficient information to make personnel decisions and to effectively support teacher growth and development. Texas is among the states that have identified teacher evaluation and support as a high priority for education policy. In 2014/15 the Texas Education Agency piloted the Texas Teacher Evaluation and Support System (T-TESS) in 57 school districts. The pilot was followed by a refinement phase in 2015/16 and statewide rollout in 2016/17. During the pilot year teacher overall ratings were based solely on rubric ratings on 16 dimensions across four domains (planning, instruction, learning environment, and professional practices and responsibilities), although T-TESS also includes a student growth measure that was piloted simultaneously. The study examined the statistical properties of the T-TESS rubric from the 2014/15 pilot year to explore the extent to which it differentiates teachers on teaching quality and to investigate its internal consistency and efficiency. The study also explored the relationships between rubric ratings and school characteristics to investigate whether certain types of schools have teachers with higher or lower ratings. The study's findings have several implications for practice and research. The findings suggest that the T-TESS rubric demonstrates potential to be an effective, consistent, and efficient evaluation rubric. Thus, the Texas Education Association and local education agencies have a promising tool for providing evidence-based feedback and targeted professional development. Future research could try to validate ratings based on the T-TESS rubric against other measures of teacher effectiveness (for example, student growth). Such validation could shed light on whether a relationship exists between rubric ratings and a teacher's contribution to student achievement. Moreover, future studies could explore whether relationships exist between the T-TESS rubric and classroom and district characteristics. Such analysis may unearth the extent to which effective teachers are equally distributed within schools and within and across districts. Finally, further research could explore whether implementing teacher evaluation systems translates into improvements in teacher effectiveness or in long-term outcomes, such as teacher retention and student achievement.

Recruiting and retaining effective teachers are serious concerns throughout Oklahoma. The Oklahoma State School Boards Association (2016) reported 500 teacher vacancies at the beginning of the 2015/16 school year, according to a survey of school districts, and 53 percent of respondents said the teacher shortage was worse than in the previous year. For years, Oklahoma rural school district administrators have reported difficulty retaining teachers who could cross state lines for higher pay and lower class sizes or seek employment in other industries. In 2013 the Oklahoma State Superintendent of Public Instruction established the Oklahoma Educator Workforce Shortage Task Force to recommend measures to alleviate the "significant and widespread shortages" of classroom teachers. The task force was succeeded in September 2015 by the Teacher Shortage Task Force, which was established to identify and recommend successful strategies for curbing the statewide teacher shortage crisis and which recommended several strategies for placing highly qualified teachers in all Oklahoma classrooms. The state's teacher shortage, as well as the unique context of rural schools in Oklahoma, led members of the Regional Educational Laboratory Southwest Oklahoma Rural Schools Research Alliance to seek information about factors associated with successful teacher recruitment and retention in Oklahoma. The goal was to develop effective strategies for recruiting and retaining teachers in rural schools. In response, this study identified factors that can support teacher recruitment and retention, particularly malleable factors that can be controlled through policies and interventions. This report refers to these factors as indicators of the characteristics of teachers or districts that predict successful teacher recruitment and retention. While associations between indicators and outcomes cannot be interpreted as causal--a specific indicator is not necessarily the cause of a related outcome--the results from this study can be used to pinpoint potential problems and inform future policies. The results can also provide a rationale for experimental evaluations of programs aiming to improve teacher recruitment and retention. The study first explores patterns of teacher job mobility in Oklahoma, including teachers' probability of remaining employed in the same district for a given number of years, the proportion of teachers who leave rural school districts and move to another rural school district, the proportion of teachers who receive tenure, and the one year retention probability for each successive year of employment. Patterns of teacher job mobility are examined for any differences between rural and nonrural school districts. The study was designed to identify teacher, district, and community characteristics in rural Oklahoma that predict which teachers are most likely to be successfully recruited (defined as having completed a probationary period of three years and obtained tenure in their fourth year of teaching) and retained longer term (defined as the duration of employment of tenured teachers in a given school district). This study covers the 10 school years between 2005/06 and 2014/15 and uses teacher and district data from the Oklahoma State Department of Education, Oklahoma Office of Educational Quality and Accountability, and community characteristics from data in federal noneducation sources and publicly available geographic information systems from Google Maps.

Empirical Education conducted rapid-cycle evaluations (RCEs) in three districts in the southcentral and southeastern United States. Each study evaluated the impact of the Dynamic Learning Project (DLP) coaching on teachers' and students' impactful use of technology and student achievement. The research questions, samples, and analytic methods varied between districts, due to the availability of data. They found strong evidence of positive impacts of DLP coaching on teacher and student technology usage, as measured by Hapara Analytics measurement of Google Suite usage, in two districts where this data was available. These findings were particularly evident, and consistent in their increased frequency and duration, for metrics that captured digital interactions between students and teachers. In all three districts studied, they found evidence of positive impacts of DLP coaching on student achievement in math and ELA. These findings, across three districts in three states with different outcome measures and varying grade levels, demonstrate the promise of the DLP coaching model. They also see little evidence of differential impacts of DLP coaching for varying student subgroups. This finding demonstrates promising evidence that DLP coaching equitably impacts students across, gender, race, ELL status, and economically disadvantaged status. The impact of DLP coaching on both the short-term and long-term outcomes hypothesized in the program logic model, as well as evidence that the associations between mediating teacher and student edtech outcomes and longer-term student learning outcomes are positive and significant. These findings are consistent with the goals of the DLP project. However, given the small sample sizes and other limitations of the data collection, the authors caution that many results were inconclusive and warrant further investigation.

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness research but are pertinent to understanding the effectiveness and scalability of the program. In the current study, a direct comparison of school achievement was not possible since the outcome measure used in the RCT was researcher-administered. However, the authors are able to look at teacher characteristics associated with higher student achievement in the RCT and they can look at teacher (and principal) characteristics associated with school-level gains in program participation in Scale-up. The authors found that RCT schools had lower levels of program-related characteristics associated with scale-up. Scale-up schools had similar levels of program-related teacher characteristics associated with greater achievement. Schools that were successful in growing participating internally were not more likely to have the characteristics associated with achievement gains. This suggests that program effectiveness in a controlled study may not be indicative of the program's prospects of wider adoption and, in fact, such studies may not create the best conditions for future program sustainability. Tables and figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

This study explored the relationships among the components of the Arizona Department of Education's new teacher evaluation model, with a particular focus on the extent to which ratings from the state model's teacher observation instrument differentiated higher and lower performance. The study used teacher-level evaluation data collected by the Arizona Department of Education from five participating pilot LEAs during the 2012/13 school year. The study relied primarily on descriptive statistics calculated from the results of the different component metrics piloted in these LEAs, as well as analysis of the correlations among these components. Results indicated that teachers' observation item scores tended to concentrate at the Proficient level (the second-highest score on a four-point scale: Unsatisfactory, Basic, Proficient, and Distinguished), with this level accounting for 62 percent of all observational item scores. In addition, while the strength of the correlation between results from observations and the state's student academic progress metric was generally low, the correlation varied significantly between high- and low-performing teachers, as well as between certain teacher subgroups. The following are appended: (1) Arizona Teacher Evaluation Model; (2) Pilot Local Education Agency Information; (3) Study Methods; (4) Detailed Arizona Teacher Evaluation Model Pilot Results; 2012/13; and (5) Detecting nonlinear Relationships between Observation Item Scores and Student Academic Progress Metrics.

Teacher evaluation is currently a major policy issue at all levels of the K-12 system driven in large part by current US Department of Education requirements. The main objective of this study is to explore the patterns of relationship between observational scores and value-added measures of teacher performance in math classrooms and the variation in these relationships across grade levels. While the MET analyses used a single composite score consisting of a simple average of the eight component scores of the protocol, in our work we treated each component separately since each measures a separately definable aspect of classroom practice. Specifically, across all the components, the authors pose the following questions: (1) Do the relationships between observation scores of math teachers and their value-added scores tend to be non-linear?; (2) Is there a difference in the relationships associated with developmental changes as indicated by grade levels (four through eight)?; and (3) To what extent the observed changes are associated with increasing departmentalization (in upper elementary and middle school)? The relationship between classroom observation instruments and teacher effectiveness is often presumed to be a simple linear mapping. The authors find that the relationship is complex: it is typically non-linear and the shape of this relationship appears to evolve with the grade level progression. Careful analysis of such relationships will improve understanding of teacher effectiveness and result in more accurate teacher evaluation systems. Better evaluation systems will in turn generate more useful administrative data for use in effectiveness research. Tables and figures are appended.

Efficacy studies of educational programs often involve mediator analyses aimed at testing empirically appropriate theories of action. In particular, in the studies of professional teacher development programs, the intervention targets presumably teacher performance while the ultimate outcome is the student achievement measured by a standardized test. The main objective of this study is to develop a methodology for creating an optimal aggregate teacher performance metric from domain scores for use as mediators in the analyses of student outcomes. The authors use one particular observation rubric, PLATO, to answer the question of how an aggregate teacher performance metric can be constructed from domain scores that would be best aligned with a selected measure of student achievement. The preliminary analysis determined that only three out of six domains of PLATO used in the study were significantly associated with the value-added measure of student achievement. Of the three significant domains, two had linear relationship with the value added. For the only domain with non-linear relationship, log transformation appeared to be sufficient to linearize the relationship. A figure is appended.

We report on the scaling up of a high school content literacy framework, Reading Apprenticeship, over a period of four years as part of the independent evaluation of an Investing in Innovation grant from the U.S. Department of Education to WestEd's Strategic Literacy Institute (SLI). Our goal was to understand the school processes that support successful implementation and promote scaling of an innovation. We focused on teachers, principals and other school-level program leadership who were also the focus for SLI's innovation designed to support scale. Our work was guided by research literature that provided insights into scale-up as increasing local ownership and depth of commitment, as well as growth in numbers. We provided formative feedback to SLI while contributing to the empirical methods and evidence for studying scale-up processes. Over the five years of scaling up, the RAISE project (for Reading Apprenticeship Improving Secondary Education) reached approximately 533,000 students, and 1,719 teachers in 239 schools in Pennsylvania, Michigan, Indiana, and Utah. Surveys of teachers and principals, as well as records of participation in project activities provided a rich longitudinal dataset. In addition to survey analysis, four schools served as case studies where school leaders and program support staff were interviewed over two years to offer a concrete picture of decision processes and resources. We used descriptive statistics to track changes over time in features of program implementation, as well as resulting changes in attitudes and commitment among teachers. In the first year, teachers were successfully integrating Reading Apprenticeship practices and cited collaboration and support from other teachers as most effective for capacity building. We found that uptake of RAISE activities leveled off by the third year. We also found a greater spread in responses across schools by the third year, calling for further investigation into school-level processes and the characteristics of schools that may be associated with the gain or loss of RAISE participants over time. Regression methods were used to identify predictors of increased or decreased participation. We found that teacher participation in team meetings during the first year and school-wide commitment predicted the increase in number of teachers participating in a school. Increases were unrelated to demographics and resources available to the school. We hypothesized that a process, consistent with SLI's innovation for scaling Reading Apprenticeship, which involved a cross-disciplinary teacher team within the school, led to increased and sustained program participation. The scale-up study was conducted in parallel to a [randomized controlled trial] RCT in which 22 treatment and 20 control schools from Pennsylvania and California participated for three years. We used the predictors of increased participation in comparing the scale-up and RCT treatment group schools as contexts for the implementation and impact of Reading Apprenticeship. Scale-up schools had more of the characteristics associated with growth suggesting greater long term sustainability. We raise the issue of whether the implementation under the constraints of the RCT is a good model for measuring impact in the context of scaling up an innovation. The following are appended: (1) Detailed RAISE Scale-up Logic Model; (2) State Maps of RAISE Scale-up, By Year; (3) Results from GL Analysis; and (4) Survey Questions Used in Chapter 8.

Classroom observation is an important component of teacher evaluation systems. Most states are implementing systems that assign a composite score to each teacher based on weights assigned to several different measures. Policy discussions often address this weighting, with many states adopting formulas with high weights for the summative scores from observations conducted by school principals or other administrators. Given the weighting of this one measure, it is important to ensure the validity of observation rubrics and equitability of the resulting teacher rankings. This paper addresses the problem of observation scores being affected by characteristics of the students in the class being taught, and explores the problem in two phases: (1) the demonstration of an alternative to the common (often implicit) assumption that the components or elements of the observation score are measuring a single underlying concept and all have the same relevance to any personnel decision that is to be based on the evaluation score; and (2) showing how the multifaceted nature of observations can be used to better understand how observation scores are affected by class characteristics. Most observation rubrics in wide use, such as the Framework for Teaching (FFT), have been designed, and are used as universal instruments. They are applied without any modifications in classrooms at different grades levels, in different subjects, and with students of widely different abilities, backgrounds, and resources. This implicit assumption of instrument invariance is questionable. The nature of the invariance may be different for different components of the instrument. The goal of the analyses in this report is to provide a stronger basis for making observations a useful part of teacher evaluation by addressing these facets of variability. Adapting the underlying meaning of instrument categories to specifics of various classrooms may require more experience than can be obtained in the course of a single academic study or in one or two rounds of annual observation for evaluation purposes. [This paper was presented at the Annual Meeting of the Association for Education Finance and Policy (AEFP) on February 27, 2015, in Washington, DC.]

NCLB waiver requirements have led to development of teacher evaluation systems, in which student growth is a significant component. Recent empirical research has been focusing on metrics of student growth--value-added scores in particular--and their relationship to other metrics. An extensive set of recent teacher-evaluation studies conducted by the Measures of Effective Teaching (MET) project yielded a body of empirical evidence on the correlations among various teacher effectiveness metrics, including scores from several widely-used classroom observation instruments, student surveys, and estimates of teachers' value-added contributions to student test achievement. However, recent studies, including the MET project, consistently show that the non-achievement measures (observations and surveys) have only limited correlation with the academic achievement measures, indicating that they may be measuring facets of teaching other than the teacher's ability to foster higher test scores. This paper will attempt to make the link from knowing what we are measuring in our evaluations to how we would use the information in personnel decisions. This link is viewed in terms of three steps, represented by the questions: (1) What are we measuring with observations and surveys? Can we identify a small set of underlying factors that are being measured? (2) What impact might these other factors have on the overall effectiveness of our education system? (3) How can the factors underlying the evaluation results be used in personnel decisions? The authors present some preliminary results of their analysis that addresses the first question and discuss the approaches to the second and third questions. An appendix contains figures and tables.

A quasi-experimental study designed to provide evidence of the effectiveness of the "Jamestown Reading Navigator" ("JRN") program for students in grades 9-10 was conducted in the Miami Dade Public School District (MDCSD). "Jamestown Reading Navigator" is a reading intervention program developed by McGraw-Hill Education to raise reading competencies of struggling middle school and high school students. The publishers state that the program "motivate[s] reluctant readers with highly engaging online and print-based content written exclusively for adolescents" ("Jamestown Reading Navigator," 2009). The current study tests two research questions: (1) Do students participating in the "Jamestown Reading Navigator" program attain higher reading achievement scores than comparable students who are not participating in "JRN"?; (2) Are there discernible differences in the size of impact on children of different gender, ethnicity, disability, English learner status, and pretest score? This study demonstrates that "JRN" had a positive impact on student achievement in reading for students in the Miami Dade Public School District's Intensive Reading classes. It also indicates a potential for "JRN" to benefit students with disabilities. Data availability limited the impact evaluation to the first year of program implementation and precludes assessing "JRN's" long term benefits. (Contains 3 tables and 3 figures.)