Search

Background/Context: Many states, school districts, and third-party platforms report measures of school performance. Such school ratings are widely consulted by parents and educators alike. The ratings appear to affect families' choices of where to live and where to enroll students (Bergman and Hill, 2018; Hasan and Kumar, 2019), as well as districts' decisions to close or restructure schools (Rockoff and Turner, 2010; Abdulkadiroglu et al., 2016; Cohodes et al., 2021). School ratings are often strongly correlated with the racial make-up of student bodies. In urban districts with racially diverse student populations, highly-rated schools tend to disproportionately enroll white and Asian students. This pattern often raises concerns that published ratings contribute to ongoing racial segregation (NFHA, 2006; Yoshinaga, 2016). Purpose/Objective/Research Question: This paper studies the relationship between widely-used public school ratings and student racial composition, drawing broader implications for school assessment systems. We explore whether the correlation between rankings and schools' racial make-up reflects differences in true school quality or whether this relationship is an artifact of selection bias. Popular ratings that fail to adjust for differences in student preparedness or ability likely suffer from such bias (Angrist et al., 2017; Abdulkadiroglu et al., 2020b). Rating schemes that reward family background rather than educational effectiveness are likely to direct households to low-minority rather than higher-quality schools, while penalizing schools that improve achievement for less-advantaged groups. Better understanding these relationships and developing improved measures can generate more equitable and transparent ratings. Setting: We use data from Denver Public Schools (DPS) and the New York City Department of Education (NYCDOE). These two districts are important in ongoing discussions of segregation and school access. Importantly for our research design, both districts use centralized assignment systems--with some element of quasi-random tiebreaking--to match students and schools. Population/Participants/Subjects: Our DPS analysis sample includes students applying for sixth-grade seats for the 2012-2013 through 2018-2019 school years. Our NYC analysis sample includes sixth-grade applicants for the 2016-2017 through 2018-2019 school years. Intervention/Program/Practice: We leverage quasi-experimental variation in centralized school assignment systems to quantify the relationship between school quality, popular school ratings, and the racial composition of schools. Research Design: We first derive a simple theoretical characterization of the link between the predictive accuracy of a school rating and its racial imbalance. Predictive accuracy is given by a rating's (squared) correlation with a school's true causal effect on achievement (i.e., quality). Racial imbalance is given by the regression of school ratings on white (or white and Asian) enrollment shares. Our characterization shows that if true school quality is uncorrelated with student race there is a kind of "free lunch" -- racial imbalance can be removed by a simple statistical procedure at no cost to predictive accuracy. School quality is not directly observed, so this theoretical framework is not immediately applicable. As in Abdulkadiroglu et al. (2017, 2020a), we surmount this identification challenge by using the random variation in school attendance generated by centralized school assignment systems. Building on this framework and the instrumental variables value-added model (IV VAM) approach from Angrist et al. (2021), we derive feasible estimators of the relationships between causal value-added, racial composition, and conventional school ratings. Data Collection and Analysis: We received detailed student-level data from DPS and the NYCDOE. IV VAM estimates quantify the trade-off between predictive accuracy and racial imbalance as in our theoretical framework, leveraging quasi-random assignment variation. We apply this framework to two ratings: one based on achievement levels (the average share of enrolled students proficient in math and English language arts), and a "progress" rating based on achievement growth. Findings/Results: We first find that ratings based on achievement levels are highly correlated with the share of enrolled students who are white. Progress ratings are much less correlated with student race. IV VAM estimates further reveal that the racial make-up of a school's student body is largely unrelated to school quality in both cities (see Figure 1). This suggests the "free lunch" in our theoretical framework applies. We next find that progress ratings predict school quality much more accurately than levels ratings. Levels ratings are only weakly related to quality due to selection bias. Though progress measures better predict quality than levels, some selection bias remains (see Figure 2). Thus, the relationship between school ratings and race is an artifact of selection bias and adjusting school ratings to reduce racial imbalance may thus come at little cost. We confirm this prediction by showing that a conventional progress-based rating adjusted to be uncorrelated with race has predictive accuracy no worse than (and sometimes better than) that of the corresponding unadjusted measures. In both cities, this new "race-balanced progress" rating essentially coincides with an optimal rating constructed to best predict causal value-added as a function of conventional progress ratings, student race, and school sector (see Figure 3). Conclusions: The oft-noted correlation between school ratings and racial composition raises the concern that such ratings promote segregation and penalize schools that serve minority students. At the same time, demographic differences in ratings may also signal important disparities in school quality. Our analysis uses the random assignment embedded in centralized assignment mechanisms to disentangle the relationship between school ratings, school quality, and race. We show that for middle schools in Denver and New York City, the fact that schools with more white students are highly rated reflects selection bias rather than educational quality. As a result, ratings purged of their correlation with racial shares predict school quality as well or better than standard measures based on achievement levels and progress. Our new "race-balanced progress" measure is constructed using a simple adjustment that analysts could easily implement. Denver and NYC share important features with other large urban districts, suggesting the patterns uncovered here may not be unique to these cities. We plan to explore the trade-off between predictive accuracy and racial imbalance in other urban districts in the near future.