Statistical Misunderstandings of the Properties of School Scores and School Accountability

This chapter provides an overview of statistical issues arising in work on educational assessment and school accountability, organized around some of the most consequential misunderstandings. The article proceeds via a series of nine vignettes describing misunderstandings of statistical uncertainty in educational assessment and school accountability. The flow of the exposition is from assessments of the statistical uncertainty in a group summary (school-score) to the consequences of uncertainty for properties of accountability decisions to attempts to adjust accountability criteria for the effects of uncertainty. The first two vignettes describe misunderstandings of the accuracy of scores. The next three vignettes depict the consequences of statistical uncertainty for properties of accountability systems (including wayward claims of a substantial "diversity penalty" and "small school advantage"). The next set of three vignettes concentrates on the often-cited "margin of error" and its misapplication to accountability, as in the NCLB confidence intervals. The final vignette describes the frequent misinterpretation of correlation coefficients to indicate the strength of the relation between student demographics and educational performance. (Contains 1 figure and 10 tables.)