Including Students with Disabilities and English Language Learners in NAEP: Effects of Differential Inclusion Rates on Accuracy and Interpretability of Findings

The paper initially describes the sources of uncertainty in National Assessment of Educational Progress (NAEP) data and standard errors. As NAEP sample sizes have increased, greater precision has been attained by the program. For this reason, exclusion effects are increasingly important. Two scenarios of revised NAEP results are presented (for New York City and for the nation) that reflect the possible results if all excluded students had been included in the data analysis: the overall NAEP results from the two recalculation scenarios vary considerably. Even where exclusion rate is constant, exclusion may affect score comparisons. When exclusion rates are not constant over time, the effects of exclusions on data comparisons can be significant. NAEP results can be affected by the percentage of students identified as Students with Disabilities (SD) or limited English proficient (LEP) in states or districts, as well as exclusion rates. The paper presents estimated recalculated results for the Trial Urban District Assessments in the two scenarios above to show how the rank orders of the districts' performance might have changed substantially. Student subgroup results may also change with increased inclusion. The effects of exclusions on NAEP data reliability can be minimized: (1) by minimizing exclusions; (2) by establishing exclusion criteria that are as clear and objective as possible and working to assure that those criteria are adhered to; and (3) making practices and criteria across states as uniform as possible. Remedies for the general effect of exclusions include: (1) efforts to minimize exclusions should continue; (2) NAEP users should be reminded more often that the students tested do not represent the entire population; and (3) research should continue on the utility of imputation models that might be used to adjust for effects of exclusions. Consequences of differential exclusion policies may be serious. If such policies vary for the two time points, groups, or jurisdictions compared, then that first, fundamental inference is compromised. The observed change, contrast, or performance gap in fact represents some mixture of differences in actual student achievement distributions and differences in decision rules determining whom to test. The increasing accuracy of NAEP statistics has made even small distortions more important than they once were. For the most part, effects of exclusions on reliability can be offset by increasing sample sizes. Effects of exclusions on validity are more problematical. It is important that NAEP continues to keep exclusions to a minimum, and that efforts be made to work toward more uniform policies and practices for determining which students should be excluded from NAEP and which should be tested. (Contains 12 footnotes and 3 tables.) [This paper was commissioned by NAGB to serve as background information for conference attendees at the NAGB Conference on Increasing the Participation of SD and LEP Students in NAEP.]