Search

This paper provides the National Center for Education Statistics (NCES), National Assessment Governing Board (NAGB), and the National Assessment of Educational Progress (NAEP) community with information that may help maintain the validity and utility of the NAEP assessments for civics and U.S. history as revisions are planned to the NAEP frameworks that will inform these assessments now scheduled for 2030. The authors addressed the overarching questions below: (1) How old are the NAEP assessment frameworks for civics and U.S. history? Why does this matter? (2) What important developments in the fields of civics and U.S. history have occurred since the current frameworks were written? (3) To what extent are the NAEP assessments aligned or disconnected to what is, and will likely be, occurring in the states? (4) Based on the findings, what recommendations should those who will be reviewing the frameworks consider to ensure the validity and utility of NAEP assessments in civics and U.S. history? [This report was commissioned by the NAEP Validity Studies (NVS) Panel.]

During the past decade, the NAEP Validity Studies (NVS) Panel has been monitoring, studying, and commenting on potential issues with the validity of the National Assessment of Educational Progress (NAEP) arising from changes that have been brought about by the adoption of rigorous state college and career readiness standards, such as the Common Core State Standards (CCSS). NAEP is meant to be reflective of the entirety of what is taught in the United States, and the many changes to standards in the past 10 years have led to questions about the extent to which NAEP continues to meet this objective. The NVS Panel has conducted several studies to investigate this issue and has found some variations in the alignment between state and NAEP standards across different NAEP grades and subjects. Given the prominent attention that the NAEP results receive, states and participating districts are especially interested in learning how well those results relate to what is taught and tested in schools. States and districts have informally posited that, if the alignment between NAEP frameworks and their own content standards were closer, then their NAEP scores might be higher. Stated another way, there are concerns that NAEP may be underreporting the actual abilities of their students--and trends in achievement--because of some degree of misalignment. When the 2017 NAEP Mathematics TUDA (Trial Urban District Assessment) results were reported and it appeared that student performance trends on NAEP were not similar to student performance trends on the state assessments that were aligned to college and career ready standards, several leaders in the affected TUDAs called for what amounted to a "recount." The results of student performance on the state assessments from 2013 to 2017 were showing more positive trends than the results of student performance on NAEP during the same period. In the reanalysis discussed here (Dogan, 2019; reproduced in the appendix), the study author noted that most of the negative NAEP trends observed in districts in recent years (i.e., 2015 to 2019) coincide with major changes in states' learning standards and assessments, raising the legitimate question: Can these trends be a function of the differences between NAEP assessment content and states' transition to new college and career readiness learning standards, such as the Common Core State Standards (CCSS), and the corresponding shift in the content of state assessments to be aligned to these newer standards? It was determined that this issue of mismatch trends or the misalignment of results should be examined for at least two reasons: (a) Urban districts are held accountable for students' performance on state assessments that are aligned to state-mandated content standards; however, (b) urban districts are not held directly accountable for performance on NAEP, which historically was regarded by many as the standard against which the adequacy of state assessments is judged. This report serves as a response from the NVS Panel to the analysis conducted by Dogan (2019). Selected panel members were asked to provide comment, and their responses have been edited together into this report. The conclusion of the report is that the secondary analysis done by Dogan for the NAEP TUDA scores is important and worthy of further exploration as part of ongoing efforts to monitor the validity of NAEP. [This report was commissioned by the NAEP Validity Studies Panel.]

Development of the Common Core State Standards (CCSS), and the creation of the Smarter Balanced Assessment Consortium (Smarter Balanced) and the Partnership for Assessment of Readiness for College and Careers (PARCC), changes the pattern of accountability testing. These changes raise the question: "How should NAEP's validity and utility be maintained?" The assessments planned by the consortia may be different enough from current state assessments to raise questions as to whether NAEP can continue to play its historic role as an independent monitor or "check" on the validity of state assessments. It is also clear is that computer-based assessment is coming to K-12 education, and both consortia plan to include more varied item types than have been commonly used in the past. Computerization of NAEP is inevitable and already planned by the National Assessment Governing Board. Computerized NAEP assessments may appear more similar to future statewide assessments. Comparability of results can usually be maintained as a test makes the transition from paper-and-pencil to computerized administration, but computerization may have an effect on results for some subgroups of the population. Computerization of NAEP is best approached in the same way as other changes to NAEP assessments have been approached: A bridge study should insure the comparability of results across the transition unless an a priori decision is made to "break trend" regardless. Assessments developed by Smarter Balanced and PARCC may reduce the number of statewide tests to the low single digits, thus making linkage feasible. Associations between the results of disparate educational assessments tend to change over time, so any linkage between the NAEP scale and the consortia statewide tests will need to be maintained regularly. A singular opportunity exists in a short window of time--essentially right now--to design the data collection for linkage between the NAEP scale and the consortia assessments while the latter are under development. Two appendices present: (1) Membership in the PARCC and Smarter Balanced Consortia; and (2) Computer-Based Assessment: A Review of the Last 15 Years of Comparability Research. [For the main report, "Examining the Content and Context of the Common Core State Standards: A First Look at Implications for the National Assessment of Educational Progress," see ED545237.]

The relationship between test preparation and academic performance on a high school graduation test was studied using data from the 1995-96 Louisiana Graduation Exit Examination (GEE). Test preparation was measured based on students' responses concerning whether they prepared well for the test or not. Academic performance was measured through five subjects: English Language Arts, Mathematics, Written Composition, Science, and Social Studies. Each subject was examined based on two test uses: students' passing status and students' scaled scores. A series of chi square tests were used for each subject to examine the relationship between test preparation and passing status. In addition, Pearson correlation analysis was used for each subject to examine the relationship between test preparation and scaled scores. Results show that there is a significant relationship between preparation and academic performance, but that the magnitudes of the relationships are different across the five subjects. The strongest relationship was found for mathematics. The patterns of the relationships between test preparation and academic performance were generally consistent across subgroups with reference to gender and ethnicity. (Contains eight tables and seven references.) (SLD)

Gender differences of gifted and talented students in mathematics performance were examined using the Louisiana Educational Assessment Program (LEAP) of 1995. The LEAP test is a statewide criterion-referenced test administered to all Louisiana public school children in Grades 3, 5, 7, 10, and 11. In this study, the database was restricted to seventh-grade students identified as gifted and talented. Gender, ethnicity, and socioeconomic status served as independent variables. Dependent variables were the nine subskill areas in the mathematics test which measure different aspects of mathematical ability. These subskill areas are: (1) numeration; (2) whole number operations; (3) fraction operations; (4) decimal numbers and operations; (5) percent, ratio, and proportion; (6) measurement; (7) geometry; (8) graphs, probability, and statistics; and (9) pre-algebra. A slight female superiority was found in two mathematics subskill areas; however, the difference was too small to have a practical meaning. No male superiority was found in the nine different mathematical categories. The patterns of gender differences were consistent across ethnic and socioeconomic background. Because LEAP tests were not designed specifically for gifted and talented students, the test may not be a good measure to discriminate among highly intelligent students. (PVD)