Search

The first step in the Validity and Reliability Study summarizes the history and purpose of California's quality rating and improvement system (QRIS), reviews findings from other QRIS evaluation studies, and describes the approach to validating the system in California. The majority of this report focuses on providing context for the California QRIS and assessing the validity and reliability of the system. The structure of the study, including the thematic questions that organize the report, are presented in Exhibit 1. To provide context for the study, the researchers offer a snapshot of the status of the implementation of the QRIS using the Hybrid Rating Matrix and the issues and challenges faced by Consortia. To understand how well the QRIS defines quality, researchers reviewed the content of the Hybrid Rating Matrix, including an examination of the research base for the quality elements and rating approaches and common practices related to them across other QRISs. Researchers then conducted three activities to examine QRIS validity and reliability. To assess how well the QRIS functions as a measure of quality, the distribution of the ratings obtained from the Common Data Elements for the RTT-ELC QRIS, and the measurement properties of the rating were assessed. Drawing on their own classroom observations conducted for the study as well as data shared by the Consortia, researchers compared classroom observation scores for programs with different ratings to assess the degree to which the ratings capture quality differences as determined by an independent measure of quality. Sensitivity analyses were also conducted which compare the distribution and validity of ratings calculated using several different strategies including the local modifications to the Hybrid Rating Matrix that individual Consortia have adopted. Finally, the report summarizes the key preliminary findings of the validation study addressing six primary research questions and offering considerations for next steps.