Introduction
Research Questions
- To what extent are QRIS ratings associated with measures of observed quality?
- To what extent are QRIS ratings associated with measures of children’s development?
Quality Rating and Improvement Systems (QRIS) are initiatives implemented in states to promote improvement in the quality of early care and education (ECE) programs. Although systems vary in their specific features, QRIS typically include a process for measuring and rating ECE program quality, sharing ratings with parents and the public, and providing supports (including financial incentives) to help programs improve their quality. Because the number of states with a QRIS and the proportion of ECE programs participating in voluntary QRIS have increased in recent years, it is important to learn about whether and how QRIS activities are working to achieve intended goals.
QRIS validation studies are one type of QRIS evaluation that examine a set of questions about how well the quality measurement and rating processes are working to differentiate meaningful levels of ECE program quality. Validation studies analyzing how QRIS ratings are associated with measures of quality and preschool children’s development were required by states that received Race to the Top – Early Learning Challenge grants. The availability of recent research results addressing similar research questions in 10 different states offers a rare opportunity to synthesize findings across multiple contexts and discuss the implications for design, implementation, and future research on state ECE quality initiatives.
Purpose
The purpose of this report is to compile and analyze findings from 10 validation studies examining quality ratings of ECE programs participating in state QRIS. Overall, the report is intended to update state administrators and other stakeholders about the effectiveness of current QRIS quality ratings in distinguishing meaningful levels of quality. The report also addresses issues of interest to researchers conducting evaluations of quality initiatives.
Key Findings and Highlights
Looking across findings from the 10 studies, QRIS ratings appear to be a helpful tool for state early childhood systems to differentiate programs at lower and higher levels of quality. Overall, QRIS ratings reflect differences in environments, interactions, and activities between ECE programs at higher and lower rating levels.
- Although statistically significant, the differences in observed quality scores between QRIS rating levels were generally small. Findings for family child care programs had mixed results.
- The studies yielded inconsistent evidence of small positive associations between ratings and patterns of children’s development. Some selective positive associations were found in some states, but not across all developmental domains examined, nor across all measures within a domain.
- Three of six studies found evidence that QRIS ratings were associated with some measures of executive function, and four found selective associations between ratings and measures of social-emotional development.
- Results documenting observed quality at medium and low levels across many QRIS programs highlight the need for continued investment and innovation in quality improvement supports for ECE programs.
Methods
The process used to produce this report included collaboration across the research teams for the ten states included in the synthesis. The group developed the plan for the synthesis and the outline for the report, and contributed analysis and writing.
The collaborative process for the synthesis included several steps of document review, meetings with research teams, and review and discussion of findings across studies. Descriptive data and findings from multivariate analyses were mostly included exactly as they appeared in the state reports; however, for certain analyses, some state teams produced new tables to align their analysis and facilitate comparison across states.
Recommendations
Several activities could build upon the studies and results described in this report to enhance quality measurement and QRIS:
- It will be important to build the literature on family child care programs in QRIS and understand how current quality measures are working in these settings.
- Validation studies (specifically) and quality improvement studies (generally) need to include children with special needs, infants and toddlers, and children who speak languages other than English and Spanish.
- Measures of children’s experiences in early care and education—beyond traditional school-readiness skills—should be included.
- As described in this report, validation studies examine how ratings are associated with measures of observed quality. However, quality ratings incorporate different domains of quality that may not be assessed by current observational measures.
- Finally, QRIS ratings are used for a variety of purposes. Clarifying the theory of change for each QRIS can help identify more accurate hypotheses about which quality levels and quality indicators should be differentiating observed quality and children’s development.
Citation
Tout, K., Magnuson, K. Lipscomb, S., Karoly, L, Starr, R., Quick H., …& Wenner, J. (2017). Validation of the Quality Ratings Used in Quality Rating and Improvement Systems (QRIS): A Synthesis of State Studies. OPRE Report #2017-92. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.