Introduction
Throughout the Youth At-Risk of Homelessness (YARH) multiphase demonstration grant program, the Children’s Bureau (CB) and Mathematica have supported grantees in developing and fine-tuning interventions and thinking about what kind of evidence they could produce about the effectiveness of their interventions. This work is culminating in a federally-led summative evaluation of Colorado’s Pathways to Success comprehensive service model, the result of a multistep process Mathematica led to identify and recommend a YARH site prepared for rigorous evaluation. YARH has also provided continued evidence-building and dissemination support to grantees who were not yet ready for a summative evaluation.
This brief presents the approach YARH used to identify the YARH grantee who is the summative evaluation partner. The approach started in YARH-1, the first phase in the multiphase evidence-building process known as YARH. It continued through YARH-2, culminating in a series of meetings with ACF and consultants in YARH-3.
Purpose
YARH has conducted a unique multiphase evidence-building process where 18 grantees received funding to develop a comprehensive service model designed to prevent homelessness among youth and young adults who have been involved in the child welfare system. Six of those grantees received additional funding in YARH-2 to conduct initial implementation and formative evaluations of the comprehensive service models designed in YARH-1. During YARH-3, one grantee was selected as the focus of a rigorous summative evaluation, including an impact and an implementation study. The remaining YARH-2 grantees received additional support to continue their evidence-building in YARH-3.
This brief describes the multiphase evidence-building process YARH undertook to select the Colorado Pathways to Success comprehensive service model for the YARH-3 summative evaluation and to support YARH-2 grantees not yet ready for summative evaluation in identifying lessons learned and possible next evidence-building steps. It should be of interest to evaluators, funders, child welfare agencies, and program developers and implementers.
Key Findings and Highlights
YARH convened two evidence-building meetings to support selection of an intervention ready for summative evaluation. Four factors guided discussions for recommending a YARH-2 intervention for the summative evaluation:
1. Interest of the field in the intervention
2. Readiness of the intervention for a summative evaluation
3. Rigor of evidence that would result from the proposed design for the summative evaluation
4. Likelihood of detecting statistically significant favorable impacts
Following these meetings, Mathematica recommended moving forward with a summative evaluation of Colorado’s Pathways to Success comprehensive service model.
Mathematica also led an evidence-building meeting focused on providing feedback to the YARH sites not participating in the summative evaluation and identifying opportunities to continue their evidence-building work.
YARH’s evidence-building process offers several lessons learned, including:
- Grantees and funders are eager to participate in summative evaluations to understand what works, for whom, and how.
- Successfully supporting grantees requires careful consideration of evaluation-focused activities while they are simultaneously providing services.
- Transparency about what information would inform decisions about site readiness to participate in a summative evaluation was vital to the success of the multiphase program and the grantees.
Methods
Information for this brief comes from the YARH funding opportunity announcements and Mathematica memos and notes from YARH-3 evidence-building meetings.
Citation
M. McCormick, K. Chesnut, and M.C. Bradley (2023). Innovation and Support for Evaluating Assessing Summative Evaluation Readiness: Lessons Learned, OPRE Report # 2023-121, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- OPRE:
- Office of Planning, Research & Evaluation
- YARH:
- Youth At-Risk of Homelessness, acronym used to represent the initiative funded by Administration for Children & Families to support communities in addressing homelessness among youth and young adults with child welfare involvement