Building Evidence for a New Intervention or Program Takes Time

Throughout the Youth At-Risk of Homelessness (YARH) grant program, the Children’s Bureau and Mathematica helped grantees develop and fine-tune comprehensive service models (referred to as interventions). The Children’s Bureau and Mathematica also helped grantees consider how they could contribute to the evidence base for other programs, including child welfare agencies interested in developing interventions to reduce homelessness among youth and young adults. Over a decade, YARH grantees engaged external evaluators, funders, local service providers, youth, and young adults to gather evidence. Efforts to build evidence culminated in the decision to fund a federal impact and implementation study of Colorado’s Pathways to Success (Pathways) intervention. This brief uses YARH as an example of how evidence is built in steps. Readers—including program staff, funders, and evaluators—can draw on the YARH  examples to inform future evaluation efforts.

This brief uses YARH as an example of how evidence is built through the following steps:

1. Design an intervention
2. Implement, refine, and document the intervention 
3. Produce indications of outcomes
4. Produce evidence of impacts
5. Replicate and produce strong evidence of positive impacts

In step 1, YARH grantees designed new interventions based on evidence from a range of sources, such as administrative data, case record reviews, and focus groups with youth and young adults. In step 2, they implemented pilot versions of their interventions, conducted small-scale tests to examine intervention fidelity and intervention processes, and established continuous quality improvement (CQI) processes; they used this evidence to refine their interventions and document key elements in intervention manuals. In step 3, grantees conducted several formative evaluation activities to determine if staff implemented the revised interventions with fidelity and to produce indications of outcomes for their interventions. In step 4, ACF and its partners selected Colorado’s Pathways intervention for a federally-led rigorous summative evaluation to produce evidence of impacts for this intervention. Step 5 is not currently part of YARH, but the summative evaluation in step 4 sets the stage for potential replication and further study of Pathways’ impacts. 

As YARH demonstrates, building evidence takes years, but there are lessons learned along the way, and they all help create evidence. This process requires contributions from various parties, including staff, evaluators, funders, and community members. Staff and evaluators need to remain flexible, patient, and involved throughout implementation. Funders help by investing time and dedicating financial resources to support building evidence. Community members—including program participants, service providers, and youth and young adults—may inform intervention design and dissemination of evidence. The work of YARH has shown that building evidence takes time, but during that time, multiple types of evidence can help programs inform and better understand their interventions.

Lynn-Green, R., E. Mugo,  N. Fung, and M.C. Bradley. “Building Evidence for a New Intervention or Program Takes Time.” OPRE Report #2024-166. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2024.

Kim, J., M. J. McCormick, and N. Islam. “The Creative and Collaborative Progression of YARH.” OPRE Report #2024-167. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2024. https://www.acf.hhs.gov/opre/report/creative-and-collaborative-progression-yarh