The Sexual Risk Avoidance Education (SRAE) grant program—a federally funded adolescent pregnancy prevention initiative administered by the Administration for Children and Families (ACF), Family and Youth Services Bureau—teaches youth how to voluntarily refrain from non-marital sexual activity. The SRAE National Evaluation (SRAENE) team from Mathematica developed three toolkits with resources to support SRAE grant recipients in designing and conducting implementation evaluations.
Grant recipients can use an implementation evaluation (which is sometimes called a process evaluation) to study how they delivered their program activities. For example, they can assess whether program facilitators implemented the program as intended with high quality; whether youth received the expected dosage; or whether outside events, policies, or programs affected program implementation. The findings from an implementation evaluation can help identify areas for improvement and provide useful context for the findings from an outcome or impact evaluation. The three toolkits on implementation evaluations are—
Purpose
The three toolkits intend to support SRAE grant recipients in designing and conducting implementation evaluations. The SRAENE team from Mathematica developed the toolkits.
The primary audience for these resources is practitioners and evaluators supporting SRAE and related youth programming. The resources also may be of interest to researchers and evaluators more broadly, as they provide a variety of resources for conducting implementation evaluations.
Key Findings and Highlights
This webinar and resource (PDF) provide an overview of implementation evaluations, including what topics can be studied in an implementation evaluation, examples of research questions, possible data sources, and differences between implementation evaluations and impact and outcome evaluations.
The SRAENE team developed three toolkits with resources to support grant recipients as they plan and conduct implementation evaluations.
- Toolkit 1 includes 12 resources on planning for an implementation evaluation. The resources cover topics including developing an evaluation plan, identifying research questions and data sources, obtaining institutional review board (IRB) approval, building staff buy-in for evaluation activities, and identifying procedures for managing and storing data.
- Toolkit 2 includes 11 resources on collecting data for an implementation evaluation. The resources cover data collection through focus groups, interviews, fidelity logs, surveys, and observations, as well as best practices for virtual data collection.
- Toolkit 3 includes 10 resources about analyzing and reporting on data for an implementation evaluation. The resources cover analysis methods, dissemination strategies, and using findings from an implementation evaluation in program decision-making.
This video provides information on the three toolkits.
Methods
The SRAENE team compiled resources for the toolkits from various sources. The team also developed new resources within each toolkit. The resources in the toolkits were refined through a pilot with SRAE grant recipients in 2022.
Citation
Angus, Megan, Brittany Tabora, and Betsy Keating. “Introduction to SRAENE Implementation Evaluation Toolkits“ OPRE Report No. 2023-197. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2023.
Keating, Betsy, Megan Angus, and Brittany Tabora. “Summary of Toolkit 1: Planning for an Implementation Evaluation.” OPRE Report No. 2023-200. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2023.
Angus, Megan, Brittany Tabora, and Betsy Keating. “Summary of Toolkit 2: Data Collection for an Implementation Evaluation.” OPRE Report No. 2023-199. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2023.
Tabora, Brittany, Betsy Keating, and Megan Angus. “Summary of Toolkit 3: Analysis and Reporting for an Implementation Evaluation.” OPRE Report No. 2023-198. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2023.