Introduction
Research Questions
- What is the impact of each PACE program and of HPOG 1.0 programs collectively on educational progress and labor market outcomes?
- To what extent do impacts vary across selected subpopulations?
- For PACE programs that generated positive earnings impacts, do the benefits exceed the costs from the standpoint of participants, employers, government, and society?
This Analysis Plan presents our plan for evaluating the impacts of the Pathways for Advancing Careers and Education (PACE) project and the first round of Health Profession Opportunity Grants (HPOG 1.0) on key outcomes nine years after random assignment into the program.
The purpose of this plan is to document the study approach and publicly commit to specified outcomes in hypothesis testing and an estimation approach prior to inspecting treatment-control differences on newly collected data. It describes the operationalization of outcome measures, including their source data. It refers to prior documents for data, measures, and methods previously established for the evaluation.
Purpose
The purpose of this report is to describe a plan for answering the study’s key research questions using nine-year outcomes. By specifying these details in advance, this document serves as a public commitment to the planned analysis.
Key Findings and Highlights
The report identifies one confirmatory outcome, which is the primary indicator of program success after nine years: average quarterly earnings over follow-up quarters 35 and 36.
The report also identifies secondary and exploratory outcomes. Secondary outcomes are additional important outcomes identified in the programs’ logic model. Exploratory outcomes include additional outcomes of interest and alternative measures of the confirmatory and secondary outcomes.
The report describes the data sources used to carry out the nine-year impact analysis, including administrative data from the National Directory of New Hires and the National Student Clearinghouse.
Methods
The impact evaluations of PACE and HPOG 1.0 both use an experimental evaluation design. The nine-year impact analyses will use administrative data to measure impacts as differences in mean outcomes between the randomly assigned treatment and control groups at each program. As elaborated in prior analysis plans, impacts will be estimated using a regression model that adjusts the difference between average outcomes for treatment versus control group members by controlling for characteristics measured at baseline.
Citation
Walton, Douglas, and Laura R. Peck. (2024). Pathways for Advancing Careers and Education (PACE) Evaluation and Health Profession Opportunity Grants (HPOG 1.0) Impact Study: Joint Nine-Year Follow-Up Analysis Plan. OPRE Report 2024-077. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- ACF:
- Administration for Children and Families
- HPOG:
- Health Profession Opportunity Grants
- HPOG 1.0:
- First round of Health Profession Opportunity Grants
- OPRE:
- Office of Planning, Research, and Evaluation
- PACE:
- Pathways for Advancing Careers and Education
- TANF:
- Temporary Assistance for Needy Families