Introduction
Research Questions
- What are the training experiences of study members past the three-year mark? Do study members eventually receive longer-term credentials? Does the share of members with any credential grow?
- What are the impacts on longer-term earnings?
- Does longer follow-up change our interpretation of the estimated impact on earnings?
- How does healthcare employment and the impact of the offer of HPOG 2.0 on healthcare employment evolve with longer follow-up?
- How does the impact of the offer of HPOG 2.0 on earnings for recipients of Temporary Assistance for Needy Families (TANF) evolve with longer follow-up periods?
Running from 2015 to 2021, the second round of the Health Profession Opportunity Grants Program (“HPOG 2.0”) funded grantees to provide participants with support services and occupational training according to career pathways principles.
HPOG was authorized as a demonstration program with a mandated federal evaluation. That evaluation includes an Impact Evaluation, Descriptive Evaluation, and Cost-Benefit Study. A Longer-Term Impact Report (LTIR) expected in 2025 will report impacts of HPOG 2.0 on employment, earnings, and other outcomes of well-being about six years after study entry. This Analysis Plan describes the methodology for answering the key research questions for that report. It serves as a guide for the statistical and programming staff conducting the analyses.
Purpose
The purpose of the Analysis Plan for the HPOG 2.0 Longer-Term Impact Report is to describe a plan for answering the study’s key research questions. The Analysis Plan operationalizes key outcome measures, briefly describes key sources of data for the study, and provides details of the methods that will be used. By specifying these details in advance, this document serves as a public commitment to the planned analysis.
The document also improves the transparency and replicability of study findings by committing the research team to make consequential decisions prior to inspecting estimates of program effects. Most methods and operationalizations of outcomes measures continue from earlier HPOG 2.0 impact analyses. This Analysis Plan therefore primarily focuses on specifying analytic methods and presentation strategies specific to these longer-term analyses.
Key Findings and Highlights
The study’s estimation of impacts will build on random assignment conducted by all 27 non-Tribal HPOG 2.0 grantees.
The Analysis Plan describes the outcomes that will be examined for program impact; how these outcomes will be measured; and how impacts on multiple outcomes will be prioritized in the summary of program effectiveness.
It also describes plans for the format of the Longer-Term Impact Report. We plan for a short main body (less than 25 pages), associated appendix volume, and spreadsheet reporting additional detail.
Methods
Between February 2016 and September 2021, eligible applicants to local HPOG 2.0 programs were assigned randomly either to a treatment group that was offered HPOG 2.0 training and associated services or to a control group that did not have access to HPOG during the study period but did have access to other training and services in the community. Comparisons of mean outcomes for the treatment group versus mean outcomes for the control group give strong estimates of the impact of the program. Regression adjustment provides more precise estimates. A cluster-corrected estimation strategy gives appropriate measures of precision.
Outcomes are measured in three surveys and in three administrative data systems. The three surveys are the Short-Term Follow-Up Survey initiated about 15 months after study entry, the Intermediate-Term Follow-up Survey initiated about three years after study entry, and the Longer-Term Follow-Up Survey initiated about 66 months (five and a half years) after study entry. The three administrative data systems are the Participant Accomplishment and Grant Evaluation System (PAGES) recording information at application and activities funded by the HPOG 2.0 program, the National Student Clearinghouse recording college outcomes, and the National Directory of New Hires recording earnings outcomes.
Citation
Judkins, David Ross, Jacob Alex Klerman, Eleanor Harvill, and Larry Buron. (2024). Analysis Plan for the HPOG 2.0 National Evaluation Longer-Term Impact Report, OPRE Report 2024-023. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- ACF:
- Administration for Children and Families
- OPRE:
- Office of Planning, Research, and Evaluation
- HPOG:
- Health Profession Opportunity Grants
- IEDP:
- Impact Evaluation Design Plan
- STIR:
- Short-Term Impact Report
- ITIR:
- Intermediate-Term Impact Report
- CCIR:
- COVID-Cohort Impact Report
- LTIR:
- Longer-Term Impact Report