Health Profession Opportunity Grants (HPOG) Impact Study’s Three-Year Follow-Up Analysis Plan

January 22, 2019
Topics:
Self-Sufficiency, Welfare & Employment
Projects:
Career Pathways Intermediate Outcomes (CPIO) Study, 2014-2019 | Learn more about this project, Health Profession Opportunity Grants (HPOG) Evaluation Portfolio | Learn more about this project
Types:
Reports
Health Profession Opportunity Grants (HPOG) Impact Study’s Three-Year Follow-Up Analysis Plan cover
Download report (pdf)
  • File Size 1mb
  • Published 2019

Introduction

In 2010, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services awarded the first round of five-year grants for the Health Profession Opportunity Grants (HPOG 1.0) Program to 32 organizations in 23 states, including five tribal organizations. The purpose of the HPOG Program is to provide education, training, and supportive services to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the healthcare field that pay well and are expected to either experience labor shortages or be in high demand.

The ACF Office of Planning, Research, and Evaluation is using a multipronged research and evaluation strategy to assess the success of the HPOG Program. To assess its effectiveness, the first round of HPOG grantee programs is being evaluated using an experimental design in which program applicants were assigned at random to a “treatment” group that could access the program or to a “control” group that could not and then their outcomes compared in the short- (about 15 to 18 months after randomization), intermediate- (three years), and long-term (six years).

This report presents a detailed plan for evaluating the intermediate impact of the HPOG 1.0 grants on various outcomes. It describes the operationalization of outcome measures and provides key details of the methods to be used in the intermediate impact analysis.

Primary Research Questions

  1. 1 What impacts do HPOG programs have on the outcomes of participants and their families?
  2. 2 To what extent do impacts vary across selected subpopulations?
  3. 3 To what extent do the education and employment experiences of HPOG participants over time suggest that they are following a career pathway?

Purpose

The purpose of this report is to describe a plan for answering the research questions using three-year outcomes. By specifying these details in advance, this document serves as a public commitment to the planned analysis.

Key Findings and Highlights

  • The report identifies two confirmatory outcomes, which are the main indicators of the HPOG Program’s progress toward its goals after three years. The confirmatory outcomes will be (1) training completion and (2) average earnings in the 12th and 13th quarters after random assignment.
  • The report also describes secondary and exploratory outcomes. Secondary outcomes are additional important outcomes identified in the HPOG logic model. Exploratory outcomes include additional outcomes of interest that may be affected by the program but are not identified in the logic model, or alternative specifications of the confirmatory and secondary outcomes.
  • Finally, the report describes the data sources that will be used to carry out the study, including the Three-Year Survey and administrative earnings data from the National Directory of New Hires.

Methods

The impact evaluation of HPOG 1.0 uses an experimental evaluation design. It will estimate the impact of HPOG using a regression model that adjusts the difference between average outcomes for treatment versus control group members by controlling for exogenous characteristics measured at baseline. The analysis will use a three-level model to account for variation at the individual participant, administrative division (intake location), and program levels. Missing data will be accounted for with multiple imputation and nonresponse weighting.

Citation

Litwok, Daniel, Douglas Walton, Laura R. Peck, and Eleanor Harvill. (2018). Health Profession Opportunity Grants (HPOG) Impact Study’s Three-Year Follow-Up Analysis Plan. OPRE Report #2018-124, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Last Reviewed: January 18, 2019