Health Profession Opportunity Grants (HPOG 1.0) Impact Study Interim Report: Program Implementation and Short-Term Impacts

Publication Date: June 18, 2018
Current as of:
Health Profession Opportunity Grants (HPOG 1.0) Impact Study Interim Report: Program Implementation and Short-Term Impacts Cover

Download Report

Download PDF (3,161.56 KB)
  • File Size: 3,161.56 KB
  • Pages: N/A
  • Published: 2020

Introduction

Research Questions

  1. What impacts did the locally implemented HPOG programs as a group have on the outcomes of participants and their families?
  2. To what extent did those impacts vary across selected subpopulations?
  3. Which locally adopted program components influenced average impacts?

In 2010, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services awarded the first round of five-year HPOG grants (HPOG 1.0) to 32 organizations in 23 states; five were tribal organizations. The purpose of the HPOG Program is to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the healthcare field that pay well and are expected to either experience labor shortages or be in high demand. HPOG 1.0 grantees designed and implemented programs to provide eligible participants with education, occupational training, and support and employment services to help them train for and find jobs in a variety of healthcare professions. The ACF Office of Planning, Research, and Evaluation supports a multipronged research and evaluation strategy to assess the success of the HPOG Program. To assess its effectiveness, the first round of HPOG programs was evaluated using an experimental design in which program applicants were assigned at random to a “treatment” group that could access the program or a “control” group that could not, and then their outcomes compared. This document reports on the programs’ implementation and short-term impacts, those that arose roughly five calendar quarters after random assignment. It reports an average impact across the diverse HPOG 1.0 programs.

Note: This report was originally released in May 2018. Errors in the analysis of subgroup impacts were identified after publication. These errors were corrected and the report reissued in November 2019. See footnote 41 in the main report for more information.

Purpose

The HPOG 1.0 Impact Study is making an important contribution to the field’s collective knowledge about sector-based and career pathways programs. Because it found that HPOG had a positive impact on short-term “educational progress” (having completed or being currently enrolled in program training), the evaluation also may be able to detect impacts on longer-term employment and earnings in subsequent follow-up findings. Most other evaluations focus on a single program usually selected for its promise, and the results of those evaluations are generalizable to programs that are similar to the one evaluated. In contrast, the HPOG 1.0 Impact Study considers a large collection of diverse locally implemented programs, all operating in their own way under broad ACF guidelines. The benefit of this approach is that it helps to assess whether the general HPOG model—across many implementations of it—is effective in achieving its goals. That assessment could have implications for policy and program design, as well as for future research.

Key Findings and Highlights

According to a follow-up survey (initiated 15 months after randomization) and national administrative data (pinned to the fifth follow-up quarter):

  • About 68 percent of the treatment group versus about 60 percent of the control group had completed or were currently enrolled in training. This measure of educational progress was the study’s confirmatory outcome; as such, the favorable impact is a sign that HPOG is on track toward its main objective of increasing labor market success.
  • The treatment group was no more likely than the control group to be employed, but the treatment group was more likely to be employed in the healthcare sector (about 53 percent versus 41 percent).
  • The treatment group earned slightly more than the control group. The $137 difference represents a 4 percent increase in the fifth follow-up quarter.
  • These observed impacts arose because of the differences between the training and supportive services available to the treatment group versus the control group. Indeed, the major difference was the treatment group’s greater access to supportive services. The treatment group reported receiving more academic, personal, and financial services than did the control group.
  • Although HPOG was effective for a wide variety of people (e.g., across age, race/ethnicity, parental status), generally it appears to be more effective for those who enter the program with the advantages of some college or a degree, fewer barriers to school/work, or employment. Individuals with these characteristics benefited more from HPOG than did participants who were less advantaged at entry.

Methods

The HPOG 1.0 Impact Study used an experimental evaluation design to assess the impacts of 42 local HPOG programs in 23 of 32 first-round grantees nationwide. By randomizing eligible applicants to treatment and control groups, the evaluation provides rigorous evidence to inform the adult training field about sector-based and career pathways programs. In 19 of the 42 programs, the evaluation also analyzed how certain program enhancements—emergency assistance to address unexpected needs (such as a car repair or utility shutoff), non-cash incentives to encourage certain behaviors (such as perfect attendance) or meeting certain program milestones (such as completing a certification), and facilitated peer support to foster social and emotional connections among students and with faculty and staff—might improve on impacts of the standard HPOG program. The evaluation also used non-experimental methods to examine the relative contributions of various program characteristics.

The study’s impact analysis used administrative data from the first five quarters after study participants were randomized and data from a follow-up survey initiated 15 months after they were randomized. The implementation analysis used data from a variety of sources, including administrative data from HPOG programs, qualitative data collected through in-person interviews during site visits, and various surveys of HPOG grantees, staff, management, and partners/stakeholders.

Citation

Peck, Laura R., Alan Werner, Eleanor Harvill, Daniel Litwok, Shawn Moulton, Alyssa Rulf Fountain, and Gretchen Locke. (2018). Health Profession Opportunity Grants (HPOG 1.0) Impact Study Interim Report: Program Implementation and Short-Term Impacts, OPRE Report 2018-16. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.