Which Program Characteristics Are Linked to Program Impacts? Lessons from the HPOG 1.0 Evaluation

Publication Date: May 13, 2019
HPOG Cover March 2019

Download Report

Download Report PDF (1,557.56 KB)
  • File Size: 1,557.56 KB
  • Pages: N/A
  • Published: 2019

Introduction

Research Questions

  1. What characteristics of HPOG programs associate with any impacts on four key outcomes?
  2. How do program characteristics associate with impacts for various outcomes?

In 2010, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services awarded the first round of five-year grants from the Health Profession Opportunity Grants (HPOG 1.0) Program to 32 organizations in 23 states; five were tribal organizations. The purpose of the HPOG Program is to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the healthcare field that pay well and are expected to either experience labor shortages or be in high demand. HPOG 1.0 grantees designed and implemented programs to provide eligible participants with education, occupational training, support, and employment services to help them train for and find jobs in a variety of healthcare professions.

The impact evaluation of HPOG 1.0 randomized 13,717 individuals into treatment or control groups across 42 HPOG programs operated by 23 non-tribal grantees. Members of the treatment group could access the HPOG Program; members of the control group could not. The difference in outcomes between the treatment and control groups is HPOG’s impact. Each program developed and implemented its own HPOG model based on ACF guidelines and its own decisions about which program features and supports would be most effective to help its participants complete training.

This paper uses variation in program characteristics—including program components, implementation features, local context, and participant traits—to explore which characteristics are associated with the size of HPOG’s short-term impact on participant outcomes. We examine the relationship between program characteristics and impacts on four key HPOG outcomes—educational progress, employment, employment in healthcare, and earnings.

Many stakeholders—including practitioners, policymakers, funders, and researchers—may be interested in which combination of program components, implementation strategies, participant characteristics, and local context make important contributions to a program’s impact on individuals. Insights from this kind of research can help inform future program design and implementation.

Purpose

The purpose of this paper is to identify which HPOG program characteristics associate with short-term impact magnitude. It aims to contribute to a better understanding of the HPOG Program by assessing how variation in program impacts associates with variation in various program characteristics, including design, implementation, and context.

From a methodological perspective, this work also provides a model for how to analyze cross-site data from multi-site experiments. It is often not practical, feasible, or even desirable to randomize all the program characteristics that one might want to learn about. Controlling for as many characteristics as possible enables researchers to isolate at least some individual program characteristics that might enhance or suppress overall impact. These observations can prove important for program practice and also generate causal hypotheses that can be tested in future evaluations.

Key Findings and Highlights

A variety of program characteristics associate with the size of short-term impacts (measured at 15 to 18 months after program entry). The set of characteristics that contributes to impacts varies by outcome:

  • For education outcomes, HPOG programs that offer greater access to tuition and financial assistance, childcare, transportation, employment supports, and emergency assistance produced larger impacts on educational progress.
  • For employment outcomes, no HPOG program components or implementation features are associated with larger impacts on overall employment in the short term. However, access to employment supports and social and other services is associated with larger impacts on employment in the healthcare sector.
  • The analysis also finds that access to employment supports is associated with a larger impact on earnings.

Although many participants had completed their training by the short-term follow-up point, about 20 percent were still enrolled in training. Future research will explore impacts at three years and six years after program enrollment, as part of the ACF-funded Career Pathways Intermediate Outcomes and Long-term Outcomes research projects.

This analysis does not allow us to make causal claims about the relationship between program characteristics and impacts. Instead, the analysis associates variation in impacts with variation in program characteristics, providing suggestive evidence of these relationships.

The paper also considers descriptively how program characteristics vary across the 42 HPOG programs examined, and identifies three “typologies”—“service-rich” programs, “education-focused” programs, and “employment-focused” programs.

Methods

To relate program characteristics to impact size, we use a multi-level model that analyzes variation across programs to identify which program characteristics influence program impacts. We consider a wide set of measures of program characteristics—including program components, implementation features, participant composition, and local context measures—and use an empirical approach to identify the characteristics most associated with impact magnitude. We conduct the empirical selection separately for each outcome, because the set of characteristics associated with impacts on one outcome might not necessarily be the same characteristics associated with impacts on another outcome.

We conducted a related analysis in the HPOG 1.0 Program Implementation and Short-Term Impacts Report ("Impacts Report") using a different process for choosing which characteristics to analyze: certain program characteristic measures were included based on their theorized relationship with impact magnitude, and other measures were selected empirically. Further, the Impacts Report selected program components based on their association with the impact only on educational progress, and then used the same characteristics to estimate models for all other outcomes. That analysis found that access to each of tuition assistance and other financial services, childcare, and transportation is associated with larger impacts on educational progress. Further, that analysis generally did not find evidence that the program components and implementation features of interest were associated with impact on employment-related outcomes, primarily because that analysis considered only those variables that associated with educational progress. By using a fully empirical approach and enabling model selection to vary by outcome, this paper provides a more flexible analysis of the impact of program characteristics on impact magnitude than was the case in the Impacts Report.

To classify programs into groups with similar characteristics, we performed a cluster analysis. This analysis identifies three main types of programs among the 42 programs analyzed, partitioning the sample according to the programs’ components and implementation features, the two types of characteristics that are most under program managers’ control.

Citation

Walton, Douglas, Eleanor L. Harvill, and Laura R. Peck (2019). Which Program Characteristics Are Linked to Program Impacts? Lessons from the HPOG 1.0 Evaluation. OPRE Report 2019-51, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Current as of: