Summary and Insights from the Ten PACE and HPOG 1.0 Job Training Evaluations: Three-Year Cross-Site Report

Publication Date: December 3, 2021
Ten PACE and HPOG 1.0  Job Training Evaluations cover image

Download Report

Download Report PDF (8,404.29 KB)
  • File Size: 8,404.29 KB
  • Pages: 87
  • Published: 2021

Introduction

Research Questions

  1. After three years, what were the impacts of PACE and HPOG 1.0 programs on: Educational outcomes? Entry into career-track employment and higher earnings? Individual and family well-being, including income and other life outcomes?
  2. What differences between program elements, context, or participant characteristics might help to explain differences in impacts across programs?

This report summarizes three-year impact findings from the Pathways for Advancing Careers and Education (PACE) project and the Health Profession Opportunity Grants (HPOG 1.0) Impact Study. These two large-scale projects evaluated education and training programs for Temporary Assistance for Needy Families (TANF) recipients and other low-income adults. The programs, to varying degrees, represent a range of strategies within the career pathways framework to support getting people started and then advancing in careers.

Each of the nine PACE evaluations (four of which were of HPOG-funded programs) and the HPOG 1.0 Impact Study (which pooled across 42 HPOG-funded programs) used experimental designs to assess impacts on postsecondary training, earnings and employment, and other life outcomes. Analyses in this report indicate that after three years, most programs substantially increased credential receipt, especially for short-term credentials. Few programs increased overall employment, but four of the five evaluations of HPOG-funded programs (three of the four PACE evaluations of HPOG-funded programs and the HPOG 1.0 Impact Study) found increased employment in the healthcare field. Only one program, Year Up, increased earnings—and its impact, $1,857 per quarter, is among the largest reported from randomized evaluations of training programs for low-income adults to date.

Further follow-up is already underway. Planned PACE and HPOG 1.0 impact analyses will examine outcomes through six years of follow-up. Those findings will be shared in future reports.

Purpose

There has been a good deal of research on particular aspects and strategies within the career pathways model, but PACE and HPOG 1.0 are among the earliest evaluations to use career pathways as a framework for program effectiveness research. The analyses presented in this report were undertaken to evaluate whether job training programs that incorporated elements of the career pathways framework successfully provided training to low-skilled adults and whether the programs’ efforts led to impacts on credentials, earnings, employment, and other life outcomes. This report synthesizes three-year findings from across the PACE project’s nine program-level evaluations and the HPOG 1.0 Impact Study.

Key Findings and Highlights

Most programs had large impacts on credential receipt and more modest impacts on training duration.
Out of the 10 evaluations (9 PACE evaluations plus the HPOG 1.0 Impact Study), nine found increases in credential receipt, ranging from 6 percentage points (36 percent) to 32 percentage points (145 percent). Most of the increase was for short-term credentials such as a Certified Nursing Assistant certificate.

Only one program, Year Up, increased earnings at the three-year mark.
With one exception—Year Up—the impact on quarterly earnings three years after random assignment (Q12-Q13) was not statistically significant. The large and statistically significant impact in Year Up, in contrast, is among the largest reported from randomized evaluations of training programs for low-income adults to date.

Few programs increased employment overall, but four of the five evaluations of HPOG-funded programs found increased employment in the healthcare field.
Only 2 of the 10 evaluations detected improvements in employment, and those impacts were modest. In contrast, four of the five evaluations of HPOG-funded programs, each of which aimed to increase healthcare employment, found favorable impacts on employment in the healthcare field.

There is little evidence that PACE and HPOG 1.0 programs, other than Year Up, affected participants’ career progress or well-being.
Consistent with the lack of impact on earnings, there is little evidence that PACE or HPOG 1.0 programs reduced financial distress or public assistance receipt. Neither did they consistently affect child development or well-being. Some programs reduced personal student debt. Year Up improved several measures of career progress and well-being.

Year Up’s program includes many strongly implemented elements that plausibly contribute to its success.
The evaluations do not provide definitive evidence on why Year Up is more effective than other programs. One conjecture focuses on Year Up’s funding strategy: Because Year Up is largely funded by employers (in the form of payments for program interns), it must prioritize satisfying employers’ needs, including tailoring selection of trainees and the content of training to meet those needs. Year Up is a mature organization that embodies stronger and more innovative organizational qualities and practices than does the typical workforce organization. This organizational capability is central to Year Up’s ability to strongly implement other key elements of the model that may contribute to the program’s success: screening applicants for their ability to benefit from the program, providing comprehensive wraparound services, emphasizing practical skills rather than credentials, and focusing solely on young adults who are less likely to have children or other responsibilities that interfere with their training.

Methods

Each program in PACE was evaluated separately using an experimental design to measure effects on education, employment and earnings, and other life outcomes. Program applicants were assigned at random to a treatment group that could access the program or to a control group that could not access the program but could access other programs in the community. Such a design ensures that any estimated differences between the treatment and control group (i.e., impacts) can be attributed to program access rather than to unmeasured differences between eligible study sample members with access (the treatment group) and without access (the control group). The impact estimates for the PACE project are based on samples ranging in size from about 500 to about 2,500 study participants randomly assigned between November 2011 and December 2014.

HPOG 1.0 used an experimental design to evaluate a collection of 42 diverse locally implemented programs, all funded by the Administration for Children and Families and operating under its broad guidelines. Impact estimates are based on a sample of about 13,800 study participants randomly assigned between March 2013 and November 2014. Four of the nine sites evaluated in the PACE project were included among the 42 programs in the HPOG 1.0 Impact Study. Therefore, though PACE reported program-specific findings for these programs, the four also contributed to the pooled HPOG 1.0 findings. Both the PACE and HPOG 1.0 projects used data from a follow-up survey of study participants conducted about three years after random assignment and quarterly wage data from the National Directory of New Hires. PACE also used data from college administrative records and the National Student Clearinghouse.1

1. See three-year reports listed in Appendix A for more detail on the methodology.

Citation

Juras, Randall, and Larry Buron. 2021. Summary and Insights from the Ten PACE and HPOG 1.0 Job Training Evaluations: Three-Year Cross-Site Report. OPRE Report 2021-155. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Glossary

ACF:
Administration for Children and Families
HPOG:
Health Profession Opportunity Grants
PACE:
Pathways for Advancing Careers and Education
TANF:
Temporary Assistance for Needy Families