Summary and Insights from the Ten PACE and HPOG 1.0 Job Training Evaluations: Three-Year Cross-Site Report

There has been a good deal of research on particular aspects and strategies within the career pathways model, but PACE and HPOG 1.0 are among the earliest evaluations to use career pathways as a framework for program effectiveness research. The analyses presented in this report were undertaken to evaluate whether job training programs that incorporated elements of the career pathways framework successfully provided training to low-skilled adults and whether the programs’ efforts led to impacts on credentials, earnings, employment, and other life outcomes. This report synthesizes three-year findings from across the PACE project’s nine program-level evaluations and the HPOG 1.0 Impact Study.

Most programs had large impacts on credential receipt and more modest impacts on training duration.
Out of the 10 evaluations (9 PACE evaluations plus the HPOG 1.0 Impact Study), nine found increases in credential receipt, ranging from 6 percentage points (36 percent) to 32 percentage points (145 percent). Most of the increase was for short-term credentials such as a Certified Nursing Assistant certificate.

Only one program, Year Up, increased earnings at the three-year mark.
With one exception—Year Up—the impact on quarterly earnings three years after random assignment (Q12-Q13) was not statistically significant. The large and statistically significant impact in Year Up, in contrast, is among the largest reported from randomized evaluations of training programs for low-income adults to date.

Few programs increased employment overall, but four of the five evaluations of HPOG-funded programs found increased employment in the healthcare field.
Only 2 of the 10 evaluations detected improvements in employment, and those impacts were modest. In contrast, four of the five evaluations of HPOG-funded programs, each of which aimed to increase healthcare employment, found favorable impacts on employment in the healthcare field.

There is little evidence that PACE and HPOG 1.0 programs, other than Year Up, affected participants’ career progress or well-being.
Consistent with the lack of impact on earnings, there is little evidence that PACE or HPOG 1.0 programs reduced financial distress or public assistance receipt. Neither did they consistently affect child development or well-being. Some programs reduced personal student debt. Year Up improved several measures of career progress and well-being.

Year Up’s program includes many strongly implemented elements that plausibly contribute to its success.
The evaluations do not provide definitive evidence on why Year Up is more effective than other programs. One conjecture focuses on Year Up’s funding strategy: Because Year Up is largely funded by employers (in the form of payments for program interns), it must prioritize satisfying employers’ needs, including tailoring selection of trainees and the content of training to meet those needs. Year Up is a mature organization that embodies stronger and more innovative organizational qualities and practices than does the typical workforce organization. This organizational capability is central to Year Up’s ability to strongly implement other key elements of the model that may contribute to the program’s success: screening applicants for their ability to benefit from the program, providing comprehensive wraparound services, emphasizing practical skills rather than credentials, and focusing solely on young adults who are less likely to have children or other responsibilities that interfere with their training.

Each program in PACE was evaluated separately using an experimental design to measure effects on education, employment and earnings, and other life outcomes. Program applicants were assigned at random to a treatment group that could access the program or to a control group that could not access the program but could access other programs in the community. Such a design ensures that any estimated differences between the treatment and control group (i.e., impacts) can be attributed to program access rather than to unmeasured differences between eligible study sample members with access (the treatment group) and without access (the control group). The impact estimates for the PACE project are based on samples ranging in size from about 500 to about 2,500 study participants randomly assigned between November 2011 and December 2014.

HPOG 1.0 used an experimental design to evaluate a collection of 42 diverse locally implemented programs, all funded by the Administration for Children and Families and operating under its broad guidelines. Impact estimates are based on a sample of about 13,800 study participants randomly assigned between March 2013 and November 2014. Four of the nine sites evaluated in the PACE project were included among the 42 programs in the HPOG 1.0 Impact Study. Therefore, though PACE reported program-specific findings for these programs, the four also contributed to the pooled HPOG 1.0 findings. Both the PACE and HPOG 1.0 projects used data from a follow-up survey of study participants conducted about three years after random assignment and quarterly wage data from the National Directory of New Hires. PACE also used data from college administrative records and the National Student Clearinghouse.¹

Juras, Randall, and Larry Buron. 2021. Summary and Insights from the Ten PACE and HPOG 1.0 Job Training Evaluations: Three-Year Cross-Site Report. OPRE Report 2021-155. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.