Analysis Plan for the PACE Intermediate (Three-Year) Follow-up Study

November 2, 2018
Topics:
Self-Sufficiency, Welfare & Employment
Projects:
Career Pathways Intermediate Outcomes (CPIO) Study, 2014-2019 | Learn more about this project, Health Profession Opportunity Grants (HPOG) Evaluation Portfolio | Learn more about this project, Pathways for Advancing Careers and Education (PACE), 2007-2018 | Learn more about this project
Types:
Reports
Analysis Plan for the PACE Intermediate (Three-Year) Follow-up Study Cover
Download report (pdf)
  • File Size 1mb
  • Pages 68
  • Published 2018

Introduction

The Pathways for Advancing Careers and Education (PACE) evaluation is a study of nine promising programs that use a “career pathways” framework for increasing education, employment, and self-sufficiency among low-income individuals and families. Funded by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services, PACE will include three points of participant follow-up—at 18 months, three years, and six years after random assignment. The first round of reports, covering program implementation and impacts at 18 months after random assignment, were produced in 2017-2018, and published on the ACF Office of Planning, Research, and Evaluation (OPRE) website (www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-...).

This Analysis Plan is for the second round of reports, covering three years after random assignment. This is the second supplement to the Evaluation Design Report (Abt Associates 2014), which provided general plans for the PACE evaluation. The first supplement (Abt Associates 2015) was the Analysis Plan for the PACE Implementation and Early Impact Study, covering each program’s implementation and impacts in the first 18 months after random assignment. This Analysis Plan provides more details than the earlier documents for the third-year analyses, including detailed specification of the participant outcomes measured.

A long-term study and third round of reports, covering six years after random assignment, is underway.

Primary Research Questions

  1. 1 What are the intermediate-term effects of the PACE programs on earnings, educational outcomes, household income, adult well-being, and other life outcomes?
  2. 2 How do effects of PACE programs vary over time, across outcomes or domains, and by participant characteristics?
  3. 3 What are the common patterns of work and education over time, particularly with respect to alternating between the two?

Purpose

This Analysis Plan describes the methodology for answering the above research questions in subsequent impact reports. It will serve as a guide for the statistical and programming staff conducting the analyses. The document will also improve the transparency and replicability of study findings by committing the project team to make consequential decisions prior to inspecting estimates of program effects.

Key Findings and Highlights

  • The most prominent outcome in each report will be earnings in the 12th and 13th quarters following random assignment. This will be the confirmatory outcome in all nine reports—that is, the most critical and consequential outcome the programs seek to affect.
  • In the five PACE programs that primarily support training in colleges (i.e., degree-granting institutions) instead of other types of postsecondary training institutions, receipt of a college credential requiring at least one year’s worth of study will also be a confirmatory outcome.
  • We specify a variety of secondary outcomes in the areas of earnings and employment, education, and other life outcomes. These all have an expected direction of effects and will be tested with one-sided hypothesis tests.
  • We also specify in less detail a variety of exploratory outcomes and analyses.
  • The Analysis Plan includes detailed methods for covariate control, nonresponse adjustment, and imputation of answers to questions that a respondent skipped.

Methods

In general, the impact analysis will compare mean outcomes between the treatment and control groups at each PACE site. We will use a variety of information sources to measure the outcomes, including data from participant follow-up surveys, program and college records (where available), and administrative data from the National Directory of New Hires and National Student Clearinghouse. We will make adjustments for baseline imbalances, survey nonresponse, item nonresponse, and attendance at institutions of higher education from which records are unavailable. We arrived at these procedures through careful consideration of the evaluation literature, the designs of the PACE programs, and the particularities of local record systems.

Last Reviewed: November 13, 2018