Evaluation of Employment Coaching for TANF and Related Populations: Evaluation Design Report

September 19, 2019
Topics:
Self-Sufficiency, Welfare & Employment
Projects:
Evaluation of Employment Coaching for TANF and Related Populations, 2016-2021 | Learn more about this project
Types:
Reports
Cover Evaluation of Employment Coaching for TANF and Related Populations: Evaluation Design Report
Download report (pdf)
  • File Size 3mb
  • Pages 294
  • Published 2019

Introduction

An important goal of Temporary Assistance for Needy Families (TANF) and other assistance programs is to help their participants become economically secure and achieve self-sufficiency. To meet this goal, some programs offer services to address challenges to employment such as those caused by lack of education or occupational skills, mental health issues, or lack of transportation or child care. However, for some participants, these services are not enough to help them become economically secure.

Recently, some researchers and policymakers have argued that people may not succeed if they have difficulty applying the many “self-regulation skills” needed to get, keep, and advance in a job. Self-regulation includes the ability to finish tasks, stay organized, and control emotions and behaviors. Research has shown that self-regulation is important in attaining goals and in determining life outcomes, including those related to employment (Almlund et al. 2011). Moreover, evidence suggests that facing poverty, and the multiple stresses that accompany it, makes it particularly difficult to use self-regulation (Mullainathan and Shafir 2013). However, evidence has also shown that interventions can both strengthen self-regulation and encourage its use, including among populations facing poverty (Almlund et al. 2011; Kautz et al. 2014). Consistent with the notion that improved self-regulation could improve employment outcomes, some employment programs have begun to use trained coaches to help clients set individualized goals and to provide motivation, support, and feedback as the clients strengthen and use self-regulation skills, which in turn will help them pursue those goals. In doing so, they aim to help the clients succeed in the labor market and move toward economic security.

To learn more about the potential of coaching to help TANF recipients and other low-income individuals reach economic security, the Office of Planning, Research, and Evaluation (OPRE) of the Administration for Children and Families (ACF) contracted with Mathematica Policy Research and Abt Associates to conduct an evaluation of employment coaching interventions––the Evaluation of Employment Coaching for TANF and Related Populations. The evaluation will use an experimental research design to examine the effectiveness and implementation of coaching interventions that aim to help low-income individuals succeed in the labor market. It will also provide the information necessary to allow other organizations to replicate and enhance employment coaching interventions.

Primary Research Questions

  1. 1 Do the employment coaching interventions improve the outcomes of low-income people, including (1) intermediate outcomes related to goal pursuit and other skills associated with labor market success, and (2) employment and economic security outcomes?
  2. 2 Do the impacts of the coaching-focused interventions change over time or vary for different groups of participants?
  3. 3 How were the employment coaching interventions implemented and what factors appear to have impeded or facilitated implementation?

Purpose

This report describes the design of the Evaluation of Employment Coaching. It identifies the types of employment coaching interventions that are the focus of this evaluation and how coaching is expected to affect participants’ economic security. Next, it discusses the process for selecting employment coaching interventions to study in the evaluation. It then provides details on the design of the impact study, including the process for conducting random assignment, data needs and sources, and the analytic approach to estimating intervention impacts. It also describes the implementation study, including the research questions to be addressed, the data collection strategy, and the analytic approach. The report concludes with an overview of the evaluation and reporting schedule.

Key Findings and Highlights

The research design for the Evaluation of Employment Coaching will provide rigorous estimates of the effectiveness of employment coaching in improving the economic security of low-income populations, and important lessons on the implementation of employment coaching. The impact study will estimate intervention impacts using an experimental design. The implementation study will document and analyze the implementation of the employment coaching interventions with three key purposes: (1) to describe the program design and operations of each employment coaching intervention and the conditions necessary for replication; (2) to help interpret the impact analysis results; and (3) to identify lessons learned for purposes of program refinement and replication.

Methods

To address the evaluation’s research questions, the Evaluation of Employment Coaching will include both impact and implementation studies. The impact study will use an experimental research design to assess the effectiveness of each employment coaching intervention in improving employment-related outcomes, economic security, self-regulation, and other measures of well-being. People who are eligible for coaching will be randomly assigned either to a “treatment group” and have access to the employment coaching intervention or to a “control group” and not have access to the employment coaching intervention. The effectiveness of the employment coaching intervention will be assessed based on differences in outcomes between members of the treatment and control groups. Intermediate outcomes of interest will include measures of improvement in self-regulation skills and outcomes that reflect steps toward meeting employment goals, such as preparing a resume and participating in job search activity. Longer-term outcomes examined will include measures of success in obtaining and retaining employment, career advancement, earnings, receipt of TANF and other financial assistance, other measures of economic security, and other indicators of personal and family well-being. Outcomes will be measured using data collected from (1) two follow-up surveys of study participants, (2) administrative employment and unemployment insurance records from the National Directory of New Hires, and (3) when readily available, administrative records pertaining to TANF and Supplemental Nutrition Assistance Program benefit receipt.

The implementation study will help interpret the impact study findings and document the interventions; discuss the implementation challenges and successes; and support future replication of employment coaching interventions. It will draw on the following data sources: (1) program documents; (2) semi-structured interviews with coaches and other program staff; (3) a survey of coaches and other staff; (4) video recordings of coaching sessions; (5) in-depth interviews with participants who work with a coach; and (6) the two follow-up surveys of study participants.

Citation

Evaluation of Employment Coaching for TANF and Related Populations: Evaluation Design Report, OPRE Report #2019-65. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Last Reviewed: October 11, 2019