Introduction
Research Questions
- How were programs implemented?
- What is the context in which each program was implemented?
- What is the design of each program?
- How and how well was each program implemented?
- What is the cost to implement the programs?
- What is the total cost per participant per month for each program?
- What is the total cost per participant for each program?
- Do the programs improve outcomes?
- Does each program affect participants’ employment outcomes?
- Does each program affect participants’ economic independence?
- Does each program affect the amounts and types of services participants receive?
- Does each program affect participants’ health and other outcomes?
Many people face complex challenges to obtaining, retaining, and advancing in employment that in turn affect their ability to be economically independent. Although current research suggests some promising strategies for people facing complex challenges, there is still much to learn about how to best serve this population.
To identify and study innovative employment programs for people facing complex employment challenges, the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families, U.S. Department of Health and Human Services is sponsoring, and Mathematica is conducting, the Next Generation of Enhanced Employment Strategies (NextGen) Project. The NextGen Project is part of OPRE’s Innovative Strategies for Addressing Employment Barriers Portfolio, which seeks to rigorously evaluate the “next generation” of employment strategies for individuals with low incomes, and is partnering with the Social Security Administration (SSA) on select evaluations.
This report describes the design of the NextGen Project, which is evaluating the effectiveness of four programs that provide services for people with physical, mental, or emotional health challenges. 1 Many of these people are potential applicants for Supplemental Security Income (SSI). The four programs are:
Bridges from School to Work (Bridges), which serves young adults (17—24) with disabilities who are transitioning out of high school. The program aims to meet the needs of both the young adults it serves and local employers. Bridges staff use a strengths-based approach focused on the young adults’ skills, interests, and abilities rather than a deficit-oriented framing focused on their disabilities.
Individual Placement and Support for Adults with Justice Involvement (IPS-AJI), which provides assistance to adults with mental health issues who are reentering the community after incarceration or who have received an alternative sentence. The program uses the IPS model to offer participants mental health treatment and employment assistance at mental health centers.
Philadelphia Workforce Inclusion Networks (Philly WINs), which serves adults with low incomes and chronic physical, mental, or emotional conditions or disabilities that could limit their employment. The program develops relationships with and provides technical assistance to a network of employers that provide inclusive workplaces and prepares program participants for jobs at these and other employers.
Western Mass Mental Health Outreach for MotherS PartnershipSM (Western Mass MOMS), which serves adult caregivers who identify as women or nonbinary, have low incomes, and exhibit depressive symptoms. The program is based on the MOMS Partnership® model, which is designed to reduce depressive symptoms, improve social connections, and promote economic well-being among mothers. For Western Mass MOMS, the NextGen Project team worked with the MOMS Partnership model developers and the service provider to add employment services.
The results of the NextGen Project will inform policymakers and practitioners interested in helping people with complex challenges to employment become economically independent. The findings also are intended to help SSA better understand the types of programs that effectively connect or reconnect potential SSI applicants to work before they apply for benefits.
Purpose
This report describes the designs of the descriptive, cost, and impact studies of the four programs included in the NextGen Project. It first discusses how the NextGen team identified and selected programs for the study and prepared those programs for participation. Next, it provides a descriptive study design, including the research questions addressed, the data collection strategy, and the analytic approach. It then describes the cost study, including its goals, the data collection strategy, and the analytic approach. Finally, it details the impact study design, including research questions, the experimental design, data needs and sources, and the analytic approach to estimating program impacts. A separate appendix for each program provides more details on the program and the design of its evaluation.
Key Findings and Highlights
The NextGen Project will provide rigorous estimates of programs’ effectiveness in improving the economic independence of people facing complex challenges to employment. It will also offer important lessons about the implementation and cost of these programs. The descriptive study will document the programs and their operations, assess whether they have been implemented as designed, and help interpret findings from the impact studies. The cost study will estimate the total cost and the cost per program participant to help decision makers allocate resources. The impact study will estimate program impacts using an experimental design.
Methods
To address the NextGen Project’s research questions, each of the four evaluations will include descriptive, cost, and impact studies. The descriptive studies will document the programs and their operations. The descriptive study will draw on multiple data sources, including ongoing discussions with program leaders and staff as part of technical assistance in implementing the study; semi-structured interviews with program leaders and staff; surveys of program leaders and staff; program observations; in-depth interviews with participants; program management information systems; program documents; and surveys of study participants.
The cost study will estimate the cost of each program, overall and per participant. To develop these estimates, the NextGen team will use data collected from the study’s Random Assignment, Participant Tracking, Enrollment, and Reporting (RAPTER®) system and programs’ management information systems, as well as financial data requested from the programs.
The impact study will assess the programs’ effectiveness. People eligible for each program and who consent to be in the study will be randomly assigned to either a program group or a comparison group. People in the program group will be offered the program’s services. Those in the comparison group will not be offered the program services but will be offered other services provided in the community. The NextGen team will assess the effectiveness of programs based on differences in outcomes between members of the program and comparison groups. For all programs, the study team will examine impact estimates for measures of service receipt, employment, earnings, and application for or receipt of SSI. Depending on the program, other outcomes might include mental health, perception of social support, receipt of public assistance, and involvement in the criminal justice system. The team will measure outcomes using data collected from two follow-up surveys of study participants and administrative records.
Citation
April Wu, Kristen Joyce, Annalisa Mastri, Sheena McConnell, Jody Schimmel Hyde, and Jacqueline Kauff. (2024). Next Generation of Enhanced Employment Strategies Project: Evaluation Design Report. OPRE Report #2024-154. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.