How can researchers use existing evidence to answer timely questions that have the potential to inform policy and practice? At OPRE, we are using information collected by systematic evidence reviews, such as the Pathways to Work Evidence Clearinghouse, to answer questions about emerging priorities.
The Pathways to Work Evidence Clearinghouse is a systematic review of the evidence on interventions that aim to improve employment outcomes, reduce employment challenges, and support self-sufficiency for populations who have low incomes, especially TANF and other public assistance recipients. A core goal of the Pathways Clearinghouse is to make that research evidence useful to decision makers. To achieve that goal, the Pathways Clearinghouse looks for opportunities to summarize the rich information contained in the Clearinghouse to meet the needs of decision makers and address pressing questions in the field of economic self-sufficiency. We recently harnessed the power of these data to expand our understanding of how broader contextual factors matter for the effectiveness of employment and training programs that serve job seekers who have low incomes. Such broader contextual factors include the economic conditions under which programs are operating. In other words, what is the supply of available jobs? Are program participants competing for jobs against more or fewer people? Are employment conditions generally stable, more challenging (like those associated with a recession), or improving (like those associated with a recovery)? These questions have become particularly salient as a result of the COVID-19 pandemic.
In light of the economic impacts of the COVID-19 pandemic, especially for sectors of the economy that tend to employ individuals with low incomes, we wanted to better understand which types of interventions and services might be most impactful during periods of economic recession and recovery. We asked: How well might we expect employment and training programs to perform in the kinds of economic conditions we’re currently experiencing? What advice might we be able to give practitioners serving job seekers with low incomes during economic times like these?
To answer these questions, the Pathways Clearinghouse research team at Mathematica conducted a meta-analysis that drew on the information catalogued by the Clearinghouse — such as details on intervention characteristics, study timing, and the size of the effects interventions had on employment, earnings, and related outcomes. As a first step, the Pathways Clearinghouse team referred to the existing research literature to define what counted as a period of economic recession, a period of recovery, and a period of economic stability. A recession was defined as a period when the unemployment rate is increasing, a recovery was defined as a period when the unemployment rate was falling, and a period of economic stability was defined as a period of relative stability in the unemployment rate. The team then went through 141 interventions previously reviewed by the Clearinghouse that had the information needed to address the research questions and identified those that were implemented during periods of recession, recovery, and economic stability. Ultimately, the team found 30 interventions implemented during recessions, 95 interventions implemented during periods of recovery, and 33 implemented during stable economic periods.[1]
The meta-analysis of the evidence on these interventions found that, on average, interventions improve employment outcomes among people with low incomes during both economic recessions and recoveries. This was an encouraging finding, as we had some reason to believe that during periods of economic turbulence it might be harder for the individuals with low incomes participating in these programs to improve their employment or earnings outcomes. This finding indicates, however, that the extra supports interventions provide might be helpful during more challenging economic times.
The meta-analysis also found that specific types of interventions were especially effective during either periods of recession or periods of recovery.
- During periods of recession, interventions that primarily focused on case management or other supports, employment services, and work and work-based learning had evidence of improving outcomes. The largest effects were seen among interventions focused on services in the first two categories: case management or other supports and employment services. In fact, case management interventions tended to have larger effects during recessions than they did during economically stable conditions. Meanwhile, education and training interventions tended to have smaller effects during recessions than during stable economic conditions.
- When we turned to look at interventions during periods of recovery, we found several similarities: first, interventions primarily focused on case management or other supports, employment services, and work and work-based learning had evidence of improving outcomes; and second, case management interventions had larger average effects than when implemented during stable economic conditions. But we also found important differences: during recoveries, interventions focused on incentives and sanctions also showed evidence of improving outcomes, and the largest effects during recoveries were among interventions focused on education and training and work and work-based learning.
The six types of primary services
- Case management or other supports interventions focus on assessing clients’ needs, linking clients to other available services, and providing other supports to overcome barriers, such as substance abuse counseling or classes to promote financial literacy.
- Education and training interventions focus on providing or supporting an individual through education and training programs.
- Employment retention services focus on helping employed people maintain their jobs and progress in their careers.
- Employment services help people prepare for, find, apply to, and obtain jobs.
- Incentives and sanctions interventions focus on providing, or taking away, cash or noncash benefits, such as public assistance benefits or funding for child care.
- Work and work-based learning interventions focus on providing clients with work and on-the-job learning opportunities.
These findings indicate that considering the broader economic conditions in which an intervention is being implemented could matter for the effectiveness of that intervention. They also indicate that some types of interventions may be more likely to be effective under certain kinds of economic conditions than others. Findings such as these enhance our understanding of the factors that affect an intervention’s effectiveness. They have meaningful implications for practitioners, as well, providing guidance on the kinds of services that may be more likely to help program participants during different economic conditions. Looking ahead, the Pathways Clearinghouse is continuing to use meta-analysis and related techniques to better understand how the context in which interventions are implemented may shape outcomes. Stay tuned!
Amelie Hecht is a National Poverty Fellow at OPRE whose work focuses on improving the well-being of families with low incomes through research, evaluation, and technical assistance projects. She is particularly interested in evaluation capacity building and community-engaged research that addresses disparities.
Kim Clum is a Senior Social Science Research Analyst whose work focuses on adults with low incomes, economic self-sufficiency, the application of behavioral sciences to human services, and racial and ethnic disparities in the human services. She has a particular interest in qualitative methods and equitable approaches to conducting research and evaluation.
Footnote
[1] These counts sum to greater than 141 because for some findings, the study enrollment period included both recession and recovery years. In these cases, we classified the study as having an enrollment period that included a recession year and as having an enrollment period that included a recovery year.