Table of Contents | Previous | Next |
EXECUTIVE SUMMARY
This report uses meta-analysis, a set of statistically based techniques for combining quantitative findings from different studies, to synthesize estimates of program effects from random assignment evaluations of welfare-to-work programs and to explore the factors that best explain differences in the programs' performance. The analysis is based on data extracted from the published evaluation reports and from official sources. All the programs included in the analysis targeted recipients of Aid to Families with Dependent Children (AFDC; now called Temporary Assistance for Needy Families, TANF1). The objective of the analysis is to establish the principal characteristics of welfare-to-work programs that were associated with differences in success, distinguishing between variations in the services received, differences in the characteristics of those who participated in each program, and variations in the socio-economic environment in which the programs operated.
Meta-analysis is a powerful instrument for analyzing the combined impacts of comparable policy interventions, while controlling for a range of factors pertaining to these interventions or the environment, in which they took place. However, like other statistical techniques, meta-analysis can be subject to data limitations that adversely affect its capacity to produce robust and reliable results. Multicollinearity of variables (resulting from small sample size), inconsistencies in the information provided in different evaluation reports, and omitted or misspecified variables are some of the data analysis risk that we sought to minimize, for instance, by verifying data entries and carefully considering the specification of the regression equations that are estimated. Yet, it would have been impossible, as well as impractical, to eradicate all risk of error in the analyses, much of which would have beyond the control of this study and could be traced back to the original evaluations. In the light of such limitations, many of the conclusions that are reached are subjected to sensitivity tests. These tests were conducted to establish the robustness of the meta-analyses’ key findings.
Separate meta-analyses of both voluntary and mandatory programs were conducted. Voluntary programs provide services (e.g., help in job search, training, and remedial education) for those who apply for them and they sometimes provide financial incentives to encourage work. Mandatory programs are targeted at recipients of government transfer payments. They also provide employment-orientated services and sometimes provide financial work incentives, but differ from voluntary programs by requiring participation in the services by potentially subjecting individuals assigned to the program to fiscal sanctions (i.e., reductions in transfer payments) if they do not cooperate.
This study uses a unique database, assembled specifically for synthesizing findings from evaluations of welfare-to-work programs. The data used in the study are from 27 random assignment evaluations of mandatory welfare-to-work programs for AFDC applicants and recipients and four random assignment evaluations of voluntary welfare-to-work programs for AFDC recipients. The evaluations in the study sample were conducted similarly. AFDC applicants and recipients were randomly assigned to either a program group that participated in the welfare-to-work program being evaluated or to a control group, which was eligible to receive any services that existed prior to the introduction of the welfare-to-work program. Relying mainly on administrative data, various measures of outcomes (such as earnings and the percentage receiving AFDC) were computed on the members of the program and control groups over time. Once this follow-up information was available, each program effect was estimated as the difference in the mean outcome for the program group and the control group, a measure that is often referred to as the “program impact.”
The database contains four measures of program impacts:
- average earnings,
- the percentage in employment,
- the average amount of AFDC received, and
- the percentage in receipt of AFDC.
Program impacts are available for up to twenty calendar quarters after random assignment, along with the levels of statistical significance for each of these impact measures. Findings from cost-benefit analyses are also included when available. In addition, the database contains the values of a number of explanatory variables. These include the characteristics of the program population (gender and ethnic mix, age distribution, family structure, and education levels, and so forth), measures of program effects on participation in various activities (job search, basic education, vocational training, and work experience), whether each evaluated program tested financial incentive and time limits, program effects on sanctioning, and socioeconomic data for each of the program sites and for each of the evaluation years (site unemployment and poverty rates, the percentage of the workforce in manufacturing employment, median household income, and the maximum AFDC payment for which a family-of-three was eligible).
Because controls often receive services similar to those received by persons assigned to the evaluated programs, but from other sources, it is important to measure the net difference between the two groups in their receipt of services – that is, the program’s impact on participation in services. These net differences indicate that a typical mandatory welfare-to-work program puts much more emphasis on increasing participation in relatively inexpensive activities, such as job search, than on increasing participation in more costly activities, such as basic education and vocational training. Nonetheless, it cost the government almost $2,000 (in year 2000 dollars) more per member of the program group to operate the evaluated mandatory programs than to run the programs serving controls. Voluntary programs typically put more emphasis on expensive services than mandatory programs do and, hence, are usually more costly to run.
The four impacts mentioned above were examined in four separate calendar quarters (the 3rd, 7th, 11th, and 15th after random assignment). Between 64 and 79 estimates were available for each impact measure during the two earlier quarters and between 44 and 56 estimates were available during the later two quarters. The analysis suggests that welfare-to-work programs, on average, had the intended positive impact on the four indicators, although these averages were usually small. There was considerable variation among the individual programs, however, suggesting that some performed much better than others.
Much of the analysis was devoted to determining why some programs were more successful than others. Among the more important conclusions concerning mandatory welfare-to-work programs that were reached are the following (findings for voluntary programs are described later):
-
Three program features appear to be positively related to the effectiveness of mandatory welfare-to-work interventions: increased participation in job search, the use of time limits, and the use of sanctions. The latter relationship is only important in the first couple of years after entry into a program.
-
Financial incentives decrease impacts on whether AFDC is received and on the amount of AFDC that is received, but do not improve impacts on labor market outcomes.
-
The evidence is somewhat mixed over whether increases in participation in basic education, vocational education, and work experience increase program effectiveness. However in general, the findings do not support putting additional resources into these activities.
-
It is unclear as to whether the effectiveness of mandatory welfare-to-work programs has improved over time.
-
Mandatory welfare-to-work programs appear to do better in strong labor markets than in weak ones.
-
Because generous state AFDC programs (represented in the analysis by the size of the maximum AFDC payment for which a family of three is eligible) reduce incentives to leave the welfare rolls, it was anticipated that the relationship between AFDC generosity and program impacts on the receipt of AFDC would be negative. However, the evidence on this relationship is mixed, varying with the statistical procedures used to test the hypothesis.
-
A typical mandatory welfare-to-work program appears to have a positive effect on all four program impact measures for five to seven years after random assignment, although the impacts begin to decline after two or three years.
-
In general, mandatory welfare-to-work programs appear to be more effective in serving relatively more disadvantaged caseloads than more advantaged caseloads—for example, AFDC recipients (rather than applicants), program group members without recent employment experience (rather than program group members with recent employment experience) and long-term (rather than short-term) participants in AFDC. However, similar evidence of a differential impact for program group members with and without high school diploma is lacking. Moreover, there is some evidence of a positive relationship between program impacts and the average age of persons in the caseload.
The findings listed above are based on weighted regressions in which the dependent variables are estimates of program impacts and the weights, as prescribed by meta-analysis, are the inverse of the standard errors of the impact estimates. The report provides evidence suggesting that these regressions can be used to assess whether it is likely that a particular mandatory welfare-to-work program is performing better or worse than an average mandatory program. Although this information is not as reliable as that provided by a full evaluation, it can serve as a partial substitute for such an evaluation.
The net operating costs of a typical mandatory welfare-to-work program (i.e., the cost to the government of providing program services, excluding income transfers, such as AFDC payments), were around $1,800 per program group member (in year 2000 dollars). These costs are larger for programs that substantially increase participation in basic education and vocational education. Increases in participation in work experience do not seem to increase costs, perhaps because work experience participants are often assigned to agencies other than those operating the welfare-to-work programs, and whatever costs are involved may not get incorporated into estimates of net operating costs. Increases in participation in job search appear to result in very small increases in cost, while financial incentives appear fairly costly to administer. Increases in sanction rates engender considerable costs, presumably because of government expenditures required for administering and enforcing sanctions.
Benefit-cost analyses were conducted as part of the evaluation of many, but far from all, of the evaluations of the welfare-to-work programs in the database. Findings from these analyses indicate that the net benefits (i.e., benefits less costs) of a typical mandatory welfare-to-work program are surprisingly small. According to the findings, which attempt to capture total net benefits over several years (often five), society receives net benefits of around $500 per program group member from a typical mandatory welfare-to-work program; savings to the government are around $400 per program group member, on average; and those assigned to a typical program are barely affected. It is likely that the net benefits from a typical mandatory welfare-to-work program are actually even smaller than these estimates imply because, as shown in the report, benefit-cost analyses are less likely to be conducted for those programs with especially small impacts on earnings.
Unsurprisingly, the net benefits received by participants are higher for program group members in programs that offer financial incentives than for those assigned to programs that do not. However, the increases in participant net benefits are fully or nearly offset by reductions in government net benefits. Thus, the social cost of financial incentives appears to be small or negligible. Because the findings also suggest that they do little to increase employment or earnings, financial incentives that are provided through welfare-to-work programs are perhaps best viewed as simply transferring income from the government to low-wage welfare recipients who find jobs.
We used meta-analysis to systematically identify interventions with very high positive or negative impacts. In doing so, we defined “very high” positive or negative impacts as those at least one standard deviation above or below, respectively, the mean for all interventions. We did this for all four impact measures for quarters 3, 7, 11, and 15. We required interventions to have at least two quarterly outliers before classifying them as having had exceptionally high positive or negative outliers because we were interested in identifying programs that repeatedly, perhaps persistently, under- or over-performed. This was to avoid highlighting isolated instances of above average positive or negative performance that may not be sustained over time. We conducted two types of analysis: (1) we compared each impact estimate in a given calendar quarter to the weighted mean of all impact estimates available for that quarter (Type A); (2) we used the weighted regressions and their explanatory variables to control for factors that influence the effectiveness of welfare-to-work interventions (Type B). We find, as expected, that interventions are more likely to produce Type A than Type B outliers. Additional analysis that may well be worth undertaking, but that would require information beyond that in our database, would be required to determine why interventions that produced Type B outliers were over- or under-performing.
As indicated by the fact that they produced multiple positive outliers for different impact measures, the GAIN evaluation interventions in California’s Riverside and Butte Counties and the NEWWS evaluation’s program in Portland Oregon are among those that repeatedly over-performed. Other interventions with repeated positive outlier impacts include the California Work Pays Demonstration program and the New York State Child Assistance Program. Both record more Type B than Type A outliers; that is, their status as over-performing interventions becomes more apparent after the factors that influence program impacts have been taken into account. Repeatedly under-performing programs include Minnesota's Family Investment Program (MFIP), Vermont's Welfare Restructuring Project (WRP), and GAIN's Tulare County intervention. MFIP especially under performed with respect to reducing AFDC payments or the number of AFDC recipients, perhaps because it offered financial incentives.
Seven of the random assignment evaluations of mandatory welfare-to-work programs in our database provided sufficient information on child outcomes for analysis as part of this report. Even though the data severely limited in depth analysis of program impacts for children, several findings are noteworthy. Overall, program impacts on children were small, but there is evidence that the considerable variation across programs in their estimated impacts on children is not entirely due to sampling error, but is partially attributable to systematic differences among the interventions. However, with the exception of impacts on the emotional and behavioral problems of children, we were unable to determine what these systematic differences might be.
There is no support in our data for the proposition that increasing the net incomes of welfare families improves child outcomes. However, the welfare-to-work programs that were examined did not produce large changes in the incomes of those assigned to them. When various program characteristics are controlled, impacts on emotional and behavioral problems are less positive for school age children than for young children. Additionally, three program features appear to positively affect the impact of welfare-to-work interventions on children’s behavioral and emotional outcomes: sanctions, participation in basic education, and participation in unpaid work. Two features of welfare-to-work programs exert negative influences on impacts on childhood behavioral or emotional impacts: financial incentives and time limits. Finally, increasing expenditures on welfare-to-work programs has a positive effect on their impacts on childhood behavioral and emotional problems.
There have been four evaluations of ten interventions that paid a stipend to AFDC recipients who volunteered for temporary jobs that were intended to help them learn work skills. These voluntary welfare-to-work interventions increased the earnings and decreased the AFDC payments of participants by modest, but non-trivial, amounts. However, there is fairly substantial variation in these impacts. A partial explanation for this variation appears to be that more expensive voluntary welfare-to-work programs produce larger impacts on earnings and AFDC payment amounts than less expensive programs. We found no evidence of a similar relationship between program costs and program impacts for mandatory welfare-to-work programs.
The research that is presented in this report suggests a number of conclusions about welfare-to-work programs. One that particularly stands out is that although there are a few welfare-to-work programs that may be worth emulating, by themselves, most such programs are unlikely to reduce the size of welfare rolls by very much or to improve the lives of most program group members and their children very substantially. Thus, they must be coupled with other policies, such as earnings subsidies.
| Table of Contents | Previous | Next |