Table of Contents | Previous | Next |
15. SUMMARY OF FINDINGS AND CONCLUSIONS
As previously indicated, a key goal of this research has been to increase information about what sorts of welfare-to-work programs work best under different labor market conditions and for different types of welfare recipients. Although the methodology we used is somewhat technical, our goal was a practical one: to provide information to help policymakers to make informed decisions in designing welfare-to-work programs.
We have analyzed 27 random-assignment evaluations of mandatory welfare-to-work programs and four random-assignment evaluations of voluntary welfare-to-work programs, covering nearly 100 welfare-to-work interventions, exploring what program features increase or decrease an intervention’s effectiveness and cost-effectiveness. In the following sections, we summarize our key findings and present our conclusions.
15.1 IMPACTS OF MANDATORY WELFARE-TO-WORK INTERVENTIONS
Overall, mandatory welfare-to-work interventions had the desired impacts, although these were typically of modest size. Quarterly earnings impacts, for instance, averaged around $75 (in year 2000 dollars) in the 3rd quarter rising to $115 in the 15th quarter. Over a year, the earnings gain for program group members from a typical welfare-to-work intervention, therefore, amounted to around $500, about a 10 percent increase. However, there was considerable variation among the interventions. Similarly, the percentage of AFDC participants who were employed in a given quarter was around 10 percent (or three percentage points) higher among program group members in the mandatory reform program than among participants in traditional AFDC programs. Again, there was considerable variation among interventions impacts.
The amount of AFDC payments received by control families in the evaluations of mandatory welfare-to-work programs declined sharply between quarters, from an average of $1,033 in the 3rd quarter to $458 in the 15th quarter. The additional decreases attributable to reform programs rose from $38 to $75 over the same period. A very similar pattern was apparent for AFDC participation, which declined from 81 percent to 41 percent between the 3rd and the 15th quarter among control group members. It decreased by a further 1.5 percentage points in the 3rd quarter and 4.4 percentage points in the 15th quarter among persons assigned to the reform programs. However, the trends over time may be exaggerated by the inclusion in the analysis of a greater proportion of high-impact interventions in the later quarters.
15.2 WHAT MAKES WELFARE-TO-WORK INTERVENTIONS SUCCESSFUL?
Using meta-analysis, this study has been able to go beyond conventional comparisons of the impacts of selected mandatory welfare-to-work programs by identifying what makes a welfare-to-work program “successful” or “unsuccessful”. In particular, we have been able to examine differences among program characteristics, the characteristics of program caseload, and the characteristics of the local environment.
The analyses consistently indicated that sanctions and job search had strong, positive effects on program impacts. The imposition of time limits was also found to increase impacts, albeit not for earnings impacts. In contrast, activities intended to improve a program group member’s human capital (e.g., basic or vocational education and work experience) did not have a consistent positive effect on program impacts. In fact, in some instances, their effects may have been negative. Financial incentives reduced program impacts on AFDC participation and payment, as might be expected, but, contrary to intentions, failed to have positive effects on program impacts on earnings and employment.
The impacts of mandatory welfare-to-work programs seem to be larger when labor markets are strong than when they are weak. It was unclear whether interventions were more effective for one-parent or two-parent families. We were also unable to draw firm conclusions as to whether the performance of welfare-to-work interventions has improved over time and, hence, whether program implementers have learned from experience and improved the content and administration of programs.
A series of sensitivity tests generally confirmed the results of our meta-analyses. Although the tests changed the size of some of the regression coefficients, the signs of the coefficients mostly remained the same, especially when they were statistically significant. A further analysis suggested that the regression models are useful for assessing whether a particular mandatory welfare-to-work program is performing better or worse than an average program.
15.3 EFFECTIVENESS FOR SUB-GROUPS
The impacts of mandatory welfare-to-work programs seem to be greater for more disadvantaged groups, particularly for long-term participants in AFDC (as compared to short-term participants). Other subgroups that particularly benefited from these interventions were program group members without employment experience during the year before random assignment (as compared to those with recent employment experience) and existing recipients of AFDC (as opposed to new applicants). However, in these latter two instances, the differences between the two subgroups that were compared were not always statistically significant. Moreover, no statistical significance at all could be established for differences in the impacts on program group members with and without a high school diploma. In addition, there was some evidence that program impacts were larger for older caseloads than for younger caseloads.
15.4 PROGRAM IMPACTS OVER TIME
Regression analysis suggests that the impacts of mandatory welfare-to-work programs typically linger for between five and seven years, but begin to decline after two to three years. Employment was the first of the impacts to decline, followed by AFDC payments and earnings, while the impact on the percentage receiving AFDC was the last to decline.
15.5 ANALYSIS OF OUTLIERS
A small number of mandatory interventions stand out for their very large or small impacts at multiple times during the evaluation follow-up period. Programs with multiple, exceptionally large impacts included the already celebrated welfare-to-work interventions in Riverside County, California and Portland, Oregon. Alongside these are two lesser known programs, namely, the California Work Pays Demonstration and the New York State Child Assistance Program. The extent to which they out performed other programs became particularly apparent after the analysis controlled for factors that affect program impacts.
Vermont’s Welfare Restructuring Project and Minnesota’s Family Investment Program were repeatedly among the lowest performing interventions, possibly because they provided financial incentives that tended to reduce program impacts on AFDC participation rates and payment amounts while having little positive effect on employment rates or on earnings.
15.6 COSTS AND BENEFITS
The net costs to the government of operating the evaluated mandatory welfare-to-work interventions averaged around $1,800 (in year 2000 dollars). Job search activities were, by far, the lowest cost contributors to governments’ net operating costs, whereas basic and vocational education and the administration of sanctions and financial incentives all added substantially to net operating costs.
An analysis of benefit-cost data, which are available for 49 mandatory welfare-to-work interventions, indicated that the median net benefit to society from these interventions was around $500 per program group member (in year 2000 dollars) and the median net benefit to the government was approximately $400 per program group member. Program group members themselves seemed to gain little from being assigned to most welfare-to-work programs. As benefits and costs are measured over a number of years after random assignment, the fiscal gains from welfare-to-work programs appear small.
15.7 CHILD OUTCOMES
Even though welfare-to-work programs do not directly target child outcomes, it is likely that changes to family income and maternal employment can affect child development. To examine these relationships, seven evaluations of mandatory welfare-to-work programs that provided information on child outcomes were analyzed in this report. In general, effects on children were small and mixed. No clear widespread harm or benefits to children could be found. While several individual program evaluations have found that increasing the incomes of welfare recipients increases positive outcomes for children, we found no evidence of that in these data. However, the welfare-to-work programs that were examined did not produce large changes in the incomes of those assigned to them. Several program characteristics were found to affect program impacts on the behavioral and emotional outcomes of children, but not other impacts on children. Sanctions, participation in basic education, participation in unpaid work, and program net operating cost were found to have positive effects on program impacts on children’s behavioral and emotional outcomes, while financial incentives and time limits appear to have a negative effect. Finally, after controlling for various program characteristics, it was clear that these programs were less effective at impacting the behavioral and emotional outcomes of school age children than for young children.
Findings from this analysis are tentative because we were unable to control for important social and environmental indicators that may also affect child wellbeing. Indeed, many questions in the “black box” remain unanswered. Still and all, policy makers should pay particular attention to the negative effect financial incentives had on impacts on behavioral and emotional outcomes for children, as well as the fact that financial incentives certainly did not improve other types of impacts. On the other hand, increasing sanctions did improve program impacts on children’s behavioral and emotional wellbeing, as well as producing the desired effects on program group members.
15.8 VOLUNTARY PROGRAMS
There were four evaluations of voluntary welfare-to-work interventions that paid a stipend to AFDC recipients who volunteered for temporary jobs intended to provide them with work experience and skills. These interventions had positive, but moderate, impacts on earnings and AFDC payments, with more costly interventions having larger impacts than less expensive interventions.
15.9 CONCLUSIONS: LESSONS FOR POLICY AND ANALYSIS
Several conclusions of relevance to policy-making emerge from our analysis.
First, the meta-analysis, especially the analysis of findings from benefit-cost studies, suggests that the effects of a typical welfare-to-work program are extremely modest. Although these programs are probably worth running, and, as discussed below, can be improved, by themselves they will do little to reduce the size of welfare rolls or improve the lives of most persons assigned to them. Thus, they must be coupled with other policies, such as earnings subsidies.
Second, the meta-analysis clearly supports the focus of welfare-to-work programs on job search. Activities that emphasize human capital development appear less effective than job search activities and are much more costly to provide.
Third, welfare-to-work interventions seem to produce better results for more disadvantaged groups, but the evidence was consistent only for long-term recipients of AFDC. With respect to other sub-groups, the evidence was somewhat ambiguous. The greater benefit of program assignment accruing to long-term AFDC recipients might, in fact, be the result of greater efforts to target these recipients by some interventions. Arguably, there is a case for extending welfare-to-work programs to other disadvantaged subgroups, particularly if further analysis shows that they are indeed served less well than other similarly disadvantaged subgroups.
Fourth, we also found financial incentives, or at least the current structure of these incentives, have perverse effects on program impacts. Financial incentives tended to increase AFDC participation and payment. Importantly, they also may have had a negative effect on employment and earnings impacts, at least in the early quarters after random assignment. At best, they do not have the intended positive effects. Moreover, the use of financial incentives seems to be one of the factors that caused some specific interventions to markedly under-perform. Furthermore, financial incentives appeared to have had no effect on most impacts of welfare-to-work programs on children and a negative effect on program impacts on behavioral and emotional outcomes for children. Thus, although financial incentives do transfer income to the working poor, they appear to fail to have their intended effects on program group members and their children, and they are costly to administer. Consequently, policy makers may want to reconsider offering financial incentives or at least the way in which they are offered. Perhaps earning subsidies that appreciably increase family income will produce the desired effects on parents and children.
Fifth, the analysis suggests that welfare-to-work programs are more likely to be effective in locations that are enjoying job-growth. Thus, it may make sense to allocate additional resources to welfare-to-work programs when job growth is occurring.
Sixth, there is evidence that the impacts of mandatory welfare-to-work program linger for five to seven years after random assignment. However, this evidence is not definitive because most evaluations of welfare-to-work programs do not provide impact estimates for this many years. Better information on this important topic would be possible by extending the length of time over which impacts are estimated.
Finally, our analyses have found little evidence that the performance of welfare-to-work programs has been improving over time. This might indicate an absence of systematic policy learning. It is important that lessons from welfare-to-work programs are shared and that the results of analyses, such as the one reported here, are widely studied. In particular, we found that while the effects of most welfare-to-work programs are quite modest, a few stand out. However, our meta-analysis could determine only some of the factors contributing to these success stories. Other methods, such as more detailed surveys and focus groups, might be used to investigate these programs in more detail to attempt to unravel the sources of their success.
| Table of Contents | Previous | Next |