2. BACKGROUND

2.1 PREVIOUS RELATED RESEARCH

In a 1997 summary of training and employment program evaluations, Friedlander, et al. suggested that welfare-to-work programs typically result in modest, but sometimes in substantial, positive effects on the employment and earnings of one-parent families headed by females. They also noted that the programs are often found to reduce the receipt of welfare and welfare payment levels of these families, but these effects are usually modest and tend to decrease over time. The evidence is less clear for two-parent families.

Friedlander, et al. (1997) make a useful distinction between voluntary and mandatory programs. Voluntary programs provide services (e.g., help in job search, training, and remedial education) for those who apply for them and they sometimes provide financial incentives to work for individuals who apply for them. Mandatory programs are targeted at recipients of government transfer payments. They also provide employment-orientated services and sometimes provide financial work incentives, but they formally require participation in the services by potentially subjecting individuals assigned to the program to fiscal sanctions (i.e., reductions in transfer payments) if they do not cooperate.

The Friedlander, et al. (1997) review also indicates that there is considerable variation in the effectiveness of different training and employment programs. As previously indicated, a key objective of the research described in this report is to examine the extent to which this variation is attributable to the characteristics of the programs themselves, the characteristics of participants in the programs, and the economic environment in which the programs are conducted.

In previous, closely related work, we conducted a meta-analysis of 24 random-assignment evaluations of mandatory welfare-to-work programs that operated in over 50 sites between 1982 and 1996 and were targeted at one-parent families on AFDC. Published papers on this research (Ashworth et al., 2004, and Greenberg et al., 2005 forthcoming) highlighted the effects of the receipt of program services and participant and site characteristics on program impacts on earnings and the receipt of AFDC. For example, the findings suggest that higher levels of program sanctioning rates result in larger impacts on earnings and leaving the welfare rolls. They also imply that if a program can increase the number of participants who engage in job search, it will, as a result, have larger effects on their earnings and on their ability to leave the welfare rolls. On the other hand, increases in participation in basic or remedial education, vocational training, or work experience and the provision of financial work incentives do not appear to result in larger increases in earnings. Additional findings from the meta-analysis indicate that program impacts on earnings are larger for white than for non-white participants and for older participants than for younger participants. They also appear to be larger when unemployment rates are relatively low. In another published paper, Greenberg et al. (2004) examined how program impacts on earnings change over time and found that the effects of a typical welfare-to-work program appear to increase after random assignment for two or three years and then disappear after five or six years.

Of all the welfare-to-work programs that have been evaluated by random assignment in the U.S., the two that operated in Riverside, California and Portland, Oregon produced the most dramatic impacts. As a result, these programs have become very well known. Greenberg et al. (forthcoming) examined the factors that contributed to Riverside and Portland’s exceptional success. More specifically, they first measured the difference in impacts for these two programs and the impact for an average site. They then determined whether the estimated regression could explain a substantial proportion of these differences (it typically could). The findings suggest that only part of this success can be attributed to the design of the programs that operated in Riverside and Portland. The social and ethnic mix of the programs' participants and the economic conditions prevailing at Riverside and Portland at the time of evaluation were also important.

There have been several recent studies in addition to our own that have also attempted to unravel the factors that cause program effectiveness to vary across training, employment, and welfare-to-work programs. For example, the National Evaluation of Welfare-to-Work Strategies, which provides a comparative analysis of eleven welfare-to-work programs over a five-year period, was particularly concerned with comparing the effectiveness of employment-focused and education-focused programs (Hamilton et al., 2001). However, although this study compared impacts across different programs, it did not control for differential exogenous factors, such as variations in the mix of program participants or in economic conditions. It found that welfare-to-work programs that emphasize labor market attachment were more effective in reducing welfare spending, increasing earnings, and facilitating the return to employment of participants than programs that emphasized human capital development.

In a path-breaking re-analysis of random assignment evaluations of California's Greater Avenues for Independence (GAIN) program, Florida's Project Independence and the National Evaluation of Welfare-to-Work Strategies, Bloom, Hill and Riccio (2003) pooled the original survey data for over 69,000 members of program and control groups who were located in 59 different welfare offices. The resulting hierarchical linear analysis, which utilized unpublished qualitative data on the program delivery processes, found that the way in which welfare-to-work programs were delivered and the emphasis that was placed on getting the “work-first” message across strongly affected the second-year impacts of the programs included in the analysis. The results also indicated that welfare-to-work programs were less effective in environments with higher unemployment rates.

Greenberg, Michalopoulos, and Robins (2003) have recently completed a meta-analysis of the impacts of voluntary training programs on earnings. In doing this, they systematically took account of differences in program design, the characteristics of program participants, and labor market conditions. Their analysis indicates that program effects are greatest for adult women participants (many of whom received welfare), modest for adult men, and negligible for youth. They also found race to be an important determinant of program impacts on earnings and that, at least for adults, more expensive programs were not more effective than otherwise similar less expensive programs.

2.2 THE DATABASE

All the studies listed above provide useful information. However, the database on which we based our previous research offered several distinct advantages over the data sources utilized in the other studies. In particular, the database:

1. Provided the widest coverage of mandatory welfare-to-work program evaluations by including all those available by the end of 2000 that used a random assignment design to assess programs that provided job search, training services, or financial incentives to encourage work to AFDC recipients.

2. Recorded all the quarterly and annual program impact estimates that were published in various reports from these evaluations by the end of 2000.

3. Included variables that pertain to the receipt of program services, the characteristics of program participant, and the characteristics of the sites in which the programs operated.

The database that we previously used for the studies described above has been greatly updated and expanded for use in the research described in this report. First, data for new impact estimates for previously evaluated mandatory welfare-to-work programs have been added. Second, data from several more recently initiated random assignment evaluations of mandatory welfare-to-work programs have also been incorporated into the database. Third, the database that we previously used contained information on only mandatory welfare-to-work programs. Random assignment evaluations of voluntary programs that were targeted specifically at AFDC recipients have now been added. Fourth, indicators of the evaluated programs’ impact on the well-being of the children of program participants, as well as outcome information on the children of controls, have been added to the database for those programs for which they are available. Fifth, the original database recorded the net effect of mandatory welfare-to-work programs on the proportion of program participants who were sanctioned (that is, it indicates the experimental-control difference in percent sanctioned) and indicated whether each evaluated program provided financial work incentives. The information in the database about sanctions and financial incentives has been greatly expanded. For example, it now records the duration of the sanctions and whether the sanctions required the complete or only the partial withdrawal of AFDC benefits. For those evaluated programs that offered financial work incentives to welfare recipients, the database now indicates the dollar amount of the financial incentive that would be received by an individual with two children who has been in a full-time minimum wage job for two months. A similar calculation is available for an individual who has worked full-time for 13 months. All financial information that is recorded in the database – and, by implication, available for use in this report – has been inflated to year 2000 US dollars.

Further information about the database and how it can be accessed is provided in Appendix A.

The random assignment welfare-to-work evaluations that are included in the database are listed in Table 1.3 Some of the listed evaluations were conducted at more than one site. Moreover, some of these sites experimented with more than one type of welfare-to-work program (or intervention). In other words, an evaluation may have reported the impacts of several interventions, undertaken at several sites. This is reflected in our database, which records the impacts for each site and intervention separately. For example, the National Evaluation of Welfare-to-Work Strategies (NEWWS) pertains to 11 interventions at seven sites. The majority of the evaluations of mandatory welfare-to-work programs in our database were conducted in the 1990s. For 77 of the 116 interventions recorded in the database, random assignment commenced between 1990 and 1998. A further 38 interventions were evaluated in the 1980s, including a few for which the random assignment extended into the following decade. One evaluation, MDRC’s study of the Supported Work program, was completed in the 1970s. Only one evaluation, Indiana’s Welfare Reform program, was conducted after the introduction of TANF, which replaced AFDC4 and took effect from 1 July 1997.5 TANF ended federal entitlement to assistance and created block grants to fund State expenditures on benefits, administration, and services to needy families. It also introduced time limits on the receipt of welfare assistance and changed work requirements for benefit recipients. 6

All the evaluated welfare-to-work programs listed in Table 1 were intended to encourage employment and also, in most cases, to reduce dependency on welfare. The evaluations are divided between those that assessed mandatory programs and those that examined voluntary programs. There are a few evaluations listed in Table 1 that assessed programs that provided financial work incentives but not services. We classify those “pure” financial work incentive programs for which individuals had to apply as “voluntary” and those for which welfare recipients were made eligible, regardless of whether they applied, as “mandatory.” Individuals assigned to “pure” financial work incentive programs that are classified as “mandatory” were obviously not subject to sanctions for refusal to participate in services, as no services were offered. However, the manner in which eligibility to participate in the program was determined was similar to that of mandatory programs that did provide services.

The older experiments listed in Table1 (for example, SWIM in San Diego and the Employment Initiatives program in Baltimore) tended to be of demonstration programs, which were run for the express purpose of seeing how well they functioned. The study sites volunteered for this purpose and may not have been very representative. For example, funding levels may have been high and the staff exceptionally motivated However, there is little evidence on whether the findings were distorted by such factors. Most of the more recent random assignment evaluations (for example, the Virginia Independence Program and the Indiana Welfare Reform program) resulted because a state desired to implement a new program and as a condition of obtaining Federal waivers was required to evaluate the intervention using random assignment. These evaluations often took place when state AFDC programs were undergoing many changes. Consequently, staff was probably less motivated and had less time to focus on the innovations being evaluated than was the case in the earlier evaluations.

2.3 COMPONENTS OF THE CURRENT RESEARCH

The research described in this report attempts to exploit the greater diversity of programs and greater number of impact estimates that are available in the updated and expanded database. For example, we examine voluntary welfare-to-work programs, as well as those that are mandatory. In our prior work, we were only able to study the latter. In all, 27 evaluations of mandatory welfare-to-work programs and four evaluations of voluntary welfare-to-work programs, which together cover nearly 100 interventions, are used in the analysis.

A key objective of the research described in this report, like the original analyses that were described earlier, is to explore whether and how program impacts are affected by various program, participant, or site characteristics. To determine how robust our conclusions are, we subject much of our statistical analysis to sensitivity tests. These tests will be described when they are presented.

Our previous analysis was limited to program impacts among one-parent families. In this report, we examine program impacts on two-parent families, as well as those on one-parent families. Only two measures of program impacts – earnings and the percentage of program group members receiving AFDC – were examined in our earlier research. In addition to utilizing the updated and expanded database to re-examine program impacts on earnings and AFDC receipt, the research presented in this report includes an analysis of additional measures of program effects including impacts on employment status and the amount of AFDC benefits received. The estimates of net program benefits, which were obtained from the cost-benefit analyses that were part of many (but not all) of the welfare-to-work evaluations listed in Table 1, are also examined. In addition, an analysis is conducted of the various measures of the effects of welfare-to-work programs on child well-being, which, as previously mentioned, have been added to the database.

The database contains up to 20 calendar quarters of impact estimates for the evaluated programs (although fewer quarters of estimates were available from many evaluations). Using these estimates, we examine how the impacts of welfare-to-work programs vary over time. The additional data allow for a longer-term follow-up of impacts. As previously mentioned, we also explored this issue in our earlier research. However, the updated database contains substantial numbers of additional quarters of measured impacts, especially from the later calendar quarters, that were not previously available to us, permitting somewhat more precise estimates of how program effects change over time.

Current understanding of what constitutes a “successful” welfare-to-work program or a “failed” one is mainly based on simple comparisons of the impacts of selected welfare-to-work programs that rarely attempt to standardize for differences in participant or site characteristics that exist across programs. In this report, we attempt to identify especially successful and unsuccessful programs after controlling for the effects of measurable program features and target group and site characteristics. The objective of this analysis is to identify welfare-to-work programs that are highly successful or unsuccessful ceteris paribus                         . In other words, the goal is to distinguish programs that still record positive or negative impacts even after accounting for factors that might be expected to increase or decrease impacts, such as a program design, and advantageous or disadvantageous labor market conditions and target group characteristics. Once identified, one can speculate as to what accounts for the remaining over- or underperformance of these programs.

It has been recognized that welfare reform and welfare-to-work programs might affect different population subgroups differentially (Walters, 2001). Thus, a few studies have begun to explore the effects of, for example, different amounts of education or differences in ethnic origin on program impacts (Michalopoulos, et al., 2000 and Harknett, 2001). In this report, we add to this research by presenting comparisons of the separate impact estimates for sub-groups contained in the database.