4. FINDINGS OF THE DESCRIPTIVE ANALYSIS

As noted earlier, the analyses in this report focus on four indicators of program impacts. These are: increases in the earnings received by members of the program group, increases in the percentage of those in the program group in employment, decreases in the amount of AFDC payments that those in the program group received, and decreases in the percentage of those in the program group in receipt of AFDC payments. Table 2 presents basic descriptive statistics for these four indicators, measured at the 3^rd, 7^th, 11^th and 15^th quarter after random assignment for mandatory welfare-to-work interventions. Both weighted and unweighted estimates are shown. However, unless we specifically indicate otherwise, in discussing Table 2, we focus on the weighted estimates.

Some caution is required in comparing statistics for different quarters because the number of evaluations and, therefore, the composition of the evaluations upon which the statistics are based, changes.9 This is illustrated by the increasing importance over time from random assignment of the weighted mean impacts relative to their median counterparts. With the exception of the impact measuring the percentage employed, by the 15^th quarter, the means are higher than their corresponding medians. The results for later quarters, in particular the 15^th quarter, are, thus, based on a greater proportion of relatively high-impact programs than appears to have been the case during earlier quarters.

The most striking result in Table 2 is the modest sizes of the impacts, whether measured as means or medians. For example, the weighted impacts on quarterly earnings are all around $100 or less than $500 annually (in year 2000 dollars) and the weighted impacts on AFDC payments tend to be even smaller. However, the standard deviations of the mean impacts are quite large, suggesting that some of the evaluated interventions were much more effective than others. Thus, it is useful to explore the reasons why success differs among welfare-to-work programs. Much of the rest of this report is devoted to such an exploration.

4.1 EARNINGS

The largest number of observations is available for impacts on the earnings of the program group. Individuals taking part in traditional welfare programs (the control group) earn, on average, $675 (weighted) in quarter 3. This rises to nearly $1360 (weighted) in quarter 15 (in year 2000 dollars). Similarly, mean impacts—that is, the difference between the control groups' and the program groups' earnings—average $74 in quarter 3 and $115 in quarter 15. Program group members, therefore, earn, on average, around ten percent more per quarter than control group members. The proportion of extra earnings, however, declines somewhat in later quarters as mean impacts rise less from one quarter to the next than the mean control group earnings do. In fact, the unweighted mean impact declines between the 11^th and the 15^th quarter, while mean earnings for the control group continue to increase. Large standard deviations highlight the variability of both weighted and unweighted impacts among the evaluated interventions.

4.2 PERCENTAGE EMPLOYED

Around a third of control group members are employed in each quarter, although this fraction increases somewhat over time. Welfare-to-work programs appear to increase employment among those assigned to them by about three percentage points. However, while this is an 11.3 percent increase over the control groups’ mean employment rate in the third quarter after random assignment, it is only a 7.5 percent increase in the 15^th quarter. Again, high standard deviations indicate considerable variation among programs. The weighted and unweighted mean employment figures for controls are fairly similar, as are the weighted and unweighted impact estimates.

4.3 AMOUNT OF AFDC PAYMENT

The weighted amount of AFDC payments received between the 3^rd and the 15^th quarter by members of the control groups of welfare-to-work programs declines, on average, from $1,033 in quarter 3 to under $460 in quarter 15 as individuals increasingly leave the AFDC rolls without the intervention of a welfare-to-work program. It will be recalled that a reduction in AFDC payment is recorded as a positive impact; that is, positive values for impact indicate a reduction in the receipt of AFDC. Thus, in the 3^rd quarter, as control group members receive $1,033, on average, individuals assigned to the program group receive approximately $38 less (i.e., about $995). In the 15^th quarter, the control mean is only $458, or less than half of the mean recorded for the 3^rd quarter, while the weighted mean impact reaches $75, or about twice the amount recorded for a typical site after three quarters. AFDC payments to program group members in quarter 15, therefore, average around $383.

As a proportion of control group AFDC payments, mean impacts increase from less than four percent ($37.8/$1,032.8) in the 3^rd quarter to over 16 percent ($75.1/$458.0) in the 15^th quarter. However, the mean and median impacts change differentially between quarters. For example, the median impact value declines from $89 in the 11^th quarter to $41 in the 15^th quarter. Thus, the increase in the mean impact between these quarters may be affected by the greater presence of a number of very high-impact programs among the declining total of observations available in the final quarter.

4.4 PERCENTAGE PARTICIPATING IN AFDC

A similar trend can be observed for the percentage of individuals still receiving AFDC payments after random assignment. For the control group, this proportion decreases from nearly 81 percent in the 3^rd quarter to 41 percent in the 15^th quarter; that is, it is approximately halved. The additional reduction in the receipt of AFDC due to welfare-to-work programs averages 1.5 percentage points in the 3^rd quarter and 4.4 percentage points in the 15^th quarter. Hence, as the AFDC caseload among those randomly assigned in welfare-to-work experiments declines, both the relative and the absolute program impact increases. The decline in the median impact from the 11^th to the 15^th quarter again suggests that the greater mean impact of welfare-to-work programs in the later quarters after random assignment, at least in part reflects the increasing importance of high-impact programs in the remaining sample of evaluated interventions.

Overall, the descriptive statistics from Table 2 suggest that, on average, welfare-to-work programs had the intended positive impact on all four indicators and that these positive impacts were maintained in all four quarters we have examined. However, the programs’ absolute and relative impacts appear to be sustained for longer with respect to AFDC payments and AFDC receipt than earnings and employment. In all instances, standard deviations matched or, indeed, exceeded the mean impact values, thus suggesting considerable variation between individual programs. For later quarters, as the number of evaluations declines, their composition also changes, with the inclusion of a greater proportion of high-impact interventions.

4.5 DESCRIPTION OF THE TESTED INTERVENTIONS, TARGET POPULATIONS, AND SITES

As mentioned earlier, much of the analysis in this report relies on estimating regressions for examining the relation between the four impact measures described above and measures of program design, the characteristics of the target population, and social and economic conditions at the sites of the evaluated programs. A list of variables that are in the database and, thus, could potentially be used as explanatory variables in these regressions appear in Table 3, along with their means and standard deviations.

These means and standard deviations pertain to the subset of 79 observations for which estimates of impacts on earnings were available in the 7^th calendar quarter. As indicated by Table 2, the sample size and hence the sample composition varies by impact measure and by quarter. In addition, the values of the site socio-economic condition measures are specific to the year during which the impacts were measured. Hence, the means and standard deviations of the variables listed in Table 3 also vary to some extent by impact measure and quarter. However, the values that appear in the table are representative of the values for the other impact measures and quarters.

As indicated by Table 3, some of the variables listed in Table 3 are not available for every observation. As discussed below, we attempted to minimize this problem by selecting explanatory variables for the regressions that have relatively few missing values. When an explanatory variable was nevertheless missing in running the regressions, we used its mean value. Later we report the results of sensitivity tests in which we compare our findings with those from regressions in which observations with missing values are dropped.

A key indicator of program design is how it affects the receipt of the services it provides. Because controls often receive services similar to those received by persons assigned to the evaluated programs, but from outside the program, it is important to measure the net difference between the two groups in their receipt of services – that is, the program’s impact on participation in services. The measures of impacts on participation that are reported in Table 3 typically indicate whether participation has occurred by around a year after random assignment, although some evaluations record participation impacts later than that. The data indicate that a typical mandatory welfare-to-work intervention in our sample put much more emphasis on increasing participation in relatively inexpensive activities, such as job search, than on increasing participation in more costly activities, such as basic education and vocational training. Nonetheless, it costs the government almost $2,000 (in year 2000 dollars) more to operate the evaluated programs than to run the programs serving controls.

Arguably, the singularly greatest contribution of the evaluated mandatory welfare-to-work interventions was to increase participation in job search activities by an average of 21 percentage points. The programs’ net contributions to other activities, including those aimed at promoting human resource development, were considerably smaller, increasing participation in basic education by an average of just seven percentage points and in vocational training and work experience by less than three percentage points. Indeed, some individual programs with a work-first emphasis actually had a negative impact on participation in these activities. The mandatory nature of the programs covered by Table 3 is exemplified by the six percentage point average net increase in sanctions that resulted from them.

About 15% of the 79 interventions that comprise the sample for Table 3 tested time limits and nearly one-third tested financial incentives. Nearly half of the latter interventions (“pure” financial incentive programs) were designed to test financial incentives alone. The mean financial incentive amount of $82.75 that appears in Table 3 is computed by averaging over all 79 interventions, those that provided financial incentives and those that did not. Thus, the interventions that did provide financial incentives paid about $250, on average, to a single mother with two children during her 13^th month in a full-time job.

The mid-point of the random assignment of the earliest evaluations listed in Table 1 occurred in 1983. The mid-point of random assignment of the typical mandatory welfare-to-work intervention in our sample took place about eight years later.

In a typical evaluated intervention, the average age of family heads in the target population was 31, with about one-quarter being under 25. The number of children in these families was about two. Around half the families had at least one child less than six years of age. On average, 36 percent of the target population was black, 41 percent was white, and 17 percent was Hispanic. Just over half of the family heads in the target population for a typical evaluation had obtained a high school degree or diploma, and this varied little across the evaluated intervention. Finally, slightly less than half the family heads had been employed during the year before random assignment, with some variation between sites.

Unemployment rates, which serve as indicators of the availability of jobs, averaged 6.4 percent across the sample of interventions in Table 3 but, as indicated by a standard deviation of 2.3, varied considerably. An alternative measure of the availability of jobs is the annual percentage change in manufacturing employment, which was just over one percent, and, as the standard deviation of 4.5 implies, was often negative. Poverty rates, which are indicative of a range of factors reflecting both individual characteristics (e.g. lone parenthood, lower educational attainment) and area characteristics (lower job availability in deprived areas, less commercial investment, greater risk of segregation), averaged 14.6 percent. Annual median household income, which averaged $40,237 (in year 2000 dollars) across the evaluation sites, provides an alternate measure of local living standards. Manufacturing employment accounted for 13 percent of total employment at the sites, on average.

Two measures of the characteristics of the AFDC programs at the evaluation sites appear in Table 3. The first indicates the generosity of AFDC payments across the program sites. Averaged across the interventions, single mothers with two children and no other income were eligible for a monthly payment of $603 (in year 2000 dollars). The standard deviation of just under $200 of the maximum AFDC payment confirms the considerable state-to-state variation in generosity. The second measure attempts to capture the “toughness” of sanctions at the sites as exemplified by either specifying a minimum sanction length at the first sanction (the alternative is to sanction until compliance) or terminating full family AFDC benefits during the first sanction (the alternative is a partial reduction in benefits). Only six percent of the sites had at least one of these provisions.