6. BASIC REGRESSION FINDINGS

The regressions that examine the hypotheses discussed above are reported in Tables 4 through 7. The regressions for each of the four impact measure are contained in a separate table. Each table reports regressions for the 3^rd, 7^th, 11^th, and 15^th calendar quarters. Although it is useful and frequently interesting to compare the regressions across the impact measures and calendar quarters, and we do so below, it is important to keep in mind that sample composition varies among the impact measures and quarters and this may account for some of the differences in findings.

The regression for each impact and quarter was computed twice, once with only the nine intervention characteristic measures and once with the four socio-economic contextual measures also included. The F-test for the first of these computations indicates whether the coefficients on the nine intervention characteristic variables are jointly statistically significant, while the F-test for the second set of computations indicates whether the coefficients on the four socio-economic contextual characteristic variables are jointly significant when these variables are added to the original nine. The F-tests indicate that, with one exception,12 the first set of coefficients are always jointly highly significant at conventional levels, but that the second set of coefficients frequently are not jointly significant, even when some of the individual coefficients are significant. This suggests that differences in design among programs contribute importantly to why some interventions are more effective or less effective than others, but that once program characteristics are taken into account, differences among target populations and site characteristics often do not play an important role. However, there are important exceptions to the latter conclusion, as suggested by the fact that the F-test for the coefficients on the contextual variables is sometimes statistically significant and, even when it is not, some of the individual coefficients are significant.

Because two regressions are reported for each of four different calendar quarters and four different impact measures, the findings in Tables 4-7 are complex to interpret. In presenting the results, we treat all four impacts as positive values, including impacts on AFDC receipt and payments, although, unlike the earnings and the employment impacts, they record reductions rather than increases in participation and financial receipts. The reporting all four impacts as positive values results in their meta-regression coefficients having the same sign if an explanatory variable has similar effects on different impact measures. This should aid in the interpretation of the regression results.

In the discussion of the regression results, we first consider each explanatory variable separately. This is followed by a brief overall summary of key findings.

6.1 SANCTIONS

Increasing the sanction rate appears to have a positive effect on program impacts, especially those on whether AFDC is received and on the amount of AFDC received. Although some of the coefficients on sanctioning are negative, these negative coefficients are never statistically significant, while the positive coefficients are often significant. The coefficients on sanctions are usually largest in the 7^th calendar quarter, when they are always positive and highly statistically significant at conventional levels. After that the importance of sanctioning appears to fade, suggesting that sanctions help get welfare recipients off the welfare rolls and into jobs initially, but do not necessarily keep them there over the longer term.

6.2 JOB SEARCH

Tables 4-7 indicate that increasing the use of job search has a positive effect on the impact of welfare-to-work programs, regardless of how the impact is measured. The coefficients on the job search variable are almost always positive and are usually statistically significant. Moreover, the contribution of job search does not seem to diminish with time from random assignment.

6.3 BASIC EDUCATION

Tables 4 and 5 imply that increasing participation in basic education does not improve labor market outcomes. The coefficients on the basic education variable are sometimes positive and sometimes negative in Tables 4 and 5, but they are small relative to the coefficients on job search and statistically significant in only one regression. Although Table 6 suggests that increasing participation in basic education may decrease the amount of AFDC payments the welfare-to-work program group receives, Table 7 implies that this does not reduce the welfare rolls, except possibly in the 15^th quarter. It is difficult to reconcile the divergent implications of Tables 6 and 7; but, in general, there appears to be little evidence in support of making basic education a major component of welfare-to-work programs.

6.4 VOCATIONAL EDUCATION

The coefficients on vocational education are statistically insignificant at conventional levels more often than not. Moreover, they are typically negative in sign, even when significant. The major exceptions occur in the regressions on impacts on AFDC payments during the 7^th and 11^th quarters. There, the coefficients on vocational training are both positive and highly significant, but only in the regressions that contain the socio-economic contextual variables. These coefficients are not very robust, however. For example, if the variable for maximum AFDC payments is dropped from the regression specification, they become non-significant and shrink greatly in size, with the 11^th quarter coefficient becoming negative. We conclude that increasing participation in vocational training does not exert a positive influence on the impacts of welfare-to-work programs and may even have a negative effect.

6.5 WORK EXPERIENCE

The coefficients on the work experience variable are seldom statistically significant and, when they are significant, they are usually only marginally so. The major exceptions occur for the 7^th quarter impacts in the regressions on the receipt of AFDC and the amount of AFDC received, when the coefficients are negative. Overall, the evidence does not seem to indicate that program impacts improve very much, if at all, with an increase in participation in work experience.

6.6 FINANCIAL INCENTIVES

Tables 6 and 7 indicate that including financial incentives in a welfare-to-work program exerts a negative influence on its impacts on the receipt of AFDC and on the amount of AFDC received. These negative coefficients are highly statistically significant through the 11^th quarter and are often large relative to the mean impacts (see Table 2). However, they appear to diminish over time as individuals leave the welfare rolls and no longer qualify for financial incentive payments. This finding is unsurprising as financial incentives typically operate by increasing the earnings disregarded in computing AFDC benefits. What is surprising is that the financial incentive coefficients are usually negative in sign and occasionally statistically significant in the labor market regressions reported in Tables 4 and 5. As noted earlier, this may be the result of earnings disregards reducing the work effort of some employed AFDC recipients who decide to work fewer hours while maintaining their overall income. The objective of financial incentives is, of course, to encourage employment and thereby increase earnings, but they do not appear to do so.

6.7 TIME LIMITS

The coefficients on the dummy variable for time limits are almost always positive in sign and they are often statistically significant. Moreover, they appear to grow through the 11^th quarter (about three years after random assignment) as AFDC recipients approach the time limits or even reach them. Thus, either through threat or direct implementation, time limits seem to increase the impacts of mandatory welfare-to-work interventions. However, as indicated by Table 4, they do not seem to have much effect on earnings, except possibly in the 11^th quarter.

6.8 NUMBER OF YEARS SINCE 1982

The purpose of this variable is to determine whether policy makers have learned from past experiences so that newer programs are more successful than older ones. Unfortunately, a clear conclusion cannot be drawn. The coefficients on the years since 1982 variable are always positive in the regressions on labor market impacts (see Tables 4 and 5), but they are rarely statistically significant. They are more often statistically significant in the regressions on the AFDC impacts (see Tables 6 and 7), but are about as likely to be negative as positive.13

6.9 ONE-PARENT FAMILIES VERSUS TWO-PARENT FAMILIES

All evaluations of welfare-to-work programs assess their effects on one-parent families as this group constitutes over 90 percent of all families who received AFDC. Some also include two-parent families in the evaluation, and when they do, one- and two-parent families are usually evaluated separately. Thus, a dummy variable was included in the regression specification to distinguish between impacts estimated for the two types of families, with impacts for two-parent families assigned a value of one and impacts for one-parent families assigned a value of zero. The findings are difficult to interpret because the coefficients on this variable are positive about as often as they are negative, and a subset of both the positive and negative coefficients are statistically significant.

6.10 AVERAGE AGE OF THE TARGET GROUP

Young welfare recipients, many of whom are teenagers or in their early twenties, may face greater disadvantages in the labor market than older recipients both because of their age and because their children tend to be younger. Thus, caseload age may be positively associated with impacts of welfare-to-work interventions. It could instead be negatively related, however, if the programs help younger recipients overcome barriers that would otherwise exist, but older recipients can find jobs on their own. In addition, older recipients tend to have more children.

As it turns out, the coefficients on the average age of the target group are positive much more often than they are negative. Moreover, none of the negative coefficients are statistically significant, while a few of the positive coefficients are significant. Thus, there is weak evidence that program impacts are larger when the average age of the target group is greater.

6.11 PERCENT OF THE TARGET GROUP EMPLOYED THE YEAR PRIOR TO RANDOM ASSIGNMENT

AFDC recipients with recent employment experience, most of whom have been on the welfare rolls for relatively short periods of time, are much more likely to find employment readily and leave the welfare rolls than their counterparts who have little or no work experience and are long-term recipients. Moreover, as pointed out earlier, such individuals also tend to have more education and are more likely to be white, which may also make it easier for them to obtain a job.

However, if these more job-ready recipients can find employment on their own, without the aid of welfare-to-work programs, but less job-ready recipients need the help of such programs, then the relation between our measure of recent employment and program impacts may be negative. As it turns out, except for the 3^rd calendar quarter, the coefficients on the recent employment variable are always negative. However, most of these negative coefficients are not statistically significant. Thus, there is only weak evidence that welfare-to-work programs do more to aid recipients without recent employment experience than recipients who can more readily obtain employment on their own.

6.12 ANNUAL PERCENTAGE CHANGE IN LOCAL MANUFACTURING EMPLOYMENT

This variable reflects the state of the labor markets at the intervention sites at the time the evaluations were conducted by indicating whether jobs were being added or lost. The coefficients on the variable are usually positive. Furthermore, they are often statistically significant, and they are only significant when they are also positive. This suggests that welfare-to-work programs work best when there are job openings.14 The fact that the coefficients in the earnings and employment regressions are only positive and statistically significant in the 3^rd and 7^th calendar quarters suggests that the availability of jobs is most important during the first couple of years after AFDC recipients are assigned to welfare-to-work programs.

6.13 POVERTY RATE

The poverty rate is only included in the earnings and employment regressions and was intended to capture both the availability of jobs at the places and times the evaluated interventions operated and the quality of those jobs that were available. As noted earlier, a high poverty rate is also indicative of target populations that are less likely to have a high school degree and more likely to be non-white. With only one exception, the coefficients on this variable are negative, suggesting that earnings and employment are lower when there are fewer jobs available,15 those that are available are unattractive, and program target populations are more disadvantaged. However, the coefficients are statistically significant only in the 7^th quarter.

6.14 MAXIMUM AFDC PAYMENTS

We suggested earlier that the generosity of AFDC (as represented by the maximum payment for which a single mother with two children and no earnings is eligible) could be either positively or negatively related to program impacts on AFDC payment amounts. Table 6 indicates that this relationship is positive and, except for the 15^th quarter, highly statistically significant. However, we expected AFDC generosity to be negatively associated with program impacts on the size of the welfare rolls, both because program break-even levels are higher under more generous programs and because welfare recipients may be more hesitant to leave the rolls when benefits are more generous. Table 7 suggests that this hypothesis finds weak support in the 3^rd and 15^th quarters when the coefficients on maximum AFDC payments are negative but insignificant, but not in the 7^th and 11^th quarters when the coefficients are both positive and statistically significant.

6.15 SUMMARY OF KEY FINDINGS

Conclusions based on the regressions results discussed above are summarized below. We examine the robustness of these conclusions in Section 7; until then, they should be considered tentative.

• Three program features appear to be positively related to the effectiveness of welfare-to-work interventions: increased participation in job search, the use of time limits, and the use of sanctions. The latter relationship is only important in the first couple of years after entry into a program.

• Financial incentives decrease impacts on whether AFDC is received and on the amount of AFDC that is received, but do not improve impacts on labor market outcomes.

• The evidence is somewhat mixed over whether increases in participation in basic education, vocational education, and work experience increase program effectiveness. However in general, the findings do not support putting additional resources into these activities.

• It is unclear as to whether the effectiveness of welfare-to-work programs has improved over time.

• Welfare-to-work programs appear to do better in strong labor markets than in weak ones, especially in the first year or two after individuals enter these programs.

• Impacts on the size of AFDC payments and (possibly) the receipt of AFDC are larger in locations where AFDC systems are more generous.

• Based on the regression results described above, it is not clear whether the welfare-to-work interventions tend to be more effective or less effective in serving relatively more disadvantaged caseloads than more advantaged caseloads. The typically negative relation between program impacts and recent job experience suggests that they are more effective in serving a relatively disadvantaged caseload, but the generally positive relation between impacts and caseload age and the negative association between impacts and the poverty rate implies the opposite. However, none of these relationships are very often statistically significant. In Section 14, we directly compare subgroups of relatively disadvantaged AFDC recipients with subgroups that are relatively less disadvantaged and find some evidence that welfare-to-work program impacts tend to be larger for the former than for the latter, although this evidence is still not entirely conclusive.