9. PROGRAM IMPACTS OVER TIME

In this section, we look at how long program impacts last. Most of the studies listed in Table 1 measured program impacts for several calendar quarters after random assignment took place. This allowed us to investigate how impacts might change over time across a range of program evaluations. A variable was constructed for this purpose that equals one for program impact estimates that pertain to the first post-random assignment quarter, two for estimates that pertain to the second post-random assignment quarter, and so forth. We refer to this variable as the “quarters since random assignment” variable.

The findings described earlier suggest that job search and sanctions make key contributions to the effectiveness of welfare-to-work programs. It seems possible that such programs may give the experimental group a competitive advantage in the labor market at first, but that this advantage will diminish over time as the control group catches up. Thus, we expected that while the relation between the quarters since random assignment variable and the program impact measures might initially be positive, it would eventually turn negative.18

To investigate this possibility, we computed regressions with two time variables: the quarters since random assignment variable in its original form (number of quarters) and the square of the quarters since random assignment. We expected the coefficients on the first variable to be positive and the coefficient on the second to be negative.

The coefficients are reported in Table 13. All the quarters of impact estimates that were provided by each mandatory evaluation in our database were included in computing the regressions. The regressions were estimated both with and without the inclusion of other explanatory variables. The purpose of including the other explanatory variables in the regressions is to determine whether the relation between impacts and quarters since random assignment changes after controlling for other factors. The variables we use to control for other factors are identical to those used in the regressions reported in Tables 4 through 7.

Both time variables are statistically significant at the 1 percent level in all eight regressions, regardless of whether control variables are included in the regression or not. However, unsurprisingly, the inclusion of the control variables considerably increases the explanatory power of the regression model, as indicated by the increased value of the adjusted R-squared. This increase is particularly apparent with respect to the amount of AFDC payment, where the adjusted R-squared increases from .048 to .484 as a result of inclusion of the control variables. Nevertheless, this has no effect on the statistical significance of the two time variables, although the size of the coefficients on both variables increases a bit.

The linear variable for the number of quarters is positively associated with all four impacts, indicating that, indeed, program impacts increase over time. However, the coefficient on the squared time variable is negative. In other words, while it is correct to say that program impacts increase with time, this is only true initially as, after some point, they begin to shrink.

As indicated by the penultimate row in Table 13, the coefficients in the regressions including control variables imply that the impacts of a typical welfare-to-work program begins to diminish between the eighth and fourteenth quarter, depending on the impact indicator. After about two years (8.4 quarters), the average program’s employment impact peaks and begins to decline, followed about a year later by the amount of AFDC (10.7 quarters) and earnings (11.9 quarters) impacts. The decline in the impact on the percentage receiving AFDC is the last to set in, doing so after approximately three and a half years (13.6 quarters).

The regressions that exclude the control variables imply that the impact peaks occur somewhat later, although the difference between the two types of regression is less than one calendar quarter with respect to earnings and the percentage employed. However, this rises to three and four quarters, respectively, for the AFDC payment and percentage receipt impacts.

The regressions also imply that the impacts of a typical welfare-to-work program disappear five to seven years after random assignment.19 Again, except for the percentage employed, the regressions that exclude the control variables predict that this point occurs somewhat later. Some caution is required in interpreting these findings because they are based on extrapolation somewhat beyond the sample range of five years. In addition, as previously discussed, welfare-to-work interventions with large positive impacts tend to be evaluated for a longer period of time than less successful interventions. This could cause our estimates of how long program impacts last to be exaggerated. An investigation of this possibility in earlier research, which was limited to earnings impacts (Greenberg et al., 2004), suggests that the exaggeration is only around half of one-year.

In sum, the regressions show that a typical welfare-to-work program has a positive effect on all four program performance measures for five to seven years after random assignment, although the impacts begin to decline after two or three years.