7. SENSITIVITY ANALYSES

It is important to determine whether the list of key findings that appears above is robust. In this section, we examine whether these findings change when the assumptions that underlie the regressions reported in the previous section are modified. Specifically, we tested the sensitivity of the findings to each of the following modifications:

1. Running unweighted regressions. This is especially important for regressions on impacts that are measured as continuous variables (i.e., earnings and the amount of AFDC payments) because, as explained earlier, some strong assumptions were needed to derive the weights used in computing the weighted regressions for these impacts.

2. Excluding the observations with the two highest and two lowest impact estimates from those used to run the regressions. This sensitivity test indicates whether the findings were influenced by extreme values.

3. Dropping observations with missing value for any of the explanatory variables from the regressions. In the regressions reported in the previous section, we instead used the sample mean of a variable when it was missing for an observation.16

4. When the same control group was used in estimating the impacts of two different interventions at the same site, each impact estimate was given only half the weight they were given in computing the previously reported regressions. The purpose of this exercise was to see if the findings are sensitive to possible dependency between the two impact estimates.

To minimize the number of regressions involved, we limited these sensitivity tests to the 7^th quarter after random assignment and to regressions that included the full list of explanatory variables. The 7^th quarter allows us to maximize sample size, while also examining a period well after most individuals who were assigned to the evaluated programs are no longer receiving program services.

The results from the sensitivity tests for each impact estimate appear in Tables 8 through 11. For purposes of comparison, the first column repeats the 7^th quarter regressions that appear in Tables 4-7. The remaining four columns contain regressions based on each of the four modifications described above.

With a few exceptions, which are discussed in the following paragraph, the key findings from the earlier regressions appear quite robust, although the magnitudes of some of the individual coefficients on the different variables sometimes change substantially. The adjusted R-squared for the unweighted regressions are much smaller than those for the weighted regressions and the coefficients are considerably less likely to be statistically significant. However, the coefficients are generally of the same sign and even of the same order of magnitude as the original estimates, especially when the latter are statistically significant. The coefficients in the fourth column in which observations containing missing information are dropped vary the most from the original estimates. These differences probably occur both because the observations used for the two computations differ considerably from one another and because the smaller sample size on which the estimates in the fourth column are based causes them to be less precise. Thus, there are fewer significant coefficients in the fourth column than in the first. Nonetheless, most of our earlier conclusions are still supported.

Turning now to our individual conclusions, Table 10 provides some indication that increases in participation in basic education may increase program impacts on the amount of AFDC that is paid out, which is contrary to the third bullet point that appears at the end of the previous section. However, the sensitivity tests do not suggest that the three remaining impacts increase as basic education increases. Furthermore, the sensitivity analyses provide additional support for our earlier contention that the significant positive relation between increases in participation in vocational education and impacts on AFDC payments found in the first column is not robust. The fourth columns of Table 9 and 11 exhibit negative (but insignificant) relations between program impacts and the annual percentage change in manufacturing employment, thereby weakening our conclusion that welfare-to-work programs are most effective when labor market conditions are good. However, the relationship continues to be positive in the fourth columns of Tables 8 and 10, supporting the conclusion. Moreover, the other sensitivity tests are also consistent with the conclusion. Perhaps, the strongest case for changing one of the original conclusions in light of the sensitivity analysis concerns the relations between AFDC generosity and program impacts on the receipt of AFDC and the size of AFDC payments. Our original estimates indicate that these relationships are positive, while three of the sensitivity tests suggest that they are negative. For previously discussed reasons, a negative relationship between AFDC generosity and program impacts on the receipt of AFDC seems more plausible than a negative one.