Table of Contents | Previous | Next |
5. HYPOTHESES TESTED IN THE REGRESSION ANALYSES
A number of hypotheses are possible about the relation between the intervention impact estimates and the variables described in the previous section. We consider some of these hypotheses in this section. As will be seen, in a number of instances, there are plausible contradictory hypotheses, one of which implies a negative relationship between a given explanatory variable and intervention impacts and the other of which implies a positive relationship.
5.1 INTERVENTION CHARACTERISTICS
It is, of course, difficult, if not impossible, to capture the essence of a welfare-to-work intervention with a few quantitative measures. However, the best available measures are probably the net participation and sanction rate estimates that appear in Table 3. In general, we anticipated that an increase in any activity, holding program effects on other activities constant, would be positively related to program impacts, if the activity were at all effective. However, some training activities, such as basic education, work experience, and (especially) vocational training, require a number of weeks to complete. If these activities are not very effective in increasing earnings, but those participating in them believe that they will be, they may hold some individuals on the welfare rolls and out of the labor market longer than would otherwise be the case. Thus, net participation rates for basic education and vocational training could be negatively related to the impacts of welfare-to-work interventions.
Sanctioning would be expected to have an indirect effect on welfare-to-work intervention impacts by increasing participation in program services. These indirect effects should, in principle, be captured by the measures of net participation in program activities. However, sanctions may also have positive direct effects on program impacts. In those instances in which an AFDC grant is entirely eliminated families are terminated from the welfare rolls and, as a consequence, some of the heads of these families will presumably seek employment. Even when the grant is only partially reduced, the reduction may cause some individuals to decide to leave the AFDC rolls and seek employment.
We expected that financial incentives would be positively related to program impacts on employment and earnings. However, because these incentives usually raise the earnings level at which families can continue to receive AFDC, we also anticipated that they would be negatively related to intervention impacts on whether AFDC was received. Moreover, it is also conceivable that greater earnings disregards reduce the work effort of AFDC recipients who would work even in their absence, as these persons are able to maintain their standard of living while working fewer hours (see Blank, Card, and Robins, 2000). Whenever this is the case, total income would remain roughly the same and earnings disregards would reduce earnings.
Although few members of the treatment groups who were assigned to the interventions that tested time limits actually reached these limits during the earlier calendar quarters after random assignment, the very existence of time limits may create pressures to leave the welfare rolls and replace transfer payments with earnings. For instance, the lengths of different welfare spells are summed under most time limit provisions to determine whether a family has reached the limit. Thus, to the extent possible, families may wish to conserve months on welfare for those times when they need financial aid the most. Consequently, we anticipated that time limits would be positively related to intervention impacts in all the calendar quarters we examine, but especially the later quarters as members of welfare-to-work program groups begin to approach them.
The purpose of the number of years since 1982 variable that appears in Table 3 is to test whether welfare-to-work programs have improved over time because more has been learned about running them effectively. If so, the relation between this variable and the program impact estimates would be positive. As previously discussed, however, older random assignment evaluations tended to be of demonstration programs, while more recent evaluations were typically of new welfare reform programs that states desired to implement. This difference could have caused the estimated relation to be negative.
5.2 CHARACTERISTICS OF THE TARGET POPULATION
Two opposing hypotheses can be formulated about the relationship between program impacts and the extent to which the caseload in a welfare-to-work program faces disadvantages or barriers in obtaining employment and, hence, in increasing earnings and leaving the AFDC rolls. On the one hand, program impacts may be smaller the more disadvantaged the participant caseload, because persons in such a caseload will have greater difficulty in obtaining employment. On the other hand, members of more advantaged caseloads may be better able to obtain employment on their own, without the aid of a welfare-to-work program, while such an intervention may be needed to help those with barriers to employment overcome them. If so, impacts will be larger the more disadvantaged the caseload is. To illustrate, impacts could be larger for program group members with recent work experience because it is easier for these persons than for program group members with a long-term welfare dependency to find jobs. The contrary possibility is that such individuals may be better able than persons with a long-term welfare dependency to succeed in the labor market on their own without help from a program. If so, the impacts of the interventions will be smaller as the percentage of the target population that worked during the year prior to random assignment becomes larger.
5.3 SOCIO-ECONOMIC CONDITIONS AT THE SITES
The impacts of welfare-to-work programs on earnings and welfare receipt are likely to be influenced by the socio-economic conditions that prevailed at the times and places the programs operated. The evaluations that are included in our database measured program impacts under a wide variety of socio-economic conditions. Although each individual study estimated training effects over only a few years, taken together, they cover a time span of nearly two decades. Moreover, the evaluated programs operated in varied communities. For example, the welfare-to-work programs included in this study include programs from high-benefit states, such as California and Connecticut, alongside states with relatively low AFDC benefit levels, such as Arkansas, Virginia, and Florida. However, although this means that the study captures a range of benefit regimes and associated work incentive conditions, the included programs are not necessarily nationally representative.
In theory, measures of the availability of jobs at the evaluation sites (e.g., the unemployment rate or the annual percentage change in manufacturing employment) could be either positively or negatively related to the impacts of welfare-to-work interventions. On the one hand, if jobs are scarce, then those who are assigned to a welfare-to-work program may enjoy a competitive advantage over similar persons who enter the program when jobs are difficult to find. If so, the relationship between the unemployment rate and program impacts would be positive and that between the change in manufacturing employment and program impacts negative. On the other hand, a welfare-to-work program may be most helpful when jobs are plentiful. If so, the relationship between the unemployment rate and program impacts would be negative and that between the change in manufacturing employment and impacts positive. Finally, there could be a quadratic relationship between the two measures of job availability and program impacts. This would occur if when there are few job openings, there is little that a welfare-to-work program can accomplish; but when jobs are in abundance, welfare recipients can readily obtain employment without the aid of a program. Under these circumstances, program impacts would be greatest when job availability is between these two extremes.
Similar reasoning suggests that the relationship between the poverty rate or median household income and the impacts of welfare-to-work initiatives could be either positive or negative. On the one hand, a welfare-to-work program could be especially helpful to persons who reside in areas with limited economic opportunities, providing them an advantage over similar persons who do not receive services from such a program, resulting in a positive relation between impacts and poverty rates and a negative relation between impacts and median household income. On the other hand, neither persons assigned to such a program nor similar persons who are not assigned may have decent employment opportunities in highly disadvantaged areas. Thus, a welfare-to-work program may have relatively little impact in such areas. In addition, if a job is obtained, it should pay less where poverty rates are high or median household income is low. This suggests that the poverty rate at the evaluation sites should be negatively related to program impacts on earnings and median household income should be positively related.
The generosity of welfare payments at the evaluation sites (which is represented in the analysis by the size of the maximum AFDC payment for which a family of three is eligible) is expected to reduce the impact of a welfare-to-work program on the receipt of AFDC by reducing the incentive of some welfare recipients to leave the rolls. In addition, the earnings level at which families can continue to receive AFDC increases with the generosity of welfare payments, making it more feasible to remain on the welfare rolls while working.
Welfare payment generosity could either reduce or increase the impact of a welfare-to-work intervention on the amount of AFDC payments. If individuals are more likely to remain on the welfare rolls when the system is more generous, this will, of course, also increase the amount of AFDC that is paid out. Once individuals leave the AFDC rolls, however, the reductions in transfer payments will be greater.
5.4 SELECTING EXPLANATORY VARIABLES FOR THE REGRESSION ANALYSIS
Because the number of impact estimates that are available in each quarter that we analyze is limited, especially in the 11th and 15th quarters, multicollinearity was a serious potential problem in conducting the regression analysis. Thus, it was necessary to restrict the explanatory variables to a subset of those appearing in Table 3.
We used the following strategy to do this. First, with one minor exception (discussed below), we use the same regression model throughout, rather than “tailoring” the set of explanatory variables to each impact measure. Relying in part on the hypotheses discussed above, the variables were mainly selected for conceptual reasons. However, the number of missing values was also considered. It was also necessary to drop variables that were highly correlated with other variables that were in the model in order to minimize multicollinearity.
Second, and most importantly, because policy makers have control over the design of welfare-to-work programs, but little control over most contextual factors at the program sites, and thus would presumably be more interested in how the former affects program impacts than the latter, we attempt to capture the characteristics of the intervention being evaluated as completely as possible. With this in mind, the regressions include all the intervention characteristic variables listed in Table 3 but three. We did not include the net cost of operating the evaluated welfare-to-work programs because it is missing for about a third of our observations and it tend to be collinear with the participation measures (e.g., programs that substantially increase basic and vocational education tend to be more expensive). A preliminary examination suggested, however, that the net cost of welfare-to-work programs is virtually unrelated to their impacts. To conserve the number of explanatory variables, we also did not distinguish between pure incentive programs and those that provided both financial incentives and services and we did not include variables that measured the size of the financial incentive package. A preliminary investigation suggested that the pure financial incentive programs may be less effective than mixed programs, holding other factors constant, but this result was rarely statistically significant. The preliminary analysis also included a measure of the amount financial incentive that would be received by an AFDC recipient who found a minimum wage job that she or he kept for over a year. The coefficient on this measure was small and never approached statistical significance, indicating that the generosity of the financial incentive did not matter.
Third, because we use a large number of intervention characteristics variables and were concerned about the number of available observations and multicollinearity, the regressions include a minimal number of variables representing target population and site characteristics. While being parsimonious, however, we attempted to control for socio-economic conditions as best as possible so we could isolate the true effects of program characteristics on program impacts.
Specifically, we included the following three socio-economic contextual variables in all the regressions: the average age of the target population, the percentage of the target population that were employed the year prior to random assignment, and the annual percentage change in manufacturing employment. In addition, the regressions on impacts on earnings and employment include the poverty rate and the regressions on impacts on AFDC payments and the receipt of AFDC include the maximum AFDC payment available to a family of three. The first variable is most likely to capture the state of the labor market at the evaluation sites and the second the generosity of the welfare system.
Hypotheses concerning all the included variables were discussed earlier and need not be repeated here. It is important to point out, however, that the included socio-economic variables proxy many of the variables that were left out. For example, the hypotheses that we developed concerning the annual percentage change in manufacturing employment and the unemployment rate are similar, as are those pertaining to the poverty rate and median household income. Moreover, the poverty rate also captures the racial composition and the education level of the target population to a considerable degree. For example, using the same set of observations as those on which Table 3 is based, the simple correlation between the poverty rate and the percentage of the target population that is white is -.68 and the simple correlation between the poverty rate and the percent of the target population with a high school degree or equivalent is -.59. Similarly, employed the year prior to random assignment is highly correlated with length of time on AFDC (-.66) the percentage of the target population that is white (.52), and the percentage of the target population with a high school degree or equivalent (.52). In addition, the average age of the target population is highly correlated with number of children in the families in the target population (.78) and the percentage of families having a child under six (-.66). Most of the sites that had “tough” sanctions were also testing time limits (.64).10 Thus, only one of these variables could be used in the regressions and we choose the latter.
When explanatory variables were highly correlated with one another, we usually selected the one with the fewest missing values for inclusion in the estimated regression equations. If we had selected the variables we ended up excluding in their place, some of the conclusions drawn from the regressions would have differed. For example, instead of emphasizing the effects of time limits on program impacts, we would have discussed how impacts were influenced by strong sanctions.
5.4 OMITTED AND MISSPECIFIED VARIABLES
Like virtually all non-experimental empirical work that estimates relationships, this study is potentially subject to biases resulting from omitted variables. Such biases result if an omitted factor is correlated with both the dependent variable and at least one of the included explanatory variables. Ideally, therefore, the hypotheses discussed above should be tested holding everything constant that may affect both the dependent variables and the explanatory variables. As noted in the previous section, one limitation in doing this is multicollinearity. However, controlling for all potential influences is also both impossible because information is not available about all the factors that may be germane, and impractical because there are such a wide variety of possibilities. For example, the data needed to measure staff morale, cooperation among organizational units, employer attitudes towards welfare recipients, and the quality of leadership at welfare-to-work program sites does not exist. A number of the more recent welfare-to-work experiments (e.g., Minnesota’s MFIP, Delaware’s ABC program, Iowa’s FIP, and Virginia’s VIEW) tested a wide variety of provisions including changes in rules affecting assets, rules pertaining to exemptions from participating in welfare-to-work requirements, sanctions for not complying with non-work requirements (such as cooperating with child support and ensuring children receive immunizations and attend school), family caps that did not allow family benefits to increase with the birth of an additional child, and requirements for minors with children of their own. Whether these provisions are correlated with both the program impacts and the explanatory variables that we focus on in this study is unclear. However, even if they are, it is difficult to construct measures of some of them because of their complexity (e.g., requirements for minor parents), while others are specific to only one or two of the experiments in our database (e.g., family caps).
Biases may be caused by misspecified explanatory variables, as well as omitted variables. To the extent variables are misspecified or measured with error, their coefficient estimates are generally biased towards zero.
One example of explanatory variables that may be misspecified to some degree are the measures of participation in program activities that are commonly available in reports on the evaluations of welfare-to-work programs and that are used in this study. To illustrate, welfare recipients are usually counted as participants in a particular program activity such as job search or basic education if they take part in these activities for as little as one day. Intensity of participation is not measured. Nor is the order in which services are provided. For instances, so-called “work-first” programs require participants seek jobs first and provide them with education and training only if they fail to find employment, while programs that emphasize human capital development provide education an training first and job search afterward. However, work-first programs usually have a greater measured impact on participation in job search than those that emphasize human capital development, while the latter have a greater impact on participation in basic education and vocational training. Such considerations are not captured by the measures of program participation produced by most welfare-to-work evaluations.
There are other difficulties in appropriately specifying various explanatory variables. For example, available estimates of site environmental variables often do not correspond to the exact geographic area in which program target populations live and work, although we attempted to make as close a match as possible. Although we have information about the percentage of program target populations that were employed prior to random assignment, it might be more useful to know the average number of months of prior employment but the required data are not available. More generally, we are limited to aggregate information about the characteristics of members of program and control groups. It would be better to have information at the individual level—for example, the number of weeks each individual in the evaluation sample population worked prior to random assignment.11 It is especially difficult to construct a variable, or even a set of variables, that adequately captures the use of sanctioning by welfare-to-work programs because sanctioning is so multi-dimensional, including for example how easy it is to be reinstated after being sanctioned, the extent to which families who are sanctioned are followed up, the services available to those who are sanctioned, and the number of times individuals are sanctioned. As previously discussed, we did construct measures of several dimensions of sanctioning for use in this study but because of multicollinearity we were restricted to a simple measure of program impact on the percentage of program group households that were sanctioned.
| Table of Contents | Previous | Next |