12. ANALYSIS OF SUBGROUPS

A minority of evaluations of random assignment welfare-to-work programs report impacts for subgroups of those assigned and not assigned to programs. In this section, we summarize and analyze the impacts for the four pairs of subgroups most commonly recorded in welfare-to-work evaluations:

(1) participants in AFDC who, at the time of random assignment, were either applicants for AFDC or already recipients of AFDC;

(2) AFDC participants who had been in employment sometime during the year prior to random assignment or who had not been employed during that time;

(3) AFDC participants who, at the time of random assignment, had obtained a high school diploma or General Education Degree or who had not; and

(4) long-term AFDC participants who, at the time of random assignment, had received AFDC for two or more years or short-term participants who had received AFDC for less than two year or were new AFDC applicants.

While some evaluations report separate impacts for both AFDC applicants and AFDC recipients, a small number of evaluated welfare-to-work programs specifically targeted only one of these subgroups. We include both types of evaluations in our analysis. Our analysis of subgroups is limited to evaluations of the effects of welfare-to-work programs on one-parent families, as impacts are very rarely reported for subgroups of two-parent households.

Table 21 lists the evaluations that report specific subgroup impacts, including programs solely targeted at applicants or recipients of AFDC. The characteristics that distinguish the four subgroup pairs that appear in the table are all known or believed to influence the chances of AFDC participants finding employment and leaving welfare. For instance, chances of leaving the welfare rolls typically decline with the duration of AFDC participation. This may result from eroding occupational and social skills that are rated highly by employers or simply because employers doubt that long-term welfare recipients have much of a commitment to the work force. Hence, long-term recipients of AFDC would be expected to be more difficult to place into work than short-term recipients. Similarly, AFDC applicants are more likely to leave welfare after just a short time than existing recipients of AFDC. AFDC participants with recent employment experience might find it easier to gain employment and to leave welfare than AFDC participants who have not worked for a year or more. Employers not only value recent work experience as a demonstration of acquired or retained occupational skills, but also as an indication of the AFDC recipients’ willingness and ability to hold down a job. Skills and educational qualifications, of course, also matter. In principle, employment chances are augmented by higher levels of education. AFDC participants with a high school diploma may spend less time on welfare and more time in employment than participants without a high school diploma. Their more advanced level of education should make them more attractive to employers. We anticipate that program impacts will be smaller for those subgroups that are able to find employment without the aid of welfare-to-work interventions (i.e., short-term AFDC recipients, AFDC applicants, and AFDC participants with recent work experience or with a high school diploma) than for subgroups requiring assistance from interventions (i.e., long-term AFDC recipients, AFDC recipients, and AFDC recipients without recent work experience or a high school diploma).

12.1 CAVEATS

The evaluations listed in Table 21 report separate impact estimates for subgroups. As indicated by the table, because the reporting of subgroup impact estimates is selective and irregular, the number of estimates that are available for the meta-analysis varies among subgroups and among types of impact measures. As before, we report impacts for the 3^rd, 7^th, 11^th and 15^th quarter after random assignment.

Evaluation reports typically contain less comprehensive data for subgroups than for the total target population participating in a welfare-to-work experiment. They frequently cover fewer calendar quarters or the data are limited to annual impacts.23 Moreover, the evaluation reports do not provide the same detail, if they provide any at all, on the characteristics of subgroups or their rates of participation in program services. In the absence of specific data about the subgroups, this information is taken from data provided for the overall evaluation sample.

As explained in Section 3, sample size is used in imputing standard errors when they are not otherwise available in order to weight program impacts. Because we do not have data on the size of individual subgroup samples in our database, we use the size of the overall evaluation sample for this purpose. Values for individual subgroups are, of course, smaller than those for the evaluation sample as a whole. It is important to bear this mind in interpreting the results of the meta-analysis.

12.2 THREE TYPES OF ANALYSES

Three types of analysis of subgroup impacts are reported. The first two analyses use only subgroup impacts; this analysis is referred to as the “pure-subgroup analysis.” Dummy variables, coded 0 and 1, are added to the data to indicate the subgroup within a subgroup pair to which each impact estimate refers. The third analysis includes both subgroup and main group impacts and is referred to as the “mixed-group analysis.”

Pure-subgroup analyses. In the first pure-subgroup analysis, mean impacts are calculated for each subgroup. This is done for all four quarters and for all four impact measures, using the impact estimates available in each quarter.

The second pure-subgroup analysis computes the difference between the mean impacts for each pair of subgroups, using regression analysis to adjust these differences for a range of explanatory control variables. In other words, the regression-adjusted differences can be viewed as an attempt to control for the fact that the two subgroups in each pair may differ from one another in numerous ways such as in age and in the services they received. This exercise is limited to impacts from the 7^th quarter after random assignment, the quarter that usually contains the largest number of observations.

The regression analysis follows the same principles outlined earlier for the main group analyses. In fact, all the subgroup regressions take the independent variables of the main group analyses as their starting point, but also include the subgroup-dummy variable. Because fewer impact estimates are available for the subgroup analyses than for the main-group analyses, however, multicollinearity between explanatory variables is more serious. This reduces the number of explanatory variables that can be use in the meta-analysis of subgroups relative to those employed in the main-group analyses. Thus, control variables that are highly correlated with one another are dropped following two basic principles. First, variables describing program characteristics are the last to be removed, because they describe program conditions that administrators are able to influence and are, therefore, of particular practical interest. Second, some variables that are highly correlated with others are removed, while retaining as many explanatory variables as possible. For instance, if one variable is highly correlated with another two, but the latter two are not highly correlated with one another, the former variable is dropped and the latter two are retained.

Mixed-group analysis. The mixed-group analysis integrates subgroup and all other one-parent impacts into one data file. As before, impact estimates that pertain to pure subgroups (e.g., AFDC recipients or AFDC applicants) are coded 0 and 1, respectively. In addition, when separate impact estimates for subgroups are not reported for an evaluated intervention, impact estimates that pertain to individuals from both subgroup categories (e.g., recipients and applicants) are coded within this range as appropriate. For instance, an impact for a program with 30 percent of the program group being AFDC applicants and the remainder recipients is coded as 0.3 in constructing the subgroup variable used for comparing recipients with applicants. Evaluations that do not report the proportion of subgroup members within their total sample are excluded from the mixed-group analysis. As before, the regression seeks to reduce multicollinearity among independent variables, while retaining as many variables that describe program characteristics as possible.

12.3 FINDINGS FOR THE COMPARISON OF UNADJUSTED MEANS FOR THE PURE-SUBGROUPS

Table 22 presents weighted impact means for the four pairs of pure-subgroups, four impact measures, and four quarters.

Applicants and recipients. The simple means comparisons of the pure-subgroup of AFDC applicants and recipients that appear in Table 22 indicate that impacts for recipients are larger than those for applicants with respect to the percentage receiving AFDC and the percentage employed in each of the four reported quarters. The results are less consistent with respect to the amount of AFDC payment and the amount of earnings. The latter is higher for recipients than applicants in the early quarters, but this is reversed in the later quarters, when fewer evaluations are available for analysis. The relative sizes of the amount of AFDC payment impacts alternate between the two subgroups from quarter to quarter. Moreover, the impact means for the percentage receiving AFDC are negative for both subgroups in the 11^th quarter. This is unusual, but the number of available impact estimates is small. This highlights the sensitivity of the analyses to changes in the number and the composition of the impact estimates that are available.

Employed and not employed in year prior to random assignment. Relatively few evaluations estimate separate impacts for program group members who were, and were not, employed in the year prior to random assignment, thereby reducing the robustness of comparisons between these two subgroups. The number of observations that are available for each subgroup reaches double-digit figures in only two instances. In both cases, the mean impact is larger for welfare-to-work program group members without employment in the previous year than for those with previous employment.

Program group members with and without a high school diploma. The analysis of the mean impacts of welfare-to-work program group members with and without a high school diploma also suffers from a scarcity of observations. This said, comparisons in the four instances in which there are at least 20 observations all suggest a greater impact of welfare-to-work programs for program group members with high school degrees.

Long-term and short-term participants in AFDC. Comparisons of impact means for long-term and short-term participants in AFDC are again severely curtailed by the small number of observation available for all but the 7^th quarter. Both the AFDC payment and the earnings data for the 7^th quarter suggest a greater positive impact for long-term AFDC participants than for short-term participants. Although earnings impacts are greater for short-term AFDC participants in the 3^rd and 11^th quarters, many fewer observations are available for analysis in these two quarters.

Summary of findings from the comparison of unadjusted means. The comparison of these impact means is hampered by the small number of observations available for analysis. In many instances, findings are inconsistent from one quarter to the next or from one impact measure to the next. It is, therefore, difficult to identify generalizable patterns across all four subgroups and impacts. Nevertheless, with only one exception, whenever ten or more observations are available for each of the subgroup pairs, impacts are more positive for the more disadvantaged subgroup – that is, for AFDC recipients (rather than applicants), for program group members without recent employment experience (rather than program group members with recent employment experience) and for long-term (rather than short-term) participants in AFDC. The sole exception occurs in the comparison of welfare-to-work program group members with and without a high school diploma. Program impacts average higher for the former than the latter.

12.4 FINDINGS FOR QUARTER 7 DIFFERENCES IN MEANS BETWEEN SUBGROUPS

The pattern of relatively larger impacts for the more disadvantaged subgroup in each pair becomes more apparent when the analysis focuses on the 7^th quarter. Table 23 shows the differences in weighted mean impacts for the pure-subgroups without regression adjustment and the differences in weighted mean impacts for the pure-subgroups and for the mixed groups after regression adjustment. The full regression results are reported in Tables A.1 through Table A.4 in the Appendix. Positive differences indicate greater mean impacts among the first listed subgroup in each pair (i.e., the relatively more advantaged subgroup) and negative differences imply the converse.

As explained above, the mixed-group analyses add data from evaluations for which the proportion of welfare-to-work program group members belonging to subgroups is known, but impact estimates for separate subgroups were not estimated, to data for the pure-subgroups. This addition substantially increases the number of observations available for analysis, particularly for the comparisons between the subgroups of program group members who were and were not employed during the year prior to random assignment and program group members who had and did not have a high school diploma.

Applicants and recipients of AFDC. With the exception of the amount of AFDC payments, the results reported in Table 23 indicate that a consistently greater program impact is achieved for AFDC recipients (indicated by the negative sign) than for AFDC applicants. The differences are statistically significant for all three types of analyses with respect to impacts on employment. The mixed-group analysis also suggests that welfare-to-work programs perform significantly better for recipients than for applicants with respect to their impacts on AFDC payments and earnings. However, this is not the case with respect to participation in AFDC, which is only statistically significant in the pure-subgroup analyses.

Employed or not employed in year prior to random assignment. The comparisons reveal consistently greater impacts for program group members who have not been recently employed than for those who have. However, statistical significance only occurs for earnings and the percentage employed in the pure-subgroup analyses and for the amount of AFDC payments in the mixed group analysis.

Program group members with and without high school diplomas. Findings are more mixed in the comparison between the subgroup of welfare-to-work program group members with a high school diploma and the subgroup of program group members without a high school diploma. Whereas welfare-to-work programs produce greater impacts among program group members without high school diplomas in terms of their participation in AFDC, the reverse is true for the other program impacts. However, with only one exception (earnings in the regression-adjusted pure-subgroup analysis), none of the differences in means are statistically significant.

Long-term and short-term participants in AFDC. Long-term participants in AFDC achieve significantly greater impacts in quarter 7 than short-term participants in all three types of analysis for three of the four impacts—namely, AFDC payments, earnings, and the percentage employed. Moreover, the level of statistical significance of these differences is at the 1-percent level. This consistently high level of significance sets the meta-analysis results for the subgroup of long-term and short-term participants apart from those of any of the other subgroups. Only with respect to participation in AFDC are the comparisons of means statistically non-significant for the pure subgroups, although statistical significance is achieved for the mixed-groups.

Summary of findings from the comparison of quarter 7 impact differences. The comparison of mean impacts between subgroups seven quarters after random assignment most clearly reveals the strong and positively greater effects of welfare-to-work programs on long-term participants in AFDC than on short-term recipients. At the other extreme, the subgroup analyses found little evidence of a statistically significant difference in the performance of welfare-to-work programs for program group members with and without high school diploma.

The evidence is more diverse for the remaining two subgroups. First, there is evidence that welfare-to-work programs benefit program group members without recent employment experience more than those with recent employment experience. However, this finding is statistically significant in less than half of the comparisons and, it is never significant in all three types of analyses, suggesting once again that the regression findings are highly sensitive to changes in the availability of impact estimates.

Second, the analyses suggest welfare-to-work program achieve greater impacts for AFDC recipients than applicants. But, again, although this finding is replicated across the four impact measures and the three types of analyses, statistical significance is achieved across all three types of analysis in just one instance, for the impact on the percentage employed.

When statistical significance of differences can be established, however, it is evident that welfare-to-work programs are more likely to benefit individuals from more disadvantaged subgroups than individuals from less disadvantaged subgroups. More specifically, AFDC recipients, program group members not employed in the previous year, and long-term participants in AFDC are more likely to gain from being assigned to welfare-to-work programs than their more advantaged counterparts. Similar evidence of a differential impact for program group members with and without high school diploma is lacking.

Comparing the results of the pure-group and the mixed-group analyses. Methodologically, the pure subgroup analyses are the more appropriate ones, as they use impact estimates separately for each subgroups. Were it not for their small numbers, pure subgroups would provide the more appropriate analysis framework. However, because the mixed-group analysis is based on a larger number of cases, including those cases for which impacts were measured separately by subgroup, its results should be more robust. Moreover, it is possible to include a larger number of explanatory control variables in comparisons of subgroup differences.

Despite their differences, the pure-group and the mixed-group analyses produced remarkably similar results. With just two exceptions (among 48 mean estimates), the estimated impacts were of the same direction. Although there was less agreement between the two analyses concerning the statistical significance of impacts, all three of the differences between impacts for short-term and long-term AFDC recipients are highly statistically significant.