Table of Contents | Previous | Next |
Chapter 3
Impacts on Service Receipt and Supportive Service Payments
In 2001, the Riverside County, California, Department of Public Social Services (DPSS) created and implemented the Post-Assistance Self-Sufficiency (PASS) program as part of the Employment Retention and Advancement (ERA) project in an effort to promote job retention and advancement among working individuals who had recently left the Temporary Assistance for Needy Families (TANF) program.
This chapter uses data from the DPSS “P3” automated program tracking system to describe the impacts of PASS on sample members’ service receipt and supportive service payments. (The activities of sample members in both research groups — the PASS group and the DPSS control group — that were pursued outside the PASS program were not recorded in the P3 system.) The chapter also discusses participation findings based on the ERA 12-Month Survey of clients. In addition, the chapter presents the program’s impacts on transitional child care payments as calculated from the DPSS child care tracking system. (Box 3.1 explains how to interpret the impact tables used in the ERA evaluation.)
Impacts on Service Receipt
Program Services Recorded in the P3 System
Table 3.1 shows the percentages of sample members in the PASS group and in the control group who ever received postemployment services. The top row of the table shows that, during the 26-month follow-up period after random assignment, 47 percent of the PASS group received postemployment services, compared with only 8 percent of the control group — a statistically significant difference of 39 percentage points.
Table 3.1 also displays the types of services and activities in which individuals participated. Among PASS group members, case management and counseling was the most common program service (32 percent), followed by job search activities (15 percent) and referrals to and support in education and training programs (8 percent).1 In contrast, few control group members participated in any activity. About 5 percent received transportation assistance — the most frequently used service for this group.
Box 3.1How to Read the Tables in the ERA EvaluationMost tables in this report use a similar format, illustrated below. The two leftmost columns show a series of supportive service outcomes for the PASS group and the control group. For example, the table shows that about 14 (13.6) percent of the PASS group members and about 6 (5.8) percent of the control group members received a supportive service payment. Because individuals were assigned randomly either to the PASS program or to the control group, the effects of the program can be estimated by the difference in outcomes between the two groups. The “Difference” column in the table shows the differences between the two research groups’ rates of receiving supportive service payments — that is, the program’s impacts on supportive service payments. For example, the impact on receiving a supportive service payment can be calculated by subtracting 5.8 percent from 13.6 percent, yielding an impact of 7.8 percentage points. Differences marked with asterisks are “statistically significant,” meaning that it is quite unlikely that the differences arose by chance. The number of asterisks indicates whether an impact is statistically significant at the 1 percent, 5 percent, or 10 percent level. (The lower the level, the less likely that the impact is due to chance.) For example, as shown below, the PASS program had a statistically significant impact of 7.8 percentage points, at the 1 percent level, on receiving a supportive service payment. (Three asterisks correspond to the 1 percent level; two asterisks, the 5 percent level; and one asterisk, the 10 percent level.) The p-value shows the exact level of significance. The bottom row of the table shows the average total supportive service payments among those sample members who received a payment. Measures shown in italics are considered “nonexperimental” because they include only a subset of the full report sample. Because participants in the PASS group may have different characteristics than participants in the control group, differences in these outcomes may not be attributable to the PASS program. Statistical significance tests are not conducted for nonexperi-mental measures.
|
|||||||||||||||||||||||||||
| Outcome (%) | PASS Group | Control Group | Difference (Impact) | P-Value | |
|---|---|---|---|---|---|
| Received any services | 47.0 | 8.3 | 38.7 *** | 0.00 | |
| Service type | Case management, counseling | 32.3 | 2.4 | 29.9 *** | 0.00 |
| Job search/employment referrals | 14.9 | 0.0 | 14.9 *** | 0.00 | |
| Education and training (referral and support) | 7.9 | 0.0 | 7.9 *** | 0.00 | |
| Life skills training | 0.7 | 0.0 | 0.7 *** | 0.00 | |
| Transportation-related supporta | 4.1 | 4.7 | -0.6 | 0.48 | |
| Adult and child care supporta | 2.3 | 1.0 | 1.3 ** | 0.02 | |
| Other financial supporta | 4.1 | 0.4 | 3.8 *** | 0.00 | |
| Crisis intervention | 0.3 | 0.0 | 0.3 * | 0.06 | |
| Other | 1.7 | 2.3 | -0.6 | 0.26 | |
| Received any services, by provider | Center for Employment Training | 48.6 | 9.1 | 39.5 *** | 0.00 |
| Volunteer Center | 60.2 | 9.8 | 50.4 *** | 0.00 | |
| Valley Restart | 39.1 | 8.8 | 30.3 *** | 0.00 | |
| Riverside Community College | 44.2 | 5.3 | 38.9 *** | 0.00 | |
| DPSS Rancho Mirage | 31.6 | 13.2 | 18.3 | 0.11 | |
| Sample size (total = 2,770) | 1,627 | 1,143 | |||
| SOURCE: MDRC calculations from Riverside DPSS P3 automated program tracking system records. NOTE: See Appendix C. In rare cases, negative values among the control group were changed to zero. This occurs because the impact analysis uses ordinary least squares regression, which can result in estimates slightly below zero, even when this is impossible. aThis is assistance provided by PASS service provider staff to address these client needs. |
Rates of service receipt among the PASS sample members ranged widely according to which of the five service providers they were assigned to: DPSS Rancho Mirage (32 percent); Valley Restart (39 percent); Riverside Community College (RCC) (44 percent); the Center for Employment Training (CET) (49 percent); and the Volunteer Center (60 percent).2 The rates of service receipt at the Volunteer Center and RCC corroborate the field researchers’ on-site assessments (presented in Chapter 2), which suggest that these providers implemented relatively robust program interventions in the PASS study.
Differences in service receipt rates according to research group represent the effects (or impacts) of PASS. The Volunteer Center’s program led to the largest impact on service receipt (50 percentage points). Impacts on service receipt at CET and RCC were also large (about 40 percentage points), but the impacts were smaller at Valley Restart (30 percentage points) and Rancho Mirage (18 percentage points). (Appendix Table E.2 presents a breakout of service receipt rates and impacts analyzed by service provider.) Provider-specific analysis of the participation data reveals that nearly all of CET’s participation involved case management and counseling services, whereas the Volunteer Center and RCC (and, to a lesser degree, Valley Restart and DPSS Rancho Mirage) provided clientele with a broader range of services, including job search activities, education and training referrals, and supportive service payments. (Appendix Tables E.3 through E.7 present provider-specific participation outcomes broken out by component activities.)
Based on field research and the P3 program tracking data, PASS group sample members most often received services within the first few months following random assignment. About one-third of the PASS group received services during the quarter in which they were randomly assigned; about one-fifth received services in the next quarter; and only about one-eighth received services in the third quarter. Although service receipt rates for the PASS group fell rapidly after that point, Appendix Table E.1 shows that statistically significant differences in the service receipt rates for the two research groups were apparent in every quarter of the follow-up period.
Services Measured Through the ERA 12-Month Survey
As mentioned in Chapter 1, MDRC conducted a small-scale, one-year client survey of people who were randomly assigned into the study between October and December 2002 (Quarter 4). The survey asked a series of questions about, among other topics, respondents’ experiences related to employment and to participation in work-related activities (both as part of, and independent of, the PASS program).
In contrast to the P3 program tracking data, the ERA 12-Month Survey data indicate that respondents in both research groups participated in services at very high, and nearly equal, rates overall and across most types of services and activities. The survey indicates, for example, that — over the 12 months following random assignment — 77 percent of the PASS group and 70 percent of the control group participated in job retention and advancement activities, either as part of the PASS program or outside it. (For more survey-based participation outcomes, see Appendix Table E.8.)
There are several reasons why the survey findings might run counter to the P3 findings. The survey captured any employment-enhancing activities in which respondents participated during a 12-month follow-up period. In contrast, the P3 database measured only the postemployment services that were supplied by PASS providers or DPSS. In addition, if sample members lost their jobs and returned to TANF, they could have received services through the DPSS Phase 1 program (for unemployed TANF recipients) or Phase 2 program (for working TANF recipients) — services that would be captured in the survey but not in the P3 database. As discussed in Appendix H, the survey might not be as reliable as other data sources used in this report.3 For these reasons, the analysis of postemployment activities in this chapter gives more weight to the service receipt patterns and impacts that are derived from the P3 data than to those derived from the ERA 12-Month Survey. (See Appendix H for additional detail on the survey response analysis.)
Impacts on Child Care and Other Supportive Service Payments
PASS offered clients several types of supportive service payments — such as transportation and ancillary assistance (to pay for books, uniforms, tools, and so on) — in order to help them stay in their jobs and engage in advancement activities. In addition, all sample members in both research groups who found work and remained off TANF were eligible for transitional child care payments.
Child Care Payments
Sample members in both the PASS group and the control group were eligible for Stage 2 child care payments for up to 24 months after they left the TANF rolls.4 However, the Riverside County Office of Education (RCOE) — the agency that administered the Stage 2 child care program — instituted a waiting list for Stage 2 child care payments during the sample buildup and follow-up period (from 2002 to 2004) because of the high demand among people who were eligible for these payments. To ensure that this waiting list would not act as a barrier to program participation and employment for sample members in the PASS study, RCOE agreed to give priority to them — regardless of their research group assignment — for Stage 2 child care slots, which DPSS agreed to pay for out of its Stage 1 child care budget.
As shown in Table 3.2, substantial proportions of individuals in both the PASS group (41 percent) and the control group (38 percent) received child care payments within 26 months of random assignment. (Almost all these payments were made to sample members while they were registered in the PASS program.) Members of both research groups received an average of two months of child care payments, for an average total payout of about $1,800 during the follow-up period. (This average includes zero dollars for sample members who did not receive this type of assistance.) Thus, over a 26-month follow-up period, there were no impacts on the receipt of child care payments, on the average length of time that payments were received, or on the total amount of child care payments. Furthermore, none of the five PASS service providers individually produced impacts on the receipt of child care payments or on the total amount of payments.
Other Supportive Service Payments
PASS produced a modest impact on the receipt of supportive service payments for purposes other than child care. As shown in Table 3.2, nearly 14 percent of PASS group members received such payments at some point in the 26 months following random assignment, compared with 6 percent of control group members, yielding a statistically significant difference of 8 percentage points. The average total amount of assistance was low for both groups, however: $18 for PASS group members and $9 for control group members, yielding an impact of only $9. (These averages include zero dollars for sample members who did not receive this type of assistance.)
Appendix Table E.2 shows that two of the five PASS service providers — RCC and the Volunteer Center — produced statistically significant impacts on the receipt of supportive service payments for purposes other than child care as well as impacts on the average total amount of such payments over the 26-month follow-up period. Again, however, these impacts are small.
| Outcome | PASS Group | Control Group | Difference (Impact) | P-Value | |
|---|---|---|---|---|---|
| Child carea | Received a child care payment (%) | 40.7 | 37.9 | 2.8 | 0.12 |
| Average number of months with a child care payment | 2.1 | 2.0 | 0.1 | 0.52 | |
| Average total child care payments ($) | 1,804 | 1,792 | 12 | 0.94 | |
Average number of months with a child care payment among those sample members who received a payment |
5.1 | 5.3 | -0.1 | NA | |
| Average total child care payment among those sample members who received a payment ($) | 4,434 | 4,729 | -295 | NA | |
| Other supportive servicesb | Received a supportive service payment (%) | 13.6 | 5.8 | 7.8 *** | 0.00 |
| Average number of months with a supportive service payment | 0.3 | 0.2 | 0.0 | 0.49 | |
| Average total supportive service payments ($) | 18 | 9 | 9 *** | 0.00 | |
| Average number of months with a supportive service payment among those sample members who received a payment | 1.9 | 3.8 | -2.0 | NA | |
| Average total supportive service payment among those sample members who received a payment ($) | 130 | 154 | -25 | NA | |
| Sample size (total = 2,770) | 1,627 | 1,143 | |||
| SOURCES: MDRC calculations from Riverside DPSS P3 automated program tracking system and child care payment records. NOTES: See Appendix C. These averages include all sample members, including those who never received those payments. aMeasures calculated from child care payment records. bMeasures calculated from Riverside DPSS P3 automated program tracking system and child care payment records. |
1“Referrals to and support in education and training programs” covers any services or payments that connect participants to classes and programs as well as any services that help clients persist in these activities. In addition to referrals to remediation and training programs, this cluster of services includes financial and educational counseling as well as payments for books and other school supplies. (back)
2Chapter 2 fully describes the five PASS service providers. (back)
3MDRC gave more weight to the P3 data for several additional reasons related to the fielding of the 12-month survey. First, the P3 system covers all sample members, while the survey covers only 224 respondents (8 percent of the sample). MDRC removed the survey from the field at an early stage of the fielding effort because the initial implementation research suggested that PASS program group members, on average, were unlikely to be receiving substantially more services than control group members. In addition, as discussed in Appendix H, a survey response analysis uncovered some evidence of bias: On several pre-random assignment characteristics, survey respondents appear different from survey nonrespondents. In addition, while the original sample was slated to cover more cohorts, the sample that was actually fielded was selected from among report sample members who were randomly assigned from October through December 2002, which covers only one quarter of the four-quarter, full-sample intake period. This raises concerns about how generalizable the survey results are to the full sample. Impacts measured through administrative records on such key economic outcomes as earnings were weaker for sample members in the one-quarter cohort than for the other random assignment cohorts. (back)
4Stage 1 dollars pay for the child care needs of unemployed TANF clients in Phase 1 (Greater Avenues for Independence, or GAIN) and for working TANF recipients in Phase 2. Stage 2, which is sometimes called “transitional child care,” pays for the child care needs of TANF leavers for up to 24 months after they go off the welfare rolls. (back)
| Table of Contents | Previous | Next |