Table of Contents | Previous | Next |
Appendix H
Survey Response Analysis
This appendix assesses the reliability of impact results for the Riverside County, California, Post-Assistance Self-Sufficiency (PASS) program based on the Employment Retention and Advancement (ERA) 12-Month Survey. It describes how the survey sample was selected, reports the response rates for survey sample members, and compares the impacts calculated with results from the administrative records data for the survey sample and the report sample. Finally, the appendix discusses discrepancies in employment outcomes among sample members having both survey data and administrative records data.
MDRC removed the survey from the field at an early stage of the fielding effort because the initial implementation research suggested that PASS program group members, on average, were unlikely to be receiving substantially more services than control group members. Thus, the possibility that PASS could produce economic impacts was viewed as slim. The number of surveys in the field at the time, however, was sufficient to assess the implementation of the program. Given that economic impacts were seen as doubtful at the time, the remaining individuals who were slated to be surveyed — whose responses would have brought the overall survey sample size up to a point suitable for measuring economic impacts — were not sought. Measuring economic impacts (as opposed to implementation indicators) with a small sample is problematic because the confidence intervals around the group estimates make it difficult to detect statistically significant impacts.
In addition, as is shown below, this response analysis uncovered some evidence of survey response bias, which further diminishes the utility of the survey. Based on several pre-random assignment characteristics, survey respondents appear to be different from survey non-respondents.
Finally, while the original sample was slated to cover more cohorts, the sample that was actually fielded was selected from among report sample members randomly assigned from October through December 2002, which covers only one quarter of the four-quarter, full-sample intake period. This raises concerns about how generalizable the survey results are to the full sample. As discussed below, impacts on such key economic outcomes as earnings were weaker for those in the October through December 2002 cohort than for the other random assignment cohorts.
As a result of the above factors, the ERA 12-Month Survey results are not used in this report to gauge economic impacts.
Survey Selection
The report sample in the Riverside PASS study includes 2,770 single parents who were randomly assigned to the PASS group (N = 1,627) and to the DPSS group (N = 1,143) from July 2002 through June 2003.
A two-step process was used to select the 12-month survey sample. First, the survey-eligible sample was selected (N = 605). This sample includes individuals in single-parent families in the research sample who were age 18 or older at the time of random assignment, were not randomly assigned in the Rancho Mirage service area, could speak English or Spanish, and were randomly assigned from October through December 2002. Next, 50 percent (N = 300) of the sample members in this survey-eligible sample were randomly selected to be interviewed, split between the PASS group (N = 154) and the control group (N = 146). This sample is referred to as the fielded survey sample.
Appendix Table H.1, which compares the pre-random assignment characteristics of the survey-eligible sample and the fielded survey sample, shows that people with certain characteristics had a higher likelihood of being sampled for the survey. Sample members who did not have a high-school diploma or General Educational Development (GED) certificate or those who had employment in the prior year were more likely to be selected for the survey. It is not surprising that there are some differences between these samples, given the survey selection criteria.1
Survey Response Rates
The survey interviews that were conducted took place 13 to 21 months after the sample members entered the study. The overall response rate was about 75 percent (N = 224 out of 300) among sample members who were fielded for the 12-month survey. The risk for nonresponse bias in Riverside PASS is further complicated by the differences in response rates among members of the PASS group (78 percent) and members of the control group (71 percent).2 Out of the 224 total respondents, 120 were PASS group members, and 104 were control group members.
| Survey Sample | ||
|---|---|---|
| Parameter Estimate | P-Value | |
| ERA group (PASS group) | -0.1117 *** | 0.0093 |
| Age of the youngest child | 0.0015 | 0.8237 |
| Number of children | 0.0226 | 0.2890 |
| Black, non-Hispanic | -0.0965 | 0.5101 |
| White, non-Hispanic | -0.1499 | 0.2994 |
| Hispanic | -0.0884 | 0.5401 |
| No high school diploma or GED | 0.1363 *** | 0.0030 |
| Employed in the quarter prior to random assignment | -0.0805 | 0.2780 |
| Female | -0.0190 | 0.7785 |
| Month of sample intake | -0.0146 | 0.5605 |
| 21-30 years of age | -0.0442 | 0.6052 |
| 31-40 years of age | -0.1013 | 0.3230 |
| 41 years old or older | -0.1102 | 0.3484 |
| Employed in the prior year | 0.1595 * | 0.0972 |
| Received food stamps in the prior year | -0.0936 | 0.1644 |
| Earnings in the prior 3 years | 0.0000 | 0.5428 |
| Number of quarters employed in prior 3 years | 0.0027 | 0.7853 |
| Center for Employment Training | -0.0366 | 0.5576 |
| Valley Restart | 0.0087 | 0.8810 |
| Volunteer Center | -0.0283 | 0.6461 |
| R-square (0.054) | ||
| F-statistic (1.66) | ||
| P-value of F-statistic (0.0348) | ||
| Sample size | 605 | |
| SOURCE: MDRC calculations from California Employment Development Department unemployment insurance records. NOTES: Statistical signficance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. This regression analysis was conducted among the survey-eligible sample (N = 605). |
Appendix Table H.2 shows that survey respondents are statistically different from non-respondents on several pre-random assignment characteristics. Sample members who are in the PASS group, female, or living in the Valley Restart service area were more likely to respond; sample members without a high-school diploma or GED, or from other racial/ethnic groups, were less likely to respond.3
Overall, PASS group respondents are not statistically significantly different from control group respondents. However, Appendix Table H.3 shows that members of the PASS group (N = 120) and members of the control group (N = 104) differ on some individual characteristics; control group members are more likely to be Hispanic and to have received food stamps during year prior to random assignment, while PASS group members were more likely to be working in a job covered by unemployment insurance (UI) during the quarter prior to random assignment. Because of these differences between ERA group respondents and nonrespondents, and because of the small sample sizes, the survey-based impacts in Riverside PASS are unusually sensitive to regression adjustment.
Impacts from the Administrative Records Data Across the Survey Sample and the Report Sample
Appendix Table H.4 shows the impacts on UI-covered employment, public assistance, and income among sample members in the report sample, the survey-eligible sample, the fielded sample, and the respondent sample.4 The table shows that the impacts on such key economic outcomes as earnings are weaker among the fielded cohort. However, the impacts among survey respondents are numerically larger than the impacts among the fielded sample or the report sample. The relatively large impacts among the respondent sample are often not statistically significant, due to the sample’s very small size.
| Survey Sample | ||
|---|---|---|
| Parameter Estimate | P-Value | |
| ERA group (PASS group) | 0.0825 | 0.1038 |
| Age of the youngest child | -0.0014 | 0.8692 |
| Number of children | -0.0163 | 0.5208 |
| Black, non-Hispanic | 0.3314 * | 0.0647 |
| White, non-Hispanic | 0.3617 ** | 0.0413 |
| Hispanic | 0.4367 ** | 0.0131 |
| No high school diploma or GED | -0.1216 ** | 0.0318 |
| Employed in the quarter prior to RA | -0.0355 | 0.6746 |
| Female | 0.1782 ** | 0.0355 |
| Month of sample intake | 0.0224 | 0.4671 |
| 21-30 years of age | 0.0250 | 0.8017 |
| 31-40 years of age | -0.0274 | 0.8320 |
| 41 years old or older | 0.0798 | 0.5802 |
| Employed in the prior year | -0.1755 | 0.1347 |
| Received food stamps in the prior year | -0.0014 | 0.9854 |
| Earnings in the prior 3 years | 0.0000 | 0.2531 |
| Number of quarters employed in prior 3 years | 0.0247 ** | 0.0420 |
| Center for Employment Training | -0.0416 | 0.5814 |
| Valley Restart | 0.1215 * | 0.0923 |
| Volunteer Center | -0.0496 | 0.5096 |
| R-square (0.111) | ||
| F-statistic (1.74) | ||
| P-value of F-statistic (0.0273) | ||
| Sample size | 300 | |
| SOURCE: MDRC calculations from California Employment Development Department unemployment insurance records. NOTE: Statistical signficance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. |
| Characteristic | PASS Group | Control Group | |
|---|---|---|---|
| Female (%) | 90.8 | 92.3 | |
| Race (%) | Hispanic | 48.3 | 56.7 |
| Black | 18.3 | 18.3 | |
| White | 30.8 | 25.0 | |
| Other | 2.5 | 0.0 | |
| Age (%) | 20 or younger | 8.3 | 9.6 |
| 21-30 | 53.3 | 48.1 | |
| 31-40 | 25.8 | 28.8 | |
| 41 or older | 12.5 | 13.5 | |
| Average age (years) | 30 | 30 | |
| High school diploma (%) | 59.5 | 53.1 | |
| Employed during the quarter prior to random assignment (%) | 82.5 | 74.0 | |
| Employed during the year prior to random assignment (%) | 89.2 | 87.5 | |
| Number of quarters employed in the prior 3 years | 7.2 | 6.8 | |
| Earnings in the 3 years prior to random assignment ($) | 17,194 | 15,537 | |
| Number of children (%) | 1 | 43.3 | 44.2 |
| 2 | 29.2 | 25.0 | |
| 3 or more | 27.5 | 30.8 | |
| Average number of children | 2.0 | 2.1 | |
| Age of youngest child (%) | Under 3 years | 45.8 | 41.3 |
| 3-5 years | 19.2 | 26.0 | |
| 6 years or older | 35.0 | 32.7 | |
| AFDC receipt history (%) | Never | 5.2 | 4.1 |
| Less than 3 months | 44.0 | 43.9 | |
| 3 months or more and less than 2 years | 12.9 | 12.2 | |
| 2 years or more and less than 5 years | 19.0 | 26.5 | |
| 5 years or more and less than 10 years | 10.3 | 6.1 | |
| 10 years or more | 8.6 | 7.1 | |
| Received food stamps in the prior year (%) | 81.7 | 91.3 ** | |
| Sample size (total = 224) | 120 | 104 | |
| SOURCE: MDRC calculations from California Employment Development Department unemployment insurance records. NOTES: Results are for sample members randomly assigned from October through December 2002. Chi-square (categorical) and two-tailed T (continuous) tests were used to assess the differences in characteristics across research groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. |
| Outcome | PASS Group | Control Group | Overall | Difference (Impact) | ||
|---|---|---|---|---|---|---|
| Quarters 2-5 | Ever employed (%) | Report sample | 80.1 | 77.1 | 78.8 | 3.0 ** |
| Eligible sample | 80.8 | 80.4 | 80.7 | 0.4 | ||
| Fielded sample | 81.2 | 78.7 | 80.0 | 2.5 | ||
| Respondent sample | 85.3 | 79.5 | 82.6 | 5.7 | ||
| Average quarterly employment (%) | Report sample | 64.5 | 61.5 | 63.2 | 2.9 ** | |
| Eligible sample | 67.0 | 63.0 | 65.3 | 4.0 | ||
| Fielded sample | 66.2 | 60.4 | 63.4 | 5.7 | ||
| Respondent sample | 70.8 | 61.9 | 66.7 | 8.9 * | ||
| Employed 4 consecutive quarters (%) | Report sample | 47.7 | 44.6 | 46.4 | 3.1 * | |
| Eligible sample | 52.6 | 46.3 | 49.9 | 6.3 | ||
| Fielded sample | 48.4 | 44.1 | 46.3 | 4.3 | ||
| Respondent sample | 52.9 | 45.8 | 49.6 | 7.1 | ||
| Number of quarters employed | Report sample | 2.6 | 2.5 | 2.5 | 0.1 ** | |
| Eligible sample | 2.7 | 2.5 | 2.6 | 0.2 | ||
| Fielded sample | 2.6 | 2.4 | 2.5 | 0.2 | ||
| Respondent sample | 2.8 | 2.5 | 2.7 | 0.4 * | ||
| Earnings ($) | Report sample | 9,178 | 8,261 | 8,800 | 916 *** | |
| Eligible sample | 9,771 | 9,162 | 9,512 | 609 | ||
| Fielded sample | 9,502 | 8,839 | 9,180 | 663 | ||
| Respondent sample | 10,365 | 9,220 | 9,833 | 1,146 | ||
| Ever received TANF (%) | Report sample | 40.7 | 43.5 | 41.8 | -2.8 | |
| Eligible sample | 36.5 | 35.8 | 36.2 | 0.8 | ||
| Fielded sample | 34.7 | 33.3 | 34.0 | 1.4 | ||
| Respondent sample | 34.8 | 34.9 | 34.8 | -0.1 | ||
| Amount of TANF received ($) | Report sample | 1,563 | 1,581 | 1,570 | -18 | |
| Eligible sample | 1,382 | 1,279 | 1,338 | 102 | ||
| Fielded sample | 1,365 | 1,273 | 1,320 | 92 | ||
| Respondent sample | 1,274 | 1,453 | 1,357 | -179 | ||
| Ever received food stamps (%) | Report sample | 46.2 | 47.5 | 46.8 | -1.3 | |
| Eligible sample | 41.5 | 41.9 | 41.7 | -0.5 | ||
| Fielded sample | 37.7 | 36.9 | 37.3 | 0.8 | ||
| Respondent sample | 38.3 | 38.5 | 38.4 | -0.1 | ||
| Amount of food stamps received ($) | Report sample | 971 | 964 | 968 | 7 | |
| Eligible sample | 828 | 756 | 798 | 72 | ||
| Fielded sample | 769 | 601 | 687 | 168 | ||
| Respondent sample | 738 | 682 | 712 | 56 | ||
| Total measured income ($) | Report sample | 11,712 | 10,807 | 11,338 | 905 *** | |
| Eligible sample | 11,981 | 11,198 | 11,647 | 783 | ||
| Fielded sample | 11,637 | 10,713 | 11,187 | 923 | ||
| Respondent sample | 12,377 | 11,355 | 11,903 | 1,023 | ||
| SOURCE: MDRC calculations from California Employment Development Department unemployment insurance records. NOTES: The report sample includes 2,770 sample members; PASS: 1,627; control: 1,143. The survey-eligible sample includes 605 sample members; PASS: 347; control: 258. The fielded sample includes 300 sample members; PASS: 154; control: 146. The respondent sample includes 224 sample members; PASS: 120; control: 104. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. |
Comparison of UI-Reported and Survey-Reported Employment Among Survey Respondents
In this section, outcomes from UI data are compared with respondent-reported employment at time of the 12-month survey interview. Appendix Table H.5 presents unadjusted out comes for UI-reported and survey-reported employment. Overall, the UI records and the survey data provide consistent information about employment status for three-fourths of the combined research sample (approximately 60 percent of the employed sample and 18 percent of the unemployed sample). However, the control group members were 11 percentage points more likely than the PASS group members to report employment that is not captured by the UI wage records; this is the main source of the discrepancy between UI-based and survey-based employment impacts.
Conclusion
Because of very small sample sizes, patterns of survey response bias, and a difference in impacts between the survey cohort and the larger report sample, the survey was deemed not to be a reliable source of information about such economic outcomes as employment and earnings for the PASS program as a whole. Since measuring participation outcomes typically involves less precision, the survey data are used in this report to provide information about program implementation.
| Outcome | PASS Group | Control Group | Overall | Difference (Impact) | |
|---|---|---|---|---|---|
| Reported employment, by data source (%) | UI records |
72.0 | 68.0 | 70.0 | 4.0 |
| ERA survey | 68.0 | 75.0 | 71.0 | -7.0 | |
| Reported employment, by combination of data sources (%) | UI records and ERA survey | 61.0 | 57.0 | 59.0 | 4.0 |
| UI records but not ERA survey | 11.0 | 11.0 | 11.0 | 0.0 | |
| ERA survey but not UI records | 7.0 | 18.0 | 12.0 | -11.0 | |
| Neither UI records nor ERA survey | 22.0 | 14.0 | 18.0 | 8.0 | |
| Sample size (total = 224) | 120 | 104 | |||
| SOURCES: MDRC calculations from California Employment Development Department unemployment insurance records and ERA 12-Month Survey. NOTE: Estimates are not regression-adjusted. |
1For this analysis, MDRC created an indicator of being sampled for the survey and related the indicator to pre-random assignment characteristics in a multivariate analysis. Note that, in addition to other background characteristics (such as age, race/ethnicity, and number of children), research status was included in this model. The first column of Appendix Table H.1 provides the parameter estimates indicating the effect of each variable on the probability of being sampled for the survey. The asterisk and P-value show the statistical significance of this relationship. These findings would normally be surprising, due to the random sampling methodology that was used to select the fielded survey sample. However, in Riverside PASS, a stratified sampling strategy was applied, and the sample was selected using random assignment ratios that differ from the ratios in the report sample. The stratification appears to account for the sampling bias. (back)
2The reasons given for nonresponse among the 76 nonresponding sample members were (1) that they could not be located (N = 53); (2) that they could not be located before the fielding period expired (N = 11); (3) that they refused to be interviewed (N = 11); or (4) that they were incapacitated (N = 1). (back)
3This analysis examines whether certain pre-random assignment characteristics are associated with higher or lower incidence of survey response. For this analysis, MDRC created an indicator of survey response and related the indicator to pre-random assignment characteristics in a multivariate analysis, using the same model as to estimate the probability of being sampled for the survey. (back)
4Within each sample, all the impacts are regression-adjusted to control for differences in background characteristics, prior earnings, prior employment, prior public assistance receipt, location or residence, and period of sample intake. (back)
| Table of Contents | Previous | Next |