Table of Contents | Previous | Next |
Appendix H: ERA 12-Month Survey Response Analysis for the NYC PRIDE Program
This appendix assesses the reliability of impact results for the ERA 12-Month Survey. It also examines whether the impacts for the survey respondents can be generalized to the impacts for the research sample. First, the appendix describes how the survey sample was selected. Second, it discusses the response rates for the survey sample for the three research groups. Next, the appendix examines differences in background characteristics between survey respondents and survey nonrespondents, and then it analyzes differences among survey respondents according to research group. Finally, the appendix compares the impacts on employment, earnings, and receipt of public assistance across the survey samples and the research sample, as calculated using administrative records data.
This appendix concludes, with some caution, that the PRIDE survey is reliable and that the survey respondent sample can be generalized to the research sample. Survey respondents and nonrespondents do not differ in key pre-random assignment characteristics. Among the respondents, no systematic differences were found between research groups, except for the month of sample intake. The impacts on welfare receipt for respondents are similar to the impacts for the research, survey-eligible, and fielded samples (defined below). However, the employment impacts for the research sample are larger and statistically significant, whereas the impacts for the other samples are not statistically significant.
Survey Sample Selection
As noted in the Introduction of this report, the research sample includes 3,188 sample members who were randomly assigned between December 4, 2001, and December 31, 2002.
MDRC used a two-step process to select the sample for the ERA 12-Month Survey. First, the survey-eligible sample was selected. It includes 1,704 single-parents who were randomly assigned from July through December 2002, were age 18 or older at the time of random assignment, and were able to speak English or Spanish. The random assignment period for the survey-eligible sample covers half the entire sample intake period, which raises some concern about the generalizability of the findings.
Next, MDRC randomly selected 1,043 survey-eligible sample members to be interviewed. This sample is referred to as the fielded sample, and it includes 524 PRIDE group members and 519 control group members.
Box H.1Key Analysis SamplesResearch sample. Single-parents randomly assigned between December 2001 and December 2002. Survey-eligible sample. Sample members in the research sample who were randomly assigned from July through December 2002 and who met the criteria for inclusion. Fielded sample. Sample members who were randomly selected from the survey-eligible sample to be interviewed for the survey. Respondent sample. Sample members in the fielded sample who completed the ERA 12-Month Survey. Nonrespondent sample. Sample members in the fielded sample who were not interviewed because they were not located or they refused to be interviewed or because of other reasons. |
Survey Response Rates
Sample members who completed the ERA 12-Month Survey are referred to as “survey respondents,” or the respondent sample, while sample members who were not interviewed are known as “nonrespondents.” or the nonrespondent sample. Approximately 73 percent of the fielded sample, or 759 sample members, completed the survey. The response rates were identical between the research groups. Almost two-thirds of the nonrespondent sample refused to be interviewed or could not be located.1
A response rate of nearly 80 percent inspires confidence that findings calculated from survey responses may be generalized to all members of the research sample. However, response bias may occur even with a relatively high response rate — typically, when respondents from different research groups vary in background characteristics that may affect employment and welfare receipt. In addition, survey results would be less reliable if a large proportion of members of a key subgroup did not complete an interview.
Comparison of Respondents and Nonrespondents Within the Survey Sample
In order to examine whether there are systematic differences between those who responded to the survey and those who did not, an indicator of survey response status was created, and then multivariate analysis was used to identify which pre-random assignment characteristics are significantly related to the indicator.
Appendix Table H.1 shows the estimated regression coefficients for the probability of being a respondent to the ERA 12-Month Survey. As can be noted from this table, besides background characteristics such as race/ethnicity, age, employment history, and other measurable qualities, a research status indicator was included in the model. The first column of the table provides the parameter estimates that indicate the effect of each variable on the probability of completing the survey. The asterisks and p-values show whether each relationship is statistically significant.
In general, the results show no consistent differences in background characteristics between survey respondents and nonrespondents. A few measures predict greater or smaller likelihood of responding, and they attain statistical significance — for example, having a child age 6 or older and history of food stamp receipt. However, the R‑square statistic suggests that only approximately 4 percent of variance is explained by these significant factors, meaning that knowing a fielded sample member’s background characteristics would not help much in predicting whether that individual would respond to the survey.
| Outcome | Survey Sample | |
|---|---|---|
| Parameter Estimate | P-Value | |
| ERA group | -0.004 | 0.885 |
| Age | 0.002 | 0.385 |
| Age of youngest child in household | -0.008 | 0.153 |
| Youngest child age 3 to 5 years old | 0.089 * | 0.054 |
| Youngest child age 6 or older | 0.069 | 0.274 |
| Number of children | -0.011 | 0.644 |
| Black, non-Hispanic | 0.226 | 0.157 |
| White | 0.057 | 0.731 |
| Hispanic | 0.221 | 0.164 |
| Provider: Brooklyn Bureau of Community Service | -0.032 | 0.377 |
| Provider: Federation Employment and Guidance Service (FEGS) | 0.018 | 0.647 |
| Provider: Goodwill Industries and Fedcap | 0.033 | 0.501 |
| Rents and lives in a public housing development/complex | 0.221 | 0.110 |
| Rents apartment/house and receives subsidized housing/Section 8 | 0.163 | 0.243 |
| Rents apartment/house and does not live in public housing | 0.170 | 0.212 |
| Lives in emergency or temporary housing | 0.146 | 0.291 |
| Average number of people in household | 0.005 | 0.819 |
| Family assistance | -0.049 | 0.117 |
| Employed in the prior year | -0.026 | 0.597 |
| Employed in the prior quarter | -0.032 | 0.590 |
| Employed 2 quarters prior | 0.082 | 0.176 |
| Ever employed in past 3 years | 0.038 | 0.377 |
| Number of quarters employed in prior 3 years | 0.010 | 0.170 |
| Received food stamps in the prior year | 0.156 ** | 0.040 |
| Received ADC in the prior year | -0.048 | 0.647 |
| Relative month of random assignment | -0.007 | 0.408 |
| R-square (0.0452) | ||
| F-statistic (1.85) | ||
| P-value of F-statistic (0.006) | ||
| Sample size | 1,043 | |
| SOURCES: MDRC calculations from the New York Welfare Management System (WMS) and unemployment insurance (UI) wage records from the State of New York and public assistance records from New York City. NOTE: Employment and earnings in jobs does not include employment outside New York or in jobs not covered by unemployment insurance, for example, “off the books” jobs, some agricultural jobs, and federal government jobs. |
Comparison of the Research Groups in the Respondent Sample
Random assignment designs minimize the possibility of potential biases in a study’s results. However, the possibility remains that the characteristics of the respondent sample in each research group differ due to the nonrespondent sample. If this is true, the impact estimates for the respondent sample may be affected.
Appendix Table H.2 shows baseline characteristics of the ERA (PRIDE) group and control group members. In general, differences between the groups are relatively small and are not statistically significant. The only exception to this finding is that a larger percentage of PRIDE group members received Section 8 housing. MDRC ran a more rigorous test of differences in background characteristics, using ordinary least squares (OLD) regression, and a similar finding was obtained (not shown).
| Outcome (%) | PRIDE Group | Control Group | |
|---|---|---|---|
| Age | 20 years or younger | 1.3 | 0.5 |
| 21 to 30 years | 23.2 | 19.3 | |
| 31 to 40 years | 33.9 | 39.1 | |
| 41 years or older | 41.6 | 41.2 | |
| Race/ethnicity | Hispanic | 54.0 | 52.1 |
| Black | 40.7 | 41.0 | |
| White | 5.3 | 5.8 | |
| Other | 0.0 | 1.1 | |
| Age of youngest child in household (%) | Less than 3 years | 23.7 | 19.7 |
| Between 3 to 5 years | 21.5 | 18.7 | |
| More than 6 years | 54.8 | 61.6 | |
| Receiving family assistance | 57.9 | 56.5 | |
| Average size of household | 3.1 | 3.1 | |
| Housing status (%) | Rent, public housing | 21.1 | 19.0 * |
| Rent, subsidized housing | 18.9 | 13.7 | |
| Rent, other | 44.5 | 45.1 | |
| Emergency/temporary housing | 14.7 | 21.4 | |
| Other housing arrangements | 0.8 | 0.8 | |
| PRIDE provider | Brooklyn Bureau of Community Service | 36.6 | 30.9 |
| Federation Employment and Guidance Service (FEGS) | 22.6 | 23.2 | |
| Goodwill Industries/Fedcap | 10.5 | 12.4 | |
| National Center for Disability Services | 30.3 | 33.5 | |
| Employed during the quarter prior to random assignment | 10.8 | 10.3 | |
| Employed during the year prior to random assignment | 27.6 | 28.2 | |
| Employed during the 3 years prior to random assignment | 51.1 | 53.6 | |
| Number of quarters employed in the prior 3 years | 2.4 | 2.7 | |
| Sample size (total = 759) | 379 | 380 | |
| SOURCES: MDRC calculations from the New York Welfare Management System (WMS) and unemployment insurance (UI) wage records from the State of New York and public assistance records from New York City. NOTES: In order to assess differences in characteristics across research groups, Chi-Square tests were used for categorical variables and T-tests were used for continuous variables. Significant levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Employment and earnings in jobs does not include employment outside New York or in jobs not covered by unemployment insurance, for example, “off the books” jobs, some agricultural jobs, and federal government jobs. |
Comparison of the Respondent Sample with the Fielded Sample, the Survey-Eligible Sample, and the Research Sample
Using administrative records data, this section discusses whether respondents’ impacts can be generalized to the fielded, survey-eligible, and research samples. Consistency of impact findings among the samples is considered to be the best result, suggesting that impacts on measures calculated from survey responses can be generalized to the research sample. Survey results may be considered unreliable because of response bias when impacts for survey respondents that are calculated with administrative data differ in size and direction from results for all other samples. Other patterns of inconsistency point to additional problems with the survey findings. Limiting sample selection to certain months of sample intake may introduce a “cohort effect” — a pattern of impacts that also occurs in the fielded and survey-eligible samples but that differs from the pattern when all members of the research sample are included. Alternatively, an unlucky sample draw may be inferred when impacts for the respondent sample resemble results for the fielded sample but when findings for both samples vary from findings for the eligible and research samples from which they were drawn.
Appendix Table H.3 shows the adjusted means and impacts on employment and welfare outcomes for the research sample, fielded sample, and respondent sample during the first year of the follow-up period.2 This comparison is useful in assessing whether the story changes when using the different samples. The analysis found that the employment and employment stability impacts for the research sample are larger than for the other samples and are statistically significant. With one exemption, impacts on receipt and payments of public assistance for the respondent sample are similar to the impacts for the research, survey-eligible, and fielded samples. The PRIDE program led to a reduction of food stamp receipt for the fielded and respondent samples, whereas the program did not have a significant effect for the survey-eligible or research samples. It is important to note that although the magnitude of the impacts may be slightly larger and statistically significant, the direction of the impacts across the samples remains the same.
| Outcome | PRIDE Group | Control Group | Difference (Impact) | ||
|---|---|---|---|---|---|
| Quarters 2-5 | Ever employed (%) | Research sample | 23.0 | 18.7 | 4.3 *** |
| Eligible sample | 22.0 | 19.5 | 2.5 | ||
| Fielded sample | 21.0 | 20.1 | 0.9 | ||
| Respondent sample | 22.1 | 20.9 | 1.2 | ||
| Average quarterly employment (%) | Research sample | 13.2 | 11.3 | 1.9 * | |
| Eligible sample | 12.3 | 12.0 | 0.2 | ||
| Fielded sample | 11.4 | 12.5 | -1.1 | ||
| Respondent sample | 11.9 | 12.6 | -0.7 | ||
| Employed 4 consecutive quarters (%) | Research sample | 5.5 | 4.7 | 0.8 | |
| Eligible sample | 5.3 | 5.2 | 0.1 | ||
| Fielded sample | 4.6 | 5.2 | -0.6 | ||
| Respondent sample | 4.2 | 4.5 | -0.2 | ||
| Number of quarters employed | Research sample | 0.5 | 0.5 | 0.1 * | |
| Eligible sample | 0.5 | 0.5 | 0.9 | ||
| Fielded sample | 0.5 | 0.5 | 0.0 | ||
| Respondent sample | 0.5 | 0.5 | 0.0 | ||
| Employed Quarter 5 (%) | Research sample | 15.7 | 13.9 | 1.8 | |
| Eligible sample | 15.5 | 15.1 | 0.4 | ||
| Fielded sample | 13.6 | 15.1 | -1.5 | ||
| Respondent sample | 14.2 | 15.1 | -0.9 | ||
| Ever received TANF (%) | Research sample | 99.3 | 99.3 | -0.1 | |
| Eligible sample | 99.1 | 99.5 | -0.4 | ||
| Fielded sample | 99.0 | 99.5 | -0.5 | ||
| Respondent sample | 99.2 | 99.8 | -0.7 | ||
| Amount of TANF received ($) | Research sample | 5,806 | 6,100 | -293 *** | |
| Eligible sample | 5,850 | 6,141 | -290 *** | ||
| Fielded sample | 5,833 | 6,214 | -381 *** | ||
| Respondent sample | 5,990 | 6,298 | -308 ** | ||
| Ever received food stamps (%) | Research sample | 99.0 | 98.8 | 0.1 | |
| Eligible sample | 98.7 | 99.2 | -0.5 | ||
| Fielded sample | 98.9 | 99.0 | -0.2 | ||
| Respondent sample | 99.0 | 99.2 | -0.2 | ||
| Amount of food stamps received ($) | Research sample | 3,299 | 3,336 | -37 | |
| Eligible sample | 3,356 | 3,430 | -74 | ||
| Fielded sample | 3,353 | 3,456 | -103 * | ||
| Respondent sample | 3,354 | 3,525 | -171 ** | ||
| SOURCE: Administrative records from NYC PRIDE. NOTES: See Appendix E. The research sample includes 2,648 sample members; PRIDE group: 1,553; control group:1,095 The eligible sample includes 1,704 sample members; PRIDE group: 939; control group: 765. The fielded sample includes 1,043 sample members; PRIDE group: 524; control group: 519. The respondent sample includes 759 sample members; PRIDE group: 380; control group: 379. |
1 The remaining third of the nonrespondent sample were not interviewed because they were incapacitated or institutionalized or were located after the fielding period expired. (back)
2 All the impacts within each sample are regression-adjusted, to control for differences in background characteristics, prior employment, prior public assistance, PRIDE provider, and period of sample intake. (back)
| Table of Contents | Previous | Next |