Appendix E: Minnesota ERA 12-Month Data Issues and Survey Response Analysis

The ERA 12-Month Survey provides information on respondents’ participation in various activities and services, health care coverage, job characteristics, household composition, and other measures presented in this report. This appendix assesses the reliability of impact results for the survey and for those that completed a baseline assessment. First, a description of how the survey sample was selected is provided. The response rates for the survey sample and the two research groups are then discussed. Second, differences between survey respondents and survey nonrespondents are examined, followed by a comparison between the research groups among the survey respondents. Afterwards, a comparison is provided between sample members who completed a baseline assessment and sample members who did not. Finally, administrative records data are used to compare the impacts across survey samples, the baseline assessment sample, and the report sample to determine the extent to which the impacts observed for the survey sample and the baseline assessment sample are representative of the report sample. Examining the results between the assessment sample and the report sample is particularly important, since the baseline assessment sample is used to describe the report sample’s work barriers and to identify key subgroups.

Overall, there is little evidence to suggest that the survey is not reliable or that the survey respondent sample results cannot be generalized to the report sample. The response rates were high for the full survey sample and across research groups. Furthermore, respondents and nonrespondents do not differ in key pre-random assignment characteristics. A comparison between research groups among the survey respondents shows no systematic differences between the groups. The results also show that the respondents’ impacts on employment and welfare receipt are similar to the impacts for the report sample and the survey-eligible sample.

Among the baseline assessment sample, no differences in pre-random assignment characteristics were found between the research groups. However, the administrative records impacts for this sample were found to be larger when compared with the impacts for the report sample. This may be the result of a “cohort” effect or “nonresponse” bias. For this reason, caution should be exercised when interpreting impacts for subgroups defined using the baseline assessment.

Survey Sample Selection

As noted earlier in this report, the report sample includes 1,962 single parents who were randomly assigned from January 2002 to April 2003.

A two-step process was used to select the sample for the ERA 12-Month Survey. First, the survey-eligible sample was selected. It includes 725 sample members who were randomly assigned from October 2002 to March 2003 and who met the eligibility criteria for the survey.1 Sample members younger than age 18, in a two-parent family, or who did not speak English or Spanish2 were excluded from the survey-eligible sample, which makes up about 37 percent of the single-parent sample and covers 60 percent of the entire sample intake period.

From the fielded sample, a random sample of 657 sample members was chosen to be interviewed. This sample is referred to as the fielded survey sample. To ensure representation of individuals across the total sample, the survey sample is also split equally between ERA and control group members.

Survey Response Rates

Sample members who were interviewed for the ERA 12-Month Survey are referred to as “survey respondents,” or the respondent sample, while sample members who were not interviewed are known as “nonrespondents,” or the nonrespondent sample. A total of 503 sample members, or 77 percent of the fielded sample, completed the survey. Most of the nonrespondent sample (88 percent) refused to be interviewed or could not be located.3 The response rates of the research groups were very similar: 78 percent of the Tier 2 group members completed the survey, compared with 75 percent of the (Tier 1) control group members.

Although the overall response rates are high, whenever the response rate is lower than 100 percent, nonresponse bias may occur. Differences may exist between the respondent sample and the larger, fielded sample, owing to differences between the sample members who completed a survey and those who did not. Furthermore, the estimates may be biased if the background characteristics differ between the research groups.

Comparison of Respondents and Nonrespondents within the Survey Sample

In order to examine whether there are systematic differences between those who responded to the survey and those who did not, an indicator of survey response status was created, and then multivariate analysis was used to identify which pre-random assignment characteristics are significantly related to the indicator.

Appendix Table E.1 shows the estimated regression coefficients for the probability of being a respondent. As can be noted from this table, besides background characteristics such as race, age, and number of children, a research status indicator was included in the model. The first column of the table provides the parameter estimates that indicate the effect of each variable on the probability of completing the survey. The asterisks and p-values show the statistical significance of this relationship.

The age of the youngest child and month of intake were statistically significant in predicting whether or not someone would complete a survey. People with younger children and people randomly assigned in the later cohorts were less likely to respond to the survey. The F-statistic, along with the p-value of the F-statistic (at the bottom of Appendix Table E.1), shows that the differences between the survey respondents and the survey nonrespondents are not statistically significant. Furthermore, the R-square suggests that less than 4 percent of variance is explained by these significant factors.

Comparison of the Research Groups in the Survey Respondent Sample

Random assignment designs minimize the possibility of potential biases in the results. Although the response rates are high across both research groups, there is still the possibility that the characteristics of each research group differ due to the nonrespondent sample. If this is true, the impact estimates for the respondent sample may be affected.

Appendix Table E.2 shows baseline characteristics of the ERA and control group members. The differences between the groups are relatively small and not statistically significant. Furthermore, a multivariate regression analysis was performed to further test whether or not there was a relationship between the background characteristics and the research status. A 0/1 dummy indicating the research status was regressed on pre-random assignment characteristics — many of which are shown in Appendix Table E.2. None of the background characteristics was found to be related to research status.

Comparison Between Sample Members With Baseline Assessment Data and Those Without Baseline Assessment Data

As noted in this report, starting in October 2002, the random assignment process was streamlined, and staff began collecting baseline data over the phone. As a result, fewer people completed an assessment; a total of 60 percent of the single-parent sample members have baseline assessment data. Sample members who completed the baseline assessment are referred to baseline assessment sample. This section examines whether the results for single-parents with baseline assessment data are reliable and can be generalized to the full single-parent sample.

First, there were not large differences in response rates between research groups: 58 percent of the Tier 2 and 62 percent of the Tier 1 group members completed a baseline assessment. The presence of large differences would have been a potential source of bias in research group comparisons. When estimating impacts for the baseline assessment sample, the main concern is that research groups may differ in background characteristics that affect future employment and other outcomes. Appendix Table E.3 shows the baseline characteristics for the Tier 1 and Tier 2 sample members who have baseline assessment data. As shown, there are no statistically significant differences between the research groups. An additional analysis was performed to determine whether there are any observable program-control differences within the baseline assessment sample. An indicator of research group status was regressed on pre-random assignment demographic characteristics. The results (not shown) indicated that only the month of sample intake was significantly different between both groups. It was found that a slightly larger percentage of control group members completed the baseline assessment after October 2002. However, this difference is small (39 percent versus 33 percent), and the p-value of regression is not statistically significant.

Using a multivariate analysis, the analysis also examined whether there are systematic differences between those who completed the baseline assessment and those who did not (not shown). The month of sample intake was statistically significant in predicting whether or not someone would complete a survey. This is not surprising since, prior to October 2002, almost everyone completed a baseline assessment before being randomly assigned. Being in the control group was also statistical significant in predicting whether someone would have completed a baseline assessment. The F-statistic and the p-value of the F-statistic show that the differences between the sample members who completed a baseline assessment and those who did not are statistical significant. The R-square suggests that 37 percent of the variance is explained by these significant factors.

Comparison of Survey Respondents with the Fielded Sample, the Report Sample, and the Baseline Assessment Sample

Using administrative records data, this section discusses whether impacts for the survey respondents and for the baseline assessment sample can be generalized to the report sample. Given the differences between those who completed the baseline assessment and those who did not, and between respondents and nonrespondents, it is important to assess whether findings for the baseline assessment sample and the survey sample can be generalized to the report sample. There might be other reasons besides nonresponse bias that may affect the ability to generalize these samples to the report sample. As discussed above, the fielded sample includes sample members who were randomly assigned during a period of time that does not cover the full random assignment period. Similarly, not everyone who was randomly assigned after October 2002 completed the baseline assessment. As a result, a “cohort effect” may have been introduced. This could affect the impact estimates, because the survey sample or the baseline assessment sample might differ from sample members who were randomly assigned in other cohorts.

Appendix Table E.4 shows the adjusted means and impacts on several employment and public assistance outcomes for the report, fielded, respondent, and baseline assessment samples.4 This comparison is useful in assessing whether the story changes when the different samples are used. This table shows that the impacts for the fielded and respondent samples are consistent with the impacts for the report sample. Although the magnitude and statistical significance of the impacts occasionally vary across samples, in general the story is similar. For example, statistically significant impacts on the average quarterly employment rate were found for the first year of follow-up for the baseline assessment sample, but the impacts are not significant for the other samples. However, the size of the impacts is similar across the samples: The impact is 2.3 percent for the report sample, 2.4 percent for the fielded sample, 2.7 for the respondent sample, and 4.9 percent (statistically significant) for the baseline assessment sample. The biggest differences between the samples are found in the TANF receipt outcome.

Note that the employment impacts among the baseline assessment sample tend to be larger when compared with the other samples, which appears to be the result of a cohort effect and nonresponse bias. For this reason, caution should be exercised when interpreting impacts for subgroups defined using the assessment data.