Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

Appendix F: Texas ERA 12-Month Survey Response Analysis

Corpus Christi Survey Response Analysis

This section of the appendix assesses the reliability of impact results from the one-year survey of the Employment Retention and Advancement (ERA) evaluation in Corpus Christi, Texas. It examines whether the impacts for respondents to the ERA 12-Month Survey can be generalized to the impacts for the report sample in Corpus Christi. After introducing how the survey sample was selected, it discusses the response rates for survey sample members and for sample members in the research groups. It also examines the differences in background characteristics between survey respondents and nonrespondents and between research groups among survey respondents. Finally, this section compares the impacts across the survey sample and the report sample in Corpus Christi on measures created from the administrative records.

Survey Selection in Corpus Christi

The report sample in Corpus Christi includes 1,309 sample members who were randomly assigned to the ERA group (N = 656) or to the control group (N = 653) from October 2000 through June 2002.

A two-step process was used to select the ERA 12-Month Survey sample in Corpus Christi. First, the survey-eligible sample was selected. This sample includes 357 sample members who met the survey eligibility criteria. Research sample members who were randomly assigned from January through June 2002, who were age 18 or older, who were in a single-parent family, and who spoke either English or Spanish were eligible for the survey. It is particularly notable that the survey sample was drawn from a narrow 6-month segment of the sample intake period; this raises concerns about the generalizability of results to the full report sample (which covered a 26-month period in Corpus Christi).

In Corpus Christi, all sample members who were eligible for the survey were selected to be interviewed. The interview sample is referred to as the fielded survey sample and is split equally between members of the ERA group (N = 180) and members of the control group (N = 177). Box F.1 describes the key analysis samples in Corpus Christi.

Survey Response Rates in Corpus Christi

Sample members who were interviewed for the ERA 12-Month Survey are referred to as survey respondents, while sample members who were not interviewed are known as nonrespondents.

Although the overall response rate in Corpus Christi was 82 percent, nonresponse bias may occur whenever the response rate is below 100 percent, because there may be differences in the average background characteristics of the respondent sample and the fielded sample. The response rate was slightly lower among the ERA group (78 percent) than among the control group (85 percent). 1

The primary reasons that some sample members in Corpus Christi were not interviewed (N = 67) are that they could not be located (N = 54), they refused to be interviewed (N = 9), they were incapacitated (N = 3), or they were ineligible to be surveyed (N = 1).

Box F.1 Key Analysis Samples in Corpus Christi Research sample. Sample members who were randomly assigned during the sample intake period, which started in October 2000 and ended in December 2002. Report sample. Single parents who were randomly assigned from October 2000 through June 2002. Survey-eligible sample. Sample members in the research sample who were randomly assigned from January through June 2002 and who met the criteria for inclusion in the survey. Fielded sample. Sample members who were chosen from the survey-eligible sample to be interviewed for the survey. In the case of Corpus Christi, the fielded sample and the survey-eligible sample are the same. Respondent sample. Sample members in the fielded sample who completed the ERA 12-Month Survey. Nonrespondent sample. Sample members in the fielded sample who were not interviewed because they were not located or they refused to be interviewed or because of other reasons.

Comparison of Respondents and Nonrespondents Within the Survey Sample: Corpus Christi

This section examines whether there are differences in pre-random assignment characteristics between respondents and nonrespondents within the survey-eligible sample in Corpus Christi. In order to examine differences between those who completed the survey and those who did not, MDRC created an indicator of survey response and related the indicator to pre-random assignment characteristics in a multivariate analysis.

Appendix Table F.1 shows estimated regression coefficient for the probability of being a respondent in Corpus Christi, based on background characteristics. The first column of the table provides the parameter estimates that indicate the effect of each variable on the probability of completing the survey. The stars (asterisks) and p-values show whether the relationships are statistically significant.

The Employment Retention and Advancement Project

Appendix Table F.1

Estimated Regression Coefficients for the Probability of Being a Respondent on the ERA 12-Month Survey, Corpus Christi

	Survey Sample
	Parameter Estimate	P-Value
ERA group	-0.054	0.199
Age of the youngest child	-0.001	0.889
Number of children	-0.016	0.442
Black, non-Hispanic	0.074	0.593
White	0.102	0.350
Hispanic	0.132	0.209
No high school diploma or GED	-0.091 **	0.042
Employed in the quarter prior to random assignment	-0.003	0.953
Female	-0.122	0.145
Month of sample intake	0.001	0.956
21-30 years of age	0.021	0.772
31-40 years of age	0.085	0.353
41 years old and over	0.105	0.304
Employed in the prior year	0.055	0.422
Received food stamps in the prior year	0.142 ***	0.004
Earnings in the prior 3 years	0.000	0.413
Number of quarters employed in prior 3 years	0.001	0.947
R-square (0.081)
F-statistic (1.75)
P-value of F-statistic (0.033)
Sample size	357

Appendix Table F.1 shows that survey respondents and nonrespondents in Corpus Christi were similar at random assignment: Most measures of background characteristics are unable to differentiate between them. However, food stamp receipt in the year prior to random assignment and not having a high school diploma or General Educational Development (GED) certificate are both statistically significant characteristics in predicting whether someone would complete a survey in Corpus Christi. Survey respondents were more likely to have received food stamps in the year prior to random assignment and to have a high school diploma or GED. It is not surprising to see that food stamp receipt in the year prior to random assignment is a good predictor of survey response. Other recent response analyses have found this relationship as well. This is likely attributable to better and more updated contact information for these sample members. It is less clear why having no high school diploma or GED is statistically significant in predicting survey completion.

At the bottom of Appendix Table F.1, the F-statistic and the p-value of the F-statistic show that the overall difference between survey respondents and nonrespondents in Corpus Christi is statistically significant (p-value = 0.003). Thus, caution should be exercised in generalizing these results to the report sample.

Comparison of the Research Groups in the Survey Respondent Sample: Corpus Christi

The random assignment design essentially eliminates the possibility of selection bias. However, the risk of differences between the background characteristics of ERA and control group members is reintroduced due to the survey sampling and response process. Specifically, if different types of ERA group members respond (compared with control group members), the integrity of the experiment would be compromised for the survey analysis. If this is true to a large extent, the ability to generalize from the respondent sample may be affected.

Appendix Table F.2 shows whether there may be any statistically significant differences in background characteristics between ERA group and control group respondents in Corpus Christi. Overall, there are few differences between them. The main differences are that ERA group respondents at random assignment were younger, had lower earnings, and were more likely to have received food stamps in the prior year.

Comparison of Survey Respondents with the Fielded Sample and the Report Sample: Corpus Christi

This section examines whether impacts among survey respondents in Corpus Christi can be generalized to the fielded sample and the report sample and whether any cohort effects may have been introduced.2

The Employment Retention and Advancement Project

Appendix Table F.2

Background Characteristics of Survey Respondents, Corpus Christi

Variable		ERA Group	Control Group
Female (%)		5.7	8.1
Race/ethnicity (%)	Hispanic	73.8	74.3
	Black	5.7	4.7
	White	19.1	17.6
	Other	1.4	3.4
Age (%)	20 or younger	12.1	8.7*
	21-30	53.9	47.7*
	31-40	26.2	25.5*
	41 or older	7.8	18.1*
	Average age (years)	28	30*
High school diploma (%)		48.6	57.0
Employed during the quarter prior to random assignment (%)		49.6	47.0
Employed during the year prior to random assignment (%)		75.9	71.1
Number of quarters employed in the prior 3 years (%)		6.3	6.5
Earnings in the 3 years prior to random assignment ($)		11,629	15,571**
Number of children (%)	0	0.7	0.7
	1	43.3	49.7
	2	27.0	28.2
	More than 3	29.1	21.5
	Average number of children	2.0	1.8
Age of youngest child (%)	Under 3 years	55.7	49.3
	3-5 years	22.1	20.3
	6 years and older	22.1	30.4
AFDC receipt history (%)	Never	46.1	43.6
	Less than 3 months	4.3	6.0
	3 months or more and less than 2 years	34.8	30.9
	2 years or more and less than 5 years	9.9	10.7
	5 years or more and less than 10 years	3.5	7.4
	10 years or more	1.4	1.3
Received food stamps in prior year (%)		80.1	70.5*
Sample size (total = 290)		141	149

SOURCES: Texas background information form and administative records from the State of Texas. NOTES: Results are for sample members randomly assigned from January through June 2002. Chi-square (categorical) and two-tailed T (continous) tests were used to assess the differences in characteristics across research groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Appendix Table F.3 shows the adjusted means and impacts on employment and welfare outcomes for Corpus Christi’s full sample, fielded sample, and respondent sample.3 With some exceptions, the impacts are consistent across the samples.4 In some cases, smaller impacts are statistically significant among the report sample, due to the larger sample size. Appendix Table F.3 shows that the impact on earnings is larger among the report sample than the respondent sample, for reasons that are unclear.

Conclusions About Corpus Christi

Due to the narrow cohort of months from which Corpus Christi’s survey sample was selected –– as well as some evidence of response bias –– caution should be exercised when generalizing the results among the survey sample to the broader report sample. In particular, this response analysis found that survey respondents were more likely than nonrespondents to have received food stamps in the year prior to random assignment and were more likely to have a high school diploma or GED. Furthermore, respondents in the ERA group were younger than their control group counterparts and were more likely to have received food stamps in the year prior to random assignment. Finally, there were some small differences across the three samples in the pattern of impacts on measures of administrative records.

The Employment Retention and Advancement Project

Appendix Table F.3

Comparison of Impacts for the Full, Fielded, and Respondent Samples in Corpus Christi

Outcome		ERA Group	Control Group	Difference (Impact)
Quarters 2-5
Ever employed (%)	Full sample	72.7	73.7	-1.0
	Fielded sample	73.1	71.8	1.3
	Respondent sample	75.3	73.2	2.1
Average quarterly employment (%)	Full sample	52.4	49.5	2.9 *
	Fielded sample	54.8	49.5	5.3
	Respondent sample	58.8	53.2	5.6
Employed 4 consecutive quarters (%)	Full sample	29.9	26.1	3.8 *
	Fielded sample	32.9	27.9	5.0
	Respondent sample	36.6	33.2	3.3
Number of quarters employed	Full sample	2.1	2.0	0.1 *
	Fielded sample	2.2	2.0	0.2
	Respondent sample	2.4	2.1	0.2
Earnings ($)	Full sample	3,982	3,567	415 *
	Fielded sample	4,229	3,847	382
	Respondent sample	4,518	4,353	165
Ever received TANF (%)	Full sample	81.9	78.0	4.0 **
	Fielded sample	83.7	76.6	7.1
	Respondent sample	81.8	76.6	5.3
Amount of food stamps received ($)	Full sample	2,745	2,820	-75
	Fielded sample	2,632	2,743	-111
	Respondent sample	2,644	2,789	-145
Total measured income ($)	Full sample	7,751	7,376	375 *
	Fielded sample	7,876	7,580	296
	Respondent sample	8,184	8,115	70

SOURCE: MDRC calculations from administrative records from the State of Texas. NOTES: The full sample includes 1,723 sample members; experimental: 868; control: 855. The fielded sample includes 357 sample members; experimental: 180; control: 177. The respondent sample includes 290 sample members; experimental: 141; control: 149. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Fort Worth Survey Response Analysis

This section of the appendix assesses the reliability of impact results from the one-year survey of the Employment Retention and Advancement (ERA) evaluation in Fort Worth, Texas. It examines whether the impacts for respondents to the ERA 12-Month Survey can be generalized to the impacts for the report sample in Fort Worth. After introducing how the survey sample was selected, it discusses the response rates for survey sample members and for sample members in the research groups. It also examines the differences in background characteristics between survey respondents and nonrespondents and between research groups among survey respondents. Finally, this section compares the impacts across the survey sample and the report sample on measures created from the administrative records. The survey sample in Fort Worth is the smallest of the three Texas sites and is drawn from a cohort that does not overlap at all with the report sample cohort. For this reason, it is not advisable to make inferences about Fort Worth’s broader report sample on the basis of the survey sample’s impacts.

Survey Selection in Fort Worth

The report sample in Fort Worth includes 1,163 sample members who were randomly assigned to the ERA group (N = 577) or to the control group (N = 586) from October 2000 through June 2002.

As in the other sites, a two-step process was used to select the ERA 12-Month Survey sample in Fort Worth. First, the survey-eligible sample was selected. This included 250 sample members who met the survey eligibility criteria. Research sample members who were randomly assigned from September through December 2002, who were age 18 or older, who were in a single-parent family, and who spoke either English or Spanish were eligible for the survey. It is particularly notable that the survey sample was drawn from a very narrow 3-month segment of the sample intake period that does not overlap at all with the report sample cohort; this raises substantial concerns about the generalizability of results to the full report sample (which covers a 26-month period in Fort Worth).

In Fort Worth, all sample members eligible for the survey were selected to be interviewed. The interview sample is referred to as the fielded survey sample and is split fairly evenly between members of the ERA group (N = 123) and members of the control group (N = 127). Box F.2 describes the key analysis samples in Fort Worth.

Box F.2 Key Analysis Samples in Fort Worth Research sample. Sample members who were randomly assigned during the sample intake period, which started in October 2000 and ended in December 2002. Report sample. Single parents who were randomly assigned from October 2000 through June 2002. Survey-eligible sample. Sample members in the research sample who were randomly assigned from September through December 2002 and who met the criteria for inclusion in the survey. Fielded sample. Sample members who were chosen from the survey-eligible sample to be interviewed for the survey. Respondent sample. Sample members in the fielded sample who completed the ERA 12-Month Survey. Nonrespondent sample. Sample members in the fielded sample who were not interviewed because they were not located or they refused to be interviewed or because of other reasons.

Survey Response Rates in Fort Worth

Sample members who were interviewed for the ERA 12-Month Survey are referred to as survey respondents, while sample members who were not interviewed are known as non-respondents.

Although the overall response rate in Fort Worth was about 75 percent (N = 188), this is slightly below the OMB standard for response rates (80 percent), and thus the risk of nonresponse bias is somewhat higher. Roughly equal proportions of ERA group members (76 percent) and control group members (75 percent) responded to the survey. 5

The primary reasons that some sample members in Fort Worth were not interviewed (N = 62) are that they could not be located (N = 35), they refused to be interviewed (N = 14), or they were incapacitated (N = 4).

Comparison of Respondents and Nonrespondents Within the Survey Sample: Fort Worth

This section examines whether there are differences in pre-random assignment characteristics between respondents and nonrespondents within the survey-eligible sample in Fort Worth. In order to examine differences between those who completed the survey and those who did not, MDRC created an indicator of survey response and related the indicator to pre-random assignment characteristics in a multivariate analysis.

Appendix Table F.4 shows estimated regression coefficient for the probability of being a respondent in Fort Worth, based on background characteristics. The first column of the table provides the parameter estimates that indicate the effect of each variable on the probability of completing the survey. The star (asterisk) and p-value show the statistical significance of this relationship.

Appendix Table F.4 shows that survey respondents and nonrespondents in Fort Worth were similar at random assignment: No measures of background characteristics were able to differentiate between them. However, whether a sample member had received food stamps prior to random assignment comes close to being a statistically significant predictor of survey response (p-value = 0.101). At the bottom of Appendix Table F.4, the F-statistic and the p-value of the F-statistic show that the overall difference between survey respondents and nonrespondents is not statistically significant.

Comparison of the Research Groups in the Survey Respondent Sample: Fort Worth

The random assignment design essentially eliminates the possibility of selection bias. However, the risk of differences between the background characteristics of ERA and control group members is reintroduced due to the survey sampling and response process. Specifically, if different types of ERA group members respond (compared with control group members), the integrity of the experiment would be compromised for the survey analysis. If this is true to a large extent, the ability to generalize from the respondent sample may be affected.

Appendix Table F.5 shows whether there may be any statistically significant differences in background characteristics between ERA group and control group respondents in Fort Worth. There are no statistically significant differences in background characteristics between ERA group survey respondents and control group survey respondents. In other words, the integrity of the experiment is preserved. For example, among survey respondents, ERA group members were employed, on average, 7.1 quarters in the three years prior to random assignment, which is very close to the control group’s average of 6.9 quarters.

The Employment Retention and Advancement Project

Appendix Table F.4

Estimated Regression Coefficients for the Probability of Being a Respondent on the ERA 12-Month Survey, Fort Worth

	Survey Sample
	Parameter Estimate	P-Value
ERA group	0.003	0.955
Age of the youngest child	-0.004	0.667
Number of children	0.041	0.181
Black, non-Hispanic	-0.172	0.359
White	-0.245	0.185
Hispanic	-0.180	0.346
No high school diploma or GED	0.047	0.434
Employed in the quarter prior to random assignment	-0.054	0.498
Female	0.040	0.772
Month of sample intake	0.023	0.387
21-30 years of age	-0.116	0.251
31-40 years of age	-0.111	0.381
41 years old and over	-0.004	0.981
Employed in the prior year	-0.052	0.593
Received food stamps in the prior year	0.101	0.101
Earnings in the prior 3 years	0.000	0.813
Number of quarters employed in prior three years	0.019	0.169
R-square (0.066)
F-statistic (0.96)
P-value of F-statistic (0.499)
Sample size	250

SOURCE: MDRC calculations from administrative records from the State of Texas. NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Comparison of Survey Respondents with the Fielded Sample and the Report Sample: Fort Worth

This section examines whether impacts among survey respondents in Fort Worth can be generalized to the fielded sample and the report sample and whether any cohort effects may have been introduced.6

The Employment Retention and Advancement Project

Appendix Table F.5

Background Characteristics of Survey Respondents, Fort Worth

Variable		ERA Group	Control Group
Female (%)		95.7	95.8
Race/ethnicity (%)	Hispanic	15.2	15.6
	Black	64.1	61.5
	White	19.6	19.8
	Other	1.1	3.1
Age (%)	20 or younger	13.0	10.4
	21-30	46.7	52.1
	31-40	31.5	26.0
	41 or older	8.7	11.5
	Average age years	28	29
High school diploma (%)		54.3	57.3
Employed during the quarter prior to random assignment (%)		56.5	50.0
Employed during the year prior to random assignment (%)		76.1	71.9
Number of quarters employed in the prior 3 years (%)		7.1	6.9
Earnings in the 3 years prior to random assignment ($)		19,242	18,914
Number of children (%)	1	44.6	37.5
	2	31.5	33.3
	More than 3	23.9	29.2
	Average number of children	1.9	2.0
Age of youngest child (%)	Under 3 years	55.4	54.2
	3-5 years	15.2	16.7
	6 years and older	29.3	29.2
AFDC receipt history (%)	Never	41.3	41.7
	Less than 3 months	2.2	7.3
	3 months or more and less than 2 years	54.3	42.7
	2 years or more and less than 5 years	1.1	5.2
	5 years or more and less than 10 years	1.1	3.1
	Received food stamps in prior year (%)	70.7	66.7
Sample size (total = 188)		92	96

SOURCES: Texas background information form and administative records from the State of Texas. NOTE: Results are for sample members and randomly assigned from September through December 2002. Chi-square (categorical) and two-tailed T (continous) tests were used to assess the differences in characteristics across research groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Appendix Table F.6 shows the adjusted means and impacts on employment and welfare outcomes for Fort Worth’s report sample, fielded sample, and the respondent sample.7 The comparison shows that the impacts among the fielded sample and among respondent sample are inconsistent with the impacts among the report sample on most measures of earnings, employment, and food stamp receipt. This is not surprising, given that the respondent and fielded samples do not at all overlap with the report sample. For example, among the report sample, ERA group members were 2.3 percentage points more likely to have been employed during Year 1 than control group members. Among the respondent sample, ERA group members were nearly 12 percentage points less likely to have been employed than control group members. There are large discrepancies on several other measures. Notably, ERA had a large impact on food stamp receipt among the respondent sample, which was not at all evident among the report sample.

Conclusions About Fort Worth

Due to the narrow cohort of months from which Fort Worth’s survey sample was selected –– as well as the substantially different pattern of impacts among the survey respondent sample –– great caution should be exercised when generalizing the results among the survey sample to the broader report sample. While the differences in background characteristics are not large, the impacts, in some cases, are qualitatively different across subsamples. This is likely because the survey sample does not overlap with the report sample.

The Employment Retention and Advancement Projec

Appendix Table F.6

Comparison of Impacts for the Report, Fielded, and Respondents Samples in Fort Worth

Outcome		ERA Group	Control Group	Difference (Impact)
Quarters 2-5
Ever employed (%)	Report sample	69.4	67.1	2.3
	Fielded sample	64.6	72.8	-8.2
	Respondent sample	63.7	75.4	-11.6 *
Average quarterly employment (%)	Report sample	49.1	47.1	2.0
	Fielded sample	41.2	48.4	-7.2
	Respondent sample	40.6	48.8	-8.2
Employed 4 consecutive quarters (%)	Report sample	26.4	25.4	1.0
	Fielded sample	15.0	28.8	-13.8 ***
	Respondent sample	15.4	27.9	-12.5 **
Number of quarters employed	Report sample	2.0	1.9	0.1
	Fielded sample	1.7	1.9	-0.3
	Respondent sample	1.6	2.0	-0.3
Earnings ($)	Report sample	4,486	4,248	238
	Fielded sample	3,540	4,446	-906
	Respondent sample	3,583	4,479	-896
Ever received TANF (%)	Report sample	80.8	81.1	-0.3
	Fielded sample	76.3	78.1	-1.8
	Respondent sample	81.1	77.5	3.6
Amount of food stamps received ($)	Report sample	2,698	2,672	26
	Fielded sample	3,142	2,354	788 ***
	Respondent sample	3,298	2,577	721 ***
Total measured income ($)	Report sample	8,469	8,113	355
	Fielded sample	7,906	7,771	134
	Respondent sample	8,194	8,054	140

SOURCE: MDRC calculations from administrative records from the State of Texas. NOTE: The report sample includes 1,163 sample members; experimental: 577; control: 586. The fielded sample includes 250 sample members; experimental: 123; control: 127. The respondent sample includes 188 sample members; experimental: 92; control: 96. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Houston Survey Response Analysis

This section of the appendix assesses the reliability of impact results from the one-year survey of the Employment Retention and Advancement (ERA) evaluation in Houston, Texas. It examines whether the impacts for respondents to the ERA 12-Month Survey can be generalized to the impacts for the report sample in Houston. After introducing how the survey sample was selected, it discusses the response rates for survey sample members and for sample members in the research groups. It also examines the differences in background characteristics between survey respondents and nonrespondents and between research groups among survey respondents. Finally, this section compares the impacts across the survey sample and the report sample in Houston on measures created from the administrative records.

Survey Selection in Houston

The report sample in Houston includes 1,816 sample members who were randomly assigned to the ERA group (N = 904) or to the control group (N = 912) from March 2001 through June 2002.

As in the other sites, a two-step process was used to select the ERA 12-Month Survey sample in Houston. First, the survey-eligible sample was selected. This sample includes 501 sample members who met the survey eligibility criteria. Research sample members who were randomly assigned from January through June 2002, who were age 18 or older, who were in a single-parent family, and who spoke either English or Spanish were eligible for the survey. It is particularly notable that the survey sample was drawn from a narrow 6-month slice of the sample intake period; this raises concerns about the generalizability of results to the full report sample (which covered a 16-month period in Houston).

In Houston, 370 out of the 501 sample members eligible for the survey were selected to be interviewed. The interview sample is referred to as the fielded survey sample and is split fairly evenly between members of the ERA group (N = 187) and members of the control group (N = 183). Box F.3 describes the key analysis samples in Houston.

Box F.3 Key Analysis Samples in Houston Research sample. Sample members who were randomly assigned during the sample intake period, which started in March 2001 and ended in December 2002. Report sample. Single parents who were randomly assigned from March 2001 through June 2002. Survey-eligible sample. Sample members in the research sample who were randomly assigned from January through June 2002 and who met the criteria for inclusion in the survey. Fielded sample. Sample members who were chosen from the survey-eligible sample to be interviewed for the survey. Respondent sample. Sample members in the fielded sample who completed the ERA 12-Month Survey. Nonrespondent sample. Sample members in the fielded sample who were not interviewed because they were not located or they refused to be interviewed or because of other reasons.

Survey Response Rates in Houston

Sample members who were interviewed for the ERA 12-Month Survey are referred to as survey respondents, while sample members who were not interviewed are known as nonrespondents.

Although the overall response rate in Houston was approximately 80 percent (the OMB standard), nonresponse bias may occur whenever the response rate is below 100 percent, because there may be differences in the average background characteristics of the respondent sample and the fielded sample.

The primary reasons that some sample members in Houston were not interviewed (N = 53) are that they could not be located (N = 54), they refused to be interviewed (N = 11), or they were incapacitated (N = 3).

Comparison of Respondents and Nonrespondents Within the Survey Sample: Houston

This section examines whether there are differences in pre-random assignment characteristics between respondents and nonrespondents within the survey-eligible sample in Houston. In order to examine differences between those who completed the survey and those who did not, MDRC created an indicator of survey response and related the indicator to pre-random assignment characteristics in a multivariate analysis.

Appendix Table F.7 shows estimated regression coefficient for the probability of being a respondent in Houston, based on background characteristics. The first column of the table provides the parameter estimates that indicates the effect of each variable on the probability of completing the survey. The stars (asterisks) and p-values indicate whether the relationships are statistically significant.

The Employment Retention and Advancement Project

Appendix Table F.7

Estimated Regression Coefficients for the Probability of Being a Respondent on the ERA 12-Month Survey, Houston

	Survey Sample
	Parameter Estimate	P-Value
ERA group	-0.010	0.820
Age of the youngest child	-0.001	0.937
Number of children	0.010	0.605
Black, non-Hispanic	0.103	0.426
White	0.038	0.789
Hispanic	0.118	0.377
No high school diploma or GED	-0.002	0.960
Employed in the quarter prior to random assignment	0.094 *	0.082
Female	-0.096	0.422
Month of sample intake	-0.004	0.754
21-30 years of age	-0.124	0.123
31-40 years of age	-0.129	0.184
41 years old and over	-0.152	0.226
Employed in the prior year	-0.044	0.526
Received food stamps in the prior year	0.045	0.381
Earnings in the prior 3 years	0.000	0.604
Number of quarters employed in prior 3 years	0.003	0.771
R-square (0.027)
F-statistic (0.57)
P-value of F-statistic (0.911)
Sample size	370

SOURCE: MDRC calculations from administrative records from the State of Texas. NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Appendix Table F.7 shows that survey respondents and nonrespondents in Houston were similar at random assignment: Most measures of background characteristics are unable to differentiate between them. Employment in the quarter prior to random assignment is the only statistically significant characteristic in predicting whether someone would complete a survey in Houston. Survey respondents were more likely than nonrespondents to be employed in the quarter prior to random assignment.

At the bottom of Appendix Table F.7, the F-statistic and the p-value of the F-statistic show that the full set of background characteristics is not able statistically to distinguish between survey respondents and nonrespondents.

Comparison of the Research Groups in the Survey Respondent Sample: Houston

The random assignment design essentially eliminates the possibility of selection bias. However, the risk of differences between the background characteristics of ERA and control group members is reintroduced due to the survey sampling and response process. Specifically, if different types of ERA group members respond (compared with control group members), the integrity of the experiment would be compromised for the survey analysis. If this is true to a large extent, the ability to generalize from the respondent sample may be affected.

Appendix Table F.8 shows whether there may be any statistically significant differences in background characteristics between ERA group and control group respondents in Houston. There are no differences between them. In other words, the integrity of the experiment is preserved.

The Employment Retention and Advancement Project

Appendix Table F.8

Background Characteristics of Survey Respondents, Houston

Variable		ERA Group	Control Group
Female (%)		96.7	95.2
Race/ethnicity (%)	Hispanic	28.7	29.0
	Black	61.3	63.8
	White	9.3	7.2
	Other	0.7	0.0
Age (%)	20 or younger	10.7	10.2
	21-30	54.7	59.2
	31-40	25.3	21.8
	41 or older	9.3	8.8
	Average age (years)	28	28
High school diploma (%)		41.9	42.8
Employed during the quarter prior to random assignment (%)		44.0	47.6
Employed during the year prior to random assignment (%)		70.7	76.9
Number of quarters employed in the prior 3 years (%)		6.1	6.3
Earnings in the 3 years prior to random assignment ($)		13,633	14,392
Number of children (%)	0	0.0	2.1
	1	40.0	34.9
	2	26.0	32.2
	More than 3	34.0	30.8
	Average number of children	2.1	2.1
Age of youngest child (%)	Under 3 years	55.3	60.1
	3-5 years	16.7	19.6
	6 years and older	28.0	20.3
AFDC receipt history (%)	Never	45.5	44.0
	Less than 3 months	10.3	9.9
	3 months or more and less than 2 years	32.4	31.9
	2 years or more and less than 5 years	4.8	9.9
	5 years or more and less than 10 years	4.8	2.1
	10 years or more	2.1	2.1
Received food stamps in prior year (%)		77.3	72.8
Sample size (total = 297)		150	147

SOURCES: Texas background information form and administative records from the State of Texas. NOTES: Results are for sample members and randomly assigned from January through June 2002. Chi-square (categorical) and two-tailed T (continous) tests were used to assess the differences in characteristics across research groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Comparison of Survey Respondents with the Fielded Sample and the Report Sample: Houston

This section examines whether impacts among survey respondents in Houston can be generalized to the fielded sample and the report sample and whether any cohort effects may have been introduced.8

Appendix Table F.9 shows the adjusted means and impacts on employment and welfare outcomes for Houston’s full sample, fielded sample, and respondent sample.9 The pattern of impacts differs across these samples, and so a great deal of caution should be exercised when making inferences from the survey sample to the report sample in Houston. For example, Appendix Table F.9 shows that, among the full report sample, ERA reduced earnings in Year 1 by $94, compared with a $228 increase among the fielded survey sample and a $728 increase among the survey respondent sample. Among the full sample, ERA reduced the percentage working in any given quarter by 1.0 percentage point, compared with a 3.7 percentage point increase among the fielded sample and a 5.0 percentage point increase among the survey respondent sample. Likewise, the ERA and control group differences in the percentage ever employed and food stamp receipt are larger among the respondent sample than among the fielded sample and the report sample. The fact that these differences are not statistically significant, however, may suggest that random variation rather than an actual cohort effect may explain the differences in impacts.

This potential response bias cannot be rectified by weighting based on background characteristics, because differences in measurable characteristics are not systematic. Since the overall impacts on administrative records measures are much stronger among those who were selected for the survey (and stronger still among survey respondents), it is not surprising that the survey impacts look somewhat better in Houston than would be expected from the full sample’s impacts on administrative records outcomes.

Conclusions About Houston

Due to the narrow cohort of months from which Houston’s survey sample was selected –– as well as the different pattern of impacts among the survey respondent sample –– great caution should be exercised when generalizing the results among the survey sample to the broader report sample. While the differences in background characteristics are not large, the impacts are different across subsamples, which tends to make the survey results appear to be more favorable than the full sample’s results.

The Employment Retention and Advancement Project

Appendix Table F.9

Comparison of Impacts for the Full, Fielded, and Respondent Samples in Houston

Outcome		ERA Group	Control Group	Difference (Impact)
Quarters 2-5
Ever employed (%)	Full sample	64.6	63.8	0.8
	Fielded sample	66.8	61.3	5.5
	Respondent sample	67.9	60.0	7.9
Average quarterly employment (%)	Full sample	42.8	43.9	-1.0
	Fielded sample	45.3	41.6	3.7
	Respondent sample	46.1	41.2	5.0
Employed 4 consecutive quarters (%)	Full sample	21.4	23.4	-2.0
	Fielded sample	24.2	21.7	2.5
	Respondent sample	24.7	21.1	3.6
Number of quarters employed	Full sample	1.7	1.8	0.0
	Fielded sample	1.8	1.7	0.1
	Respondent sample	1.8	1.6	0.2
Earnings ($)	Full sample	3,851	3,946	-94
	Fielded sample	3,959	3,730	228
	Respondent sample	4,322	3,594	728
Ever received TANF (%)	Full sample	85.6	84.5	1.1
	Fielded sample	82.3	83.1	-0.8
	Respondent sample	84.4	84.0	0.4
Amount of food stamps received ($)	Full sample	2,766	2,743	24
	Fielded sample	2,824	3,002	-178
	Respondent sample	2,888	3,125	-237
Total measured income ($)	Full sample	7,927	7,897	30
	Fielded sample	8,116	7,984	132
	Respondent sample	8,568	8,003	564

SOURCE: MDRC calculations from administrative records from the State of Texas. NOTES: The full sample includes 1,950 sample members; experimental: 970; control: 980. The fielded sample includes 370 sample members; experimental: 187; control: 183. The respondent sample includes 297 sample members; experimental: 150; control: 147. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

---

 

Table of Contents | Previous | Next