Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

D.2 ASSESSING AND CORRECTING FOR THE EFFECTS OF NONRESPONSE TO THE EARLY HEAD START INTERVIEWS AND ASSESSMENTS

In the previous section, we examined the baseline characteristics of program and control group members in the full analysis sample and concluded that they were similar. However, as discussed in Chapter II, not all sample members completed the follow-up interviews and assessments. The response rate was about 75 percent to the 15-month parent services interview (PSI), 70 percent to the 24-month birthday-related parent interview (PI), and 58 percent to the Bayley and video assessments. Furthermore, response rates differed somewhat across sites and subgroups defined by site and family characteristics at baseline. Thus, it was important to test whether program group members who responded to the interviews are fully representative of all program group members, and whether control group members who responded to the interviews are fully representative of all control group members. Furthermore, it was important to test whether the baseline characteristics of respondents in the two research groups differ from each other.

If not corrected, the effects of interview nonresponse could lead to two problems:

1. The impact estimates could be biased. This would occur if the differences in the average baseline characteristics of respondents in the program and control groups were correlated with the outcome variables, and hence, the impact estimates.

2. The impact estimates might not be generalizable to the study population of eligible families. This would occur if the differences between interview respondents and nonrespondents were correlated with the outcome variables (regardless of whether or not the average characteristics of program group and control group respondents were similar).

In this appendix, we assess the effects of nonresponse and discuss procedures that we used to adjust for potential nonresponse effects.

1. Assessing the Effects of Nonresponse

Our basic approach for assessing the effects of nonresponse to key data sources was to compare the baseline characteristics of (1) respondents in the program and control groups, and (2) respondents to the full sample of respondents and nonrespondents in each research group. We conducted this analysis using data from the HSFIS application and enrollment forms, and with the same methods that we used to compare the baseline characteristics of the full program and control groups (see Appendix D.1). To keep the presentation manageable, we focus our analysis on the 15-month PSIs and the 24-month birthday-related interviews and assessments.

Tables D.2A to D.2D display the following results from the nonresponse analysis, with separate tables displayed for each data source:

1. Variable distributions for interview respondents, by research status
   
2. Significance levels for testing differences between the characteristics of respondents in the program and control groups.
   
3. Variable distributions for the full sample of respondents and nonrespondents, by research status
   
4. Significance levels for testing differences between respondents and the full sample of respondents and nonrespondents, by research status

We find some differences in the characteristics of respondents and the full sample of respondents and nonrespondents for each research group and data source. Response rates for the program group were higher in center-based programs than in home-based or mixed-approach programs, and response rates for both research groups were higher in "fully implemented" programs than in programs that were not fully implemented. Response rates increased with the education level of the primary caregiver. In addition, they were higher if the primary caregiver (1) was employed at the time of random assignment, (2) was married or living with other adults, and (3) spoke English as the primary language at home. Response rates were also slightly larger for whites than for African Americans and Hispanics, for older mothers than younger ones, and for families not receiving welfare than for those receiving it. The p-values to test the hypotheses that variable means and distributions are jointly similar are less than .01 for all data sources and for both research groups. These results suggest that program group respondents are not fully representative of the full program group, and control group respondents not fully representative of the full control group.

However, we find fewer differences between the baseline characteristics of program and control group respondents. Very few of the differences in the distributions of the baseline variables for respondents in the two research groups are statistically significant. For example, the program and control group differences are statistically significant at the 10 percent level for only 6 of the 48 univariate tests for the 24-month Bayley assessment (which is close to the approximately 5 tests that would be expected by chance). Similarly, only 6 of the tests for the 24-month video assessment, 9 for the 24-month PI, and 3 for the 15-month PSI are statistically significant at the 10 percent level. Furthermore, none of the joint tests from the multivariate regression models is statistically significant at the 10 percent level. Finally, very few univariate tests for key variables are rejected at the 10 percent level across the sites, and the significant test statistics are scattered across sites and variables (not shown).

In sum, we find some differences in the characteristics of respondents and nonrespondents, but these differences are not large and in many instances are present for both the program and the control groups. Consequently, the characteristics of respondents in the two research groups are similar, which suggests that our impact estimates are likely to be unbiased.

2. Adjusting for the Effects of Nonresponse

As discussed in Chapter II of this report, the main approach we used to adjust for observed differences between program and control group respondents was to estimate program impacts using regression models. In these models, we regressed outcome variables on a program status indicator variable and a large number of explanatory variables. The explanatory variables were constructed using HSFIS data and pertain to the characteristics of families and children at baseline. An important criterion that we used to select the explanatory variables was that they should capture differences between the characteristics of respondents in the two research groups. Furthermore, to adjust for differences in response rates across sites, we assigned equal weight to each site in the analysis.

We believe that our regression approach produced unbiased estimated impacts because there were not large differences between respondents in the two research groups, and because the regression models controlled for some of these differences. However, the regression procedure does not correct for differences between respondents and nonrespondents in each research group, so the estimated impacts may not be generalizable to the full study population.

To address this problem, we constructed sample weights so that the weighted observable baseline characteristics of respondents were similar to the baseline characteristics of the full sample of respondents and nonrespondents. For each survey instrument, we constructed separate weights for program and control group members using the following three steps:

1. We estimated a logit model predicting interview response. The binary variable indicating whether or not a family was a respondent to the instrument was regressed on the full set of HSFIS variables used in the nonresponse analysis discussed above, as well as site indicator variables. Only HSFIS variables that were statistically significant predictors of response status were retained as explanatory variables in the models.1

2. We calculated a propensity score for each family in the full sample. We constructed this score, the predicted probability that a family was a respondent, using the parameter estimates from the logit regression model and the family's HSFIS characteristics. Families with large propensity scores were likely to be respondents, whereas families with small propensity scores were likely to be nonrespondents.

3. We constructed nonresponse weights using the propensity scores. Families were ranked by the size of their propensity scores and were divided into six groups of equal size. The weight for a family was inversely proportional to the mean propensity score of the group the family was assigned to.2

This propensity score procedure yielded large weights for families with characteristics that were associated with low response rates (that is, for those with small propensity scores). Similarly, the procedure yielded small weights for families with characteristics that were associated with high response rates. Thus, the weighted characteristics of respondents were similar, on average, to the characteristics of the entire research sample.

As discussed in Chapter II, our main procedure was not to include these weights in the regression models when estimating impacts per eligible applicant and per participant. The use of these weights correctly adjusts for nonresponse bias when impacts are estimated with a simple differences-in-means estimation approach. However, using weights does not correctly adjust for nonresponse bias in a regression context, because the regression-adjusted impact estimates are not weighted correctly (DuMouchel and Duncan 1983).

To check the robustness of study findings, however, we did estimate the regression models using the sample weights (see Appendix D.5). In addition, we used weights when estimating impacts using a simple differences-in-means approach (see Appendix D.5). These differences-in-means impact estimates should be unbiased and generalizable to the study population (although they are less precise than the regression-adjusted impact estimates). We inflated the standard errors of the weighted impact estimates to account for design effects due to weighting.

It is important to note that the use of weights and regression models adjusts only for observable differences between survey respondents and nonrespondents in the two research groups. The procedure does not adjust for potential unobservable differences between the groups. Thus, our procedures only partially adjust for potential nonresponse bias.

 

TABLE D.2A
COMPARISON OF THE BASELINE CHARACTERISTICS OF RESPONDENTS AND THE FULL SAMPLE OF RESPONDENTS AND NONRESPONDENTS TO THE
15-MONTH PSI, BY RESEARCH STATUS

	Respondents		Respondents and Nonrespondents
Variable	Program Group	Control Groupa	Program Groupb	Control Groupc
Site Characteristics
Program Approach
Center-based	20.5	18.6	20.2	20.6*
Home-based	47.1	47.6	46.7	45.6
Mixed	32.3	33.8	33.0	33.9
Overall Implementation Level
Full and early	34.2	34.1	34.5*	34.8*
Full but late	37.7	36.9	35.0	35.1
Never	28.1	29.0	30.5	30.0
Family and Parent Characteristics
Age of Mother at Birth of Focus Child
Younger than 20	38.4	40.1	39.0	39.5
20 to 25	33.0	32.3	33.2	32.0
25 or older	28.6	27.6	27.9	28.5
Mother Was Younger than 19 at First Birth	41.8	41.0	42.9	41.2
Highest Grade Completed
Less than 12	44.8	47.3	47.7*	47.8*
12 or earned a GED	28.8	28.7	27.3	29.8
More than 12	26.4	24.0	24.9	22.4
Race and Ethnicity
White non-Hispanic	38.2	37.8	37.3	37.1
Black non-Hispanic	34.3	35.1	34.2	35.0
Hispanic	23.3	22.2	23.8	23.4
Other (Asian or Pacific Islander,    American Indian, Eskimo, Aleut)	4.2	5.0	4.7	4.5
Primary Occupation
Employed	22.8	24.5	22.9	23.8
In school or a training program	22.2	21.8	22.0	21.4
Other	55.0	53.7	55.0	54.7
English Language Ability
Primary language is English	81.1	79.1	79.9*	78.1
Primary language is not English but the    applicant speaks English well	8.7	9.9	9.6	10.3
Primary language is not English and the    applicant does not speak English well	10.2	11.0	10.5	11.6
Living Arrangements
Living with a spouse	25.1	26.2	24.9	25.4*
Living with other adults	38.5	40.3	38.3	39.1
Living with no other adults	36.4	33.6	36.8	5.5
Adult Male Present in the Household	38.5	40.3	38.1	39.1
Number of Adults in the Householdd
1	37.6	34.4	37.8	36.6*
2	50.0	51.9	49.8	50.8
3 or more	12.4	13.7	12.4	12.6
Number of Children Less than 5 Years Old in the Household Other than the Focus Child
0	64.9	64.8	64.3	65.1
1	26.3	27.0	27.0	26.8
2 or more	8.9	8.2	8.7	8.1
Number of Children Between 6 and 17 in the Household
0	64.0	65.4	64.3	66.4
1	22.9	21.9	23.1	21.3
2 or more	13.1	12.7	12.6	12.3
Number of Moves in the Past Year
0	52.1	51.3	49.5*	49.8*
1	27.8	28.5	28.9	28.1
2 or more	20.1	20.2	21.6	22.1
Owns Home	12.8	11.8	11.0*	11.1
Household Income as a Percent of the Poverty Level (Percent)
Less than 33	29.2	29.7	30.2	30.0
33 to 67	32.3	29.1	32.5	29.2
67 to 99	24.9	26.7	24.0	26.5
100 or more	13.6	14.6	13.3	14.3
Welfare Receipt
AFDC/TANFe	35.8	34.0	35.6	34.7
Food Stamps	47.7	48.1	48.0	47.8
Medicaid	76.5	76.0	76.6	74.7*
SSI	6.8	7.7	7.0	7.0
WIC	87.7	86.0	87.5	85.9
Public housing	10.0	9.0	9.5	8.9
Has Inadequate Resources
Food	4.2	6.6*	4.9*	6.3
Housing	11.5	11.7	12.3	13.3*
Money to buy necessities	19.4	20.4	20.8*	21.7*
Medical care	13.6	13.2	14.0	14.7*
Transportation	20.5	22.0	20.9	22.4
Child care	33.1	32.6	34.4*	34.6*
Money for supplies	26.6	29.2	27.1	29.4
Support from friends	12.1	11.8	12.9	14.0*
Parent information	12.4	16.0*	12.5	16.3
Maternal Risk Indexf
0 or 1 (low risk)	19.7	18.0	18.8	17.3
2 or 3 (moderate risk)	54.5	55.4	54.2	56.4
4 or 5 (high risk)	25.9	26.6	27.1	26.3
Random Assignment Date
Before 10/96	36.8	37.5	36.0	36.5*
10/96 to 6/97	30.3	32.7	30.2	30.8
After 6/97	32.9	29.8	33.8	32.7
Previously Enrolled in Head Start orAnother Childhood Development Programe	12.8	13.5	12.8	13.4
Characteristics of Focus Child
Age (Months)
Unborn	25.6	27.5	24.2*	26.5
Less than 5	35.5	33.7	36.1	34.7
5 or more	38.9	38.7	39.7	38.7
Male	50.6	49.3	51.7	50.4
First Born	62.4	62.7	62.3	62.8
Birthweight Less than 2,500 Gramse	9.3	7.9	9.9	8.4
Born more than 3 Weeks Earlye	15.1	12.4	15.8	12.0
Stayed in Hospital After Birthe	17.6	16.4	18.3	16.0
People Concerned About the Child's Overall Health and Developmente	12.3	13.7	13.0	13.3
Received an Evaluation Because of Concerns About the Child's Overall Health and Development or Because of Suspected Developmental Delaye	5.6	7.6	6.0	6.9
Risk Categories
Has established riskse	11.2	10.4	11.6	10.6
Has biological or medical riskse	17.2	17.4	18.3	16.8
Has environmental riskse	29.6	36.4*	32.5*	36.4
Covered by Health Insurancee	90.6	91.6	90.1	89.6*
Sample Size	1,139	1,097	1,513	1,488
SOURCE: HSFIS application and enrollment forms and 15-month PSI data.

 

TABLE D.2B
COMPARISON OF THE BASELINE CHARACTERISTICS OF RESPONDENTS AND THE FULL SAMPLE OF RESPONDENTS AND NONRESPONDENTS TO THE 24-MONTH PI, BY RESEARCH STATUS

	Respondents		Respondents and Nonrespondents
Variable	Program Group	Control Groupa	Program Groupb	Control Groupc
Site Characteristics
Program Approach
Center-based	22.0	19.9	20.2*	20.6
Home-based	45.8	45.6	46.7	45.6
Mixed	32.2	34.5	33.0	33.9
Overall Implementation Level
Full and early	34.9	34.5	34.5*	34.8*
Full but late	38.2	38.3	35.0	35.1
Never	26.9	27.2	30.5	30.0
Family and Parent Characteristics
Age of Mother at Birth of Focus Child
Younger than 20	37.1	39.0	39.0*	39.5
20 to 25	33.8	32.2	33.2	32.0
25 or older	29.0	28.8	27.9	28.5
Mother Was Younger than 19 at First Birth	41.6	40.0	42.9*	41.2
Highest Grade Completed
Less than 12	44.9	46.1	47.7*	47.8*
12 or earned a GED	28.5	28.6	27.3	29.8
More than 12	26.6	25.3	24.9	22.4
Race and Ethnicity
White non-Hispanic	38.8	40.3	37.3	37.1*
Black non-Hispanic	33.6	32.4	34.2	35.0
Hispanic	23.5	22.6	23.8	23.4
Other (Asian or Pacific Islander,    American Indian, Eskimo, Aleut)	4.1	4.7	4.7	4.5
Primary Occupation
Employed	25.4	24.9	22.9*	23.8
In school or a training program	22.1	20.7	22.0	21.4
Other	52.5	54.4	55.0	54.7
English Language Ability
Primary language is English	81.3	78.6	79.9*	78.1
Primary language is not English but the    applicant speaks English well	8.2	10.2	9.6	10.3
Primary language is not English and the    applicant does not speak English well	10.6	11.2	10.5	11.6
Living Arrangements
Living with a spouse	25.2	28.4*	24.9	25.4*
Living with other adults	38.0	40.1	38.3	39.1
Living with no other adults	36.8	31.5	36.8	35.5
Adult Male Present in the Household	38.4	42.1*	38.1	39.1*
Number of Adults in the Householdd
1	37.9	32.2*	37.8	36.6*
2	49.7	53.9	49.8	50.8
3 or more	12.4	13.9	12.4	12.6
Number of Children Less than 5 Years Old in the Household Other than the Focus Child
0	64.9	63.7	64.3	65.1
1	26.6	27.5	27.0	26.8
2 or more	8.4	8.8	8.7	8.1
Number of Children Between 6 and 17 in the Household
0	64.7	66.4	64.3	66.4
1	22.5	20.4	23.1	21.3
2 or more	12.8	13.2	12.6	12.3
Number of Moves in the Past Year
0	51.5	50.1	49.5*	49.8
1	28.2	28.7	28.9	28.1
2 or more	20.3	21.2	21.6	22.1
Owns Home	12.6	11.7	11.0*	11.1
Household Income as a Percent of the Poverty Level (Percent)
Less than 33	28.0	28.5	30.2*	30.0
33 to 67	33.0	29.8	32.5	29.2
67 to 99	24.7	27.7	24.0	26.5
100 or more	14.3	13.9	13.3	14.3
Welfare Receipt
AFDC/TANFe	33.5	32.4	35.6*	34.7*
Food Stamps	46.6	46.2	48.0*	47.8*
Medicaid	76.0	73.6	76.6	74.7
SSI	6.8	7.0	7.0	7.0
WIC	88.0	86.1	87.5	85.9
Public housing	10.2	8.7	9.5	8.9
Has Inadequate Resources
Food	4.9	6.6*	4.9	6.3
Housing	12.4	11.4	12.3	13.3*
Money to buy necessities	19.6	20.9	20.8*	21.7
Medical care	12.8	14.1	14.0*	14.7
Transportation	20.1	22.1	20.9	22.4
Child care	32.9	33.6	34.4*	34.6
Money for supplies	25.0	29.8*	27.1*	29.4
Support from friends	12.7	11.5	12.9	14.0*
Parent information	12.5	15.4*	12.5	16.3
Maternal Risk Indexf
0 or 1 (low risk)	20.9	19.0	18.8*	17.3*
2 or 3 (moderate risk)	54.5	56.2	54.2	56.4
4 or 5 (high risk)	24.6	24.8	27.1	26.3
Random Assignment Date
Before 10/96	35.8	35.3	36.0*	36.5
10/96 to 6/97	28.5	31.4	30.2	30.8
After 6/97	35.7	33.3	33.8	32.7
Previously Enrolled in Head Start or Another Childhood Development Programe	12.6	13.7	12.8	13.4
Characteristics of Focus Child
Age (Months)
Unborn	24.2	25.9	24.2*	26.5
Less than 5	34.0	33.9	36.1	34.7
5 or more	41.8	40.3	39.7	38.7
Male	51.5	50.2	51.7	50.4
First Born	62.0	61.3	62.3	62.8*
Birthweight Less than 2,500 Gramse	9.2	7.8	9.9	8.4
Born more than 3 Weeks Earlye	14.9	12.6	15.8	12.0
Stayed in Hospital After Birthe	17.4	17.2	18.3	16.0
People Concerned About the Child's Overall Health and Developmente	12.5	15.5*	13.0	13.3*
Received an Evaluation Because of Concerns About the Child's Overall Health and Development or Because of Suspected Developmental Delaye	6.0	8.1	6.0	6.9*
Risk Categories
Has established riskse	11.8	10.7	11.6	10.6
Has biological or medical riskse	18.5	18.3	18.3	16.8*
Has environmental riskse	32.1	36.5*	32.5	36.4
Covered by Health Insurancee	91.3	91.1	90.1*	89.6*
Sample Size	1,092	1,021	1,513	1,488
SOURCE: HSFIS application and enrollment forms and 24-month PI data.

 

TABLE D.2C
COMPARISON OF THE BASELINE CHARACTERISTICS OF RESPONDENTS AND THE FULL SAMPLE OF RESPONDENTS AND NONRESPONDENTS TO THE 24-MONTH BAYLEY ASSESSMENT, BY RESEARCH STATUS

	Respondents		Respondents and Nonrespondents
Variable	Program Group	Control Groupa	Program Groupb	Control Groupc
Site Characteristics
Program Approach
Center-based	22.3	19.9	20.2*	20.6
Home-based	47.0	46.6	46.7	45.6
Mixed	30.7	33.5	33.0	33.9
Overall Implementation Level
Full and early	36.0