Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

D.5 ASSESSING THE ROBUSTNESS OF STUDY FINDINGS

As discussed in Chapter II, short-term Early Head Start impacts on child, parent, and family outcomes were estimated (1) using regression models to control for baseline differences between the program and control groups; (2) giving equal weight to each site; (3) not using weights to adjust for nonresponse; and (4) using the maximum sample for each outcome variable (that is, using the full sample for whom the outcome variable could be constructed). This appendix addresses the following important question: Are the impact estimates sensitive to alternative estimation strategies, weighting schemes, or sample definitions?

To test the robustness of study findings, we also estimated global impacts under the following scenarios:

1. Using Simple Differences-in-Means Estimation Techniques. Our main estimation approach was to use regression models to estimate program impacts. However, we also estimated impacts by simply comparing the mean outcomes of the program and control groups, and used t-tests to gauge the statistical significance of the estimated impacts.

2. Using Weights to Adjust for Nonresponse. As discussed in Appendix D.2, we constructed weights to adjust for potential bias in the impact estimates due to interview nonresponse. The use of these weights correctly adjusts for nonresponse using the simple differences-in-means estimation methods. Although there is no theoretical reason to use these weights in a regression context, we did include them in some models to examine how the results would change.

3. Weighting Each Site by Its Sample Size. Our main approach was to weight each site equally in the analysis regardless of sample size, because the intervention varied substantially across programs and was administered at the site level. However, we also estimated models where sites with larger sample sizes (response rates) were given larger weights than sites with smaller sample sizes (response rates). For these models, we simply pooled all observations across all sites.

4. Using Alternative Sample Definitions. Our main approach was to estimate impacts using all sample members for whom outcome measures were available. However, we also estimated impacts using alternative sample definitions: those who (1) completed a particular instrument at both data collection points (which is the sample that would be used in a growth curve analysis); (2) completed the 15-month PSI and the particular birthday-related instrument (so that the impacts on service use and receipt could be directly linked to the impacts on the child, parent, and family outcomes); and (3) completed all interviews and assessments at both data collection points.

5. Dropping Sites with Low Response Rates. We estimated impacts after dropping sample members from 4 sites with the lowest response rates, because interview respondents in these sites may not be representative of the full samples or respondents and nonrespondents in these sites.

We estimated impacts on 17 key child, parenting, and family outcomes constructed using the 24-month birthday-related instruments and the 15-month PSIs.

Our results indicate that our main global impact findings are very robust to alternative estimation strategies, weighting schemes, and sample definitions (Tables D.5A and D.5B). The regression results are very similar whether or not we use nonresponse weights and whether we weight sites equally or by their sample sizes. Interestingly, the differences-in-means estimates are very similar to the regression ones, because as discussed, the baseline characteristics of interview respondents in the two research groups are similar. The same set of policy conclusions can be drawn using impact results from the alternative sample definitions. Finally, the results do not change substantially when we drop the four sites with the lowest response rates.

In sum, we believe that our interim impact findings represent real effects and are not due to our methodological assumptions.

TABLE D.5A
IMPACT ESTIMATES PER PARTICIPANT FOR THE FULL SAMPLE ON KEY OUTCOME VARIABLES USING ALTERNATIVE ESTIMATION AND WEIGHTING STRATEGIES

	Regression-Adjusted Estimates			Differences-in-Means Estimates
Variable	Sites Weighted Equally, No Weights for Nonresponse (Benchmark)	Sites Weighted Equally, Weights for Nonresponse	Sites Weighted by Sample Size, Weights for Nonresponse	Sites Weighted Equally, No Weights for Nonresponse	Sites Weighted Equally, Weights for Nonresponse	Sites Weighted by Sample Size, Weights for Nonresponse
Bayley Mental Development Index (MDI) at 24 Months of Age	2.01***	2.14***	1.95***	2.04***	2.33***	2.05***
Percent with Bayley MDI Below 85 at 24 Months	-6.58**	-7.24***	-5.07**	-6.39**	-7.45***	-5.62**
Vocabulary Production at 24 Months	2.42**	2.77**	3.08***	2.44*	2.70**	2.95***
Sentence Complexity Score at 24 Months	0.93**	0.91**	0.97**	0.95**	0.93**	0.95**
Aggressive Behavior Problemsat 24 Months (CBCL)	-1.26**	-1.24**	-1.06**	-1.21**	-1.22**	-1.11**
Engagement of Parent at 24 Months (Three Bag)	0.09	0.11	0.12**	0.09	0.09	0.12**
Negativity Towards Parent at 24 Months (Three Bag)	-0.08	-0.06	-0.05	-0.05	-0.03	-0.04
Sustained Attention with Objects at 24 Months (Three Bag)	0.06	0.08	0.09*	0.05	0.07	0.08
Parental Supportiveness at 24 Months (Three Bag)	0.14**	0.13**	0.14***	0.15***	0.14**	0.15***
Parental Detachment at 24 Months (Three Bag)	-0.10*	-0.10*	-0.11**	-0.09*	-0.09*	-0.11**
Knowledge of Infant Development Inventory (KIDI) at 24 Months	0.05***	-0.06***	-0.05***	-0.06***	0.06***	0.06***
Family Environment Scale: Family Conflict at 24 Months	-0.06**	-0.06*	0.04	0.07**	-0.06**	-0.05
Support of Cognitive, Language, and Literary Environment (HOME) at 24 Months	0.20***	0.21***	0.21***	0.23***	0.24***	0.24***
Parenting Stress Index at 24 Months	-0.96**	-1.05**	-1.02**	-1.00**	-1.04**	-1.08**
Percentage of Children with Poor or Fair Health at 24 Months	-0.62	-0.25	-1.22	-0.26	-0.15	-1.09
Percentage of Caregivers Ever in an Education or Training Program During the 15 Months After Random Assignment	4.68**	4.89**	5.21**	5.29**	-5.84**	6.03**
Average Hours Per Week Caregivers Were in Education or Training During the 15 Months After Random Assignment	1.13***	1.12***	1.16***	1.26***	1.28***	1.32***

SOURCE: PSI and PI data and Bayley and video assessments.

*Significantly different than zero at the .10 level, two-tailed test
**Significantly different than zero at the .05 level, two-tailed test
***Significantly different than zero at the .01 level, two-tailed test

 

TABLE D.5B
IMPACT ESTIMATES PER PARTICIPANT FOR THE FULL SAMPLE ON KEY OUTCOME VARIABLES USING ALTERNATIVE SAMPLE DEFINITIONS

Variable	Completed the Relevant 24-Month or 15-Month Instrument (Benchmark)	Completed the Relevant Instrument at Both Data Collection Points	Completed the 15-Month PSI as well as the Relevant Instrument	Completed AllInstruments	Completed the Relevant Interview and Dropped 4 Sites with the Lowest Response Rates
Bayley Mental Development Index (MDI) at 24 Months of Age	2.01***	1.85**	1.76**	1.83*	1.64**
Percent with Bayley MDI Below 85 at 24 Months	-6.58**	-6.33**	-7.07***	-7.50**	-3.94
Vocabulary Production at 24 Months	2.42**	2.48**	1.97	1.33	2.23*
Sentence Complexity Score at 24 Months	0.93**	1.01**	0.92*	1.48**	0.97**
Aggressive Behavior Problems At 24 Months (CBCL)	-1.26**	-1.31**	1.49**	-2.06***	-1.21**
Engagement of Parent at 24 Months (Three Bag)	0.09	0.13*	0.11*	0.12	0.18***
Negativity Towards Parent at 24 Months (Three Bag)	-0.08	-0.09	-0.05	-0.05	-0.11**
Sustained Attention with Objects at 24 Months (Three Bag)	0.06	0.07	0.07	0.01	0.08
Parental Supportiveness at 24 Months (Three Bag)	0.14**	0.19***	0.15***	0.17**	0.15***
Parental Detachment at 24 Months (Three Bag)	-0.10*	-0.11*	-0.12**	-0.11*	-0.12**
Knowledge of Infant Development Inventory (KIDI) at 24 Months	0.05***	0.04**	0.05**	0.05	0.05***
Family Environment Scale: Family Conflict at 24 Months	-0.06**	-0.06*	-0.06*	-0.06	-0.02
Support of Cognitive, Language,and Literary Environment (HOME) at 24 Months	0.20***	0.18**	0.17**	0.14	0.18**
Parenting Stress Index at 24 Months	-0.96**	-1.22**	-0.81	-1.33*	-0.95*
Percentage of Children with Poor or Fair Health at 24 Months	-0.62	-0.55	0.10	-0.22	-2.52
Percentage of Caregivers Ever in an Education or Training Program During the 15 Months After Random Assignment	4.68**	5.88***	4.68**	7.85**	4.76*
Average Hours Per Week Caregivers Were in Education or Training During the 15 Months After Random Assignment	1.13***	1.19***	1.13***	1.96***	1.43***

SOURCE: PSI and PI data and Bayley and video assessments.

NOTE: All estimates were calculated using regression models where each site was weighted equally and where weights for
nonresponse were not used.

*Significantly different than zero at the .10 level, two-tailed test
**Significantly different than zero at the .05 level, two-tailed test
***Significantly different than zero at the .01 level, two-tailed test

 

 

Table of Contents | Previous | Next