Skip Navigation
acfbanner  
ACF
Department of Health and Human Services Administration for Children and Families
          
ACF Home   |   Services   |   Working with ACF   |   Policy/Planning   |   About ACF   |   ACF News   |   HHS Home

  Questions?  |  Privacy  |  Site Index  |  Contact Us  |  Download Reader™Download Reader  |  Print Print      

Office of Planning, Research & Evaluation (OPRE) skip to primary page content
Advanced
Search

 Table of Contents | Previous | Next

D.4 ASSESSING THE ROBUSTNESS OF STUDY FINDINGS

As discussed in Chapter II, Early Head Start impacts on child, parent, and family outcomes were estimated (1) using regression models to control for baseline differences between the program and control groups; (2) giving equal weight to each site; (3) not using weights to adjust for nonresponse; and (4) using the maximum sample for each outcome variable (that is, using the full sample for whom the outcome variable could be constructed). This appendix addresses the following important question: Are the impact estimates sensitive to alternative estimation strategies, weighting schemes, or sample definitions? To test the robustness of study findings, we also estimated global impacts under the following scenarios:

  1. Using Simple Differences-in-Means Estimation Techniques. Our main estimation approach was to use regression models to estimate program impacts. However, we also estimated impacts by simply comparing the mean outcomes of the program and control groups, and used t-tests to gauge the statistical significance of the estimated impacts.

  2. Using Weights to Adjust for Nonresponse. As discussed in Appendix D.2, we constructed weights to adjust for potential bias in the impact estimates due to interview nonresponse. The use of these weights correctly adjusts for nonresponse using the simple differences-in-means estimation methods. Although there is no theoretical reason to use these weights in a regression context, we did include them in some models to examine how the results would change.

  3. Weighting Each Site by Its Sample Size. Our main approach was to weight each site equally in the analysis regardless of sample size, because the intervention varied substantially across programs and was administered at the site level. However, we also estimated models where sites with larger sample sizes (response rates) were given larger weights than sites with smaller sample sizes (response rates). For these models, we simply pooled all observations across all sites.

  4. Using Alternative Sample Definitions. Our main approach was to estimate impacts using all sample members for whom outcome measures were available. However, we also estimated impacts using alternative sample definitions: those who (1) completed a particular instrument at all three data collection points (which is the sample that was used in the growth curve analysis); (2) completed the 26-month PSI and the particular 36-month birthday-related instrument (so that the impacts on service use and receipt could be directly linked to the impacts on the child, parent, and family outcomes); and (3) completed the 24-month PI and the particular 36-month birthday-related instrument (which is the sample that was used in the mediated analysis).

  5. Dropping Sites with Low Response Rates. We estimated impacts after dropping sample members from three sites with the lowest response rates, because interview respondents in these sites may not be representative of the full samples or respondents and nonrespondents in these sites.

We estimated impacts on 28 key child, parenting, and family outcomes constructed using the 36-month birthday-related instruments and the 26-month PSIs.

Our results indicate that our main global impact findings are very robust to alternative estimation strategies, weighting schemes, and sample definitions (Tables D.4A and D.4B). The regression results are very similar whether or not we use nonresponse weights and whether we weight sites equally or by their sample sizes. Interestingly, the differences-in-means estimates are very similar to the regression ones, because as discussed, the baseline characteristics of interview respondents in the two research groups are similar. The same set of policy conclusions can be drawn using impact results from the alternative sample definitions. Finally, the results do not change substantially when we drop the three sites with the lowest response rates.

In sum, we believe that our impact findings represent real effects and are not due to our methodological assumptions.

TABLE D.4A

IMPACT ESTIMATES PER PARTICIPANT FOR THE FULL SAMPLE ON KEY 36-MONTH OUTCOME VARIABLES USING ALTERNATIVE ESTIMATION AND WEIGHTING STRATEGIES
Variable Regression-Adjusted Estimates Differences-in-Means Estimates
Sites Weighted Equally, No Weights for Nonresponse (Benchmark) Sites Weighted Equally, Weights for Nonresponse Sites Weighted by Sample Size,Weights for Nonresponse Sites Weighted Equally, No Weights for Nonresponse Sites Weighted Equally, Weights for Nonresponse Sites Weighted by Sample Size, Weights for Nonresponse
Bayley Mental Development Index (MDI) Standard Score 1.55** 1.50** 1.54** 1.36** 1.49** 1.40**
Percentage with MDI< 85 -4.69* -4.33* -3.55 -4.27* -3.65 -4.18*
PSI: Parental Distress -0.73 -0.77 -0.64 -0.68 -0.64 -0.7
Center for Epidemiologic Studies Depression Scale (CES-D) Total Score -0.26 -0.2 -0.08 -0.24 -0.07 -0.11
Percentage of Parents Who Spanked the Child in the Previous Week -7.09*** -6.92*** -7.12*** -6.94*** -6.67*** -6.42***
Index of Severity of Discipline Strategies -0.18** -0.16** -0.17** -0.16** -0.16** -0.14*
Percentage of Parents Who Read to Their Child Every Day 4.85** 4.59* 4.46* 4.22* 4.37* 4.37*
Percentage of Parents Suggesting Only Mild Responses to Hypothetical Situations -0.18** -0.16** -0.17** -0.16** -0.16** -0.14**
Home Observation for Measurement of the Environment (HOME): Total Score 0.53** 0.55** 0.58** 0.60** 0.59** 0.57**
HOME: Support of Language and Learning 0.21** 0.22** 0.26*** 0.25** 0.28*** 0.26**
HOME: Warmth 0.08* 0.09** 0.08** 0.08* 0.08* 0.09**
Parent Supportiveness (Semistructured Play) 0.14*** 0.13*** 0.12** 0.12** 0.12** 0.12**
Parent Intrusiveness (Semistructured Play) -0.04 -0.04 -0.04 -0.03 -0.05 -0.04
Parent Detachment (Semistructured Play) -0.06* -0.06* -0.05 -0.05 -0.05 -0.05
Parent Engagement (Semistructured Play) 0.21*** 0.21*** 0.19*** 0.19*** 0.19*** 0.19***
Sustained Attention with Objects (Semistructured Play) 0.16*** 0.15*** 0.16*** 0.15*** 0.16*** 0.15***
Negativity Toward Parent (Semistructured Play) -0.08** -0.08** -0.07** -0.07** -0.07** -0.07**
Persistence (Puzzle Challenge Task) 0.07 0.05 0.07 0.05 0.06 0.04
Child Behavior Checklist: Aggressive Behavior -0.69** -0.65* -0.52 -0.55 -0.44 -0.53
Peabody Picture Vocabulary Test (PPVT-III) Standard Score 2.13** 1.97** 1.65* 2.08** 1.52* 1.96**
Percentage with PPVT <85 -6.02** -6.05** -5.86** -5.22* -5.25* -5.24*
Percentage of Caregivers Ever Employed During the 26 Months After Random Assignment 3.38* 3.24* 2.96* 3.93** 3.29* 3.70**
Percentage of Caregivers Ever in an Education or Training Program During the 26 Months After Random Assignment 8.61*** 8.85*** 8.92*** 9.37*** 9.73*** 10.01***
Average Parent-Reported Health Status of Child -0.02 -0.02 -0.01 -0.01 0 0
Continuous Biological Father Presence Child Age 14 to 36 Months -0.68 -0.39 -1.58 -1.8 -3.01 -1.55
Continuous Male Presence Child Age 14 to 36 Months -3.26 -2.81 -4.06** -4.25** -4.93** -3.98*
SOURCE: PSI and PI data and Bayley and video assessments.

*Significantly different than zero at the .10 level, two-tailed test
**Significantly different than zero at the .05 level, two-tailed test
***Significantly different than zero at the .01 level, two-tailed test


TABLE D.4B

IMPACT ESTIMATES PER PARTICIPANT FOR THE FULL SAMPLE ON KEY 36-MONTH OUTCOME VARIABLES USING ALTERNATIVE SAMPLE DEFINITIONS
Variable Completed the Relevant 36-Month or 26-Month Instrument (Benchmark) Completed the Relevant Instrument at All ThreeData Collection Points (Growth Curve Analysis Sample) Completed the 26-Month PSI as well as the Relevant Instrument Completed the 36-Month PI aswell as the Relevant Instrument (Mediated Analysis Sample) Completed the Relevant Interview and Dropped 3 Sites with the Lowest Response Rates
Bayley Mental Development Index (MDI) Standard Score 1.55** 1.67** 1.69** 1.99*** 1.34*
Percentage with MDI <85 -4.69* -8.45*** -5.96** -6.02** -3.52
PSI: Parental Distress -0.73 -1.23** -0.84 -1.15** -0.77
Center for Epidemiologic Studies Depression Scale (CES-D) Total Score -0.26 -0.2 -0.48 -0.36 -0.52
Percentage of Parents Who Spanked the Child in the Previous Week -7.09*** -8.20*** -7.44*** -7.83*** -8.93***
Index of Severity of Discipline Strategies -0.18** -0.17** -0.16** -0.18** -0.23***
Percentage of Parents Suggesting Only Mild Responses to Hypothetical Situations 4.19* 4.09* 3.69 4.35* 5.92**
Percentage of Parents Who Read to Their Child Every Day 4.85** 6.67** 5.08* 6.20** 4.98*
Home Observation for Measurement of the Environment (HOME): Total Score 0.53** 0.75*** 0.67*** 0.75*** 0.41*
HOME: Support of Language and Learning 0.21** 0.28*** 0.27*** 0.31*** 0.13
HOME: Warmth 0.08* 0.07 0.09** 0.07* 0.07
Parent Supportiveness (Semistructured Play) 0.14*** 0.20*** 0.12** 0.19*** 0.11**
Parent Intrusiveness
(Semistructured Play)		 -0.04 0 -0.04 -0.06 0
Parent Detachment
(Semistructured Play)		 -0.06* -0.08* -0.04 -0.08** -0.03
Parent Engagement
(Semistructured Play)		 0.21*** 0.20*** 0.21*** 0.27*** 0.16**
Sustained Attention with Objects
(Semistructured Play)		 0.16*** 0.12* 0.15*** 0.21*** 0.10*
Negativity Toward Parent
(Semistructured Play)		 -0.08** -0.06 -0.11*** -0.10*** -0.08**
Persistence (Puzzle Challenge Task) 0.07 0 0.05 0.12* -0.05
Child Behavior Checklist: Aggressive Behavior -0.69** -0.61* -0.74** -0.66* -0.87**
Peabody Picture Vocabulary Test (PPVT-III) Standard Score 2.13** 2.78*** 2.21** 2.77*** 2.39**
Percentage with PPVT <85 -6.02** -8.63*** -5.26* -9.27*** -5.52*
Percentage of Caregivers Ever Employed During the 26 Months After Random Assignment 3.38* 2.77 3.38* 3.38* 2.63
Percentage of Caregivers Ever in an Education or Training Program During the 26 Months After Random Assignment 8.61*** 9.22*** 8.61*** 8.61*** 7.36***
Average Parent-Reported Health Status of Child -0.02 -0.01 -0.01 0 0
Continuous Biological Father Presence Child Age 14 to 36 Months -0.68 -0.68 0.83 -0.68 -0.3
Continuous Male Presence Child Age 14 to 36 Months -3.26 -3.26 -1.72 -3.26 -3.72
SOURCE: PSI and PI data and Bayley and video assessments.

NOTE: All estimates were calculated using regression models where each site was weighted equally and where weights for nonresponse were not used.

*Significantly different than zero at the .10 level, two-tailed test
**Significantly different than zero at the .05 level, two-tailed test
***Significantly different than zero at the .01 level, two-tailed test



 

 

 Table of Contents | Previous | Next