Table of Contents | Previous | Next |
Appendix 5.1: Cognitive Domain, Estimated Impact on Program Participants
This appendix presents the impact estimates that are adjusted for the fact that some of the children assigned to Head Start failed to take advantage of this opportunity, i.e., they never participated in Head Start (the “no-shows”) and that some children assigned to the non-Head Start group managed to find their way into the program (the “crossovers”). The impact of admission into the program measures what grantees have the power to do—provide access to the program. However, the question of how much children gain from actually participating in Head Start remains an important one.
These results are summarized in Exhibit A.5.1.1 and A.5.1.21 (for children from the 3- and 4-year-old groups, respectively) focusing only on the statistically significant impact estimates, both overall and for the subgroup/moderator analyses. The first three columns of figures provide the impact estimates, and the last three columns provide the effect sizes associated with each estimate. Three different impact estimates are provided in the tables: (1) the impact of access to Head Start (the intent-to-treat estimates discussed in Chapter 5); (2) the impact of Head Start participation, adjusting only for the occurrence of “no shows”; and, (3) the impact of Head Start participation, adjusting for the occurrence of both “no shows” and “crossovers” (these are shown only for the overall average impacts). The latter two estimates, impacts on those who actually receive Head Start services, are often called “impact on the treated.”
The last three columns express each of the respective estimates as “effect sizes,” which are defined as the impact estimates divided by the standard deviation of the outcome measure in the population, providing a “yardstick” for gauging the quantitative importance of a measure impact in relation to the natural variation of the child or family outcome Head Start is seeking to effect.2 Effect sizes are important in interpreting the size of Head Start’s measured impact and, in particular, how much larger that impact may be for the average program participant as opposed to the larger group of children and families accorded access to the program.
As can be seen, all measures of the impact of participation in Head Start exceed the corresponding measures of the average impact of access to Head Start shown in the first column, regardless of which adjustment method is used. For example, the first row of Exhibit A.5.1.1 shows an estimated impact of 4.23 scale points on the Peabody Picture Vocabulary Test (PPVT) for the average child in the 3-year-old group granted access to the program through assignment to the Head Start research sample and a larger average impact of 4.50-4.79 scale points for those children who actually attended Head Start. These same increases are seen in the relationship between the calculated effect sizes for the estimates of “access” vs. “participation,” with some instances of the size of the effect moving from “small” (under 0.2) to “modest (between 0.2 and 0.5).
These tables also show the difference in the magnitude of the adjustment associated with the no-show correction alone and the combined no-show and crossover adjustment. The differences between the intent-to-treat estimates and the no-show/crossover adjusted estimates are, in most cases, larger than the differences shown using just the no-show correction alone. This is a clear indication of the dampening effect of the crossover children in the non-Head Start group, i.e., the fact that these children were able to receive some Head Start program services reduces the size of any observed difference in outcomes between the Head Start and non-Head Start groups. However, although these data represent our best estimate of this phenomenon, a more thorough analysis is needed and will be addressed in future reports. In the meantime, crossover-adjusted estimates have only been calculated for the overall average effects. Consequently, the estimates incorporating the combined no-show/crossover adjustment should only be used as a rough indication of the likely consequences of the presence of crossovers on the interpretation of the impact of Head Start on children’s school readiness.
| Outcome Measure | Impact Estimates | Effect Sizes | |||||
|---|---|---|---|---|---|---|---|
| Impact of Access to Head Start | Impact of Head Start Participation, No-Show Adjustment | Impact of Head Start Participation, Combined No-Show and Crossover Adjustment | Impact of Access to Head Start | Impact of Head Start Participation, No-Show Adjustment | Impact of Head Start Participation, Combined No-Show and Crossover Adjustment | ||
| Overall Impact | PPVT-III | 4.23* | 4.79* | 4.50* | 0.12 | 0.14 | 0.13 |
| WJ-III Letter-Word Identification-IRT/ML | 5.65** | 6.40** | 8.05*** | 0.24 | 0.27 | 0.34 | |
| Letter Naming Task | 1.30* | 1.47* | 2.05*** | 0.19 | 0.22 | 0.31 | |
| Color Naming/Identification | 0.70* | 0.79* | 0.77 | 0.10 | 0.11 | 0.11 | |
| McCarthy Drawing | 0.15* | 0.17* | 0.23** | 0.13 | 0.15 | 0.20 | |
| PELS | 0.47*** | 0.53*** | 0.65*** | 0.34 | 0.38 | 0.47 | |
| Difference in Impact 1 | PPVT-III: Depression | -0.11* | -0.12* | N/A | - 0.00 | - 0.00 | N/A |
| CTOPPP Elision: Depression | -0.13* | -0.15* | N/A | - 0.00 | - 0.00 | N/A | |
| WJ-III Applied Problems: Depression | -0.09* | -0.10* | N/A | - 0.00 | - 0.00 | N/A | |
| WJ-III Oral Comprehension: Race (White Impact Exceeds African American) | 4.73* | 5.36* | N/A | 0.33 | 0.38 | N/A | |
| Color Naming/Identification: Depression | -0.02* | -0.02* | N/A | - 0.00 | - 0.00 | N/A | |
| Counting Bears: Depression | -0.00* | -0.00* | N/A | - 0.00 | - 0.00 | N/A | |
| PELS: Depression | -0.00* | -0.00* | N/A | - 0.00 | - 0.00 | N/A | |
| Impact on Subgroup 2 | PPVT-III: Parent Married | 4.51* | 5.20* | N/A | 0.13 | 0.15 | N/A |
| PPVT-III: Spanish-English Language Group | 7.52* | 8.54* | N/A | 0.21 | 0.24 | N/A | |
| PPVT-III: Hispanic | 7.26* | 8.24* | N/A | 0.21 | 0.23 | N/A | |
| CTOPPP Elision: African American | 7.47* | 8.40* | N/A | 0.17 | 0.18 | N/A | |
| WJ-III Letter-Word Identification: No Special Needs | 5.38** | 6.01** | N/A | 0.22 | 0.25 | N/A | |
| WJ-III Letter-Word Identification: African American | 5.80** | 6.52** | N/A | 0.24 | 0.27 | N/A | |
| WJ-III Letter-Word Identification: Hispanic | 6.92* | 7.85* | N/A | 0.29 | 0.33 | N/A | |
| WJ-III Letter-Word Identification: English-English Language Group | 5.05*** | 5.72*** | N/A | 0.21 | 0.24 | N/A | |
| WJ-III Letter-Word Identification: Parent Married | 6.53** | 7.52** | N/A | 0.27 | 0.31 | N/A | |
| WJ-III Letter-Word Identification: Parent Not Married | 5.21** | 5.82** | N/A | 0.22 | 0.24 | N/A | |
| WJ-III Spelling: Hispanic | 5.61* | 6.37* | N/A | 0.25 | 0.28 | N/A | |
| Letter Naming Task: No Special Needs | 1.24* | 1.41* | N/A | 0.19 | 0.21 | N/A | |
| Letter Naming Task: Hispanic | 1.45* | 1.65* | N/A | 0.22 | 0.25 | N/A | |
| Letter Naming Task: Parent Not Married | 1.46* | 1.63* | N/A | 0.22 | 0.25 | N/A | |
| WJ-III Oral Comprehension: White | 2.82** | 3.24** | N/A | 0.20 | 0.23 | N/A | |
| WJ-III Oral Comprehension: Parent Married | 2.09* | 2.41* | N/A | 0.15 | 0.17 | N/A | |
| Color Identification: English - English Language Group | 0.87** | 0.99** | N/A | 0.12 | 0.14 | N/A | |
| Color Identification: Parent Married | 1.50** | 1.73** | N/A | 0.21 | 0.24 | N/A | |
| McCarthy Drawing: No Special Needs | 0.16* | 0.18* | N/A | 0.14 | 0.16 | N/A | |
| McCarthy Drawing: African American | 0.18* | 0.20* | N/A | 0.16 | 0.17 | N/A | |
| McCarthy Drawing: English-English Language Group | 0.17* | 0.19* | N/A | 0.15 | 0.17 | N/A | |
| PELS: No Special Needs | 0.50*** | 0.57*** | N/A | 0.36 | 0.41 | N/A | |
| PELS: White | 0.37*** | 0.43*** | N/A | 0.27 | 0.31 | N/A | |
| PELS: African American | 0.53** | 0.60** | N/A | 0.38 | 0.43 | N/A | |
| PELS: Hispanic | 0.51** | 0.58** | N/A | 0.37 | 0.42 | N/A | |
| PELS: English-English Language Group | 0.48*** | 0.54** | N/A | 0.35 | 0.39 | N/A | |
| PELS: Spanish-English Language Group | 0.46* | 0.52* | N/A | 0.33 | 0.38 | N/A | |
| PELS: Parent Married | 0.52*** | 0.60*** | N/A | 0.38 | 0.43 | N/A | |
| PELS: Parent Not Married | 0.43*** | 0.48*** | N/A | 0.31 | 0.35 | N/A | |
* = p≤0.05, ** = p≤0.01, *** = p≤0.001. 1 A total of 82 differences in impacts between subgroups were examined. The complete set of results, including differences not found to be statistically significant, appears in Appendix 5.2. Findings for depression indicate the change in Head Start’s estimated impact that accompanies a 1-point increase in mother’s baseline depression score. Findings for baseline factors other than depression indicate the amount by which Head Start’s estimated impact for the first subset of participants listed in the row label exceeds that for the second subset listed. (back) 2 A total of 99 subgroup impacts were examined. The complete set of results, including differences not found to be statistically significant, appears in Appendix 5.2. (back) |
| Outcome Measure | Impact Estimates | Effect Sizes | |||||
|---|---|---|---|---|---|---|---|
| Impact of Access to Head Start | Impact of Head Start Participation, No-Show Adjustment | Impact of Head Start Participation, Combined No-Show and Crossover Adjustment | Impact of Access to Head Start | Impact of Head Start Participation, No-Show Adjustment | Impact of Head Start Participation, Combined No-Show and Crossover Adjustment | ||
| Overall Impact | WJ-III Letter-Word Identification | 5.74* | 6.93* | 6.83* | 0.22 | 0.26 | 0.26 |
| Letter Naming Task | 2.28** | 2.75** | 2.39* | 0.24 | 0.29 | 0.25 | |
| WJ-III-Spelling | 4.14* | 4.99* | 4.74* | 0.16 | 0.19 | 0.18 | |
| Color Identification | 0.60 | 0.72 | 0.97* | 0.11 | 0.13 | 0.17 | |
| PELS | 0.41*** | 0.49*** | 0.52*** | 0.29 | 0.35 | 0.37 | |
| Difference in Impact 1 | Counting Bears: Race (Hispanic Impact Exceeds White) | 0.52* | 0.63* | N/A | 0.38 | 0.46 | N/A |
| Impact on Subgroup 2 | PPVT-III: Hispanic | 5.64* | 6.42* | N/A | 0.14 | 0.16 | N/A |
| WJ-III Letter-Word Identification: No Special Needs | 5.88* | 7.16* | N/A | 0.22 | 0.27 | N/A | |
| WJ-III Letter-Word Identification: African American | 10.56* | 14.01* | N/A | 0.40 | 0.53 | N/A | |
| WJ-III Letter-Word Identification: English-English Language Group | 7.32* | 9.30* | N/A | 0.27 | 0.35 | N/A | |
| WJ-III Letter-Word Identification: Parent Not Married | 7.92* | 9.45* | N/A | 0.30 | 0.35 | N/A | |
| Letter Naming Task: No Special Needs | 2.39** | 2.91** | N/A | 0.25 | 0.31 | N/A | |
| Letter Naming Task: White | 2.77** | 3.23** | N/A | 0.29 | 0.34 | N/A | |
| Letter Naming Task: English-English Language Group | 3.05** | 3.87** | N/A | 0.32 | 0.41 | N/A | |
| Letter Naming Task: Parent Not Married | 2.70* | 3.22* | N/A | 0.29 | 0.34 | N/A | |
| McCarthy Drawing: Parent Not Married | 0.39* | 0.47* | N/A | 0.20 | 0.24 | N/A | |
| WJ-III Spelling: No Special Needs | 4.97** | 6.05** | N/A | 0.19 | 0.24 | N/A | |
| WJ-III Spelling: African American | 9.75** | 12.94** | N/A | 0.38 | 0.50 | N/A | |
| WJ-IIII Spelling: English-English Language Group | 4.49** | 5.70** | N/A | 0.17 | 0.22 | N/A | |
| WJ-III Spelling: Parent Not Married | 6.31* | 7.53* | N/A | 0.25 | 0.29 | N/A | |
| PELS: No Special Needs | 0.43*** | 0.52*** | N/A | 0.30 | 0.37 | N/A | |
| PELS: African American | 0.75** | 0.98** | N/A | 0.53 | 0.70 | N/A | |
| PELS: English-English Language Group | 0.45*** | 0.57*** | N/A | 0.32 | 0.40 | N/A | |
| PELS: Parent Married | 0.35* | 0.43* | N/A | 0.25 | 0.30 | N/A | |
| PELS: Parent Not Married | 0.52** | 0.61* | N/A | 0.37 | 0.43 | N/A | |
|
* = p≤0.05, ** = p≤0.01, *** = p≤0.001. 1 A total of 82 differences in impacts between subgroups were examined. The complete set of results, including differences not found to be statistically significant, appears in Appendix 5.2. Findings for depression indicate the change in Head Start’s estimated impact that accompanies a 1-point increase in mother’s baseline depression score. Findings for baseline factors other than depression indicate the amount by which Head Start’s estimated impact for the first subset of participants listed in the row label exceeds that for the second subset listed. (back) 2 A total of 99 subgroup impacts were examined. The complete set of results, including differences not found to be statistically significant, appears in Appendix 5.2 (back) |
1 See Chapter 4 for an explanation of the terms used here and the analytical methods used to make the statistical adjustments. (back)
2 The standard deviation of each outcome measure is derived from data on children/families in the non-Head Start sample, excluding members of the Head Start sample, to ensure that measures of underlying variation are not affected by the intervention. (back)
| Table of Contents | Previous | Next |