Table of Contents | Previous | Next |
Chapter 2: Description of the Study Sample
Representing the National Head Start Population
Because this study, as discussed in Chapter 1, is a national probability sample of Head Start programs, an important question is “Can the study findings be generalized to the complete Head Start population?” For this purpose, the population, or universe, of interest is all newly entering 3- and 4-year-olds in all Head Start centers operating in 2002-03, except those serving only special populations (i.e., programs serving primarily only migrant, Native American, or Early Head Start children), or very new centers (see Chapter 1 for details). Ideally, all such children would have the possibility of being included in the study and the “coverage rate” would, therefore, be equal to 100 percent.
The major cause for any undercoverage is the ethical design constraint adopted that required that the selected Head Start grantees/delegate agencies and centers have more eligible applicants than could be served at their current Federal funding level. Programs that were serving essentially all the eligible children in the community (referred to as a “saturated” program or center) could not be included in the study because including them could have resulted in a reduction in the number of children being served by Head Start.
As noted in Chapter 1, there were four points in the sample selection process where grantees/delegate agencies or centers were lost due to such saturation. First, some Head Start grantees/delegate agencies were determined to be saturated before the sample was selected, and these programs were, therefore, dropped from the sampling frame. Second, after the initial sample of grantees/delegate agencies was selected, some additional programs were found to be saturated and were also deleted from the sample. At this same point in the process, two additional programs were dropped from the sample because they were Head Start Quality Research Centers (QRC)1 and were deleted so as not to be overburdened. The third point at which saturated sites were dropped from the sample was during the selection of Head Start centers. As with grantees/delegate agencies, some centers were initially determined to be saturated and were considered to be ineligible for inclusion and deleted from the study sample. Some centers were determined to be saturated during later attempts to conduct random assignment and also had to be dropped from the study sample.
Taking into account all of these opportunities for Head Start grantees/delegate and centers to be deleted due to saturation (or being a QRC site), the estimated weighted national coverage rate2 for spring 2003 data was 84.5 percent, i.e., the study sample is representative of 84.5 percent of the total universe of all newly entering 3- and 4-year-olds across the country. The weight that is used for this estimate accounts for the probability of selection for each program and center and also weights the contribution of programs and centers according to the size of their enrollment. (The small number of grantees/delegate agencies and centers that were found to be closed or merged into another program or center were properly considered as ineligible, not as undercovered.)
In addition to these fully saturated grantees/delegate agencies and centers, a number of sampled centers were found to be “partially saturated,” that is, there were enough applicants at the center to permit some children to be assigned to the control group, but the number available was insufficient to allow the selection of the full targeted sample. In such situations, treatment and control group children were selected from a “reserve” center, or/and a larger sample size was selected from another sampled center (in the same geographic cluster), to make up for the shortage of study children.
As discussed in the “Random Assignment” section of Chapter 1, additional undercoverage of children occurred because grantee-requested “high-risk” children were excluded from the study. The coverage rate of 84.5 percent cited above does not account for these few exclusions. These exclusions have negligible effect on the overall coverage rate, however, as there were only 276 exclusions out of approximately 18,000 newly entering applications received in the targeted programs.
To account for the undercoverage attributable to these different factors, the program and center weights were ratio adjusted to the total newly entering enrollment in the PSU and program, respectively (see Appendix 1.2). A weighting adjustment was used that was based on information obtained from the Head Start National Reporting System (HSNRS). This adjustment (possible only for children in the 4-year-old group) accounts for differences between the selected sample of Head Start grantees/delegate agencies and centers and the complete national program sampling frame. Appendix 2.1 provides a comparison of the characteristics of saturated and nonsaturated programs.
The Success of Random Assignment
An equally important question to ask about the study sample is “Was random assignment implemented well enough to support the intended impact analysis?” This question is addressed below from two perspectives. First, the characteristics of children randomly assigned to the Head Start and non-Head Start groups are compared using information collected for each child at the time of random assignment. Then, the extent to which children complied with their assigned status is examined, i.e., to what extent did children assigned to the Head Start group actually receive some Head Start services?
Comparing Head Start and Non-Head Start Children at Baseline
Exhibit 2.1 provides, separately for the 3- and 4-year-old age groups, a comparison of children randomly assigned to the Head Start and non-Head Start groups using weighted data3 on all characteristics that were measured and available at the time of random assignment. These data were drawn from parental applications for Head Start. As shown, there are no statistically significant differences between the two randomly assigned groups, indicating that they do not differ to any discernible extent. This suggests that the initial randomization was done with high integrity and that the samples can provide the necessary confidence in the validity of the impact estimates.
Although not related to the success of random assignment, it is interesting to note that the racial/ethnic characteristics of newly entering children in the 3-year-old group are noticeably different from the characteristics of children in the newly entering 4-year-old group. This difference shows that newly entering 3-year-olds are relatively evenly distributed between the Black and Hispanic groups (32.8% vs. 37.4%), while about half of newly entering 4-year-olds are Hispanic (51.6% vs. 17.5% Black). This difference for newly entering 4-year-olds is confirmed by an examination of data from the HSNRS.4 This ethnic difference is also reflected in the age-group differences in child and parent language.
| Characteristic | Head Start (Treatment) Group | Non-Head Start (Control) Group | Difference: (Head Start) - (Non-Head Start) |
||
|---|---|---|---|---|---|
| Child Gender: | 3-Year-Old Group | Boys | 48.5% | 48.9% | -0.4% |
| Girls | 51.5% | 51.1% | 0.4% | ||
| 4- Year-Old Group | Boys | 51.1% | 49.4% | 1.7% | |
| Girls | 48.9% | 50.6% | -1.7% | ||
| Child Race/ Ethnicity: | 3- Year-Old Group | White | 24.5% | 26.6% | -2.1% |
| Black | 32.8% | 31.8% | 1.1% | ||
| Hispanic | 37.4% | 35.7% | 1.6% | ||
| Other | 5.3% | 5.9% | -0.6% | ||
| 4- Year-Old Group | White | 26.7% | 23.3% | 3.4% | |
| Black | 17.5% | 17.0% | 0.5% | ||
| Hispanic | 51.6% | 53.8% | -2.1% | ||
| Other | 4.1% | 5.9% | -1.8% | ||
| Child Language: | 3- Year-Old Group | English | 71.1% | 69.9% | 1.2% |
| Spanish | 24.8% | 24.0% | 0.8% | ||
| Other | 3.9% | 5.7% | -1.8% | ||
| Missing | 0.2% | 0.4% | -0.2% | ||
| 4- Year-Old Group | English | 57.1% | 56.4% | 0.8% | |
| Spanish | 39.3% | 40.8% | -1.5% | ||
| Other | 3.2% | 2.3% | 0.8% | ||
| Missing | 0.4% | 0.5% | -0.1% | ||
| Parent Language: | 3- Year-Old Group | English | 74.8% | 74.8% | 0.0% |
| Spanish | 23.1% | 22.0% | 1.1% | ||
| Other | 1.5% | 1.7% | -0.2% | ||
| Missing | 0.6% | 1.5% | -0.9% | ||
| 4- Year-Old Group | English | 59.5% | 58.4% | 1.1% | |
| Spanish | 37.8% | 39.5% | -1.7% | ||
| Other | 0.9% | 0.5% | 0.5% | ||
| Missing | 1.8% | 1.6% | 0.2% | ||
| Child Income Eligible: | 3- Year-Old Group | No | 7.7% | 6.7% | 1.0% |
| Yes | 91.4% | 91.9% | -0.6% | ||
| Missing | 0.9% | 1.4% | -0.5% | ||
| 4- Year-Old Group | No | 6.0% | 10.1% | -4.0% | |
| Yes | 91.8% | 87.9% | 3.9% | ||
| Missing | 2.2% | 2.1% | 0.1% | ||
| Notes: (1) Data source: Roster information used at time of random assignment; (2) T-tests of the difference between the Head Start and non-Head Start percentage in each row were run for each characteristic; no statistically significant differences were found. With large samples, differences in means for 0/1 variables (e.g., 1=boys, 0=girls) have approximately normal distributions and follow the t distribution once divided by their standard errors. |
Deviations from Random Assignment
Random assignment rarely, if ever, results in perfect adherence to the assigned program status. That is, one would expect some children assigned to the Head Start group to not participate in the program (referred to as “no-shows”), and some of the children assigned to the non-Head Start group to enroll in the program (referred to as “crossovers”).
Such violations of pure random assignment were not, therefore, unexpected. During program recruitment, Head Start grantees and centers reported “no-shows” as a challenge they confront, with rates often in the double-digits. Absent a requirement that parents and children participate once they are accepted for Head Start enrollment, it is not surprising that some families who were randomly assigned to the Head Start group in the study subsequently opted for a different care setting for their child.5
Similarly, although every effort was made to maintain the integrity of the non-Head Start comparison group, perfect conditions could not be implemented. In a few rare instances, local staff intentionally enrolled non-Head Start children into Head Start. However, a greater threat to compliance was that parents could apply to another nearby Head Start program. This problem was particularly an issue in densely populated areas with two or more Head Start programs operating in close proximity. And, due to confidentiality restrictions, local study staff were not able to share information on participants with other nearby grantees, reducing the ability to keep control group families from Head Start enrollment.
Exhibit 2.2 provides information on the incidence of Head Start group “no-shows” and non-Head Start group “crossovers” by age group for both the total sample randomly assigned and for the children who are part of the Year 1 analysis sample that forms the basis for the findings reported in subsequent chapters of this report. The Year 1 analysis sample includes only those children (and their parents) for whom data could be collected in spring 2003 (see Chapter 4 for details on the analysis sample). In the exhibit, a child in the Head Start group is considered a “no show” if it was determined that he/she did not participate in Head Start at any time during the 2002-03 program year. A child in the non-Head Start group was deemed a “crossover” if he/she participated in Head Start at any time during the 2002-03 program year. This determination (explained in more detail in Appendix 2.2) was based on information obtained from parent surveys in fall 2002 and spring 2003, follow-back contact with all Head Start centers in the study in fall 2002 to see if individual children had attended Head Start, and care setting identified at the time of the child’s fall 2002 and spring 2003 assessments.
As shown in this table, “no-shows” accounted for 15 and 20 percent of the full randomly assigned sample for children in the 3- and 4-year-old groups, respectively, and 12 and 17 percent of the Year 1 analysis sample for 3- and 4-year-old groups once analysis weights were applied. Similarly, crossovers accounted for 17 and 14 percent of the randomly assigned group, and 19 and 17 percent of the analysis sample. The resulting differences across the two samples—a lower incidence of “no-shows” and a higher incidence of “crossovers” in the analysis sample compared to all randomly assigned children—are probably due to higher response rates among children in Head Start programs (i.e., they were probably easier to find).
| Sample Group | Any Year 1 Head Start Participation | No Year 1 Head Start Participation | Total | ||
|---|---|---|---|---|---|
| All Randomly Assigned (N=4,667): | 3-Year-Old Group | Head Start Group | 85.1% | 14.9% | 100% |
| Non-Head Start Group | 17.3% | 82.7% | 100% | ||
| 4-Year-Old Group | Head Start Group | 79.8% | 20.2% | 100% | |
| Non-Head Start Group | 13.9% | 86.1% | 100% | ||
| Year 1 Analysis Sample (N=3,898): | 3-Year-Old Group | Head Start Group | 88.2% | 11.8% | 100% |
| Non-Head Start Group | 18.5% | 81.5% | 100% | ||
| 4-Year-Old Group | Head Start Group | 83.4% | 16.6% | 100% | |
| Non-Head Start Group | 16.5% | 83.5% | 100% | ||
Chapter 4 explains how impact estimates are adjusted to account for these occurrences. At this point, it is important to note that the observed levels of noncompliance with the design, although not to be dismissed, are not atypical of what has been found in other random assignment studies and do not undermine the basic validity of the study. At worst, violations of random assignment that extend Head Start’s services to some children in the non-Head Start group and reduce the exposure to Head Start among the treatment group make it harder to detect any average impact of the program that does occur with the available sample size. These considerations should increase the confidence that any observed statistically significant impacts are real and important. The downside, of course, is that some effects of Head Start may be obscured by the loss of analytic power due to the presence of “no-shows” and “crossovers.”
Characteristics of the Year 1 Study Sample
This final section of Chapter 2 examines the characteristics of the current analysis sample that is used in this report. It comprises those children, and their parents, from whom data were collected in spring 2003. Exhibits 2.3-A and 2.3-B compare the characteristics of the children between the Head Start and non-Head Start groups for the 3- and 4-year-old groups, respectively. (The figures in these tables differ from those in Exhibit 2.1, which looked at all children randomly assigned.) The characteristics for these comparisons were all measured in fall 2002 and ideally represent the “baseline” or pre-intervention point of the study. They are also the characteristics, as discussed in Chapter 4, used as covariates in the statistical models to estimate program impacts, or to examine any variation in program impacts for particular population subgroups (e.g., are impacts higher or lower for children with disabilities?).
As demonstrated by these two tables, the Head Start and non-Head Start groups do not differ to any discernible extent except for two small differences for each of the two age groups. For the 3-year-old group, the primary caregivers of children in the Head Start group are slightly older than caregivers of children in the non-Head Start group; and the Head Start group children are somewhat more likely to have a grandparent living with them. In the case of the children in the 4-year-old group, the mothers of children in the Head Start group are more likely to have attained an educational level beyond high school, and the households of children in the Head Start group are somewhat less likely to receive public assistance through the Federal TANF program. As discussed in Chapter 4, these differences may arise from the lag in fall 2002 data collection after the point of random assignment. Because the differences are not fully accounted for by the nonresponse adjustments to the sampling weights (see Appendix 1.2), they are included in the analysis as covariates in the statistical models used to estimate program impacts (see Chapter 4).
| Characteristic | Head Start (Treatment Group) | Non-Head Start (Control) Group | Difference: (Head Start) - (Non-Head Start) | |
|---|---|---|---|---|
| Child Gender: | Boy | 47.9% | 49.1% | -1.2% |
| Girl | 52.1% | 50.9% | 1.2% | |
| Child Race/ Ethnicity: | White | 24.3% | 26.0% | -1.7% |
| Black | 33.3% | 31.4% | 1.9% | |
| Hispanic | 37.0% | 36.4% | 0.6% | |
| Other | 5.4% | 6.3% | -0.8% | |
| Child Has a Disability | 13.5% | 11.9% | 1.6% | |
| Fall-Spring Language of Child Assessment: | English-English | 75.4% | 75.9% | -0.5% |
| Spanish-English | 18.9% | 18.0% | 0.9% | |
| Spanish-Spanish | 4.3% | 4.6% | -0.3% | |
| Primary Home Language is English | 71.9% | 68.5% | 3.4% | |
| Biological Mother Was a Teen Mom | 36.2% | 37.6% | -1.3% | |
| Biological Mother Is a Recent Immigrant | 17.0% | 17.8% | -0.8% | |
| Biological Mother Is Employed | 51.4% | 57.4% | -6.0% | |
| Both Biological Parents Live With Child | 48.5% | 50.7% | -2.2% | |
| Child's Parents Are | Married | 43.7% | 45.3% | -1.6% |
| Separated or Divorced | 11.5% | 13.7% | -2.2% | |
| Primary Caregiver's Age as of 9/1/02 | 29.5 years | 28.6 years | 0.9 years * | |
| Mother's Education: | Less Than High School | 32.4% | 34.8% | -2.3% |
| High School/GED | 34.7% | 33.9% | 0.8% | |
| Beyond High School | 32.9% | 31.4% | 1.5% | |
| Grandparent Lives in Home | 3.6% | 1.7% | 1.9%** | |
| Parent's Self-Reported Health Is Excellent or Good | 85.5% | 86.5% | -1.0% | |
| Primary Caregiver — Depression Scale | 251.9 | 251.2 | 0.7 | |
| Primary Caregiver — Locus of Control Scale | 249.5 | 251.2 | -1.7 | |
| Average Household Income: | $500/month or less | 14.8% | 12.0% | 2.9% |
| $501-$1,500/month | 48.3% | 53.4% | -5.1% | |
| Over $1,500/month | 36.9% | 34.6% | 2.3% | |
| Household Receives TANF | 10.6% | 10.5% | 0.1% | |
*= p≤0.05, ** p≤0.01, *** p≤0.001. |
| Characteristic | Head Start (Treatment Group) | Non-Head Start (Control) Group | Difference: (Head Start) - (Non-Head Start) | |
|---|---|---|---|---|
| Child Gender: | Boy | 49.6% | 51.2% | -1.6 |
| Girl | 50.4% | 48.8% | 1.6 | |
| Child Race/Ethnicity: | White | 27.8% | 24.6% | 3.2% |
| Black | 25.5% | 23.3% | 2.2% | |
| Hispanic | 42.4% | 45.8% | -3.4% | |
| Other | 4.3% | 6.2% | -1.9% | |
| Child Has a Disability | 12.8% | 11.4% | 1.4% | |
| Fall-Spring Language Assessment: | English-English | 67.2% | 64.3% | 2.9% |
| Spanish-English | 25.9% | 28.3% | -2.5% | |
| Spanish-Spanish | 5.9% | 5.4% | 0.4% | |
| Primary Home Language Is English | 63.6% | 63.2% | 0.0% | |
| Biological Mother Was a Teen Mom | 38.6% | 35.2% | 3.4% | |
| Biological Mother Is a Recent Immigrant | 24.1% | 23.5% | 0.6% | |
| Biological Mother Is Employed | 48.5% | 52.0% | -3.4% | |
| Both Biological Parents Live with Child | 51.3% | 51.3% | 0.0% | |
| Child’s Parents Are | Married | 45.2% | 45.4% | -0.2% |
| Separated or Divorced | 15.9% | 14.9% | 1.0% | |
| Primary Caregiver's Age as of 9/1/02 | 29.3 years | 29.5 years | -0.2 years | |
| Mother’s Education: | Less than High School | 38.6% | 41.6% | -3.0% |
| High School/GED | 31.7% | 35.2% | -3.5% | |
| Beyond High School | 29.8% | 23.3% | 6.5%* | |
| Grandparent Lives in Home | 2.4% | 1.4% | 1.0% | |
| Parent's Self-Reported Health Is Excellent or Good | 86.6% | 86.4% | 0.1% | |
| Primary Caregiver—Depression Scale | 251.1 | 248.6 | 2.4 | |
| Primary Caregiver—Locus of Control Scale | 251.5 | 249.5 | 2.0 | |
| Average Household Income: | $500/month or less | 11.8% | 9.1% | 2.7% |
| $501-$1,500/month | 46.2% | 50.8% | -4.6% | |
| Over $1,500/month | 42.0% | 40.0% | 2.0% | |
| Household Receives TANF | 10.0% | 14.4% | -4.5%* | |
*= p≤0.05, ** p≤0.01, *** p≤0.001. |
1 The Head Start Bureau and the Office of Program, Research and Evaluation (OPRE) of DHHS awarded eight cooperative agreements under the Head Start Quality Research Center (QRC) Consortium II (2001-06) to study promoting approaches to the school readiness of Head Start children. (back)
2 An unweighted coverage rate can also be calculated, but this is a less useful measure of coverage as it estimates the proportion of children in the sample, not the universe of children served by Head Start nationally who are in programs and centers that are not saturated. (back)
3 The weights used are the same as those used for all the analyses discussed in this report. Details are provided in Appendix 1.2. (back)
4 See Appendix 2.3 for details. (back)
5 Chapter 3 presents a breakdown of the types of settings children attended. (back)
| Table of Contents | Previous | Next |