Table of Contents | Previous | Next |
Chapter 2: The Design of the Head Start Impact Study
Overall Research Design
As discussed in Chapter 1, the primary purpose of this study is to determine whether Head Start has an impact on participating children and their parents and, if so, whether such effects vary among different types of children, families, communities, and configurations of children’s early care and program experiences. By “impact” we mean a difference between the outcomes observed for Head Start participants and what would have been observed for these same individuals had they not participated in Head Start. This focus on impacts distinguishes this study from many others that seek primarily to examine relationships among participant outcomes and between participant outcomes and one or more individual or program characteristics (see, for example, the Head Start Family and Child Experiences Study (FACES)11. Instead, the present study uses information from participants and statistically equivalent children who do not participate in Head Start to determine whether Head Start causes the observed child and parent outcomes.
Given the goal of measuring program impacts, how do we determine what outcomes would have been observed if the children had not participated in Head Start That is, how do we observe children having the same characteristics in two places at the same time in Head Start and not in Head Start and compare them In many studies, researchers have addressed this problem by comparing program participants to a “participant-like” group of children who, in the ordinary course of events, do not participate in Head Start. However, even the best attempts at constructing such a comparable group of non-participants suffer from what evaluators call “selection bias.” That is, families who seek out or “select” Head Start for their children are likely to be different (on important factors that may lead to different outcomes independently of the effect of Head Start services) from those who do not. For example, parents who seek to enroll their children in Head Start may be more motivated to prepare them to start school than those parents who choose not to seek Head Start enrollment. Moreover, the reasons that these two types of parents make different decisions are both typically unobserved and likely to be related to the outcomes of interest in their own right. That is, the motivated parents do a host of things that may affect their children’s development beyond enrolling them in Head Start. Because one cannot account for all of these underlying differences, one risks mis-attributing to program participation observed differences on a particular outcome measure (e.g., emergent literacy) that may, in fact, be a result of pre-existing differences between participants and non-participants.
To avoid this problem of selection bias, the Head Start Impact Study randomly assigned a sample of newly entering 3- and 4-year-old Head Start applicants12 either to a treatment group (in which children and families received Head Start services) or to a control group (in which children were not granted access to Head Start but may have received other services chosen by their parents). Under this randomized design, a simple comparison of outcomes for the two groups yields an unbiased estimate of the impact of the treatment condition, or, in this case, the effect of Head Start participation on children’s school readiness. The advantage of this research design is that if random assignment is properly implemented with a sufficient sample size, program participants should not differ in any systematic or unmeasured way from non-participants except through their participation in Head Start services.13
The remainder of this chapter describes how this randomized design was implemented and the characteristics of the study sample.
Selection of Head Start Programs
Most randomized studies are conducted in small demonstration programs or, if done in an ongoing program, in only a small number of operating sites, usually those that volunteer to be included in the research. In contrast, the Head Start Impact Study is based on a nationally representative sample of Head Start programs and newly entering 3- and 4-year-old children. That is, both programs and children applying for entry into Head Start in fall 2002 were selected at random making results generalizable to the entire Head Start program,14 not just the selected study sample. This approach responds to the Congressional mandate that the study provide “a national analysis of the impact of Head Start” based on the selection of Head Start grantee/delegate agencies15 that “operate in the 50 states, the Commonwealth of Puerto Rico, or the District of Columbia and that do not specifically target special populations.”
Furthermore, the Advisory Committee recommended that the sample of Head Start grantee/delegate agencies should reflect variation in a variety of characteristics, including “region of the country, race/ethnicity/language status, urban/rural, and depth of poverty in communities,” and “. . .design of program as a one-year or two-year experience for children; program options (e.g., center-based, home-based, part-day, full-day); auspice (e.g., Community Action Agency, public school, nonprofit organization); community-level resources; alternative child care options for low-income children; and, the nature of the child care market and the labor market in the community studied.”
The Grantee/Delegate Agency Sampling Process
To meet the legislative requirements, the study used a multi-stage sample selection process to select Head Start programs. The process, depicted in Exhibit 1, began by selecting a large sample of grantee/delegate agencies, screening these programs for inclusion in the impact study, and then selecting a sample of Head Start centers within the sampled grantee/delegate agencies. This process is described below:
| All FY1999-2000
Head Start Grantee/Delegate Agencies in All 50 States, DC, &
Puerto Rico. Exclude “very new,” migrant, Tribal Organization and Early Head Start only grantee/delegate agencies (N=1715 programs). |
 |
| Create Geographic
Grantee Clusters Group grantee/delegate agencies by geographic proximity with a minimum of 8 per cluster (N=161 clusters) |
|
| Group Clusters
into 25 Strata. Stratify on state pre-K and childcare policy, child race/ethnicity, urban/rural location, and region. Select 1 cluster per strata with probability proportional to size. (N=261 grantee/delegate agencies) |
|
| Determine
“Eligible” Grantee/Delegate Agencies in Each Cluster.
Exclude those that are closed or merged and those that are “saturated” (have very few unserved children in the community). Eliminated 38 grantee/delegate agencies (N=223). Small grantee/delegate agencies were grouped to ensure proportional representation (N=184 groups). |
|
| Stratify
and Select Grantee/Delegate Agencies. Stratify on grantee/delegate agency characteristics including local contextual variables, and randomly select approximately 3 grantee/delegate agencies per cluster (N=76 grantee groups, 90 grantee/delegate agencies). |
|
| Contact
Grantee/Delegate Agencies for the Study. N=76 grantee/delegate agency groups (87 individual grantee/delegate agencies) |
|
| Develop
List of Centers Participating grantee/delegate agencies asked to provide list of centers to be operating in 2002-3 program year (N=1427 centers) |
|
| Determine
Eligible Centers and Create Center Groups Exclude saturated centers and create center groups by combining small centers with nearby centers. |
|
| Stratify
and Select Sample Stratify using same characteristics used with grantees. Randomly select centers and exclude saturated centers (84 grantee/delegate agencies, 383 centers) |
|
| Conduct
Random Assignment and Select Children Final Sample: 84 grantee/delegate agencies, 378 centers, 2829 T children and 1921 C children. |
- Initial grantee/delegate agency selection—The sampling process began by using the Head Start Program Information Report (PIR) to create a list of 1,715 Head Start grantee and delegate agencies operating in fiscal year (FY) 1999-2000, after excluding (1) grantee/delegate agencies serving only special populations (migrant and tribal Head Start programs, and sites serving only Early Head Start children), (2) grantees involved in the FACES 2000 study, and (3) as recommended in the Advisory Committee report (1999), grantees/delegate agencies that were “extremely new to the program.”16 This pool of Head Start programs was organized into 161 “geographic clusters” (to increase the ability to closely monitor random assignment and obtain high quality data) and then grouped into 25 strata to control for factors such as region of the country, urban/rural location, race/ethnicity, and variation in state pre-kindergarten and child care policies. One cluster was then randomly selected with probability proportional to total enrollment from each of the 25 strata, providing a total of 261 grantee or delegate agencies in the sampled clusters (to improve efficiency, random sub-sampling was done in three very large urban clusters).
- Determining grantee/delegate agency eligibility—To be eligible for inclusion in the randomized study, grantee/delegate agencies had to have enough “extra” or additional newly-entering applicants to allow for the creation of a non-Head Start control group. That is, the programs could not be serving all the eligible children in their community who wanted Head Start, a situation we refer to as “saturation.” Ethically, random assignment could only be conducted in Head Start programs where local staff were currently unable to serve all the eligible children seeking enrollment for fall 2003. This eligibility was determined from information gathered through telephone calls to all 261 grantee/delegate agencies, augmented with information provided by federal Regional Office staff and obtained from secondary sources such as local Child Care Resource and Referral Agencies. This screening process eliminated 28 grantees/delegate agencies (a reduction of 11 percent). Additionally, ten other grantee/delegate agencies had been closed or merged, further reducing the pool of eligible programs to 223 grantee/delegate agencies.
- Selecting grantee/delegate agencies—To ensure the inclusion of the full range of Head Start grantee/delegate agencies, smaller programs were combined with other agencies in the same cluster to form “grantee/delegate agency groups.” These groups (some of which consisted of a single grantee or delegate agency) were then stratified along several dimensions: urban location (central city, other urban, rural/small town); auspice (school based vs. all other agency types); percent Hispanic and percent Black enrollment; program options offered (part-day only, full-day only, both); and the percent of total enrollment represented by newly-entering 3-year-olds. Approximately three grantee/delegate agency groups were randomly selected from each of the 25 strata with probabilities proportional to the number of newly-entering children. This yielded a sample of 76 grantee/delegate agency groups comprised of 90 individual grantee/delegate agencies.
- Grantee/delegate agency recruitment—Senior project staff visited all 90 selected grantee/delegate agencies during summer 2001 to explain the study, verify information needed for study implementation, and to gain their agreement to participate in the Head Start Impact Study. Three sites were dropped at this point¾one had recently closed and two were dropped due to an overlap with a study being conducted by the federally funded Head Start Quality Research Centers leaving 87 grantee/delegate agencies in 76 grantee/delegate agency groups.
- Identifying operating Head Start centers—Because administrative data do not identify individual Head Start centers, each of the 87 grantee/delegate agencies was asked to provide a list of all centers expected to be in operation for the 2002-03 program year, and to validate basic data about the characteristics of children served, program options, and enrollment patterns. This resulted in a list of 1,427 Head Start centers in the 87 grantee/delegate agencies that could potentially be included in the Head Start Impact Study.
- Determining center eligibility and selecting a sample of study centers—The center-level data were first used to eliminate 169 centers determined to be “saturated,” as was done previously for grantee/delegate agencies. This step reduced the total eligible pool of centers from 1,427 to 1,258 across 84 separate grantee/delegate agencies in 76 grantee/delegate agency groups (a reduction of about 11 percent and the loss of three grantee/delegate agencies, but no grantee groups). Next, small centers were combined with nearby centers, and the resulting “center groups” were then stratified using the same characteristics employed in the selection of grantee/delegate agencies (excepting those that do not vary within grantee/delegate agencies such as region). A main sample consisting of an average of three center groups was selected from each eligible grantee/delegate agency, resulting in a main sample of 448 centers in 84 grantee/delegate agencies. More in-depth or up-to-date information on the initially sampled centers led to a determination that some were, in fact, ineligible for the randomized study. These included centers that: (1) had recently closed or had been merged with other centers; (2) served only Early Head Start children; (3) were collaborations between Head Start and private preschool programs that could not subject their entire pool of applicants to random assignment; or, (4) were, in fact, saturating their community with Head Start services. These findings resulted in the dropping of 103 initially sampled centers. (A “reserve” sample of an average of two center groups per program (a total of 237 centers) was also selected to be used as replacement sites if needed to achieve the expected overall study sample size. 38 of these centers were used. The final sample was 383 (448-103+38) centers).
As described below, this sample of Head Start grantee/delegate agencies and centers is designed to yield a sample of children that, when properly weighted, represents the national population of newly-entering children and their families (with the exclusions noted above).
How Representative Is the Program Sample
Two questions arise regarding such a national sample:
- How well does the study sample match the overall population from which it was drawn Specifically, how well does the sample of grantee/delegate agencies match the complete universe of Head Start programs, after excluding newly-opened programs and those serving only migrant, Native American, or Early Head Start children
- Did decisions to exclude programs from the study sample, particularly saturated programs, contribute to measured differences between the study sample and the overall population from which it was drawn
Answers to such questions speak to the external validity- or generalizeability- of the study sample.
Comparison of Grantee/Delegate Agency Sample with National Population
This first question is addressed by Exhibit 2 using data from the 1998-99 fiscal year PIR (the most recent data available at the time the study sample was selected). The first column lists important characteristics of grantee/delegate agencies that are available from the PIR, the second column provides the national distribution of all Head Start agencies on these characteristics in 1998-99, and the third and final column provides the characteristics of the initially selected study sample on these same characteristics, again as measured in 1998-99, after the sample of programs was weighted to account for how programs were selected, including adjustments for the exclusion of grantees and centers that could not be included due to saturation.
Statistically significant differences are indicated by the use of shading in Exhibit 2. As shown, the weighted study sample of Head Start programs is statistically similar to the PIR on all but three characteristics. The three observed differences indicate that the sample of grantee/delegate agencies may have overrepresented larger programs and overrepresented Hispanic/Spanish-speaking children as compared to the PIR. Two of the characteristics, Hispanic race/ethnicity and Spanish language, are highly correlated with each other. Thus, we effectively have two characteristics for which there are statistically significant differences. It should be noted, however, that these results are based on preliminary estimates of standard errors (which are used to determine statistical significance) and the more complex estimates of sampling variances that will be developed later are likely to eliminate or reduce the number of observed significant differences.
| Percent of Children by Category |
Total Head Start Grantee/Delegate Agency Population, 1998-99 PIR |
Head Start Impact Study Grantee/ Delegate Agency Sample Population, 1998-99 PIR (N=90) |
|---|---|---|
| Auspice: | ||
| Community Action | 39.2% | 38.1% |
| Agency | ||
| School System | 13.2% | 7.2% |
| Other Non-profit | 38.9% | 45.0% |
| Government Agency | 8.8% | 9.7% |
| Funded Enrollment | ||
| Less than 147 | 5.6% | 3.2% |
| 147-268 | 12.2% | 10.3% |
| 269-487 | 21.3% | 16.3% |
| More than 497 | 60.9% | 70.0% |
| Length of Day Served: | ||
| Full Day | 56.0% | 61.1% |
| Part Day | 44.0% | 38.9% |
| Options Offered: | ||
| Center based | 92.0% | 94.5% |
| Home based | 3.9% | 3.2% |
| Combination | 1.8% | 0.7% |
| Locally designed | 2.2% | 1.6% |
| Metro Status | ||
| Urban | 24.3% | 22.8% |
| Rural | 75.7% | 77.2% |
| Head Start Region | ||
| Northeast | 23.9% | 24.1% |
| North Central | 18.8% | 19.7% |
| South | 35.1% | 35.1% |
| West | 22.3% | 21.2% |
| State Pre-K Programs | ||
| Are similar to HS | 25.4% | 25.2% |
| Partly similar to HS | 18.5% | 19.1% |
| Remaining States | 56.0% | 55.6% |
| Child Age: | ||
| Under Age 3 | 0.4% | 0.4% |
| Age 3 | 34.8% | 33.5% |
| Age 4 or older | 64.8% | 66.1% |
| Child Race/Ethnicity: | ||
| White | 31.7% | 30.0% |
| Hispanic | 27.7% | 33.4% |
| Black | 36.7% | 33.0% |
| Other | 3.9% | 3.6% |
| Child Language: | ||
| English | 77.4% | 69.5% |
| Spanish | 19.0% | 25.2% |
| Other | 3.6% | 5.0% |
| Notes: (1) Shading indicates statistically significant differences. (2) Data for national population and sample are from 1998-1999 PIR. (3) Population totals exclude grantees/delegates that: serve only migrant, Native American, or Early Head Start children; those that do not serve 3- and 4-year-olds; those operating in territories other than Puerto Rico; and those newly created in FY1999-2000 or later. (4) ACF funded enrollment was used for auspice, metro status, region, and state preschool programs; total funded enrollment was used for length of day and option; actual enrollment used elsewhere (actual enrollment can exceed funded enrollment). (5) For size categories, ACF funded enrollment was used to create categories. (6) These percentages are based on total enrollment, not just newly entering 3- and 4-year-olds. |
Effects of Exclusion Due to Saturation
Exclusion of Grantee/Delegate Agencies: One factor that may account for the differences noted in Exhibit 2 is the exclusion of saturated grantee/delegate agencies. This is because saturated programs have a lower percentage Hispanic enrollment and are smaller than non-saturated programs (see Appendix B).17
As noted above, 38 (15%) of the initially selected grantee/delegate agencies were dropped from consideration for inclusion in the impact study. Of these, 28 were determined to be saturated and unable to provide “extra” children for the control group, and ten grantees/delegate agencies had either closed or had merged with another grantee/delegate agency. After eliminating these 38 programs, the final frame consisted of 223 grantee/delegate agencies in 25 geographic clusters. The 28 saturated programs represent an estimated 11 percent of the eligible Head Start program universe. However, due to the smaller than average size of the excluded programs, they represent only 3.9 percent of the total number of newly-entering 3- and 4-year-old applicants. Similarly, ten closed or merged programs are estimated to represent 60 such programs on the PIR frame containing under 3 percent of total newly entering 3- and 4-year-old applicants.
There is potential for bias due to the exclusion of these saturated Head Start programs from the sampling frame if the saturated programs are systematically different from the non-saturated programs and if the characteristics on which they differ are correlated with the outcome measures that will be examined for the impact analysis (see the next chapter). As shown in Appendix B, preliminary analyses indicate that the saturated programs in the sample are systematically different from the non-saturated programs. They are smaller, more likely to be school-based, and have a smaller percentage Hispanic enrollment than the non-saturated programs.18 As a consequence, these potential differences will have to be accounted for during the estimation of program impacts through the use of both weighting and statistical adjustment.
Exclusion of Head Start Centers: As also discussed above, within the sample of 76 grantee/delegate agency groups, all of the individual Head Start centers were screened for inclusion in the study sample. As also discussed in Appendix B, this procedure resulted in the deletion of 154 centers from a frame of 1,423 centers. The excluded 154 centers represent an estimated eight percent of the Head
Start center universe, approximately 5 percent of the total number of 3- and 4-year-old children enrolled in Head Start, and 4.7 percent of newly-entering 3- and 4-year-old applicants. As with grantee/delegate agencies, the 154 saturated centers are significantly smaller, have lower Hispanic enrollment, and have a larger percentage of newly-enrolled 3-year-olds than the non-saturated centers. Again, as noted above, these differences will have to be taken into account during the statistical analysis of program impacts.
Overall, the estimated coverage rate of the population of newly-entering 3- and 4-year-olds for the Head Start Impact Study sample is 92 percent, which is obtained by multiplying the coverage rate for Head Start grantee/delegate agencies and by the coverage rate for centers [i.e., (100-3.9% for programs)*(100-4.7% for centers)]. Stated another way, the overall under-coverage rate is 8 percent.
Understanding Saturation: To understand the context in which saturation occurs, we examined trends in child poverty from 1997 to 2001. Data from consecutive March Supplements to the Current Population Survey, collected in March 1998 through March 2002 for the previous calendar year, show that between 1997 and 2000, the estimated number of 3- and 4-year-old children in poverty in the United States fell by 475,415. The number then rose again by 94,533, in the most recent year with data, 2001. Overall, the estimated number of 3- and 4-year-old children in poverty in the United States fell about 21 percent between 1997 and 2001 (from 1,790,183 to 1,409,301). Thus, by the start of the Head Start Impact Study in 2002, Head Start grantee/delegate agencies likely had a smaller eligible population to draw from than in the mid-1990s, and this may have contributed to the observed level of service saturation.
Further evidence of this decline in the eligible population comes from the PIR data, which provide information about a grantee/delegate agency’s funded and actual enrollment19. Among the 261 grantee/delegate agencies in the initial pool selected for the Head Start Impact Study, some were under-enrolled. On average, about one in five was under-enrolled by at least five percent per year in the three years preceding the study, a slight improvement from the one in four under-enrolled in 1997-1998. Further, the PIR data suggest that for many grantee/delegate agencies, under-enrollment (defined as being at 94 percent of funded enrollment or less at a given time point) appears to be a consistent problem:
- 14.9 percent of the 261 grantee/delegate agencies experienced under-enrollment at least once during the four year period but not consistently across any year.
- 26.4 percent experienced a problem with under-enrollment consistently across at least one year but not all four years.
- 11.1 percent experienced chronic under-enrollment across all four years.
The data also indicate that challenges with under-enrollment fluctuate from year to year, i.e., grantee/delegate agencies that have problems one year may not have problems the next, and vice versa. Thus, saturation is a complicated issue for Head Start programs, as they are constantly trying to achieve the right balance among population shifts, changes in program regulations, pre-kindergarten resource availability, fluctuating economic conditions, and other factors.
Child-Level Selection and Assignment Process
Child Sample Selection
At each of the selected Head Start centers, program staff provided information about the study to parents at the time enrollment applications were distributed. Parents were told that enrollment procedures would be different for the 2002-03 Head Start year and that some decisions regarding enrollment would be made using a lottery like process.20
The study assigned local site coordinators to work with grantee/delegate agencies in each of the 25 geographic clusters to ensure that parents received this information with their applications. These site coordinators were also responsible for obtaining data from all applications for the 2002-03 program year (to ensure equal treatment of all applicants), and listing these data on a roster that was subsequently key-entered by central office study staff. Returning children, and a small number of grantee-requested “high-risk” exclusions,21 were eliminated from these lists, and checks were made for duplicate records. The high-risk exclusions were made on a case-by-case basis with each grantee/delegate agency and in close consultation with Administration for Children and Families staff. Examples of such exclusions included: children of homeless families, children in families with documented abuse and neglect, and children with severe disabilities, especially those disabilities that would make it difficult to test these children and include them in the study sample (e.g., blindness). Each grantee was limited to one exclusion per center, but in fact only 276 exclusions were taken out of a total of approximately 18,000 newly-entering applications.
At this point, local agency staff implemented their typical process of reviewing enrollment applications and screening children for admission to Head Start based on criteria approved by their respective Policy Councils. No changes were made to these locally-established admission criteria. Site coordinators recorded basic information about each applicant and what was usually a numerical score determined by local staff that signified the relative need of individual children (e.g., in some agencies, a higher score indicated a greater need for Head Start and a corresponding higher priority for admission). Using these rankings, the list of newly-entering children who would ordinarily have been enrolled was “extended” to add a specified number of children needed for the non-Head Start control group. The children added were those who would normally be “next in line” for admission if the initially targeted children could not be enrolled.
The goal was to randomly select, on average, 27 children from the expanded list at each of the sampled center groups: 16 to be assigned to the Head Start group and 11 to be assigned to the non-Head Start group. For an average center group, the 11 non-Head Start control group children represented about nine percent of total enrollment. Where necessary, stratification was used, such as in situations where the degree of saturation varied by program option (part-day vs. full-day) or age cohort. In some cases, where fewer children than expected were actually available, a smaller sample of children was selected for the study.
The original legislative mandate required that the Head Start Impact Study “to the extent practicable” address possible variation in program impact related to “the length of time a child attends a Head Start program (and) the age of the child on entering the Head Start program.” This requirement reflects the hypothesis that different program impacts may be associated with one versus two years of Head Start experience. It also reflects a trend of increased enrollment of 3-year-olds in some grantee/delegate agencies presumably due to the growing availability of preschool options for 4-year-olds (often state-sponsored programs). Consequently, the study includes samples of both newly-entering 3-year-olds (studied through two years of Head Start participation, kindergarten, and first grade) and newly-entering 4-year-olds (studied through one year of Head Start participation, kindergarten, and 1st grade). The sample of 3-year-olds is slightly larger than the sample of 4-year-olds to protect against the possibility of higher study attrition resulting from an additional year of longitudinal data collection for the younger children.22
Random Assignment of Children
Within the final set of 76 grantee/delegate agency groups (or 84 total grantees/delegate agencies), random assignment was attempted at a total of 383 randomly selected Head Start centers. Of these, only five centers (1.3 percent) did not fully cooperate with the study requirements, resulting in 378 centers with successful random assignment that can be divided into the following three groups:
- Obtained Full Sample: Random assignment was completed at 173 Head Start centers that provided the full expected sample of children.
- Obtained Smaller Sample: Random assignment was completed at 150 Head Start centers that provided a smaller than expected sample (i.e., because new application rates were lower than estimated).
- Obtained Larger Sample: Random assignment was completed at 55 Head Start centers that provided a larger than expected sample (i.e., because application rates for newly-entering children were higher than originally estimated, sample sizes were increased to compensate for other centers that were unexpectedly low).
In total, 4,750 newly entering children were randomly assigned and included in the Head Start Impact Study:
| Age Cohort | Head Start (Treatment) Group |
Non-Head Start (Control) Group |
Total Sample |
|---|---|---|---|
| 3-year olds | 1,552 | 1,045 | 2,597 |
| 4-year-olds | 1,277 | 876 | 2,153 |
| Total | 2,829 | 1,921 | 4,750 |
As indicated above, about 60 percent of the sample was assigned to the Head Start group and about 40 percent was assigned to the non-Head-Start group. This imbalance reduces the precision of the impact estimates by just two percent (compared with a balanced 50-50 design). However, it provided several important benefits: (1) it significantly increased the ability to recruit grantees and centers by decreasing the number of extra children needed for the control group; (2) decreased the loss of sites due to saturation; and (3) saved considerably on data collection costs because treatment group members (who participate in Head Start) require less effort to track and interview over time than children in the non-Head-Start control group.
Comparison of Treatment and Control Children and Their Families at the Time of Random Assignment
Was random assignment implemented well enough to support the intended impact analysis This question is addressed in Exhibit 3, which shows a comparison of children randomly assigned to the Head Start and non-Head Start groups on characteristics that were measured and available at the time of random assignment (these data were drawn from parental applications for Head Start). As shown, there are no statistically significant differences between the two randomly assigned groups indicating that they are well matched on the available characteristics. Consequently, we can conclude that the initial randomization was done with high integrity and that the samples can provide the necessary confidence in the validity of the eventual impact estimates.
| Characteristic | Head Start (Treatment) Group |
Non-Head Start (Control) Group |
Difference: (Head Start) - (Non-Head Start) |
|---|---|---|---|
| Child Age Cohort: | |||
| 3-year old | 52.8% | 53.2% | -0.4% |
| 4-year-old | 47.2% | 46.8% | 0.4% |
| Child Gender: | |||
| Boys | 49.6% | 48.7% | 0.9% |
| Girls | 50.4% | 51.3% | -0.9% |
| Child Race/Ethnicity: | |||
| Hispanic | 42.2% | 43.0% | -0.8% |
| Black | 27.0% | 25.5% | 1.5% |
| White | 28.1% | 29.1% | -1.0% |
| Other | 2.7% | 2.5% | 0.2 % |
| Child Language: | |||
| English | 65.7% | 64.7% | 1.1% |
| Spanish | 31.4% | 32.2% | -0.8% |
| Other | 2.8% | 3.1% | -0.3% |
| Parent Language: | |||
| English | 68.0% | 68.1% | -0.1% |
| Spanish | 30.5% | 30.9% | -0.4% |
| Other | 1.5% | 1.1% | 0.4% |
| Income Eligibility (for Head Start): | |||
| Yes | 93.0% | 91.7% | 1.3% |
| No | 7.1% | 8.3% | -1.3% |
| Notes: (1) Data source: Roster information used at time of random assignment; (2). These percentages reflect individual level data on newly entering 3 - and 4 -year-olds and do not include saturated grantees and centers; and (3) There were no statistically significant differences between groups. |
11 U.S. Department of Health and Human Services, Administration for Children, Youth, and Families (2003). Head Start FACES 2000: A Whole-Child Perspective on Program Performance, Fourth Progress Report. Washington, DC: Author.(back)
12The Head Start Impact Study focuses on newly-entering children to ensure that the estimated impacts are unaffected by previous program participation. Consequently, children who were returning to Head Start, as well as those previously enrolled in Early Head Start, were excluded from the study sample.(back)
13More precisely, there will be differences between individuals in the two groups, but the expected or average value of these differences is zero except through the influence of Head Start (i.e., selection bias is removed by random assignment).(back)
14As will be discussed below, some Head Start grantee/delegate agencies were intentionally excluded from the study. Study results will, therefore, be reported for both the restricted national sample and for the full Head Start population of newly entering children.(back)
15The study sample includes both Head Start grantees and their delegate agencies. Grantees are organizations that have fiscal and administrative responsibility for programs in their jurisdiction. In some cases, they can subcontract with agencies to handle administrative oversight over some or all of these programs. Throughout this report we use the term grantee/delegate agencies to refer to both types of agencies.(back)
16Defined as in operation for less than two years.(back)
17Program weights were adjusted for these exclusions, but the adjustment may not totally compensate for any potential bias.(back)
18As with the previous discussion, these estimates may change as we conduct further tests because (1) the variances at the grantee/delegate agency level are not very stable because the number of saturated programs is small, and (2) the variance estimates do not yet include the between-PSU component of variance. These factors may result in an overstating of the statistical significance of the differences.(back)
19Total funded enrollment is the number of children the grantee/delegate agency has been funded to serve, regardless of funding source. Actual enrollment is the total number of children enrolled at any time during the year.(back)
20Non-admitted (control group) children selected for the study sample were prohibited from participating in Head Start during 2002-03. Those who were 3-year-olds could, however, re-apply for Head Start in 2003-04 and may be admitted if eligible.(back)
21This decision was made because: (1) there were ethical concerns about assigning very high-risk children to the control group, especially in situations where Head Start may provide their only option for early childhood services; (2) the Field Test demonstrated that the potential exclusion of those most severely in need affected cooperation when trying to recruit study sites; and (3) there were some children who could not be assigned to the control group because of placement by the local child welfare agency.(back)
22This equal sampling of 3- and 4-year-old enrollees was done despite the fact that 4-year-olds represent about twice the proportion of all Head Start participants as do 3-year-olds. In large part, this is because the 4-year-olds include both newly entering 4-year-olds plus returning children who began Head Start as 3-year-olds and who have turned 4 years of age in their second year of program participation.(back)
| Table of Contents | Previous | Next |