Table of Contents | Previous | Next |
Chapter 1: Study Background
The Congressional Mandate
Since its beginning in 1965, Head Start’s goal has been to boost the school readiness of low-income children. The premise underlying the program is that low-income children do not receive the same level of intellectual stimulation at home as middle-class children. Based on a “whole child” model, the program provides comprehensive services that include preschool education; medical, dental, and mental health care; nutrition; and parental involvement. Head Start services are designed to be responsive to each child’s and family’s ethnic, cultural, and linguistic heritage.
In the late 1990s, the US General Accounting Office (GAO) released two reports concluding that (1) “. . .the Federal government’s significant financial investment in the Head Start program, including plans to increase the number of children served and enhance the quality of the program, warrants definitive research studies, even though they may be costly”1 and (2) this information need could not be met because “. . .the body of research on current Head Start is insufficient to draw conclusions about the impact of the national program.” 2
Based on these reports, and on the testimony of research methodologists and early childhood experts, Congress included in the 1998 reauthorization of Head Start a mandate that the US Department of Health and Human Services (DHHS) determine, on a national level, the impact of Head Start on the children it serves.3 As noted by the Advisory Committee on Head Start Research, this legislative mandate requires that the impact study address two broad research questions:4
“What difference does Head Start make to key outcomes of development and learning (and in particular, the multiple domains of school readiness) for low-income children? What difference does Head Start make to parental practices that contribute to children’s school readiness?”
“Under what circumstances does Head Start achieve the greatest impact? What works for which children? What Head Start services are most related to impact?”
The Advisory Committee set forth a framework for research on the impact of Head Start that is both scientifically credible and feasible. The Committee acknowledged that the legislative language recommended the use of a rigorous methodology, including random assignment of children to Head Start and non-Head Start groups at a diverse group of sites, selected nationally and reflecting the range of Head Start quality across the country. To implement this design, DHHS awarded a contract in October 2000 to Westat of Rockville, MD, in collaboration with The Urban Institute, the American Institutes for Research, and Decision Information Resources.
Study Objectives and Research Questions
The first broad research question noted above can be divided into two parts: (1) the direct effect of Head Start on children’s early development and (2) the extent to which Head Start has an indirect effect by improving the ability of parents to support their children’s learning and development. Though not specifically identified, it is also valuable to understand the extent to which Head Start may affect the nature, duration, and quality of children’s early care and program experiences, which may, in turn, lead to improvements in school readiness. The second broad research question recognizes the importance of also understanding how the impact of Head Start may vary for different types of children and their families and in relation to the nature, duration, or quality of a child’s early care and program experiences. These broad research questions led to the specification of the following more detailed study questions that have guided the design and implementation of the Head Start Impact Study:5
- Impacts:
-
What impact does Head Start have on school readiness including children’s approaches to learning, language development and emergent literacy, mathematical ability, physical well-being and motor development, and social and emotional development?
-
What impact does Head Start have on parental practices that contribute to children’s school readiness (e.g., time spent reading to their child)? To what extent are these parenting practices related to child development outcomes?
-
What impact does Head Start have on the nature and quality of children’s early care and program experiences (e.g., the intensity of reading instruction)? To what extent are these experiences related to child development outcomes?
-
- Variation in impacts for certain subgroups of children and families:
-
Do impacts vary according to children’s characteristics at the time of entry into Head Start? Are there some subgroups that benefit while others do not? Subgroup characteristics include gender, race/ethnicity, age at program entry (3- vs. 4-year-olds), presence of disabilities, as well as the child’s status on a number of developmental characteristics (e.g., language ability) at the point of Head Start entry.
-
Do impacts vary by characteristics of the child’s home environment at the time of entry into Head Start? What particular environments lead to positive impacts? Home characteristics include family structure (e.g., single parent, teen mother), household income, and parental practices related to school readiness before exposure to Head Start.
-
Do impacts vary by the characteristics of the community where participants reside? In which types of communities does Head Start produce clear gains? Community characteristics include characteristics of the economic and social environment (e.g., poverty, unemployment rates), and the policy environment related to the availability and quality of alternative services for low-income children (e.g., state and local government funding for preschool programs).
-
- Variation in impacts related to characteristics that may be affected by Head Start participation:
-
Do impacts vary by parent’s ability to support their children’s development or by characteristics of the home environment (e.g., does the frequency with which an adult reads to a child influence literacy outcomes)? Which subgroups based on at-home supports gain from Head Start participation?
-
Do impacts on children vary by the nature, duration, and quality of their early care and program experiences? For example, do impacts vary by the amount of language instruction they receive?
-
Overview of the Study Design and Implementation
As discussed above, the primary purpose of this study is to determine whether Head Start has an impact on participating children and their parents and, if so, whether such effects vary among different types of children, families, communities, and configurations of children’s early care and program experiences. By impact we mean a difference between the outcomes observed for Head Start participants and what would have been observed for these same individuals had they not participated in Head Start. This focus on impacts distinguishes this study from many others that seek primarily to examine relationships among participant outcomes and between participant outcomes and one or more individual or program characteristics (see, for example, the Head Start Family and Child Experiences Study (FACES)6). Instead, the present study uses information from participants and a statistically equivalent group of children who do not participate in Head Start to determine whether Head Start caused the observed child and parent outcomes.
Given this goal of measuring program impacts, how do we determine what outcomes would have been observed if the children had not participated in Head Start? That is, how do we observe children having the same characteristics in two places at the same time in Head Start and not in Head Start and compare them? In many studies, researchers have addressed this problem by comparing program participants to a “participant-like” group of children who, in the ordinary course of events, do not participate in Head Start. However, even the best attempts at constructing such a comparable group of non-participants suffer from what evaluators call “selection bias.” That is, families who seek out, or “select,” Head Start for their children are likely to be different from those who do not on important factors that may lead to different outcomes independently of the effect of Head Start services. For example, parents who apply to Head Start may be more motivated to see that their children are well prepared to start school than those parents who choose not to seek Head Start enrollment. Moreover, the reasons why these parents make different decisions are both typically unobserved and likely to be related to the outcomes of interest in their own right. That is, the motivated parents do a host of things that may affect their children’s development beyond enrolling them in Head Start. Because all of these differences cannot be accounted for, there is a risk of misattributing to program participation observed differences on a particular outcome measure (e.g., emergent literacy) that may be a result of intrinsic differences between participant and non-participant families.
To avoid this problem of selection bias, the Head Start Impact Study randomly assigned a sample of 3- and 4-year-old Head Start applicants not previously served by the program,7 either to a treatment group (in which children and families received Head Start services) or to a control group (in which children were not granted access to Head Start but may have received a range of other services chosen by their parents). Under this randomized design, a simple comparison of outcomes for the two groups yields an unbiased estimate of the impact of Head Start on children’s school readiness. The advantage of this research design is that if random assignment is properly implemented with a sufficient sample size, program participants should not differ in any systematic or unmeasured way from non-participants except through their access to Head Start services.8
Sample Selection
Most randomized studies are conducted in small demonstration programs or, if done in an ongoing program, only in a small number of operating sites, usually those that volunteer to be included in the research. In contrast, the Head Start Impact Study is based on a nationally representative sample of both Head Start programs and newly entering 3- and 4-year-old children. That is, children applying for entry into Head Start in fall 2002, from a nationally representative sample of programs, were selected at random. This makes results generalizable to the entire Head Start program, not just the selected study sample. This approach responds to the congressional mandate and recommendations of the Advisory Committee that the study provide “a national analysis of the impact of Head Start” based on the selection of Head Start grantee/delegate agencies9 that “operate in the 50 states, the Commonwealth of Puerto Rico, or the District of Columbia and that do not specifically target special populations” and that also reflect variation on a variety of characteristics, including “region of the country, race/ethnicity/language status, urban/rural, and depth of poverty in communities,” and “. . .design of program as a one-year or two-year experience for children; program options (e.g., center-based, home-based, part-day, full-day); auspice (e.g., Community Action Agency, public school, non-profit organization); community-level resources; alternative child care options for low-income children; and, the nature of the child care market and the labor market in the community studied.”
[D] |
To meet these requirements, the study used a multi-stage sampling process to select a representative group of Head Start programs. The process, depicted in Exhibit 1.1, is described below:
-
Initial grantee/delegate agency selection. The sampling process began by using the Head Start Program Information Report (PIR) to create a list of 1,715 Head Start grantee and delegate agencies operating in fiscal year (FY) 1998-1999, after excluding (1) grantee/delegate agencies serving only special populations (migrant and tribal Head Start programs and sites serving only Early Head Start children), (2) grantees involved in the FACES 2000 study, and (3) as recommended in the Advisory Committee report (1999), grantees/delegate agencies that were “extremely new to the program.”10 This pool of 1,715 Head Start programs was subsequently organized into 161 “geographic clusters” (to increase the ability to closely monitor random assignment and obtain high quality data) and then grouped into 25 strata to control for factors such as region of the country, urban/rural location, race/ethnicity, and variation in state pre-kindergarten and child care policies. One cluster of programs was then randomly selected from each of the 25 strata with probability proportional to total enrollment, providing a total of 261 grantee or delegate agencies in the sampled clusters (to improve efficiency, random subsampling was done in three very large urban clusters).
-
Determining grantee/delegate agency eligibility. To be eligible for inclusion in the study sample, grantee/delegate agencies had to have enough “extra” or additional newly entering applicants beyond their number of funded slots to allow for the creation of a non-Head Start control group. That is, the programs could not be serving all the eligible children in their community who wanted Head Start, a situation we refer to as “saturation.” Ethically, random assignment could only be conducted in communities where Head Start programs were expected to be unable to serve all the eligible children seeking enrollment for fall 2002. This eligibility was determined from information verified through telephone calls to all 261 grantee/delegate agencies, augmented with information provided by Federal Regional Office staff and with data obtained from secondary sources such as local Child Care Resource and Referral Agencies, and the PIR. This screening process eliminated 28 grantees/delegate agencies (a reduction of 11 percent) found to be operating in saturated communities. Additionally, 10 other grantee/delegate agencies had been closed or merged, further reducing the pool of eligible programs to 223 grantee/delegate agencies.
-
Selecting grantee/delegate agencies. To ensure the inclusion of the full range of Head Start grantee/delegate agencies, smaller programs were combined with other agencies in the same cluster to form “grantee/delegate agency groups.” These groups (some of which consisted of a single grantee or delegate agency) were then stratified along several dimensions: urban location (central city, other urban, rural/small town), auspice (school based versus all other agency types), percentage Hispanic and percentage African American enrollment, program options offered (part-day only, full-day only, both), and the percentage of total enrollment represented by newly entering 3-year-olds. Approximately three grantee/delegate agency groups were randomly selected from each of the 25 strata with probabilities proportional to the number of newly entering children. This yielded a sample of 76 grantee/delegate agency groups comprising 90 individual grantee/delegate agencies, across 23 states.
-
Grantee/delegate agency recruitment. Senior project staff visited all 90 selected grantee/delegate agencies during summer 2001 to explain the study, verify information needed for study implementation, and to gain their agreement to participate in the Head Start Impact Study. Three agencies were dropped at this point—one had recently closed and two were dropped due to an overlap with a study being conducted by the federally funded Head Start Quality Research Center—leaving 87 grantee/delegate agencies in 76 grantee/delegate agency groups (i.e., the overall number of grantee/delegate agency groups was not reduced).
-
Identifying operating Head Start centers. Because administrative data do not identify individual Head Start centers, each of the 87 grantee/delegate agencies was asked to provide a list of all centers expected to be in operation for the 2002-03 program year and to validate basic data about the characteristics of children served, program options, and enrollment patterns in each center. This resulted in a list of 1,427 Head Start centers in the 87 grantee/delegate agencies (76 grantee groups) that could potentially be included in the Head Start Impact Study.
-
Determining center eligibility and selecting a sample of study centers. The center-level data were first used to eliminate 169 centers determined to be “saturated,” as was done previously for grantee/delegate agencies. This step reduced the total eligible pool of centers from 1,427 to 1,258 across 84 separate grantee/delegate agencies in 76 grantee/delegate agency groups (a reduction of about 11 percent and the loss of three grantee/delegate agencies, but no grantee groups). Next, small centers were combined with nearby centers, and the resulting “center groups” were then stratified using the same characteristics used for the selection of grantee/delegate agencies (excepting those that do not vary within grantee/delegate agencies such as a region). A main sample consisting of an average of three center groups was selected from each eligible grantee/delegate agency, resulting in a main sample of 448 centers in 84 grantee/delegate agencies.
More in-depth or up-to-date information on the initially sampled centers led to a determination that some were, in fact, ineligible for inclusion in the study. These included centers that: (1) had recently closed or had been merged with other centers; (2) served only Early Head Start children; (3) were collaborations between Head Start and private preschool programs that could not subject their entire pool of applicants to random assignment; or (4) were, in fact, saturating their community with Head Start services. These findings resulted in the dropping of 103 initially sampled centers, but the addition of 38 replacement centers11 to yield a sample of 383 centers.
This sample of Head Start grantee/delegate agencies and centers, when properly weighted (see Appendix 1.2), was designed to yield a sample of children that represents the national population of newly entering children and their families (with the exclusions noted above).
Random Assignment
At each of the selected Head Start centers, program staff provided information about the study to parents at the time enrollment applications were distributed. Parents were told that enrollment procedures would be different for the 2002-03 Head Start year and that some decisions regarding enrollment would be made using a “lottery-like” process.12
The study team assigned local site coordinators to work with grantee/delegate agencies in each of the 25 geographic clusters to ensure that parents received this information with their applications. These site coordinators were also responsible for obtaining data on all applications for the 2002-03 program year (to ensure equal treatment of all applicants) and listing these data on a roster that was subsequently key-entered by central office study staff. Returning children, and a small number of grantee-requested “high-risk” exclusions,13 were eliminated from these lists, and checks were made for duplicate records. The high-risk exclusions were made on a case-by-case basis with each grantee/delegate agency and in close consultation with Administration for Children and Families staff. Examples of such exclusions included children of homeless families, children in families with documented abuse and neglect, and children with severe disabilities, especially those disabilities that would make it difficult to test these children and include them in the study sample (e.g., blindness). Each grantee was limited to one exclusion per center for its total pool of newly entering children. In fact only 276 exclusions were taken out of a total of approximately 18,000 newly entering applications.
At this point, local agency staff implemented their typical process of reviewing enrollment applications and screening children for admission to Head Start based on criteria approved by their respective Policy Councils. No changes were made to these locally established admission criteria. Site coordinators recorded basic information about each applicant and what was usually a numerical score determined by local staff that signified the relative need of individual children (e.g., in some agencies, a higher score indicated a greater need for Head Start and a corresponding higher priority for admission). Using these rankings, the list of newly entering children who would ordinarily have been enrolled was “extended” to add a specified number of children needed for the non-Head Start control group. The children added were those who would normally be “next in line” for admission if the initially targeted children could not be enrolled.
The goal was to randomly select, on average, 27 children from the expanded list at each of the sampled centers or center groups: 16 to be assigned to the Head Start group and 11 to be assigned to the non-Head Start group. For an average center group, the 11 non-Head Start control group children represented about 9 percent of total enrollment. Where necessary, stratification was used, such as in situations where the degree of saturation varied by program option (part-day versus full-day) or age cohort. In some cases, where fewer children than expected were actually available, a smaller sample of children was selected for the study.
The original legislative mandate required that the Head Start Impact Study “to the extent practicable” address possible variation in program impact related to “the length of time a child attends a Head Start program (and) the age of the child on entering the Head Start program.” This requirement reflects the hypothesis that different program impacts may be associated with 1 versus 2 years of Head Start experience. It also reflects a trend of increased enrollment of 3-year-olds in some grantee/delegate agencies presumably due to the growing availability of preschool options for 4-year-olds (often state-sponsored programs). Consequently, the study included two separate samples: a newly entering 3-year-old group (to be studied through 2 years of Head Start participation, kindergarten, and 1st grade), and a newly entering 4-year-old group (to be studied through 1 year of Head Start participation, kindergarten, and 1st grade). The 3-year-old group is slightly larger than the 4-year-old group to protect against the possibility of higher study attrition resulting from an additional year of longitudinal data collection for the younger children.14
Within the final set of 76 grantee/delegate agency groups (or 84 total grantees/delegate agencies), random assignment was attempted at a total of 383 randomly selected Head Start centers. Of these, random assignment could not be completed in only five centers (or 1.3 percent), resulting in a final sample of 378 centers with successful random assignment. However, as noted above, the full desired sample could not be obtained at each center, resulting in the following situations:
-
Obtained Full Sample. Random assignment was completed at 173 Head Start centers that provided the full expected sample of children.
-
Obtained Smaller Sample. Random assignment was completed at 150 Head Start centers that provided a smaller than expected sample (i.e., because new application rates were lower than estimated).
-
Obtained Larger Sample. Random assignment was completed at 55 Head Start centers that provided a larger than expected sample (i.e., because application rates for newly entering children were higher than originally estimated, sample sizes were increased to compensate for other centers that were unexpectedly low).
In total, 4,667 newly entering children were randomly assigned and included in the Head Start Impact Study (see Exhibit 1.2).
| Age Group | Head Start (Treatment) Group | Non-Head Start (Control) Group | Total Sample |
|---|---|---|---|
| 3-year-olds | 1,530 | 1,029 | 2,559 |
| 4-year-olds | 1,253 | 855 | 2,108 |
| Total | 2,783 | 1,884 | 4,667 |
As indicated above, about 60 percent of the sample was assigned to the Head Start group and about 40 percent was assigned to the non-Head Start group. This imbalance reduces the precision of the impact estimates by less than 2 percent (compared to a balanced 50-50 design). However, it provided several important benefits: (1) it significantly increased the ability to recruit Head Start grantees and centers by decreasing the number of extra children needed for the control group, (2) decreased the loss of sites due to saturation, and (3) saved considerably on data collection costs because treatment group members (who participate in Head Start) require less effort to track and interview over time than children in the non-Head Start control group.
Data Collection
Data collection began in fall of 2002 and will continue through the spring of 2006, following children from age of entry into Head Start through the end of the preschool years, end of kindergarten, and end of 1st grade. Comparable data are being collected for both Head Start and non-Head Start children and consist of the following:
-
Measures of children’s development that include (1) direct child assessments, (2) parent reports, and (3) teacher/care provider reports. Child outcomes are measured in the key domains of cognitive development (including assessment of skills in the areas of reading, writing, vocabulary, oral comprehension and phonological awareness, and math), social-emotional development, and health.
-
Characteristics and quality of children’s home environments are measured through (1) parental reports of beliefs and attitudes about their child’s learning and parental participation in, and satisfaction with, their child’s child care experience; (2) family household and demographic information, including parent-child relationships and the quality of the child’s home life; (3) parent ratings of their child’s behavior problems, social skills, and competencies; (4) parent’s perceptions of their child’s accomplishments; (5) parent’s perception of their relationship with their child; and (6) child and family receipt of a variety of comprehensive services.
-
Characteristics and quality of the primary preschool and child care arrangement as measured through (1) interviews with center-based directors, (2) surveys of teachers or interviews with care providers, and (3) observations of these settings.
To complete these data collection activities in the 25 geographic areas where children and families were sampled, the study uses measurement teams comprising local field interviewers/assessors and observers who work under the supervision of a site coordinator tasked with ensuring that all aspects of local data collection are completed during the field period.
This report focuses on the findings of the first year of data collection after random assignment of both Head Start and non-Head Start children. The following describes the data sources and measures used during the first year. Response rates and subsequent data collection plans are also summarized.
Fall 2002 Data Collection
Baseline data were collected in fall 2002 and included in-person interviews with the parent/primary caregiver of each study child and one-on-one child assessments conducted by the local interviewers/assessors:
-
Parent/Primary Caregiver Interviews. Parent interviews were typically conducted in the child’s home with a parent or primary caregiver living with and responsible for raising the child. Parent interviews were available in both English and Spanish versions, and bilingual English/Spanish speakers were hired for areas with Spanish-speaking families. For other languages, either interviewers/assessors fluent in these languages were hired or other local resources were asked to identify interpreters to aid in completing the parent interviews.
-
Child Assessments. Child assessments provide direct measures of how well Head Start and non-Head Start preschool programs, or other child care, are achieving the goal of assisting children to be physically, socially, and educationally ready for success in kindergarten. The assessment battery (see Exhibit 1.3) is composed of a short series of tasks that are feasible and interesting for preschoolers to carry out and that have been shown to be predictive of later school success (test citations are provided in Appendix 1.4). The 35 to 45 minute child assessment battery was typically administered one-on-one by specially trained assessors in the child’s “main” care setting, i.e., where the child spends the most time Monday through Friday between the hours of 9 AM and 3 PM.
At the time of the assessment, the interviewer/assessor asked the main care provider a series of questions to determine the appropriate language for the assessment (see Appendix 1.3). For children requiring assessment in Spanish, a bilingual interviewer/assessor administered the assessment battery in Spanish and also administered two subtests in English, i.e., the Peabody Picture Vocabulary Test (adapted) (PPVT) and the Woodcock-Johnson III Letter-Word Identification. In spring 2003, the children assessed in Spanish in fall 2002 were assessed primarily in English, along with the continued administration of two Spanish language measures: the Test de Vocabulario en Imágenes Peabody (TVIP) and the Batería Woodcock-Muñoz Identificación de Letras y Palabras. One exception is Puerto Rico where, because instruction is in Spanish, all children were assessed only with the complete Spanish battery in spring 2003. For children who could not be assessed in either English or Spanish, a bilingual interviewer/assessor or an interpreter for the child’s language were used. The interviewer/assessor (or interpreter) used the English assessment booklet, translated the instructions into the child’s language, and administered four subtests: McCarthy Draw-A-Design, Color Names and Counting, Leiter-R-Adapted, and Story and Print Concepts. For the spring assessments, these children were all tested in English.
In addition, site coordinators visited all study Head Start centers to ascertain whether children assigned to Head Start were, in fact, attending and whether any control group children had been inadvertently enrolled in Head Start.
Fall 2002 data collection was completed by mid-November for the majority of children and parents (although a small number did extend into December). The implication of this late baseline data collection is discussed in Chapter 4, along with procedures used to deal with it in the analysis of program impact.
Winter 2003 Parent Updates
In the winter of 2003, a 10-minute telephone interview was conducted with parent/primary caregivers; in some instances, in-person interviews were conducted. These short interviews were designed to obtain up-to-date contact information and child care setting information critical to determining the appropriate setting for spring 2003 data collection.
| Assessment Areas | Measure | English-Speaking Children | Spanish-Speaking Children* | Children Who Spoke Languages Other than English or Spanish | |
|---|---|---|---|---|---|
| Language Development & Literacy | Peabody Picture Vocabulary Test III (PPVT III, adapted) | X | X | -- | |
| Test de Vocabulario en Imágenes Peabody (TVIP, adapted) | -- | X | -- | ||
| Comprehensive Test of Phonological and Print Processing (CTOPPP):
|
|
X | -- | -- | |
|
X | -- | -- | ||
| Comprehensive Test of Phonological and Print Processing (CTOPP)(Spanish version): |
|
-- | X | -- | |
|
-- | X | -- | ||
| Woodcock-Johnson III: |
|
X | -- | -- | |
|
X | X | -- | ||
|
X | -- | -- | ||
| Woodcock-Muñoz R: |
|
-- | X | -- | |
|
-- | X | -- | ||
| Story and Print Concepts *** | X | X | X | ||
| Letter Naming Task *** | X | X | X | ||
| Mathematics | Counting Bears Task *** | X | X | X | |
| Woodcock-Johnson III: Applied Problems | X | -- | -- | ||
| Woodcock-Muñoz R: Problemas Aplicadas | -- | X | -- | ||
| Early Writing | McCarthy Draw-a-Design | X | X | X | |
| Woodcock-Johnson III: Spelling | X | -- | -- | ||
| Woodcock-Muñoz R: Dictado | -- | X | -- | ||
| Other Cognitive Ability | Color Names *** | X | X | X | |
| Sustained Attention | Leiter-R Attention Sustained Task (adapted) *** | X | X | X | |
| Assessor Ratings | Task persistence | X | X | X | |
| Attention span | X | X | X | ||
| Body movement | X | X | X | ||
| Attention to directions | X | X | X | ||
| Comprehension of directions | X | X | X | ||
| Verbalization | X | X | X | ||
| Ease of relationship | X | X | X | ||
| Confidence | X | X | X | ||
* The Spanish version uses the Woodcock-Muñoz R. In this version, the Dictation subtest is used in place of the Spelling subtest. Children in Puerto Rico were administered only the Spanish subtests. |
Spring 2003 Data Collection
In spring 2003, the interviewers/assessors again conducted in-person parent interviews and child assessments. Additional information was obtained from in-person interviews with directors of the Head Start and non-Head Start centers that study children attended, and teachers and other care providers completed self-administered questionnaires to rate each of the study children who were in their classroom or care. Teachers also completed questionnaires, and care providers were interviewed in person, to obtain information about themselves, the nature of the setting in which they work, and the types of services they provide to the selected study children. To further measure quality of care, direct observations of classrooms and family day care homes were conducted. To obtain comparable information on quality across all care settings, a five-question observational instrument was completed in every care setting visited, including the child’s own home. Each of these activities is described below:
-
Parent/Primary Caregiver Interview. Once again, the parent interviews were conducted in the child’s home with a parent or primary caregiver living with and responsible for raising the child. The interviews were conducted in the parent’s language with English and Spanish versions available. Parents speaking other languages were interviewed with the aid of an interpreter. Some topics added for the spring included the child’s transition from preschool to kindergarten and any information on services the family received to assist with this transition.
-
Child Assessments. In spring 2003, the same fall assessment battery was administered with the addition of a Letter Naming Task. Children previously assessed in Spanish were assessed in English, but these children were also administered two Spanish language tests (see Exhibit 1.2).
-
Care Setting Observation. Direct observations of care setting and quality were used for children in center-based and family day care home programs, including those participating in Head Start. These tools provide direct measures of the extent to which Head Start centers, and other childcare programs, employ skilled teachers and provide developmentally appropriate environments and curricula for their pupils. Trained observers conducted observations in classrooms and centers attended by the sampled children. Observers spent enough time in each class to ensure observation of a major portion of the daily schedule and a variety of classroom and center activities.
The observers used standardized observational methods and coding schemes that have been widely used in child development research and whose utility has been proven in previous large-scale studies. These include: the Early Childhood Environment Rating Scale (revised) (ECERS-R), the Classroom Observation of Teacher-Directed Activities Checklist, the Arnett Scale of Teacher/Provider Behavior, and the related Family Day Care Rating Scale (FDCRS) for observations in non-center-based settings. In the interest of having some comparable observational measure of quality across all settings, a five-question observational instrument designed for use in formal care settings as well as in the home was developed. These items are completed by interviewers regardless of whether the child’s care setting is Head Start, formal child care, or at home with a parent or other care provider. Assessors rate all care settings in five areas similar to the areas observed with the ECERS-R and FDCRS: overall safety, basic hygiene standards, availability of educational materials, and overall positive and negative interaction between provider and child.
Teacher Surveys and Care Provider Interviews. Teachers and other care providers are asked to describe themselves, the nature of the care setting in which they work, and the types of services they provide to the selected study children. This includes biographical information such as education and years of experience, characteristics of the center or child care program, quality of program management, and belief scales to assess staff attitudes about working with and teaching children. Items on the use of literacy-promoting activities are included in the teacher survey and care provider interviews, as well as in the center director interview (see below). An “other-care provider” interview is used to collect comparable information regarding child care for non-Head Start children who were in non-center based settings or at home with a relative or non-relative (other than the parent or primary caregiver15). This interview includes questions on the number of children in the care setting, types of child activities used, beliefs on how children should be taught and managed, options for parent and family involvement, staffing, and respondent demographic information.
Teacher’s/Care Provider’s Child Reports (TCRs). Teachers and other care providers are also asked to rate each of the children in their classroom or care who are participating in the study. The following scales are used: teacher/provider relationship with child, classroom behavior and conduct, problem solving and initiative, social relationships, creative representation, music and movement, language ability, and mathematical ability. Parent and teacher/other care provider ratings of children’s accomplishments and behavior are obviously not as objective as direct assessments or observations by impartial observers. Nevertheless, such ratings are an important source of information about children’s learning and behavior because parents, teachers, and care providers see children over extended periods of time and in a variety of settings, providing for more robust appraisals of children’s skills and competence.
Center Director Setting Interviews. This in-person interview is used to collect information on the operation and quality of Head Start and non-Head Start center-based programs. Issues addressed in this interview include: staffing and recruitment, teacher education initiatives and staff training, parent involvement, curriculum, classroom activities and assessment, home visits, kindergarten transition, and demographic information about the director.
Future Data Collection
In subsequent years, the fall telephone (and, where necessary, in-person) parent updates will be continued to obtain critical contact and care setting information. In the spring, through the child’s 1st grade year, in-person parent interviews will be conducted as will the direct child assessments, with the test battery modified for the kindergarten and 1st grade years. However, once the child is in elementary school, all assessments will be conducted in the child’s home.
The second spring data collection (i.e., spring 2004) has been completed, but the data have not yet been analyzed. Consequently, this report only focuses on the findings from the first study year. For the kindergarten battery, three additional Woodcock-Johnson III subtests were added: Passage Comprehension, Word Attack, and Quantitative Concepts (Concepts and Number Series). In addition, a Writing Name task has been added, and the McCarthy Draw-A-Design, Color Names & Counting, Comprehensive Test of Phonological and Print Processing (CTOPPP) Print Awareness and Story & Print Concepts have been deleted from the kindergarten battery.
Classroom observations are not being conducted in the elementary schools, but information will be collected in the spring from classroom teachers and other care providers by asking them to complete self-administered surveys and teacher reports on individual children. During the kindergarten year, the teacher survey obtains information about the kindergarten program, provisions that are made for the child’s transition to kindergarten, and whether the teacher obtained any information from the Head Start program or alternative care provider about the child’s development status or special needs. Also, at the kindergarten and 1st grade level, school-level data will be obtained by linking schools attended by study children to annual data collected from every public school in the U.S. by the Department of Education’s National Center for Education Statistics (NCES). This includes the Common Core of Data for Public and Private Elementary Schools (CCD), and the Schools and Staffing Survey—Data for Public and Private Elementary Schools. We also plan to augment the NCES data by linking to state- and district-level data that are publicly available as school “report cards” under state accountability systems.
Response Rates
As shown in Exhibit 1.4, the individual response rates for both child assessments and parent interviews completed for the two data collection periods addressed in this report have been very good. Overall, 83 percent of parents completed interviews at both points in time, and 82 percent of the children were assessed. There is some difference between Head Start and non-Head Start groups, but the gap was slightly narrowed by the spring 2003 interview.16
| Instrument | Fall 2002 | Spring 2003 | ||
|---|---|---|---|---|
| Head Start (Treatment) | Non-Head Start (Control) | Head Start (Treatment) | Non-Head Start (Control) | |
| Child Assessments | 85% | 72% | 88% | 77% |
| Parent Interview | 89% | 81% | 86% | 79% |
These response rates represent the actual (unweighted) number of interviews completed, i.e., the percentage of the sampled population that completed the interview or assessment. However, the weighted response rates are best for assessing the potential for nonresponse bias. The various levels of sampling where nonresponse can occur are: (1) nonresponding programs in fall 2002; (2) nonresponding centers in fall 2002; (3) additional nonresponding programs in spring 2003; (4) additional nonresponding centers in spring 2003; and (5) nonresponding children/parents in spring 2003. The overall response rate for impact estimates in the spring is the product of the response rates for each of these five levels.
Response rates of 100 percent were achieved among programs in both fall and spring, and there were no additional nonresponding centers in spring 2003. The fall center response rate was 98.9 percent, and the weighted spring child assessment response rate was 86.9 percent for the Head Start group and 76.5 percent for the non-Head Start group. Therefore, the product of the response rate for these five levels for child assessments is 80.9 percent (85.9% for the Head Start group and 75.6% for the non-Head Start group). The overall weighted response rate for parent interviews was nearly identical at 81.0 percent (86.9% for the Head Start group and 76.5% for the non-Head Start group).
Although the response rate is relatively high, bias in estimates of the impact of Head Start can occur to the extent that the impact differs between responding and nonresponding centers and between responding and nonresponding children and parents. As part of the weighting procedure, separate nonresponse adjustment factors were applied for categories of centers and children (see Appendix 1.2 for details). To the extent that nonrespondents and respondents within a category have similar impacts from Head Start, the application of these adjustment factors reduces the bias due to nonresponse.
Contents of This Report
This report is a preliminary examination of the impact of Head Start for children who applied to Head Start in 2002. It includes a subset of the first year of child and parenting practice outcomes as of spring 2003. This is just the precursor to the wealth of information that this study will eventually provide. This report is two volumes. In this volume, this chapter (Chapter 1) presented the study background, including an overview of the study objectives, sample design, data collection, and response rates. Chapter 2 provides further details about the study sample, including a description of child and parent characteristics measured before and after random assignment. In order to provide a context in which to understand the impact findings, Chapter 3 provides a discussion of the impact of Head Start on the types of preschool and child care settings that parents selected for their children as well as descriptive information on the characteristics of different types of early care arrangements. Chapter 4 presents an overview of the methods used for analyzing impacts on children and families.
The remaining chapters present the results of the impact analyses. The impact of Head Start on children’s cognitive development is presented in Chapter 5, focusing on six different cognitive constructs, i.e., pre-reading, pre-writing, vocabulary, oral comprehension and phonological awareness, early math skills, and parent reports of children’s literacy skills. The impact of Head Start on social-emotional development is presented in Chapter 6, focusing on parent reported measures of social competencies, social skills and positive approaches to learning, and problem behaviors. Chapter 7 presents findings on the impact of Head Start on children’s health status and access to health care. Chapter 8 presents the findings on the impact of Head Start on parenting practices in the areas of educational activities, discipline strategies, and child safety practices. In addition, a number of technical appendices present further details about the study design, study sample, and analytic techniques.
1 US General Accounting Office. (1998). Head Start: Challenges in Monitoring Program Quality and Demonstrating Results. Washington, DC: Author. (back)
2 US General Accounting Office. (1997). Head Start: Research Provides Little Information on Impact of Current Program. Washington, DC: Author. (back)
3 See Appendix 1.1 for the research-related amendments to the Head Start Act included in the 1998 reauthorization. (back)
4 Advisory Committee on Head Start Research and Evaluation (1999). Evaluating Head Start: A Recommended Framework for Studying the Impact of the Head Start Program. Washington, DC: US Department of Health and Human Services. (back)
5 For more details on the design of the Head Start Impact Study see: Administration for Children and Families. (2003). The Head Start Impact Study: Research Design and Preliminary Analysis Plan. Washington, DC: US Department of Health and Human Services. (back)
6 US Department of Health and Human Services, Administration for Children, Youth and Families. (2003). Head Start FACES 2000: A Whole-Child Perspective on Program Performance, Fourth Progress Report. Washington, DC: Author. (back)
7 The Head Start Impact Study focuses on newly entering children to ensure that the estimated impacts are unaffected by previous program participation. Consequently, children who were returning to Head Start, as well as those previously enrolled in Early Head Start, were excluded from the study sample. (back)
8 More precisely, there will be differences between individuals in the two groups, but the expected or average value of these differences is zero except through the influence of Head Start (i.e., selection bias is removed by random assignment). (back)
9 The study sample includes both Head Start grantees and their delegate agencies. Grantees are organizations that have fiscal and administrative responsibility for programs in their jurisdiction. In some cases, they can subcontract with agencies to handle administrative oversight over some or all of these programs. Throughout this report we use the term grantee/delegate agencies to refer to both types of agencies. (back)
10 Defined as in operation for fewer than 2 years. (back)
11 A “reserve” sample of an average of two center groups per program (a total of 237 centers) was also selected to be used as replacement sites if needed to achieve the expected overall study sample size of children. Thirty-eight of these centers were used. The final sample was 383 (448-103+38) centers. (back)
12 Children randomly assigned to the non-Head Start group were not to be admitted to Head Start during 2002-03. Those who were in the 3-year-old group, however, were told that they could re-apply for Head Start in 2003-04 and may be admitted if eligible. (back)
13 This decision was made because: (1) there were ethical concerns about assigning very high-risk children to the non-Head Start group, especially in situations where Head Start may provide their only option for early childhood services; (2) a previously conducted study demonstrated that the potential exclusion of those most severely in need affected cooperation when trying to recruit study sites; and (3) there were some children who could not be assigned to the non-Head Start group because of placement by the local child welfare agency. (back)
14 This roughly equal sampling of 3- and 4-year-old applicants was done despite the fact that 4-year-olds represent about twice the proportion of all Head Start participants as do 3-year-olds. In large part, this is because the 4-year-olds include both newly entering 4-year-olds plus returning children who began Head Start as 3-year-olds and who have turned 4 years of age in their second year of program participation. (back)
15 Some questions from this interview were also added to the parent/primary caregiver interview to obtain comparable quality of care information for children whose care settings are their own homes. (back)
16 A high response rate has been maintained in subsequent data collection efforts. In fall 2003, 87 percent of the Head Start group and 79 percent of the non-Head Start group were interviewed. For spring 2004, 84 percent of the Head Start parents and children and 76 percent of the non-Head Start parents and children were interviewed and/or assessed. (back)
| Table of Contents | Previous | Next |