Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

II. EVALUATION DESIGN, DATA, AND ANALYTIC APPROACHES

ACYF designed a thorough and rigorous evaluation to examine the impacts of Early Head Start on key child and family outcomes. This chapter summarizes the study design, the data sources and outcome variables used in this report, and our approach to conducting the impact analysis.

A. STUDY DESIGN

The evaluation was conducted in 17 sites where Early Head Start research programs were located. Once selected for participation in the study, programs began enrolling families and worked with MPR staff to coordinate with the requirements of random assignment.

1. Site Selection

When the 68 Early Head Start programs in the first wave were funded in late 1995, they agreed, as a condition of funding, to participate in local and national research if selected to do so. In March 1996, 41 university research teams submitted proposals to the Head Start Bureau—in partnership with Wave I Early Head Start program grantees—to conduct local research and participate in the national evaluation. ACYF purposively selected 15 research sites, using a number of criteria: (1) programs had to be able to recruit twice as many families as they could serve; (2) programs had to have a viable research partner; and (3) in aggregate, programs had to provide a national geographic distribution that represented the major programmatic approaches and settings and reflected diverse family characteristics thought to be typical of Early Head Start families nationally. Applying these criteria resulted in fewer center-based programs than desired, so in 1996 ACYF selected one additional center-based program from Wave I, and in late 1997 selected another center-based program (without a local research partner) from Wave II programs (75 of which were funded in mid-1996), resulting in the full sample of 17 programs.

Because the 17 research programs were not randomly selected, the impact results cannot be formally generalized to all Early Head Start programs funded during 1995 and 1996. Instead, the results can be generalized only to the 17 programs themselves (that is, the impact results are internally valid). However, as shown in Chapter I (Table I.2), the features of the 17 programs, as well as the characteristics of their enrolled families and children, are similar to those of all Early Head Start programs in 1995 and 1996. Thus, to the extent that the quality and quantity of services offered in the 17 programs are similar to those offered nationwide, our findings about effective program practices and their impacts on children and families are likely to pertain to Early Head Start programs more broadly.

2. Sample Enrollment

Although Wave I grantees entered Head Start with varying degrees and types of experiences (see Chapter I), all had been asked not to enroll any families until it was decided whether they would be selected for the research sample. Because all programs had agreed, in submitting their original proposals, to participate in the random assignment process if they were selected for the research sample, it was not necessary to persuade any of the programs to cooperate. Thus, as soon as the programs were selected, beginning in spring 1996, MPR staff began working with their staffs to implement the random assignment process in conjunction with each program’s regular enrollment procedures. Except for recruiting about twice as many families as they could serve, programs were expected to recruit as they would in the absence of the research, with special instructions to be sure to include all the types of families that their program was designed to serve (including those whose babies had disabilities). MPR and ACYF created detailed procedures (outlined in a “frequently asked questions” document—see Appendix E.II.A) to guide the sample enrollment process.

3. Random Assignment

As soon as programs determined through their application process that families met the Early Head Start eligibility guidelines, they sent the names to MPR, and we entered the names and identifying information into a computer program that randomly assigned the families either to the program or to the control group (with equal probabilities). Program staff then contacted the program group families, while representatives of the local research partners notified the control group families of their status.

Control group families could not receive Early Head Start services until their applicant child reached the age of 3 (and was no longer eligible for Early Head Start), although they could receive other services in the community. This ensures that our analytic comparisons of program and control group outcomes represent the effects of Early Head Start services relative to the receipt of all other community services that would be available to families in the absence of Early Head Start.

Some program staff were concerned that random assignment might, by chance, result in denial of services to families with particularly high service needs. ACYF was very clear, however, that the study findings should pertain to all families and children that Early Head Start was designed to serve, including infants and toddlers with disabilities. To address program concerns, however, ACYF and MPR established a process by which programs could apply to have a family declared exempt from participating in the research. ACYF received only one request for an exemption, and it was not considered to be warranted.

Sample enrollment and random assignment began in July 1996 and were completed in September 1998. In most sites, sample intake occurred over a two-year period, although some took less time. The extended enrollment period was due in part to the extra work involved in recruiting twice as many families as programs were funded to serve, and in part to the process of new programs working out their recruitment procedures. Two programs completed sample enrollment in late 1997, and one (the 17th site) did not begin sample intake until fall 1997. Thus, the study population for the evaluation includes Early Head Start-eligible families who applied to the program between late 1996 and late 1998.

During the sample intake period, 3,001 families were randomly assigned to the program (1,513) and control (1,488) groups (Table II.1). The samples in most sites include between 150 and 200 families, divided fairly evenly between the two research groups.

Early Head Start staff implemented random assignment procedures well. We estimate that about 0.7 percent of all control group members received any Early Head Start services (that is, were “crossovers”), and most sites had no crossovers.1 Furthermore, our discussions with site staff indicate that information on nearly all eligible families who applied to the program during the sample intake period was sent to MPR for random assignment. Program staff did not provide Early Head Start services to families who were not submitted for random assignment. Hence, we believe that the research sample is representative of the intended study population of eligible families, and that any bias in the impact estimates due to contamination of the control group is small.

Random assignment yielded equivalent groups: the average baseline characteristics of program and control group members are very similar (Appendix D). This is as expected, because MPR used computer-generated random numbers to assign families. Therefore, the only difference between the two research groups at random assignment was that the program group was offered Early Head Start services and the control group was not. Thus, differences in the subsequent outcomes of the two groups can be attributed to the offer of Early Head Start services with a known degree of statistical precision.

TABLE II.1

EVALUATION SAMPLE SIZES, BY SITE AND
RESEARCH STATUS

Site	Program Group	Control Group	Combined Sample
1	74	77	151
2	93	86	179
3	84	78	162
4	75	72	147
5	74	76	150
6	115	110	225
7	104	108	212
8	98	98	196
9	98	95	193
10	71	70	141
11	104	96	200
12	73	79	152
13	104	98	202
14	75	71	146
15	90	92	182
16	95	95	190
17	86	87	173
All Sites	1,513	1,488	3,001
NOTE: Sites are in random order.

B. DATA SOURCES AND OUTCOME MEASURES

Comprehensive data from multiple sources were used to examine the effects of Early Head Start participation on a wide range of child, parenting, and family outcomes. This section provides an overview of data sources and outcome measures used for the analysis, the response rates to the interviews and assessments, and the timing of interviews. These topics are discussed in greater detail in the Appendixes.

1. Data Sources

The follow-up data used for the analysis were collected at time points based on (1) the number of months since random assignment, and (2) the age of the focus child. Each family’s use of services and progress toward self-sufficiency were seen as likely to be a function of the amount of time since the family applied for Early Head Start services. Therefore, these data were collected at selected intervals following random assignment. Other data—particularly those related to child and family development—were more likely to be a function of the increasing age of the focus child over time. Thus, the data collection schedule for these developmental outcomes was tied to children’s birth dates. The data sources used in this report include:

1. Parent Services Follow-Up Interview (PSI) Data Targeted for Collection 6, 15, and 26 Months After Random Assignment. These data contain information on (1) the use of services both in and out of Early Head Start (such as the receipt of home visits, and of services related to case management, parenting, health, employment, and child care); (2) progress toward economic self-sufficiency (such as employment, welfare receipt, and participation in education and training programs); (3) family health; and (4) children’s health. Most PSIs were conducted by telephone with the focus child’s primary caregiver, although some interviews were conducted in person for those not reachable by phone.

2. Exit Interview When Children Reached 36 Months of Age. These interviews were conducted only with program group families when their children were 36 months old and had to transition out of Early Head Start. The exit interviews obtained information on the use of services in Early Head Start. Whenever possible, the interviews were conducted in conjunction with the 36-month parent interviews (see below), but in some cases were conducted in conjunction with the 26-month parent services interviews.

3. Parent Interview (PI) Data Targeted for Collection When Children Were 14, 24, and 36 Months Old. These interviews obtained a large amount of information from the primary caregivers about their child’s development and family functioning. These data usually were collected in person, but some PIs or portions of them were conducted by telephone when necessary.

4. Child and Family Assessments Targeted for Collection When Children Were 14, 24, and 36 Months Old. Field interviewers provided data on their observations of children’s behavior and home environments. Interviewers conducted direct child assessments (such as Bayley assessments) and videotaped structured parent-child interactions. Several measures constructed using these data overlap with those constructed from the PI data, which allowed us to compare impact findings using the two data sources.

5. Father Interviews Targeted for Collection When Children Were 24 and 36 Months Old. In addition to asking mothers about the children’s father, we interviewed the men directly about fathering issues at the time of the 24- and 36-month birthday-related interviews.2 The father study was conducted in 12 sites only. Father observational data were collected in 7 sites.

6. Baseline Data from the Head Start Family Information System (HSFIS) Program Application and Enrollment Forms. We used these forms, completed by families at the time of program application, to create subgroups defined by family characteristics at baseline, and to adjust for differences in the characteristics of program and control group members when estimating program impacts. We also used the forms to compare the characteristics of interview respondents and nonrespondents, and to construct weights to adjust for potential nonresponse bias.

7. Baseline Data from Selected Sites on Mother’s Risk of Depression. Local researchers in eight sites administered the Center for Epidemiologic Studies Depression Scale (CES-D) at baseline. These data were used in the subgroup analysis to assess whether impacts differed for mothers at risk of depression and for those who were not.

8. Data from the Implementation Study. Finally, the analysis used data from the implementation study to define subgroups based on program characteristics (such as program approach and level of program implementation) and site characteristics (such as urban or rural status and welfare regulations).

MPR prepared all the follow-up data collection instruments and trained all field staff. In all sites but one (where MPR collected the data), data collection field staff were hired by the local research teams, who were responsible, under subcontract to MPR, for collecting the data and monitoring data quality. Respondents were offered modest remuneration and a small gift to complete each set of interviews and assessments. Appendix B describes the data collection procedures in greater detail. Details about all the measures can be found in Chapter V and in Appendix C.3

It is important to recognize that linking PIs and child and family assessments to the age of the child, rather than to a fixed period after random assignment, means that at the time those instruments were administered, families were exposed to the program for different lengths of time. Nevertheless, questions about children’s development at particular ages are policy relevant. It is also of policy interest, however, to assess impacts for children and families with similar lengths of exposure to the program. Therefore, as described in Section C, we estimated impacts by doing subgroup analyses based on the child’s age at random assignment (so that program exposure times would be similar within each age group).

It is also important to recognize that at the 14-month birthday-related interviews, many families had been exposed to Early Head Start for only a short time, and especially so for families with older focus children. Thus, we did not expect impacts to appear at 14 months. In this report, we focus on the child, parenting, and family outcomes when children are 2 and 3 years old.

In sum, in this report we present impact findings using follow-up data from the 6-, 15-, and 26-month PSIs, from the exit interview, and from the 14-, 24-, and 36-month PIs and child and family assessments. Thus, our impact findings cover the first three years of the focus children’s lives. A longitudinal study is underway that will follow and interview program and control group families just before the focus children enter kindergarten to assess the longer-term effects of Early Head Start.

2. Response Rates

Table II.2 displays overall response rates for key data sources by research status,4 as well as response rates for various combinations of interviews. Interview respondents are sample members who provided data that could be used to construct key outcome variables. Nonrespondents include those who could not be located, as well as those who could be located but for whom complete or usable data were not obtained (Appendix B).

Response rates were higher for the PSIs and the PIs than for the Bayley and video assessments. Furthermore, as expected, response rates decreased somewhat over time. The rate was about 82 percent to the 6-month PSI, 75 percent to the 15-month PSI, and 70 percent to the 26-month PSI. It was 78 percent to the 14-month PI, 72 percent to the 24-month PI, and 70 percent to the 36-month PI. At 14 months, it was 63 percent to the Bayley assessment and 66 percent to the video assessment, while at 36 months, it was about 55 percent to each. About 57 percent of sample members completed all three PIs, 39 percent completed all three video assessments, and 35 percent completed all three Bayley assessments.5 The percentages who completed both the 24- and 36-month interviews were about 5 percentage points higher than those who completed all three interviews.6

Importantly, response rates were similar for program and control group members for all data sources. Although response rates were consistently 2 to 6 percentage points higher for the program group, this differential did not result in any attrition bias, as the following analyses demonstrate.

TABLE II.2

RESPONSE RATES TO KEY DATA SOURCES
(Percentages)

Data Source	Program Group	Control Group	Combined Sample
Parent Service Interviews (PSIs):
6-Month	83.9	79.3	81.6
15-Month	76.1	74.4	75.2
26-Month	71.1	67.9	69.5
15- and 26-Month	63	59.9	61.5
All three	58.6	54.4	56.5
Parent Interviews (PIs):
14-Month	79.1	77.1	78.1
24-Month	73.9	70.4	72.2
36-Month	73.2	67.4	70.3
24- and 36-Month	64.4	58.2	61.4
All three	59.4	53.9	56.7
Bayley Assessments:
14-Month	64.2	61.2	62.7
24-Month	61.5	57.1	59.4
36-Month	58.1	52.4	55.3
24- and 36-Month	46.5	40.6	43.6
All three	37	32.6	34.8
Video Assessments:
14-Month	66.5	65.2	65.8
24-Month	62.2	57.5	59.9
36-Month	57.8	52.7	55.3
24- and 36-Month	48.1	42.7	45.4
All three	40.8	37	38.9
Combinations:
PSI 15 and PI 24	65.6	63.2	64.4
PSI 26 and PI 36	63.9	58.7	61.3
PI 24 and Bayley 24	60.5	56.5	58.6
PI 24 and Video 24	61.5	57.1	59.4
Bayley 24 and Video 24	55.9	51.9	53.9
PI 24, Bayley 24, and Video 24	55.4	51.5	53.5
PI 36 and Bayley 36	57.4	52	54.7
PI 36 and Video 36	57.4	52.4	54.9
Bayley 36 and Video 36	53.2	47.9	50.6
PI 36, Bayley 36, and Video 36	52.8	47.6	50.2
PI 24 and Bayley 36	52.2	46	49.2
PI 24 and Video 36	52.4	47	49.7
Video 24 and PI 36	55.8	48.8	52.3
Video 24 and Bayley 36	47.2	40.9	44.1
Sample Size	1,513	1,488	3,001

 

In general, the same families responded to the different interviews (Table II.2). For example, among those who completed a 36-month PI, about 87 percent completed a 24-month PI, and 81 percent completed both a 14- and 24-month PI. Similarly, among those who completed a 36-month video assessment, about 99 percent also completed a 36-month PI, and about 92 percent also completed a 36-month Bayley assessment.

Response rates differed across sites (Table II.3). The rate to the 26-month PSI ranged from 55 percent to 81 percent, although it was 70 percent or higher in 11 sites. Similarly, response rates to the 36-month PI ranged from 51 percent to 81 percent; 12 sites had a rate greater than 70 percent, but 3 sites had a rate less than 60 percent (for the control group). The response rate to the 36-month Bayley and video assessments varied more, ranging from about 27 percent to 76 percent, with less than half the sites having a response rate greater than 60 percent. Response rates for the program group were substantially larger than those for the control group in some sites, although the reverse was true in a few sites.7

TABLE II.3

RESPONSE RATES TO THE 26-MONTH PSI, 36-MONTH PI AND 36-MONTH ASSESSMENTS,
BY SITE
(Percentages)

	26-Month PSI			36-Month PI			36-Month Bayley			36-Month Video
Site	Program Group	Control Group	Total	Program Group	Control Group	Total	Program Group	Control Group	Total	Program Group	Control Group	Total
1	86	73	79	86	77	81	78	65	72	84	69	76
2	62	62	62	70	57	64	55	41	48	60	44	53
3	76	77	77	76	77	77	56	53	54	62	59	60
4	60	61	61	88	67	78	65	54	60	72	56	64
5	76	67	71	80	64	72	61	36	48	59	45	52
6	54	57	56	65	65	65	49	46	48	45	42	44
7	62	69	65	51	52	51	46	46	46	35	40	37
8	80	83	81	82	72	77	63	56	60	68	62	65
9	58	52	55	53	49	51	40	35	37	27	27	27
10	61	60	60	61	64	62	61	57	59	58	59	58
11	79	68	74	69	73	71	53	55	54	55	53	54
12	79	61	70	75	67	71	52	46	49	45	46	45
13	74	73	74	82	70	76	60	57	58	65	60	63
14	67	73	68	79	73	47	65	55	47	54	50
15	73	75	74	77	76	76	59	62	60	57	63	60
16	78	74	76	77	74	75	75	71	73	74	71	72
17	91	71	81	94	68	81	78	49	64	81	54	68
Total	71	68	70	73	67	70	58	52	55	58	53	55
NOTE: Sites are in random order.

Table II.4 displays response rates for key subgroups defined by site and family characteristics at random assignment. The family subgroups were constructed using HSFIS data collected at the time of program application, which are available for both interview respondents and nonrespondents. Asterisks in the table signify whether differences in the variable distributions for respondents and the full sample of respondents and nonrespondents are statistically significant at the 10 percent level. We conducted separate statistical tests for the program and control groups. Appendix D presents detailed results from the nonresponse analysis.

TABLE II.4

RESPONSE RATES TO THE 26-MONTH PSI, 36-MONTH PI AND 36-MONTH ASSESSMENTS, BY SUBGROUPS DEFINED BY SITE AND FAMILY CHARACTERISTICS
(Percentages)

Site	26-Month PSI		36-Month PI		36-Month Bayley		36-Month Video
	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group
Site Characteristics
Program Approach			*		*		*	*
Center-based	75	67	83	69	71	56	74	59
Home-based	69	67	71	66	56	52	56	51
Mixed	72	70	70	68	53	51	50	51
Overall Implementation Level	*	*	*	*	*		*	*
Early Implementers	70	71	74	69	58	58	59	56
Later Implementers	78	72	79	69	64	53	66	56
Incomplete Implementers	65	60	65	63	52	46	48	44
Family Characteristics at Random Assignment
Mother's Age at Birth of Focus Child
Less than 20	71	67	71	66	57	55	56	54
20 or older	71	69	74	69	58	51	59	52
Mother's Education	*	*	*	*				*
Less than grade 12	68	66	69	65	57	51	57	51
Grade 12 or earned a GED	73	67	78	66	59	51	62	50
Greater than grade 12	74	75	75	77	58	57	55	60
Race and Ethnicity	*		*	*	*		*	*
White non-Hispanic	71	70	78	73	59	57	60	59
Black non-Hispanic	73	67	70	66	56	48	53	48
Hispanic	70	67	73	62	62	52	63	54
Welfare Receipt	*		*		*		*	*
Received welfare	66	65	68	66	54	50	54	48
Did not receive welfare	74	69	77	70	60	53	60	56
Primary Occupation			*		*		*
Employed	75	66	80	68	67	52	66	55
In school or training	71	67	72	66	60	52	60	50
Neither	69	76	70	69	53	53	53	53
Primary Language				*
English	70	69	73	70	57	54	57	54
Other	72	67	72	62	61	50	60	51
Living Arrangements		*					*	*
With spouse	73	72	76	72	56	54	59	57
With other adults	72	70	73	67	61	53	61	54
Alone	69	63	71	65	56	51	53	49
Random Assignment Date			*		*		*
Before 10/96	70	66	72	66	56	51	56	51
10/96 to 6/97	71	71	69	68	54	54	53	54
After 6/97	72	67	78	68	64	51	64	53
Total	71	68	73	67	58	52	58	53
SOURCE: HSFIS, 26-month PSI, 36-month PI, 26-month Bayley, and 36-month video data. *Difference between the variable distribution for interview respondents and the full sample of respondents and non respondents is statistically significant at the 10 percent level.

We find some differences in response rates across groups of sites. Response rates for the program group were higher in the center-based programs than in the home-based and mixed-approach ones, although rates for the control group were more similar across program approaches. Thus, differences in response rates between the program and control groups were largest in the center-based programs. Interestingly, rates for both research groups were higher in sites that were fully implemented than in the incompletely implemented sites.

Response rates also differed across some subgroups defined by family characteristics. They increased with the education level of the primary caregiver. In addition, they were higher if the primary caregiver was employed at random assignment (for the program group), if she was married or living with other adults, and if the family was receiving welfare. Response rates were also slightly higher for whites than for African Americans and Hispanics for some data sources, and for those randomly assigned later than earlier. The pattern of response rates across subgroups was similar for the program and control groups.

Importantly, we find fewer differences in the baseline characteristics of program and control group respondents (Appendix D). Very few of the differences in the distributions of the baseline variables for respondents in the two research groups are statistically significant for each data source. None of the p-values for testing the hypotheses that the distribution of the baseline variables are jointly similar are statistically significant. Thus, although we find some differences in the characteristics of respondents and nonrespondents, the characteristics of respondents in the two research groups appear to be similar.

Our main procedure to adjust for potential nonresponse bias was to estimate impacts using regression models that control for differences in the baseline characteristics of program and control group respondents (see Section C below). We used a large number of control variables from the HSFIS forms to adjust for observable baseline differences between the two groups. We gave each site equal weight in the analysis (regardless of the response rates in each site). In addition, as discussed in Appendix D, we calculated sample weights to adjust for nonresponse, so that the weighted characteristics of respondents matched those of the full sample of respondents and nonrespondents. We used these weights in some analyses to check the robustness of study findings (see Appendix D).

These procedures adjust for nonresponse by controlling for measurable differences between respondents and nonrespondents in the two research groups. To be sure, there may have been unmeasured differences between the groups. However, because of the large number of baseline data items in the HSFIS forms, we believe that our procedures account for some important differences between the groups. Therefore, we are confident that our procedures yielded meaningful estimates of program impacts.

3. Timing of Interviews

Most interviews were conducted near their target dates (Appendix B). For example, the average 15-month PSI was conducted 16.6 months after random assignment, and about 80 percent were conducted between 12 and 18 months. Similarly, the average 26-month PSI was conducted 28.4 months after random assignment, and about 76 percent were conducted within 30 months. The average 24-month PI was conducted when the child was 25.1 months old, and about 88 percent were conducted when the child was between 23 and 27 months old. The average 36-month PI was completed when the child was 37.5 months old, and about 82 percent were completed before the child was 40 months old. The corresponding figures for the Bayley and video assessments are very similar to those of the PIs.

On average, the 6-, 15-, and 26-month PSI interviews were conducted about 5 months before the 14-, 24-, and 36-month birthday-related instruments, respectively (Appendix B). Thus, at the 36-month birthday-related interviews and assessments, some families who remained in the program for a long period probably had received more Early Head Start services than we report here.

The distributions of interview completion times were similar for program and control group families. Thus, it is not likely that impact estimates on outcomes (such as the child language measures) were affected by differences in the ages of program and control group children at the time the data were collected.8 As discussed in Appendix C, we did not have a pertinent norming sample to age-norm some measures.

4. Outcome Variables

The Early Head Start evaluation was designed to examine the extent to which Early Head Start programs influence a wide range of outcomes. Four main criteria guided specification of the major outcome variables for the analysis: (1) selecting outcomes that are likely to be influenced significantly by Early Head Start on the basis of programs’ theories of change and the results of previous studies, (2) selecting outcomes that have policy relevance, (3) measuring outcomes reliably and at reasonable cost, and (4) selecting outcomes that could be reliably compared over time.

The primary outcome variables for the analysis can be grouped into three categories:

1. Service use

2. Child development and parenting

3. Family development

Table II.5 summarizes the key categories of outcome variables in each area, as well as the data sources used to construct them. In the analysis, we first describe the EHS experiences of program group members and examine impacts for the service use outcomes, because we would not expect meaningful impacts on the child, parenting, and family outcomes unless program group families received substantial amounts of Early Head Start services and received more and higher-quality services than the control group. Examining the services received by control group families is crucial for defining the counterfactual for the evaluation, and for interpreting impact estimates on all other outcomes. These results are presented in Chapter IV. Impact results for the child, parent, and family outcomes are presented in Chapters V, VI, and VII. A detailed discussion of the specific outcome variables for the analysis, the reasons they were selected, and the way they were constructed can be found at the start of each chapter.

TABLE II.5

CATEGORIES OF OUTCOME VARIABLES REFERRED TO IN THIS REPORT, AND THEIR DATA SOURCES

Outcome Measure	Data Source
Service Use
Home visits	6-, 15-, and 26-Month Parent Services Interviews
Case management	6-, 15-, and 26-Month Parent Services Interviews
Parenting-related services	6-, 15-, and 26-Month Parent Services Interviews
Child care and child development services	6-, 15-, and 26-Month Parent Services Interviews
Services for children with disabilities	6-, 15-, and 26-Month Parent Services Interviews
Child health services and status	6-, 15-, and 26-Month Parent Services Interviews
Family health and other family development services	6-, 15-, and 26-Month Parent Services Interviews
Father participation in program-related activities	36-Month Father Interview
Parenting Behavior, Knowledge, and the Home Environment
Knowledge of child development, discipline strategies, and safety precautions	24- and 36-Month Parent Interviews
Parent supportiveness, detachment, intrusiveness, and negative regard	Coding from Videotaped Parent-Child Semistructured Play Task (24 and 36 Months)
Parent quality of assistance, detachment, and intrusiveness	Coding from Videotaped Puzzle Challenge Task (36 Months)
Parent warmth, harshness and stimulation of language and learning	24- and 36-Month Parent Interviews
Quality of cognitive and emotional support provided in the home environment	24- and 36-Month Parent Interviews and Interviewer Observations
Father Involvement	24- and 36-Month Parent Interviews
Child Development
Child social and emotional well-being
Child engagement, negativity toward parent, and sustained attention with objects	Coding from Videotaped Parent-Child Semistructured Play Task (24 and 36 months)
Child engagement, persistence, and frustration	Coding from Videotaped Puzzle Challenge Task (36 Months)
Emotional regulation, orientation/engagement	Interviewer Observations (24 and 36 months)
Aggressive behavior	24- and 36-Month Parent Interviews
Child cognitive and language development
Bayley Mental Development Index (MDI)	Direct Child Assessment (24 and 36 months)
Vocabulary production and sentence complexity	24-Month Parent Interviews
Receptive vocabulary	Direct Child Assessment (36 Months)
Child Health Status	24- and 36-Month Parent Interviews
Family Outcomes
Parent's Health and Mental Health
Depression	24- and 36-Month Parent Interviews
Parenting stress	24- and 36-Month Parent Interviews
Family Functioning
Family conflict	24- and 36-Month Parent Interviews
Self-Sufficiency
Education and training	6-, 15, and 26-Month Parent Services Interviews
Welfare receipt	6-, 15, and 26-Month Parent Services Interviews
Employment and income	6-, 15, and 26-Month Parent Services Interviews
Father Presence, Behavior, and Well-Being
Father presence	14-, 24-, and 36-Month Parent Interviews
Father caregiving, social, cognitive, and physical play activities	36-Month Father Interview
Father discipline strategies	36-Month Father Interview
Father supportiveness and intrusiveness	Coding from Videotaped Father-Child Semistructured Play Task (36 months)
Father quality of assistance and intrusiveness	Coding from Videotaped Father-Child Puzzle Challenge Task (36 months)
Father's Mental Health
Depression	36-Month Father Interview
Parenting stress	36-Month Father Interview
Family Functioning
Family conflict	36-Month Father Interview
Child Behavior With the Father
Child engagement of the father, negativity toward the father, and sustained attention with objects	Coding from Videotaped Father-Child Semistructured Play Task (36-Months)
Child engagement of father, persistence, and frustration	Coding from Videotaped Father-Child Puzzle Challenge Task (36-Months)

5. Analysis Samples

We used different analysis samples, depending on the data source and type of analysis. The primary sample used to estimate “point-in-time” impacts on outcomes from the 24-month or 36-month PI data includes those who completed 24-month or 36-month PIs. Similarly, the primary sample for the point-in-time analysis based on the birthday-related child and family assessment data includes those who completed the assessments at each time point. In sum, we conducted separate point-in-time analyses using each of these samples in order to maximize the sample available for the analyses.

The primary sample, however, used in the analysis to examine impacts on the growth in child and family outcomes (that is, the growth curve analysis) includes those for whom data are available for all three time points. Similarly, the primary sample used in the analysis to examine the extent to which impacts on mediating (24-month) variables correlate with impacts on longer-term (36-month) outcomes (that is, the mediated analysis) includes those for whom both 24-month and 36-month data are available.

For the analysis of the service use and self-sufficiency outcomes, we used the sample of those who completed 26-month PSIs (regardless of whether a 6- or 15-month PSI was completed). Most of the service use and self-sufficiency outcomes pertain to the entire 26-month period since random assignment (for example, the receipt of any home visits, the average hours per week the child spent in center-based child care, and the average number of hours the mother spent in education and training programs), so data covering the entire 26-month period were required to construct these outcomes. About 88 percent of those who completed a 26-month PSI also completed a 15-month PSI, and 97 percent completed either a 6-month or a 15-month PSI. In the 26-month PSI, respondents were asked about their experiences since the previous PSI interview (or since random assignment if no previous PSI was completed). Thus, complete data covering the 26-month period are available for all those in the 26-month analysis sample.

We did estimate impacts, however, using alternative sample definitions to test the robustness of study findings (see Appendix D). For example, we estimated point-in-time impacts on 36-month outcomes using those who completed both the 24- and 36-month PIs (the mediated analysis sample), as well as those who completed all birthday-related interviews and assessments (the growth curve analysis sample). As another example, we estimated impacts on service use and self-sufficiency outcomes using those who completed both the 15- and 26-month PSIs. Our results using alternative samples were very similar, so, in the main body of this report, we present only results that were obtained using the primary analysis samples described above.

C. ANALYTIC APPROACHES

The Early Head Start impact analysis addresses the effectiveness of Early Head Start services on key child, parenting, and family outcomes from several perspectives. The global analysis examines the overall impacts of Early Head Start across all 17 sites combined, while the targeted analysis addresses the important policy questions of what works and for whom.

1. Global Analysis

In this section, we discuss our approach for answering the question: Do Early Head Start programs have an effect on child, parenting, and family outcomes overall? Stated another way, we discuss our approach for examining the extent to which the 17 programs, on average, changed the outcomes of program participants relative to what their outcomes would have been had they not received Early Head Start services. First, we discuss our primary approach for estimating impacts per eligible applicant. Second, we discuss our approach for estimating impacts per participant (that is, for families that received Early Head Start services). Finally, we discuss our approach for estimating impacts using growth curve models.

a. Estimating Point-in-Time Impacts per Eligible Applicant

Random assignment was performed at the point that applicant families were determined to be eligible for the program. Thus, we obtained estimates of impacts per eligible applicant by computing differences in the average outcomes of all program and control group families at each time point. This approach yields unbiased estimates of program impacts on the offer of Early Head Start services, because the random assignment design ensures that no systematic differences between program and control group members existed at the point of random assignment except for the opportunity to receive Early Head Start services.

We used regression procedures to estimate program impacts, for two reasons. First, the regression procedures produce more precise impact estimates. Second, they can adjust for any differences in the observable characteristics of program and control group members due to random sampling and interview nonresponse. However, we also estimated impacts using simple differences-in-means procedures to test the sensitivity of our findings to alternative estimation strategies (see Appendix D). The two procedures yielded very similar results; we present the regression-adjusted estimates in the main body of this report.

We estimated variants of the following regression model:

where y is an outcome variable at a specific time point, Sj (is an indicator variable equal to 1 if the) family is in site j, T is an indicator variable equal to 1 if the family is in the program group, Xs are explanatory variables measured at baseline (that include site indicator variables), e is a mean zero disturbance term, and aj and B are parameters to be estimated. In this formulation, the estimate of aj (represents the regression-adjusted impact estimate for sitej.9

An important aspect of our analytic approach was to give each site equal weight regardless of sample sizes within the sites. Early Head Start services are administered at the site level and differ substantially across programs; thus, the site is the relevant unit of analysis. Accordingly, the global impact estimates were obtained by taking the simple average of the regression-adjusted impact estimates in each site.10 The associated t-tests were used to test the statistical significance of the impact estimates.

We included a large number of explanatory variables in the regression models (Table II.6 lists the categories of variables, and Appendix Table E.II.B provides variable descriptions and means). These variables were constructed using HSFIS data and pertain to characteristics and experiences of families and children prior to random assignment. We used two main criteria to select the explanatory variables: (1) they should have some predictive power in the regression models for key outcome variables (to increase the precision of the impact estimates); and (2) they should be predictors of interview nonresponse (to adjust for differences in the characteristics of program and control group respondents).11 There was no theoretical reason to include different explanatory variables by site or to assume that the parameter estimates on the explanatory variables would differ by site. Thus, we used the same model specification for each site.12 The regression R2 values for key 36-month outcomes ranged from about .10 (for maternal depression and distress measures) to .15 (for parent-child interaction scales from the video assessments) to .30 (for measures of child cognitive and language development and the home environment) to .50 (for measures of welfare receipt).

TABLE II.6

CATEGORIES OF EXPLANATORY VARIABLES FOR REGRESSIONS

Family and Parent Characteristics

Age of Mother

Race

English-Language Ability

Education Level

Primary Occupation

Living Arrangements

Number of Children in the Household

Poverty Level

Welfare Receipt (AFDC/TANF; Food Stamps; WIC; SSI)

Has Inadequate Resources (Food, Housing, Money, Medical care, Transportation)

Previously Enrolled in Head Start or Another Child Development Program

Mobility in the Previous Year

Random Assignment Date

Child Characteristics