ACF OPRE: Early Head Start Research and Evaluation Project (EHSRE)

II. EVALUATION DESIGN, DATA, AND ANALYTIC APPROACHES

ACYF designed a thorough and rigorous evaluation to examine the impacts of Early Head Start on key child and family outcomes. This chapter summarizes the study design, the data sources and outcome variables used in this interim report, and our approach to conducting the impact analysis.

A. STUDY DESIGN

The evaluation was conducted in 17 sites where Early Head Start research programs were located. Once selected for participation in the study, programs began enrolling families and worked with MPR staff to coordinate with the requirements of random assignment.

1. Site Selection

When the 68 Early Head Start programs in the first wave were funded in late 1995, they agreed, as a condition of funding, to participate in local and national research if selected. ACYF then selected the research sites purposely from the first two waves of grantees (with 75 Wave II programs funded in mid-1996). In March 1996, 41 university research teams submitted proposals to the Head Start Bureau-in partnership with Wave I Early Head Start program grantees-to conduct local research and participate in the national evaluation. ACYF selected 15 research sites, using a number of criteria including that programs (1) had to be able to recruit twice as many families as they could serve; (2) had to have a viable research partner; and (3) in aggregate, had to provide a national geographic distribution that represented the major programmatic approaches and settings and reflected diverse family characteristics thought to be typical of Early Head Start families nationally. These criteria resulted in an underrepresentation of center-based programs, so in 1996 ACYF selected one additional center-based program from Wave I, and in late 1997 selected another center-based program (without a local research partner) from Wave II, resulting in the full sample of 17 programs.

As shown in Chapter I (Table I.2), the features of the 17 programs, as well as the characteristics of their enrolled families and children, are similar to those of all Early Head Start programs funded during 1995 and 1996. Thus, although the study results cannot be formally generalized to all Early Head Start programs, the findings about effective program practices and their impacts on children and families are likely to be indicative of Early Head Start impacts more broadly.

2. Sample Enrollment

Although Wave I grantees entered Head Start with varying degrees and types of experiences (see Chapter I), all had been asked not to enroll any families until it was decided whether they would be selected for the research sample. Because all programs had agreed, in submitting their original proposals, to participate in the random assignment process if they were selected for the research sample, it was not necessary to persuade any of the programs to cooperate. Thus, as soon as the programs were selected, beginning in spring 1996, MPR staff began working with their staffs to implement the random assignment process in conjunction with each program's regular enrollment procedures. Except for recruiting about twice as many families as they could serve, programs were expected to recruit as they would in the absence of the research, with special instructions to be sure to include all the types of families that their program was designed to serve (including those whose babies had disabilities). MPR and ACYF created detailed procedures (outlined in a "frequently asked questions" document-see Appendix E.II.A) to guide the sample enrollment process.

3. Random Assignment

As soon as programs determined through their application process that families met the Early Head Start eligibility guidelines, they sent the names to MPR, and we entered the names and identifying information into a computer program that randomly assigned the families either to the program or the control group (with equal probabilities). Program staff then contacted the program families, while representatives of the local research partners notified the control group families of their status.

Control group families were not allowed to receive Early Head Start services until their applicant child reached the age of 3, although they were told they could receive other services in the community. This ensures that our analytic comparisons of program and control group outcomes represent the effects of Early Head Start services relative to the receipt of all other community services that would be available to families in the absence of Early Head Start. Some program staff were concerned that random assignment might, by chance, result in denial of services to families with particularly high service needs. ACYF was very clear, however, that the study findings should pertain to all families and children that Early Head Start was designed to serve, particularly infants and toddlers with disabilities. To address program concerns, however, ACYF and MPR established a process by which programs could apply to have a family declared exempt from participating in the research. (No exemptions were requested.)

Sample enrollment and random assignment began in July 1996 and was completed in September 1998. In most sites, sample intake occurred over a two-year period, although some took less time. The extended enrollment period was due in part to the extra work involved in recruiting twice as many families as programs were funded to serve, and in part to the process of new programs working out their recruitment procedures. Two programs completed sample enrollment in late 1997, and one (the 17th site) did not begin sample intake until fall 1997. Thus, the study population for the evaluation includes Early Head Start-eligible families who applied to the program between late 1996 and late 1998.

During the sample intake period, 3,001 families were randomly assigned to the program (1,513) and control (1,488) groups (Table II.1). The samples in most sites include between 150 and 200 families, fairly evenly divided between the two research groups.

Early Head Start staff implemented random assignment procedures well. We estimate that about 0.7 percent of all control group members received any Early Head Start services (that is, were "crossovers"), and most sites had no crossovers. 1 Furthermore, our discussions with site staff indicate that information on nearly all eligible families who applied to the program during the sample intake period was sent to MPR for random assignment (that is, site staff did not provide Early Head Start services to families who were not submitted for random assignment). Hence, we believe that the researchsample is representative of the intended study population of eligible families, and that any bias in the impact estimates due to contamination of the control group is small.

Random assignment yielded equivalent groups: the average baseline characteristics of program and control group members are very similar (Appendix D).

This is as expected, because MPR used computer-generated random numbers to assign families. Therefore, the only difference between the two research groups at random assignment was that the program group was offered Early Head Start services and the control group was not. Thus, differences in the subsequent outcomes of the two groups can be attributed to the offer of Early Head Start services with a known degree of statistical precision.

TABLE II.1
EVALUATION SAMPLE SIZES, BY SITE AND RESEARCH STATUS
Site	Program Group	Control Group	Combined Sample
1	74	77	151
2	93	86	179
3	84	78	162
4	75	72	147
5	74	76	150
6	115	110	225
7	104	108	212
8	98	98	196
9	98	95	193
10	71	70	141
11	104	96	200
12	73	79	152
13	104	98	202
14	75	71	146
15	90	92	182
16	95	95	190
17	86	87	173
All Sites	1,513	1,488	3,001
NOTE: Sites are in random order.

B. DATA SOURCES AND OUTCOME MEASURES

Comprehensive data from multiple sources were used to examine the short-term effects of Early Head Start participation on a wide range of child, parenting, and family outcomes. This section provides an overview of data sources and outcome measures used for the analysis, the interview and assessment response rates, and the timing of interviews. These topics are discussed in more detail in the Appendixes.

1. Data Sources

The follow-up data used for the analysis are both time- and age-based. Each family's use of services and progress toward self-sufficiency were seen as likely to be a function of the amount of time since the family applied for Early Head Start services. Therefore, these data were collected at selected intervals following random assignment. Other data-particularly those related to child and family development-were more likely to be a function of the increasing age of the focus child over time. Thus, the data collection schedule for these developmental outcomes was tied to children's birth dates. The data sources used in this report include:2

Parent Services Follow-Up Interview (PSI) Data Targeted for Collection 6 and 15 Months After Random Assignment. These data contain information on (1) the use of services both in and out of Early Head Start (such as the receipt of home visits, and of services related to case management, parenting, health, employment, and child care); (2) progress toward economic self-sufficiency (such as employment, welfare receipt, and participation in education and training programs); (3) family health; and (4) children's health. Most PSIs were conducted by telephone with the focus child's primary caregiver, although some interviews were conducted in person for those not reachable by phone.
Parent Interview (PI) Data Targeted for Collection When Children Were 14 and 24 Months Old. These interviews obtained a large amount of information from the primary caregivers about their children's development and family functioning. These data were usually collected in person, but some PIs or portions of them were conducted by telephone when necessary.
Child and Family Assessments Targeted for Collection When Children Were 14 and 24 Months Old. Field interviewers provided data on their observations of children's behavior and home environments. Interviewers conducted direct child assessments (such as Bayley assessments) and videotaped structured parent-child interactions. Several measures constructed using these data overlap with those constructed from the PI data, which allowed us to compare impact findings using the two data sources.
Father Interviews Targeted for Collection When Children Were 24 Months Old. In addition to asking mothers about the children's father, we interviewed the men directly about fathering issues at the time of the 24-month birthday-related interviews. 3 The father study was conducted in 12 sites only.
Baseline Data from the Head Start Family Information System (HSFIS) Program Application and Enrollment Forms. We used these forms, completed by families at the time of program application, to create subgroups defined by family characteristics at baseline, and to adjust for differences in the characteristics of program and control group members when estimating program impacts. We also used the forms to compare the characteristics of interview respondents and nonrespondents, and to construct weights to adjust for potential nonresponse bias.
Data from the Implementation Study. Finally, the analysis used data from the implementation study to define subgroups based on program site characteristics (such as program approach and level of program implementation) and site characteristics (such as welfare regulations).

MPR prepared all the follow-up data collection instruments and trained all field staff. In all sites but one (where MPR collected the data), data collection field staffs were hired by the local research teams, who were responsible, under subcontract to MPR, for collecting the data and monitoring data quality. Respondents were offered modest remuneration and a small gift to complete each set of interviews and assessments. Appendix B describes the data collection procedures in more detail. Details about all the measures can be found in Chapters IV, V, and VI and in Appendix C.4

It is important to recognize that linking PIs and child and family assessments to the age of the child rather than to a fixed period after random assignment means that at the time those instruments were administered, families were exposed to the program for different lengths of time. Nevertheless, questions about children's development at particular ages are policy relevant. It is also of policy interest, however, to assess impacts for children and families with similar lengths of exposure to the program. Therefore, as described in Section C, we estimated impacts by doing subgroup analyses based on the child's age at random assignment (so that program participation times would be similar within each age group).

It is also important to recognize that at the 14-month birthday-related interviews, many families had been exposed to Early Head Start for only a short time, and especially so for families with older focus children. Thus, we do not expect impacts to appear at 14 months. In this report, we focus on the child, parenting, and family outcomes when children are 2 years old.

In sum, in this report we present impact findings using follow-up data from the 6- and 15-month PSIs, and from the 14- and 24-month PIs and child and family assessments. Because of this limited follow-up period, the findings should be considered short term. The final report will present longer-term findings using 26-month PSI and 36-month PI and assessment data, after program group families have had more exposure to Early Head Start services.

2. Response Rates

Table II.2 displays overall response rates for key data sources by research status,5 as well as response rates for various combinations of interviews. Interview respondents are sample members who provided data that could be used to construct key outcome variables. Nonrespondents include those who could not be located, as well as those who could be located but for whom complete or usable data were not obtained (Appendix B).

Response rates were higher for the PSIs and the PIs than for the Bayley and video assessments. Furthermore, as expected, response rates decreased somewhat over time. The rate was about 82 percent to the 6-month PSI and 75 percent to the 15-month PSI. It was 78 percent to the 14-month PI and 70 percent to the 24-month PI. At 14 months, it was 63 percent to the Bayley assessment and 66 percent to the video assessment, while at 24 months, it was about 58 percent to each. Rates were similar for program and control group members for all data sources, although they were consistently about 2 to 5 percentage points higher for the program group. In general, the same families responded to the different interviews (Table II.2). For example, about 90 percent of those who completed a 24-month PI also completed a 14-month PI. Similarly, among those who completed a 24-month video assessment, about 99 percent also completed a 24-month PI, and about 90 percent also completed a 24-month Bayley assessment. Response rates differed across sites (Table II.3). The rate to the 15-month PSI ranged from about 60 percent to 85 percent, although it was 70 percent or higher in 14 sites. Response rates to the 24-month birthday-related instruments varied more, ranging from 43 percent to 86 percent to the 24-month PI. Twelve sites had a rate greater than 70 percent, but 3 sites had a rate less than 60 percent. The response rate to the 24-month Bayley and video assessments ranged from about 40 percent to 75 percent, with less than half the sites having a response rate greater than 60 percent. Response rates for the program group were substantially larger than those for the control group in some sites, although the reverse was true in a few sites.

TABLE II.2
RESPONSE RATES TO KEY DATA SOURCES (Percentages)
Data Source	Program Group	Control Group	Combined Sample
Parent Service Interviews (PSIs)
6-Month	83.9	79.3	81.6
15-Month	75.3	73.7	74.5
Both	69.6	65.7	67.7
Parent Interviews (PIs)
14-Month	79.1	77.1	78.1
24-Month	72.2	68.6	70.4
Both	65.9	62.0	64.0
Bayley Assessments
14-Month	64.2	61.2	62.7
24-Month	60.1	55.7	57.9
Both	46.5	42.7	44.6
Video Assessments
14-Month	66.5	65.2	65.8
24-Month	60.3	55.0	57.7
Both	49.8	46.2	48.1
Combinations
PSI 15 and PI 24	63.7	61.4	62.6
PI 24 and Bayley 24	59.2	55.2	57.2
PI 24 and Video 24	59.6	54.6	57.1
Bayley 24 and Video 24	54.1	49.9	52.0
PI 24, Bayley 24, andVideo 24	53.7	49.7	51.7
Sample Size	1,513	1,488	3,001

TABLE II.3
RESPONSE RATES TO THE 15-MONTH PSI, 24-MONTH PI AND 24-MONTH ASSESSMENTS, BY SITE (Percentages)
Site	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group
	15-Month PSI		24-Month PI		24-Month Bayley		24-Month Video
1	77	65	84	69	78	65	82	65
2	74	70	66	62	48	30	46	30
3	82	91	70	83	48	59	58	68
4	77	67	87	65	69	49	81	53
5	65	61	80	72	64	55	68	57
6	65	76	68	73	57	66	50	52
7	62	59	43	44	44	38	38	36
8	84	85	82	77	59	64	74	69
9	68	73	59	52	53	44	42	36
10	62	63	48	47	44	43	42	39
11	83	78	63	64	54	55	55	51
12	78	70	82	71	66	51	64	54
13	78	79	77	78	71	67	70	72
14	80	86	77	78	64	65	57	59
15	78	80	79	80	56	69	46	59
16	81	78	82	76	82	67	81	72
17	87	71	92	81	72	58	83	66
Total	75	74	72	69	60	56	60	55
NOTE: Sites are in random order.

Table II.4 displays response rates for key subgroups defined by site and family characteristics at random assignment. The family subgroups were constructed using HSFIS data collected at the time of program application, which are available for both interview respondents and nonrespondents. Asterisks in the table signify whether differences in the variable distributions of respondents and the full sample of respondents and nonrespondents are statistically significant at the 5 percent level. We conducted separate statistical tests for the program and control groups. Appendix D presents detailed results from the nonresponse analysis.

We find some differences in response rates across groups of sites. Response rates for the program group were higher in the center-based programs than in the home-based and mixed-approach ones, although rates for the control group were similar across program approaches. Thus, differences in response rates between the program and control groups were largest in the center-based programs. Interestingly, rates for both research groups were higher in sites that were fully implemented than in the incompletely implemented sites.

Response rates also differed across some subgroups defined by family characteristics, and they increased with the education level of the primary caregiver. In addition, they were higher if the primary caregiver was employed at random assignment, if she was married or living with other adults, and if English was the primary language spoken at home. Response rates were also slightly higher for whites than for African Americans and Hispanics, for older mothers than younger ones, and for families not receiving welfare than for those receiving it. The pattern of response rates across subgroups was similar for the program and control groups.

Importantly, we find fewer differences in the baseline characteristics of program and control group respondents (Appendix D). Very few of the differences in the distributions of the baseline variables for respondents in the two research groups are statistically significant. Thus, although we find some differences in the characteristics of respondents and nonrespondents, the characteristics of respondents in the two research groups appear to be more similar.

Our main procedure to adjust for potential nonresponse bias was to estimate impacts using regression models that control for differences in the baseline characteristics of program and control group respondents (see Section C below). We used a large number of control variables from the HSFIS forms to adjust for observable baseline differences between the two groups. We gave each site equal weight in the analysis (regardless of the response rates in each site). In addition, as discussed in Appendix D, we calculated sample weights to adjust for nonresponse, so that the weighted characteristics of respondents matched those of the full sample of respondents and nonrespondents. We used these weights in some analyses to check the robustness of study findings (see Appendix D).

These procedures adjust for nonresponse by controlling for measurable differences between respondents and nonrespondents in the two research groups. To be sure, there may have been unmeasured differences between the groups. However, because of the large number of baseline data items in the HSFIS forms, we believe that our procedures account for some important differences between the groups. Therefore, we are confident that our procedures yielded meaningful estimates of program impacts.

TABLE II.4
RESPONSE RATES TO THE 15-MONTH PSI, 24-MONTH PI AND 24-MONTH BAYLEY AND VIDEO ASSESSMENTS, BY SUBGROUPS DEFINED BY SITE AND FAMILY CHARACTERISTICS (Percentages)
Site	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group	Program Group	Control Group
	15-Month PSI		24-Month PI		24-Month Bayley		24-Month Video
Site Characteristics
Program Approach		*	*		*		*
Center-based	76	67	78	66	66	54	73	56
Home-based	76	77	71	69	61	57	60	55
Mixed	74	74	70	70	56	55	54	54
Overall Implementation Level	*	*	*	*	*		*	*
Early Implementers	75	72	73	68	63	58	61	57
Later Implementers	81	77	79	75	63	55	68	60
Incomplete Implementers	69	71	64	62	54	54	51	47
Family Characteristics at Random Assignment
Mother's Age at Birth of Focus Child			*		*		*
Less than 20	74	75	69	68	56	57	57	57
20 or older	76	73	74	70	62	56	62	54
Mother's Education	*	*		*		*
Less than grade 12	71	74	68	67	57	54	58	54
Grade 12 or earned a GED	80	71	75	66	63	53	62	55
Greater than grade 12	80	80	77	78	62	65	63	60
Race and Ethnicity
White non-Hispanic	77	75	75	75	63	64	63	61
Black non-Hispanic	76	74	71	64	58	50	58	51
Hispanic	74	70	71	67	60	52	64	54
Welfare Receipt			*	*	*		*	*
Received welfare	75	72	68	66	56	54	55	52
Did not receive welfare	74	75	75	73	63	58	65	58
Primary Occupation			*		*		*
Employed	75	76	80	72	65	58	70	57
In school or training	76	76	72	67	61	56	61	55
Neither	75	73	68	69	57	56	55	55
Primary Language
English	76	75	73	70	60	57	60	57
Other	71	71	67	69	60	52	60	52
Living Arrangements		*		*		*		*
With spouse	76	76	73	77	62	62	62	60
With other adults	76	76	72	71	58	56	61	57
Alone	75	70	72	61	61	52	59	50
Random Assignment Date		*	*				*
Before 10/96	77	76	72	66	62	57	60	55
10/96 to 6/97	75	78	68	70	58	58	53	53
After 6/97	73	67	76	70	61	53	67	57
Total	75	74	72	69	60	56	60	55
SOURCE: HSFIS, 15-month PSI, 24-month PI, 24-month Bayley, and 24-month video data. * Difference between the variable distribution for interview respondents and the full sample is statistically significant at the 5 percent level.

3. Timing of Interviews

Most interviews were conducted near their target dates (Appendix B). For example, the average 15-month PSI was conducted 16.6 months after random assignment, and about 80 percent were conducted between 12 and 18 months. Similarly, the average 24-month PI was conducted when the child was 25.1 months old, and about 85 percent were conducted when the child was between 23 and 27 months old. The corresponding figures for the Bayley and video assessments are very similar to those of the PIs.

On average, the 6- and 15-month PSI interviews were conducted 5 months before the 14- and 24-month birthday-related instruments, respectively (Appendix B). Thus, at the birthday-related interviews, most families had probably received more Early Head Start services than we report here.

The distributions of interview completion times were similar for program and control group families. Thus, it is not likely that impact estimates on outcomes (such as the child language and cognitive development measures) were affected by differences in the ages of program and control group children at the time the data were collected. 6 As discussed in Appendix C, we did not have a pertinent norming sample to age-norm these measures.

4. Outcome Variables

The Early Head Start evaluation was designed to examine the extent to which Early Head Start programs influence a wide range of outcomes. Four main criteria guided specification of the major outcome variables for the analysis: (1) selecting outcomes that are likely to be influenced significantly by Early Head Start on the basis of programs' theories of change and the results of previous studies, (2) selecting outcomes that have policy relevance, (3) measuring outcomes reliably and at reasonable cost, and (4) selecting outcomes that could be reliably compared over time.

The primary outcome variables for the analysis can be grouped into three categories:

1. Service use
2. Child development and parenting
3. Family development

Table II.5 summarizes the key categories of outcome variables in each area, as well as the data sources used to construct them. In the analysis, we first examine impacts for the service use outcomes, because we would not expect meaningful impacts on the child, parenting, and family outcomes unless program group families received substantial amounts of Early Head Start services and received more and higher-quality services than the control group. Examining the services received by control group families is crucial for defining the "counterfactual" for the evaluation, and for interpreting impact estimates on all other outcomes. These results are presented in Chapter III. Impact results for the child, parent, and family outcomes are presented in Chapters IV, V, and VI. A detailed discussion of the specific outcome variables for the analysis, the reasons they were selected, and the way they were constructed can be found at the start of each chapter.

TABLE II.5
CATEGORIES OF OUTCOME VARIABLES AND THEIR DATA SOURCES
Outcome Measure	Data Source

Service Use
Home visits	6- and 15-Month Parent Services Interviews
Case management
Parenting-related services
Child care and child development services
Services for children with disabilities
Child health services and status
Family health and other family development services

Parenting Behavior, Knowledge, and the Home Environment
Knowledge of child development, discipline strategies, and safety precautions	24-Month Parent Interview
Parent supportiveness, detachment, intrusiveness, and negative regard during a parent-child structured play task	Coding from Videotaped Parent-Child Structured Play Task (24 months)
Parent-child activities to support cognitive and language development	24-Month Parent Interview
Quality of cognitive and emotional support provided in the home environment	24-Month Parent Interview and Interviewer Observations
Father Involvement	24-Month Parent Interview

Child Development
Child social and emotional well-being
Child engagement, negativity toward parent, and sustained attention with objects during a parent-child structured play task	Coding from Videotaped Parent-Child Structured Play Task (24 months)
Emotional regulation, orientation/engagement	Interviewer Observations (24 months)
Aggressive behavior	24-Month Parent Interview

Child cognitive and language development
Bayley Mental Development Index (MDI)	Direct Child Assessment (24 months)
Vocabulary production and sentence complexity	24-Month Parent Interview

Child Health Status	24-Month Parent Interview

Family Outcomes
Parent's Health and Mental Health	24-Month Parent Interview
Depression
Parenting stress

Family Functioning	24-Month Parent Interview
Family conflict

Self-sufficiency	6- and 15-Month Parent Services Interviews
Education and training
Welfare receipt
Employment and income

5. Analysis Samples

We used different samples for the impact analysis, depending on the data source. The primary sample used to estimate impacts on outcomes from the 14-month PI data includes those who completed 14-month PIs. Similarly, the primary sample for analyses based on the 24-month PI data includes those who completed these interviews, and similarly for the 14- and 24-month assessments. Thus, we conducted separate analyses using each of these samples. As discussed below, we have not estimated growth curve models, because of the relatively short follow-up period, so it was not necessary that both 14- and 24-month data be available for all sample members.

We conducted the analysis of the service use and self-sufficiency outcomes using the sample of those who completed 15-month PSIs (regardless of whether a 6-month PSI was completed). Most of these outcomes pertain to the entire 15-month period since random assignment (for example, the number of home-visiting services received during the 15 months and the number of hours spent in education and training programs), so data covering the entire 15-month period were required to construct these outcomes. About 91 percent of those who completed a 15-month PSI also completed a 6-month PSI, and the remaining 9 percent who did not complete the 6-month PSI were asked about their experiences since random assignment in the 15-month PSI. Thus, complete data covering the 15-month period are available for all those in the 15-month analysis sample.

We did estimate impacts, however, using alternative sample definitions to test the robustness of study findings (see Appendix D). For example, we estimated impacts on 24-month outcomes using those who completed both the 14- and 24-month PIs, as well as those who completed all birthday-related interviews and assessments. Similarly, we estimated impacts on service use and self-sufficiency outcomes using those who completed both the 6- and 15-month PSIs. Our results using alternative samples were very similar, so in the main body of this report we present only results that were obtained using the primary analysis samples described above.

C. ANALYTIC APPROACHES

The Early Head Start impact analysis addresses the effectiveness of Early Head Start services on key child, parenting, and family outcomes from two perspectives. The global analysis examines the overall impacts of Early Head Start across all 17 sites combined, while the targeted analysis addresses the important policy questions of what works and for whom.

1. Global Analysis

In this section we discuss our approach for answering the question, Do Early Head Start programs have an effect on child, parenting, and family outcomes overall? Stated another way, we discuss our approach for examining the extent to which the 17 programs, on average, changed the outcomes of program participants relative to what their outcomes would have been had they not received Early Head Start services. First, we discuss our approach for estimating impacts per eligible applicant. Second, we discuss our approach for estimating impacts per participant (that is, for families that received Early Head Start services).

a. Estimating Impacts per Eligible Applicant

Random assignment was performed at the point that applicant families were determined to be eligible for the program. Thus, we obtained estimates of impacts per eligible applicant by computing differences in the average outcomes of all program and control group families. This approach yields unbiased estimates of program impacts on the offer of Early Head Start services, because the random assignment design ensures that no systematic observable or unobservable differences between program and control group members existed at the point of random assignment except for the opportunity to receive Early Head Start services.

We used regression procedures to estimate program impacts, for two reasons. First, the regression procedures produce more precise impact estimates. Second, they can adjust for any differences in the observable characteristics of program and control group members due to random sampling and interview nonresponse. However, we also estimated impacts using simple differences-in-means procedures to test the sensitivity of our findings to alternative estimation strategies (see Appendix D). The two procedures yielded very similar results; we present the regression-adjusted estimates in the main body of this report.

We estimated variants of the following regression model:

where y is an outcome variable, Sj is an indicator variable equal to 1 if the family is in site j, T is an indicator variable equal to 1 if the family is in the program group, Xs are explanatory variables (that include site indicator variables), e is a mean zero disturbance term, and aj and b are parameters to be estimated. In this formulation, the estimate of aj represents the regression-adjusted impact estimate for site j.7

An important aspect of our analytic approach was to give each site equal weight regardless of sample sizes within the sites. Early Head Start services are administered at the site level and differ substantially across programs. Thus, the site is the relevant unit of analysis. Accordingly, the global impact estimates were obtained by taking the simple average of the regression-adjusted impact estimates in each site. 8 The associated t-tests were used to test the statistical significance of the impact estimates.

We included a large number of explanatory variables in the regression models (Table II.6 lists the categories of variables, and Appendix Table E.II.B provides variable descriptions and means). These variables were constructed using HSFIS data and pertain to characteristics and experiences of families and children prior to random assignment. We used two main criteria to select the explanatory variables: (1) they should have some predictive power in the regression models for key outcome variables (to increase the precision of the impact estimates); and (2) they should be predictors of interview nonresponse (to adjust for differences in the characteristics of program and control group respondents). 9 There was no theoretical reason to include different explanatory variables by site or to assume that the parameter estimates on the explanatory variables would differ by site. Thus, we used the same model specification for each site.10 11 The regression R2 values for key outcomes ranged from about .15 (for 24-month child language measures and parent-child interaction scales from the video assessments) to .30 (for Bayley assessment scores and 15-month education measures) to .50 (for measures of welfare receipt).

As discussed, we constructed weights to adjust for interview nonresponse. Our basic approach was not to use these weights in the regression models, because there is no theoretical reason to use them in this context (DuMouchel and Duncan 1983). However, to test the robustness of study findings, we estimated some regression models using the weights (see Appendix D). We also used weights to obtain all estimates of impacts using simple differences-in-means procedures. The weighted and unweighted impact results are very similar (see Appendix D).

At this stage, we have not estimated growth curve models, because of the limited follow-up period. We will estimate these models for the final report after the next rounds of follow-up data collection are completed.

TABLE II.6
CATEGORIES OF CONTROL VARIABLES FOR REGRESSIONS
Family and Parent Characteristics
Age of Mother Race English-Language Ability Education Level Primary Occupation Living Arrangements Number of Children in the Household Poverty Level Welfare Receipt (AFDC/TANF; Food Stamps; WIC; SSI) Has Inadequate Resources (Food, Housing, Money, Medical care, Transportation) Previously Enrolled in Head Start or Another Child Development Program Mobility in the Previous Year Random Assignment Date

Child Characteristics
Age of Focus Child Birthweight Less than 2,500 Grams Gestational Age Gender Evaluation History Risk Categories (Established, Biological/Medical, Environmental)
SOURCE: HSFIS application and enrollment forms.

b. Estimating Impacts per Participant

Random assignment occurred at the point of eligibility and not when families started receiving services. Hence, program and control group differences yield combined impact estimates for those who participated in Early Head Start and those who enrolled but did not participate.

An important evaluation goal, however, is to estimate impacts on those who received program services. Estimating impacts for this group is complicated by the fact that a straightforward comparison of the outcomes of program group participants and all control group members does not yield the desired impact on participants. Ideally, we would like to compare the outcomes of program group participants with control group families who would have participated in Early Head Start had they been in the program group. However, we cannot identify these control group families.

As discussed in Appendix D, we can overcome these complications by assuming that Early Head Start had no effect on families who enrolled but did not receive Early Head Start services. In this case, the impact per participant in a site can be obtained by dividing the impact per eligible applicant in that site by the site's program group participation rate (Bloom 1984). The estimated global impact per participant across all sites can then be calculated as the average of the estimated impacts per participant in each site.

A crucial issue is how to define a program participant. The key assumption that allows us to estimate impacts for participants is that the outcomes of those in the program group who enrolled but did not receive services would have been the same if they had instead been assigned to the control group (that is, the program had no effect on nonparticipants). Thus, in order to be confident that this (untestable) assumption holds, we need a conservative definition of a program participant.

A program group family was considered to be an Early Head Start participant if, during the 15 months after random assignment, the family received more than one home visit, met with a case manager more than once, enrolled its child in center care for at least two weeks, or participated in a group activity. This participation rate was 91 percent for the full program group. It ranged from 66 percent to 98 percent across the program sites, but exceeded 88 percent in 15 of the 17 sites. Because the participation rate was fairly high in most sites, the estimated impacts per eligible applicant and the estimated impacts per participant are very similar.12

c. Adjusting for Crossovers in the Control Group

As discussed, about 0.7 percent of control group members participated in Early Head Start. These crossovers were treated as control group members in the analysis to preserve the integrity of the random assignment design. Thus, the presence of these crossovers could yield impact estimates that are biased slightly downwards if the crossovers benefited from program participation.

The procedure to estimate impacts for participants can be adapted to accommodate the control group crossovers (Angrist et al. 1996). This involves dividing the impacts per eligible applicant by the difference between the program group participation rate and the control group crossover rate. These estimates, however, are very similar to the impacts per participant, because of the very small number of crossovers. For example, the impacts per participant in most sites were obtained by dividing the impacts per eligible applicant by about .90, whereas the impacts that adjust for the crossovers were typically obtained by dividing the impacts per eligible applicant by .893 (.90 - .07). Thus, for simplicity, we do not present the impacts that adjust for crossovers.

d. Presentation of Results

In Chapters IV through VII, where we report program effects on child, parenting, and family outcomes, and the effects on these outcomes for population subgroups, we present impact results for participants.13 However, in Chapter III, where we report program effects for the service use outcomes, we present results for eligible applicants to understand the extent to which Early Head Start programs are serving eligible families, and to understand the services available to eligible families in the absence of Early Head Start. This analysis is critical to understanding program operations and implementation as well as program impacts.

In the impact tables in Chapters IV to VII, we present the following statistics:

The Mean Outcome for Participants in the Program Group. This mean was calculated using the 91 percent of program group members who participated in Early Head Start (using the definition of participation discussed above).
The Mean Outcome for Control Group Members Who Would Have Been Early Head Start Participants If They Had Instead Been Assigned to the Program Group. This mean is not observed, but is estimated as the difference between the program group participant mean and the estimated impact per participant. We sacrifice technical accuracy for simplicity in the text, and refer to this mean as the "control group mean."
The Estimated Impact per Participant. As discussed, this impact was obtained by (1) dividing the regression-adjusted impacts per eligible applicant in each site by the program group participation rate in each site; and (2) averaging these site-specific impacts across sites.
The Size of the Impact in Effect Size Units. This statistic was calculated as the impact per participant divided by the standard deviation of the outcome variable for the control group times 100.
The Significance Level of the Estimated Impact. We indicate whether the estimated impact is statistically significant at the 1 percent, 5 percent, or 10 percent level. We indicate marginally significant findings at the 10 percent level, because we seek to identify patterns of program effects across the large number of outcomes and subgroups under investigation, and thus, relax the traditional 5 percent significance level threshold (see Section 3 below).

We present similar statistics in Chapter III for the impact findings on service use outcomes, except that the statistics pertain to eligible applicants rather than to participants only.

2. Targeted Analysis

The targeted analysis uses a more refined approach than the global analysis to examine the effects of Early Head Start on key outcomes. The targeted analysis addresses the important policy questions of what works, and what works for whom. It focuses on estimating whether impacts differ (1) for sites with different program approaches, implementation levels, and community contexts; and (2) for families with different characteristics at the time of program application. Specifically, the targeted analysis addresses the following questions:

Do different program approaches have different program impacts?
Do different levels of program implementation result in differential impacts?
Do different community contexts result in differential impacts?
Do program impacts differ for children and parents with different baseline characteristics?

In this interim report, we do not address questions about the effects of different intensities of services, because it is too early to define accurately which program group families received intensive services and which did not (since service participation time is limited to about 16 months). In addition, we have not yet analyzed impacts as a function of the programs' expected outcomes, since many of the programs' expectations focused on children and families at the conclusion of their Early Head Start tenure.14 We will address these questions when we have 36-month measures in the final report, in spring 2002. Future analyses will also assess the extent to which impacts on shorter-term outcomes (such as service use and some family and parenting outcomes) correlate with impacts on longer-term outcomes (such as child outcomes), and will examine the role of mediators of child, parenting, and family outcomes.

a. Program Approach, Implementation Level, and Community Context

Early Head Start programs tailor their program services to meet the needs of eligible low-income families in their communities, and select among program options specified in the Head Start Program Performance Standards. ACYF selected the 17 research sites to reflect Early Head Start sites more broadly, and thus the Early Head Start programs participating in the evaluation varied in their approach to serving families. Furthermore, they differed in their pattern of progress in implementing key elements of the revised Head Start Program Performance Standards. Accordingly, we examined how impacts varied by program approach, implementation level, and community context.

Impact results by program approach can provide important information on how to improve program services and to develop and expand the program. Variations in impacts across programs that achieved different levels of implementation may provide insights into the importance of fully implementing key program services. Because Early Head Start programs are required to tailor services to meet local community needs, it is very important to understand the conditions under which they can have various effects.

The specific subgroups defined by key site characteristics that we examined are displayed in Table II.7. The table also displays the number of sites and the percentage of research families (at the time of random assignment) who are included in each subgroup. Table II.8 displays these variables by site (so that the overlap in these site subgroups can be examined). We selected these groupings in consultation with ACYF and the Early Head Start Research Consortium. Because of the small number of sites included in the evaluation, we limited the analysis to a few key subgroups that would capture distinguishing features of Early Head Start programs that are policy-relevant and could be accurately measured.

For the analysis of impacts by program approach, we divided programs into four center-based, seven home-based, and six mixed-approach programs on the basis of their program approaches in 1997 (see Chapter I). As discussed throughout this report, because the three approaches offer different configurations of services, we expect differences in the pattern of impacts by approach (see, especially, discussions of the hypotheses relating to expected impacts in Chapters IV, V, and VI).

We used data collected from the implementation study site visits in fall 1997 and fall 1999 to assess the degree of implementation in each of the research programs (see Chapter I). We then divided programs into (1) early implementers (six sites), (2) later implementers (six sites), and (3) incomplete implementers (five sites). The early implementers became "fully implemented" by 1997 and remained so at the time of the 1999 site visits, while the later implementers were not fully implemented in 1997 but were by 1999. The incomplete implementers had still not achieved full implementation by 1999, although they demonstrated a number of strengths in particular programmatic areas.15 Some programs that were fully implemented overall were not fully implemented in one of the two areas relating to the delivery of key program services: child development and family development. Accordingly, we also examined impacts separately for groups of programs defined by their level of implementation in child development services and family development services. Though there was some overlap in these three implementation measures (Table II.8), we believe that there were enough differences to warrant separate analyses for each one.

Clearly, we expect impacts on child, parenting, and family outcomes to be larger in the fully implemented programs than in the incompletely implemented programs, because the fully implemented programs delivered services that were more intensive, more comprehensive, and of higher quality. Similarly, we expect impacts to be larger in the programs that became fully implemented earlier than in those implemented later.

Assessing impacts by the level of implementation is complicated by the fact that the fully implemented programs were not evenly distributed across the program approaches, as can be seen in Table II.8. For example, only one of the seven home-based programs was an early implementer, as compared to two of the four center-based programs and three of the six mixed-approach programs. Thus, comparing all implementers to all nonimplementers confounds impact differences by implementation level with impact differences by program approach. Therefore, we also estimated impacts for subgroups defined by interacting program approach and implementation level. Because of sample size constraints, this analysis focused on comparing estimated impacts for the three mixed programs that were early implementers to those of the three mixed programs that were not early implementers (see Appendix D).

We created an additional site-level subgroup defined by whether or not the state or county had work requirements for mothers who werereceiving TANF and who had children younger than 12 months. Hypotheses of expected impacts for these groups are discussed in Chapter VI.

The ability of the national evaluation to assess the community context was somewhat limited. A number of the local research teams conducted in-depth research in their program communities, however. Box II.1 illustrates the application of ethnographic research in the Pittsburgh and Denver sites.

**TABLE II.7 SUBGROUPS DEFINED BY PROGRAM APPROACH, IMPLEMENTATION LEVEL, AND COMMUNITY CONTEXT**
Subgroup	Number of Sites	Percentage of Families
Program Approach
Center-based	4	20
Home-based	7	46
Mixed Approach	6	34
Overall Implementation Level
Early implementers	6	35
Later implementers	6	35
Incomplete implementers	5	30
Implementation of Child Development Services
Early implementers	6	34
Single-period implementers	5	27
Incomplete implementers	6	39
Implementation of Family Development Services
Early implementers	7	43
Single-period implementers	7	41
Incomplete implementers	3	16
Whether State or County Has Work Requirements for TANF Mothers with Children Younger Than 1
State has requirements	7	42
State has no requirements	10	58
SOURCE: Data from 1997 and 1999 site visits

TABLE II.8
SUBGROUPS DEFINED BY SITE CHARACTERISTICS, BY SITE
Site	Program Approach	Overall	Child Development	Family Development	Work Requirements for TANF Mothers With Infants
		Implementation Level			Work Requirements for TANF Mothers With Infants
1	Center	Early	Early	Early	Yes
2	Home	Later	Incomplete	Single Period	No
3	Mixed	Later	Incomplete	Single Period	Yes
4	Center	Early	Early	Incomplete	Yes
5	Mixed	Incomplete	Single Period	Incomplete	No
6	Home	Incomplete	Incomplete	Single Period	Yes
7	Mixed	Early	Early	Early	No
8	Home	Later	Incomplete	Early	Yes
9	Home	Incomplete	Incomplete	Incomplete	No
10	Center	Incomplete	Single Period	Single Period	No
11	Home	Incomplete	Incomplete	Early	No
12	Mixed	Later	Single Period	Single Period	No
13	Home	Early	Early	Early	No
14	Mixed	Early	Early	Early	Yes
15	Mixed	Early	Single Period	Early	No
16	Home	Later	Single Period	Single Period	No
17	Center	Later	Early	Single Period	Yes
SOURCE: Implementation study data. NOTE: Sites are in random order.