ACF OPRE: Head Start Impact Study and Follow-up

Appendix B: Ways to Analyze Impacts without Placing a Two-Year Exclusion on Controls

Suppose randomization to the control group meant just a single year of exclusion from Head Start, not a full two-year embargo? Families of children accepted for admission to a two-year sequence (i.e., newly-entering 3-year olds) would still be served even if they were assigned to the control group at program entry, but services would be deferred for a year. In contrast, those assigned to the treatment group would be enrolled immediately and allowed to participate for the full two years.

This design option provides a direct measure of the impact of Head Start participation at age 3 years as an addition to the "standard" 1-year of services starting at age 4 years:

(1) Impact of E given S = (average outcome for Ts) - (average outcome for Ds) ,

where E represents the early, age 3, year of Head Start services and S the standard year, and where Ts and Ds are members of the treatment group and the "deferred access" control group respectively. Since both Ts and Ds participate in Head Start at age 4 and-through random assignment-are similar in background characteristics, the difference in average outcomes between the two groups reflects the impact of the earlier year of service (received by the treatment group alone) as a supplement to the standard year (received by all sample members). The same basic model can be used to obtain a measure of the impact of the "standard" year of Head Start participation as a "stand alone" option-and of a combined two years of participation. But here the derivation, and the results, are not so straightforward.

Notation and Potential Experimental Groups

Some additional notation will aid our analysis of the extra, more complex cases. Consider two additional groups of children existing for the moment only hypothetically:

Cs = children denied access to Head Start for the full two years (true controls)

Rs = children served by Head Start initially (i.e., during the early year that begins at age 3) but then removed from the program prior to the start of the pre-K year.

A matrix (shown on the next page) will help us keep track of the four experimental groups introduced to date, defined by the extent and timing of their Head Start program exposure.

Group	Participate in Head Start During Early Year (E)	Participate in Head Start During Standard Year (S)	Observed in the Data?
T [treatment]	X	X	yes
R ["removal"]	X	-	maybe
D ["deferral"]	-	X	yes
C [control]	-	-	no

It will also be useful to simplify notation for impacts and outcomes, restating Equation 1 as

(1') Impact E | S = Y(T) - Y(D) ,

where Y( ٠ ) represents the average outcome of the group in parentheses. Having measured the impact of the early year as an addition to the standard year in this way, we are also interested in the impact of a full two years of Head Start participation, the early plus standard years:

(2) Impact E + S = Y(T) - Y(C ) .

Finally, the impact of just the standard year is of some interest, both for the population of children and families now provided with two years of service and those only seeking (or only obtaining) one year.

The Challenge and a First Response

We want to obtain all these results without creating or observing Y(C ), outcomes for a fully excluded control group. This is not a problem in the first year of the analysis, when the deferral group, D, looks just like a pure control group (see matrix). Neither group participates in Head Start during that year, so Y(T ) - Y(C ) can be calculated without bias by substituting Y(D) for Y(C ) and calculating Y(T) - Y(D). However, this expedient is not available in the second and subsequent years of the research period once the deferral group D becomes "contaminated" by participating in the standard year of Head Start services. By this point, all of the three potentially observable groups-T, R (which we may or may not observe), and D-have spent at least some time in the program.

One way around this dilemma is to assume "strict additivity" of impacts across the two years of program participation. In any one year, starting with the year of "standard" Head Start participation and continuing to the end of the follow-up period (the end of 1^st grade),

the treatment group T may experience improved outcomes relative to a pure control group C for two reasons:

Ts may benefit from the Head Start assistance received during the earlier of the two years of participation as benefits of the initial "step up" to faster development continue to accrue throughout a child's early elementary years.

Head Start assistance received during the standard year of participation will also have "carry-over" benefits to succeeding years.

While both gains-those engendered by the early Head Start year and those engendered by the standard year-could occur at the same time for the same child, conceptually they can be kept separate. The question is whether the two gains interact with one another when both are present, making Head Start's total impact greater than (or, with negative synergism, less than) the sum of the parts.

To explore this option, we will start with the much simpler case of only a single year of Head Start enrollment. We can think about the impact of an early Head Start year (among those who get it) alone-i.e., absent enrollment during the standard year. We can represent this case as follows:

(3) Impact E | no S = Y(R ) - Y(C ) ,

since Rs participate in Head Start only in the earlier year and Cs not at all.

Similarly, we can think of the impact of the standard year of Head Start alone over that same period-i.e., absent participation in the early Head Start year:

(4) Impact S | no E = Y(D ) - Y(C ).

Strict additivity says that the impact of these two years of Head Start participation, when combined as a "package," equals the sum of the two individual effects, no more and no less:

(5) Y(T ) - Y(C ) = [ Y(R ) - Y(C ) ] + [ Y(D ) - Y(C ) ] .

This formulation assumes a complete lack of synergism between the two years of Head Start services: the benefit a child receives from an early year of Head Start enrollment is the same regardless of whether a second year is added, and the benefit from the standard year is the same whether or not it was preceded by an early year. No third term showing interaction or synergism between these components, either positive or negative, appears on the right-hand side of equation 5, leaving the whole exactly equal to the sum of the parts.

Equation 5 converts into

(5') Y(T ) = Y(R ) + Y(D ) - Y(C ) , which implies in turn that

(6) Y(C ) = Y(R ) + Y(D ) - Y(T ) .

Substituting back into equation 2, we get

(2') Impact E + S = Y(T ) - [Y(R ) + Y(D ) - Y(T ) ]

= 2 Y(T ) - Y(R ) - Y(D ) ,

which can be calculated without using a pure control group as long as the other three types of "exposure" to the Head Start treatment are created-full treatment for Ts, an initial year only for the "removal" group R, and the later standard year only for the "deferral" group D.

Some Caveats

Despite its advantage in eliminating the need for a pure control group, there are several reasons why we might hesitate to use equation (2') to measure Head Start's impact:

Execution of this approach requires 3-way random assignment of children applying for the early year of Head Start into separate T, R, and D groups. This complicates both the administration of random assignment and subsequent program actions with respect to children assigned to different study groups. Perhaps more importantly, it substantially increases the sampling variability of all impact estimates, both because (for a given total sample size) each study group grows smaller by one third to accommodate three rather than two groups, and because the impact formula now contains three independent terms-one even multiplied by a factor of 2-effectively quadrupling the variance of the impact estimate and thus doubling the size of the effects that can be detected with confidence.

It also means pulling some children-the Rs-out of Head Start after just one year of what is intended to be a two-year sequence. This could have two detrimental effects, adding a new potential concern regarding the ethics of random assignment (is it better to start services and then interrupt them, or to not start them at all?) and potentially engendering distorted behavior on the part of the families and/or Head Start workers who deal with the "one year and out" children. For example, knowing a child cannot stay in Head Start a full two years, a family may put the child in a non-Head Start care arrangement right away, effectively turning what was to have been an R case into a C case who receives no Head Start services at all. Similarly, Head Start staff might invest less in a young child known to be leaving the program at the end of the year than would otherwise be the case-or invest more to push his/her school readiness ahead faster in the limited time available.56

The assumption of strict additivity of effects may be wrong, depending on what one believes about complementarities between successive years of Head Start exposure. For example, one might think that the benefits of receiving the standard year of Head Start at age 4 go up if the child previously received Head Start services at age 3 to "lay the foundation" for greater gains in subsequent years. Alternatively, Head Start services may-like many other products and services in the economy-yield diminishing returns as one adds layer upon layer, making the standard year of service most productive when it is the only year of services.

This last concern not only points to the possibility that impact estimates based on equation (2') are biased, it also illustrates the difficulty one has under this approach judging the likely direction of bias. If an earlier year's enrollment makes the standard year's assistance more effective, equation (5) should have a third, positive term on the right-hand side and equation (2') underestimates the overall contribution of two years of Head Start enrollment by leaving out the positive synergism. If instead the standard year of Head Start becomes less valuable in its net contribution once a child has the initial, early year of Head Start services under her/his belt, equation (5) is missing a negative term on the right-hand side and equation (2') overestimates the contribution of the two years together by allocating full value to each year, including twice the benefit of an "initial dose" that in practice will only occur once.

A Lower-Bound Strategy

An alternative approach to estimating overall effects without a pure control group focuses on bounding the likely impact above and below as a way of narrowing our uncertainty regarding the direction and degree of bias. Two additional estimates of Head Start's impact come into play at this point, one likely biased upward and the other likely biased downward. While neither of the estimates is likely to provide the "right" answer, jointly they provide useful upper and lower bounds on that quantity. Hence, their advantage is not so much that we think them unbiased but that the direction of bias is known or can be assumed with some confidence.

Beginning in the second year of Head Start enrollment and for all subsequent years, a lower bound on program impact can be calculated as:

(7) Lower bound (Impact E + S) = [ Y(T ) - Y(D) ] + [ Y(T*) - Y(C*) ] ,

where T* and C* are the treatment and control groups created from children who apply for Head Start for the first time in the later, standard year, just a year before kindergarten entry. This latter set of children has not been considered to this point but will be an essential part of the overall evaluation, providing measures of the impact of the one-year version of Head Start for children whose families seek only a single year (or who are only admitted at the later point having applied but been excluded the previous year) The question of two years of artificial exclusion from the program never comes up for this group, so it is not directly germane to the topic of this section. It can be analyzed experimentally using standard tools each year [i.e., impact S = Y(T*) - Y(C*)]. We simply "import" that calculation here to help us deal with the thorny problems of the two-year program-specifically by approximating a parallel impact measure for the two-year service population.

To see how this is done, return again to equation (2) but now expand it by adding and subtracting a common term::

(2'') Impact E + S = [ Y(T) - Y(D) ] + [ Y(D) - Y(C ) ] .

Here, Y(D) - Y(C ) represents the impact of Head Start services that start just a year prior to kindergarten for children who normally enter the program a year earlier. Not observing Y(C ), we don't know what this quantity is. What we do know is the corresponding quantity for a somewhat different population, children who normally enter Head Start only a year before kindergarten, Y(T*) - Y(C*). The two expressions match up because the children in T* follow the same sequence of events as the children in D-no Head Start participation at age 3, followed by Head Start participation at age 4-and the children in C* mirror those in C (no Head Start participation at any time). Importantly, however, the two groups of children are not the same in their underlying characteristics, nor do they necessarily come from the same kinds of homes nor receive the same exact treatment from Head Start.

The assumption required to use Y(T*) - Y(C*) in place of Y(D) - Y(C ) in forming a lower bound is that

The children who can benefit most from Head Start are the most likely to enter the

program early (i.e., at around age 3 rather than around age 4).

This assumption will hold as long as two other seemingly plausible conditions are met:

Parents, or Head Start intake staff, or both put greater emphasis on early Head Start enrollment for children they believe will benefit most from participation.

The same parents and/or staff have at least some ability to judge reliably which children in fact will benefit most from Head Start participation.

When these conditions hold early entrants experience greater initial impacts than older entrants and

(8) Y(D) - Y(C ) > Y(T*) - Y(C*) .

Substituting the smaller of these expressions for the larger in equation (2''), we get

(2''') Impact E + S = [ Y(T) - Y(D) ] + [ Y(D) - Y(C ) ]

> [ Y(T) - Y(D) ] + [ Y(T*) - Y(C*) ] ,

showing the estimate in equation (7)-the right-hand side of this new equation-to be a lower bound on the true impact of two years of Head Start enrollment for the children normally served for that long. Equally important, all of the terms in equation (7) can be calculated from an experiment based on simple 2-way random assignment to a full treatment group (T) or a deferred treatment group (D)-for children who normally enter Head Start two years before kindergarten-and a treatment group (T*) or control group (C*)-for children who normally enter one year before kindergarten.

A Complementary Upper Bound

To complement this lower bound, we can from the same experiment calculate an upper bound on Head Start's impact on two-year participants during the second year of participation:

(9) Upper bound (Impact E + S) = [Y(T) - Y(D) ] + [ Y^ (T) - Y^ (D) ] ,

where Y^ ( · ) indicates the average outcome for the group in parentheses a year prior to the standard, second year of Head Start participation and Y( · ) again represents average outcomes during the standard, second year. Interestingly, equation (9) sums the difference in outcomes between children allowed to enter Head Start at age 3 (the Ts) and those deferred until age 4 (the Ds) across two successive years of observation. For this "sum-of-differences" estimate to be upper bound on Impact E + S, we must assume that

A given set of Head Start services has a larger immediate (i.e., same-year) effect

on a given child if begun earlier in that child's life.

This assumption accords with recent research on the relative importance of developmental inputs at different stages of a child's life, and when applied to the two years leading up to kindergarten entry implies that:

(10) Y^ (T) - Y^ (C ) > Y(D) - Y(C ) ,

Here, the left-hand term shows the benefits of Head Start if participation begins two years prior to kindergarten entry and the right-hand side shows the benefits-for the same population-of Head Start participation begun a year later. Substituting the larger of these two for the smaller in equation (2''), we get

(2'''') Impact E + S = [ Y(T) - Y(D) ] + [ Y(D) - Y(C ) ]

< [ Y(T) - Y(D) ] + [ Y^ (T) - Y^ (C ) ] .

As noted previously, in the first year of the analysis period (i.e., two years pre-K) outcomes for Cs and Ds are the same because neither group has as yet entered Head Start and the two groups are otherwise matched through random assignment. This circumstance allows us to substitute Y^ (D) for Y^ (C) in equation (2''''), turning the right-hand side of the equation into the impact estimate defined by equation (9) and thus confirming that it is indeed an upper bound on true impact under the assumption posited. As with the lower bound in equation (7), all terms in equation (9) can be calculated from a simple 2-way experiment.

Potential Problems with This Approach

First, to develop an upper bound analogous to that in equation (9) for the kindergarten and 1^st grade years of follow-up, we will have to again assume strict additivity of impacts across the two separate years of program enrollment. From equation (5), this implies that

(11) Y(T) - Y(D) = Y(R ) - Y(C ), in all years,

and a substitution equivalent to the replacement of Y^ (C ) with Y^ (D) can again be made. [The exact derivation is fairly complex and will not be presented here.] But this makes our upper bound dependent on the same assumption as our initial point estimate. If the strict additivity assumption is true, the point estimate gives us exactly the right result and we don't need bounds (upper or lower); it the assumption is not true, we are left with nothing but a lower bound unless a new means of deriving an upper bound is found.

Second, the deferred entrants (Ds) may not come back for Head Start services a year after being turned aside by random assignment. This distorts all impact estimates that rely on Y(D), though it may be possible to work around this with some added assumptions.

Finally, the calculated lower and upper bounds in equations (7) and (9) may not be well behaved (i.e., upper < lower) or of a sensible magnitude relative to the "strict additivity" point estimate.

56. For legal and ethical reasons, it is probably not an option to delay the identification of Rs until the end of the first Head Start year, grouping Rs and Ts together (as simply "program participants") during the first year. The informed consent form signed by applicant families prior to any randomization needs to indicate all the randomization that will take place. This advance announcement of a later, second round of assignment-and the "double jeopardy" it suggests- may have just as chilling an effect on family participation in Head Start as would telling a subset that the initial year will definitely be the only one. (back)

Table of Contents | Previous | Next