Table of Contents | Previous | Next |
Appendix 4.5: Tests for Lack of Impact of Head Start on Demographic and Developmental Factors Measured in Fall 2002
As discussed in Chapter 4, most of the demographic and developmental factors considered as covariates or for the impact regressions were measured with some lag following random assignment. Because these measures could have been influenced by the Head Start intervention, a statistical procedure was developed to determine if strong evidence exists that a given measure was not affected by Head Start to any appreciable degree. Where such evidence was found, the measure remained in the regression equation for the preferred impact analysis and, if of substantive interest, served as a moderator in examining how impacts varied with family and child background characteristics. This appendix describes the procedure used to make this determination and presents test results for all fall 2002 variables tested using the technique.
The procedure adopted seeks a 90 percent assurance that Head Start’s impact in the fall was small or nonexistent. Only then is it considered safe to rely on impact estimates that adjust for a given background characteristic of families or children in the study or the fall measure of the outcome variable. “Small” is defined on a relative basis that takes account of how much the fall measure varies in the population being studied, as indicated by the standard deviation of the outcome variable in fall 2002 for the non-treated comparison group. A guideline suggested by Cohen is adopted that classifies the impacts of educational and child development interventions as small, modest, or large based on their “effect size”—their average impact divided by the standard deviation in the population.1 If 90 percent certain that effect size is less than 0.2, we conclude that a small (or perhaps zero) impact has taken place, making it safe to include the variable in question in impact regressions as a covariate and/or moderator. Otherwise, where true impact may move into the range characterized by Cohen as “modest” or larger, the analysis omits the variable from the preferred set of covariates and refrains from using it as a moderator.
Formally, the test involves constructing a 90 percent confidence interval for true impact in fall 2002 calibrated in effect-size units, then checking that this interval lies entirely between -0.20 (the limit of negative impacts that Cohen would consider “small”) and 0.20 (the limit of positive impacts that Cohen would consider “small”). First, the regression procedures described in Appendix 4.3 in connection with the main impact findings in spring 2003 are applied using the fall 2002 measures of interest as dependent variables. Thus, we estimate impacts on 32 continuous variables using OLS regression models that express a fall measure, F, as the sum of an intercept term (a) and a shift in the intercept produced by a dummy variable for inclusion in the Head Start group (H):
F = a + bH
This equation is estimated using weighted data described in Appendix 1.2 to represent the national population of newly entering Head Start children in communities with more potential Head Start participants than funded Federal Head Start slots.
The coefficient on the dummy variable in this model, b, provides an estimate of B, the true (but ultimately unknown) average impact of Head Start on the fall measure F—an estimate computationally identical to the difference between the average level of the fall measure for the Head Start sample (H=1) and the average level of the measure for the non-Head Start sample (H=0). The estimate of the standard deviation of this coefficient—its standard error, s—is used to construct an interval around b that for 90 percent of all possible surveys conducted in the same manner will contain the true average impact B, given the usual OLS assumption that the dependent variable and hence all estimated coefficients have normal distributions. This interval is
b – 1.645s < B < b + 1.645s , since
- Pr (b – 1.645s < B < b + 1.645s) = Pr [(b-B)/s – 1.645 < 0 < (b-B)/s + 1.645)
-
= Pr [ - 1.645 < (b-B)/s < 1.645 ] = M(1.645) - M(-1.645) = 0.95 – 0.05
= 0.90 ,
where M( ) is the cumulative density function of a standard normal distribution used to approximate the students t distribution of (b-B)/s when sample size is large (e.g., N > 200). The endpoints of this interval are then divided by the standard deviation of F in the non-Head Start sample, d,2 to convert it to effect-size units:
[b – 1.645s] / d < B/d < [b + 1.645s] / d .
If - 0.20 < [b – 1.645s] / d and [b + 1.645s] / d < 0.20 ,
we conclude that the true effect of Head Start on F measured in fall 2002, B/d, is within Cohen’s range of “small”. By construction, there is at least a 90 percent chance that we are right in this conclusion, since the entire 90-percent confidence interval for true effect size is between -0.20 and 0.20. If instead either
[b – 1.645s] / d < - 0.20 or [b + 1.645s] / d > 0.20
we cannot be 90 percent confident that the true effect size is “small” by Cohen’s standard.
We proceed in parallel fashion for the 21 0/1 indicator variables, but use logistic regression procedures described in Appendix 4.3 to calculate a log-odds ratio and obtain an estimate of B and its standard error, s. The estimated equation expresses the natural log of the odds ratio of F=1 to F=0 as the sum of an intercept term and a shift in the intercept produced by a dummy variable for inclusion in the Head Start group:
ln [P/(1-P)] = c + dH,
where P is the probability that F = 1 (and, hence, 1-P is the probability F = 0).
The impact estimate b is then derived as the difference between two quantities:
-
log-odds ratio for children in the Head Start group, c + dH, converted into a probability by passing it through the logistic transformation,
P(H=1) = exp (c+d) / [1 + exp (c+d)] , and
-
the log-odds ratio for children in the non-Head Start group, c, converted into a probability by the same transformation,
P(H=0) = exp (c) / [1 + exp (c)] .
Thus,
b = exp (c+dH) / [1 + exp (c+dH)] - exp (c) / [1 + exp (c)] .
The standard error of b, s, and a 90 percent confidence interval for B follow from the standard probability and distributional assumptions of logit regressions, as calculated by the SUDAAN software package.
1 See Cohen, J. (1987). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Erlbaum. (back)
2 Standard deviation is calculated as the square root of the sum of squared deviations of individual values of F from the mean of F for the entire non-Head Start sample. (back)
| Table of Contents | Previous | Next |