Table of Contents | Previous | Next |
D.3 ESTIMATING IMPACTS PER PARTICIPANT
The comparison of the average outcomes of all program and all control group members yields unbiased estimates of program impacts for eligible applicants, because random assignment was performed at the point that applicant families were determined to be eligible for Early Head Start services. In Chapter II, we described our methods for obtaining regression-adjusted impacts per eligible applicant. However, some eligible families in the program group decided not to participate in the program after random assignment. This appendix describes the procedures that we used to obtain unbiased impact estimates for those who actually received some services (that is, for program participants).3
We used a two-step procedure to estimate impacts per participant for both the global and the targeted analyses. First, for each site, we divided the regression-adjusted impacts per eligible applicant by the site's program group participation rate (Bloom 1984). Second, we averaged these site-specific impact estimates giving equal weight to each site.
To illustrate how this procedure generates unbiased impact estimates for participants, we express the impact per eligible applicant on a given outcome in a site as a weighted average of the program impact for those eligible applicants who would participate in Early Head Start, given the chance, and the program impact for those who would not participate, with weights ps and (1-ps), respectively. In mathematical terms:
where IEs is the impact per eligible applicant in site s, IPs is the impact per participant (that is, the difference between the average outcome of program and control group members who would participate in Early Head Start if given the chance), and INs is the impact per nonparticipant (that is, the difference between the average outcome of program and control group members who would not participate if given the chance).
We do not know which control group families would have participated if they had instead been assigned to the program group, or which control group members would not have participated. However, this information is not necessary if we assume that all impacts were due to those who participated in Early Head Start, and that the impacts on nonparticipants were zero (that is, INs = 0). Under this assumption (or "exclusion restriction"), the impact per participant in a site can be calculated by dividing the impact estimate per eligible applicant (that is, those based on all program and control group members) by the proportion of program group members who participated in Early Head Start. In mathematical terms:
Our estimate of the impact per participant across all sites is the simple average of the site-specific impacts per participant (that is, the average of IPs over all sites). The standard errors of these impacts are larger than those for the impacts per eligible applicant, because the standard errors for the impacts per participant need to account for the estimation error in the site participation rates.
To make this procedure operational, we used PROC SYSLIN in the SAS
statistical software package to estimate the following system of equations,
using two-stage least-squares (instrumental variable) estimation techniques:
where Sj is an indicator variable equal to 1 if the family is in site j, P is an indicator variable equal to 1 if the program group family participated in Early Head Start (and is 0 for control group families and program group nonparticipants), T is an indicator variable equal to 1 if the family is in the program group, y is an outcome variable, X are explanatory variables (that include site indicator variables), e and the ujs are mean zero disturbance terms, and *j, aj, and b are parameters to be estimated.
In the first-stage regressions, we obtained estimates of *j in equation (3) for each site j. These estimates were the program group participation rates in each site.4 In the second-stage regression, we estimated equation (4) where the predicted values from the first-stage regressions were used in place of the Sj*P interaction terms. In this formulation, the estimate of aj from the second-stage regression represents the impact estimate per participant in site j. The standard errors of these estimates were corrected for the estimation error from the first-stage regressions.5
3Our definition of a program participant was discussed in Chapters II and III.(back)
4We also estimated models that included other explanatory variables (that is, that included the X variables in equation [4]). These models did not change the results and so, for simplicity, were not adopted.(back)
5This procedure uses the treatment status indicator variable (T) as an "instrument" for the program participation indicator variable (P) in each site. This is a valid instrument, because T is correlated with P but is uncorrelated with the disturbance term e due to random assignment. The instrumental variable estimates of the impacts per participant are identical to the estimates using the Bloom procedure described above (Angrist et al. 1996).(back)
| Table of Contents | Previous | Next |