Table of Contents | Previous | Next |
Appendix B: Analysis of Programs Excluded During Sampling
Comparison of Saturated and Non-saturated Head Start Programs
The frame used to select Head Start grantee/delegate agencies initially contained all the programs on the PIR in each of the 25 clusters (referred to as primary sampling units, or PSUs) that were determined to be eligible for the impact study based on PIR data. The total number of programs in the selected 25 PSUs was 355. Prior to sampling Head Start programs in each PSU, we further screened the programs by telephone to determine their eligibility for the study. To reduce the amount of telephone screening in three very large clusters, programs were sub-sampled resulting in a reduction of 86 programs; eight FACES programs were also deleted to avoid sampling them for the Head Start Impact Study. This resulted in a list of 261 grantee/delegate agencies that were screened for inclusion in the study. During this screening process, 28 programs were determined to be saturated and 10 programs were found to have closed or had merged with another program. After eliminating these 38 programs, the final frame consisted of 223 programs in the 25 PSUs.
The 28 saturated programs represent an estimated 12 percent [5.3%, 19.0%]32 of the eligible Head Start program universe, and approximately 4.3 % [2.2%, 6.4%] of the total Head Start age 3-4 year old enrollment and 3.9% [2.0%, 5.7%] of the newly-entering 3-4 year old Head Start enrollment. The 10 closed programs represent 60 closed programs on the PIR frame containing 3.0% of the total enrollment and 2.7% of first year enrollment. The total enrollment data used in these calculations were taken from the 1999-2000 PIR, and the newly-entering enrollment data came from the telephone screening.
There is potential for under-coverage bias due to the exclusion of saturated Head Start programs from the sampling frame. Newly-entering Head Start children in these saturated programs had no chance of selection for the study and therefore are not represented in the study sample. The potential for bias arises if the saturated programs are systematically different from the non-saturated programs retained on the frame, and if the characteristics on which they differ are correlated with the outcome measures for the children they enroll. If the children in these excluded programs represent only a small percentage of the Head Start population, then the potential for bias is much less.
Tables B.1 and B.2 compare saturated and non-saturated programs by the few qualitative characteristics and enrollment variables available from the PIR. The programs were weighted to account for the PSU stage of sampling and for the sub-sampling of programs in three very large clusters prior to the telephone screening. This is necessary to draw conclusions about the entire Head Start population and not merely programs in the 25 sampled PSUs. Tests of statistical significance were done using WesVar with jackknife replicate weights to account for the Head Start Impact Study sample design.
Tables B.1 and B.2 show that the saturated programs are smaller, more likely to be school-based, and have smaller percent Hispanic enrollment than the non-saturated programs. Although they appear to be more often located in the Midwest, differences in the distribution of saturated vs. non-saturated programs by Head Start regions are not statistically significant. A cautionary note is that variances at the program level are not very stable because the number of saturated programs is small. In addition, they do not include the between-PSU component of variance due to sampling PSUs33, thus they are underestimates and the p-values may be overstating the statistical significance of the differences.
| Characteristic | Saturated Programs | Non-saturated Programs | P-value for t-test of difference in means or proportions * |
|---|---|---|---|
| Percent Hispanic Enrollment | 9% | 26% | .001 |
| Percent Black Enrollment | 20% | 33% | .134 |
| Percent Age 3 Enrollment | 52% | 49% | .535 |
| Percent First Year Age 3 Enrollment | 79% | 87% | .222 |
| Average Total Enrollment | 188 | 571 | <.001 |
| Average Newly-Entering Children | 113 | 388 | <.001 |
| * Note: bold print indicates statistically significant difference. |
| Characteristic | Saturated Programs | Non-saturated Programs | P-value for Chi-square test of association * |
|---|---|---|---|
| School-based | .02 | ||
| Yes | 66% | 21% | |
| No | 34% | 79% | |
| Metro Status | .91 | ||
| Urban (MSA) | 66% | 68% | |
| Rural (Non-MSA) | 34% | 32% | |
| Level of Pre-K Services in State | .60 | ||
| Similar to Head Start | 35% | 25% | |
| Some Head Start-like | 27% | 20% | |
| Remaining States | 38% | 55% | |
| Head Start Region | .15 | ||
| Northeast | 24% | 25% | |
| Midwest | 48% | 24% | |
| South | 28% | 39% | |
| Plains | 0% | 4% | |
| West | 0% | 8% |
| * Note: bold print indicates statistically significant difference. |
Comparison of Saturated and Non-saturated Head Start Centers
Enrollment data were also obtained for all of the operating Head Start centers in each sampled grantee/delegate agency using a standardized data collection form (called the Center Information Form or CIF). This information was used to determine the level of saturation for each center. Centers that would clearly be unable to provide a sample of non-Head Start control group children were deleted from the frame prior to sampling; three sampled grantees were dropped at this stage because all of their centers were determined to be saturated.
This resulted in the deletion of 154 centers from a frame of 1,423, in addition to the seven centers in the three saturated programs. These 154 centers represent an estimated eight percent [4.7%, 11.4%] of the Head Start center universe, approximately 5.1 percent [3.0%, 7.1%] of the total Head Start age 3-4 enrollment, and 4.7 percent [2.7%, 6.8%] of newly-entering children. Overall, the estimated coverage rate of the newly-entering Head Start population for the Head Start Impact Study sample is 92 percent [89%, 95%], which is obtained by multiplying the coverage rate for programs and centers [(100-3.9% for programs)*(100- 4.7% for centers)], and the overall undercoverage rate is eight percent [5%, 11%].
Tables B.3 and B.4 compare saturated and non-saturated centers by various characteristics and by characteristics of enrolled children. As with grantee/delegate agencies, hypothesis testing was done in WesVar using jackknife replicate weights to account for the study sample design. The replicate weights do not yet incorporate the between-PSU variance component, therefore the p-values in these tables may be too liberal. In Table B.3 the chi-square test was not able to detect a significant difference with respect to type of program option offered, whether the program is based in a school, metro status, region, or level of Pre-K services available in the state. With respect to enrollment, Table B.4 shows that the saturated centers are smaller, less Hispanic, and have a larger percentage of newly-entering 3-year-olds than the non-saturated centers.
The saturation rate was calculated two ways: as the percent of centers in each program that are saturated, and as the percent of newly-enrolled children in saturated centers in each program. The average percent of saturated centers is 16.6 percent and ranges from 0 to 84 percent. The average percent of newly-entering children in saturated centers is only 13.2 percent and ranges from 0 to 69 percent, another indication that the saturated centers tend to be smaller.
| Characteristic | Saturated Programs |
Non-Saturated Programs |
P-value for Chi-square Test of Association |
|---|---|---|---|
| Program Option | .44 | ||
| Full-Day Only | 35% | 28% | |
| Part-Day Only | 52% | 50% | |
| Other | 13% | 22% | |
| School-Based | .25 | ||
| Yes | 17% | 11% | |
| No | 83% | 89% | |
| Metro Status | .64 | ||
| MSA | 74% | 70% | |
| Non-MSA | 26% | 30% | |
| Head Start Region | .38 | ||
| Northeast | 32% | 27% | |
| Midwest | 34% | 20% | |
| South | 17% | 31% | |
| Plains | 12% | 11% | |
| West | 4% | 11% | |
| Level of Pre-K Services in State | .21 | ||
| Similar to Head Start | 40% | 22% | |
| Some Head Start-like | 15% | 18% | |
| Remaining States | 45% | 60% |
| Characteristic | Saturated Programs |
Non-Saturated Programs |
P-value for t-test of difference in means or proportions* |
|---|---|---|---|
| % Hispanic Enrollment | 17% | 30% | .005 |
| % Black Enrollment | 38% | 26% | .204 |
| % Newly-Entering Children | 65% | 66% | .985 |
| % Newly-Entering 3-Year-Olds | 54% | 47% | .037 |
| % on Waiting List | 0% | 15% | <.001 |
| Average Number of Funded Slots | 37 | 48 | .036 |
| Average Total Enrollment | 26 | 47 | <.001 |
| Average Newly-Entering Children | 16 | 31 | <.001 |
| Average Number on Waiting List | 0 | 9 | <.001 |
| * Note: bold print indicates statistically significant difference. |
32The 95% confidence intervals are given in brackets, as calculated from standard errors produced by Wesvar using jackknife replication. (back)
33These adjustments will be incorporated for the spring 2003 replicate weights. (back)
| Table of Contents | Previous | Next |