ACF OPRE: Early Head Start Research and Evaluation Project (EHSRE)

D.1 COMPARING THE BASELINE CHARACTERISTICS OF PROGRAM AND CONTROL GROUP MEMBERS

In theory, randomized experimental designs ensure that differences in the average outcomes between program and control groups can be attributed to the intervention under investigation. This rigor is possible, however, only if the random assignment process generates program and control groups with similar characteristics, on average, at the time of random assignment. Thus, the benefits of the random assignment design can be realized only if random assignment is implemented correctly and produces equivalent research groups.

We believe that the process used in the Early Head Start study to randomly assign families to the program or control groups was implemented correctly. MPR staff controlled the process, random numbers generated from a computer were used to assign the families to a research status, and, to the best of our knowledge, local programs and research staff followed the specified procedures for obtaining applicants and notifying families of their group assignment.

In this appendix, we compare the characteristics of program and control group families to check that the random assignment process was implemented correctly. First, we discuss data sources and methods and then discuss analysis results.

1. Data Sources and Methods

We used data from the Head Start Family Information System (HSFIS) application and enrollment forms for the analysis. This information was collected prior to random assignment, so neither the quality of the data nor item response should differ by research status if random assignment was conducted properly. As discussed in Appendix C, the HSFIS data contain demographic information on families, primary caregivers, and focus children.

We used standard statistical tests to assess the similarity of the two research groups, including univariate t-tests to compare variable means for binary and continuous variables and chi-squared tests to compare distributions of categorical variables. In addition, we conducted a more formal multivariate analysis to test the hypothesis that variable means and distributions are jointly similar. For this analysis, we estimated logit regression models where the probability a family is in the program group was regressed on the HSFIS variables, and used chi-squared tests to assess whether the coefficients on these explanatory variables were jointly significant. This multivariate procedure adjusts for the fact that univariate tests are expected to produce some significant test statistics by chance, even when the program and control groups are identical. For example, if the hypothesis tests are conducted at the 10 percent level of significance, then we would expect that 10 percent of independent tests would be falsely rejected. The multivariate procedure also accounts for correlations across measures, whereas the univariate procedure assumes that the measures are independent.

For several reasons, our main approach was to conduct the analysis using the sample pooled across all 17 research sites rather than conduct separate analyses by site. First, pooling increases the power of the statistical tests. Second, it allows us to examine more HSFIS variables, because we cannot accurately examine program and control group differences by site for characteristics that are relatively uncommon. Finally, and most important, we used the same random assignment procedures for each site, so that we had no reason to believe that there would be differences in results across sites. However, we also conducted the analysis separately by site for selected HSFIS variables and display p-values for these tests.

2. Analysis Results

Table D.1A displays analysis results for the sample pooled across the 17 research sites. The table displays variable distributions for the program and control groups, as well as p-values for testing differences across the two groups. Table D.1B displays p-values by site for 12 selected variables.

The results indicate that random assignment produced program and control groups with equivalent characteristics. For the full sample, the program and control group differences are statistically significant at the 10 percent level for only 3 of the 47 univariate tests (which is less than the approximately 5 tests that would be expected by chance), and only 4 of the tests are statistically significant at the 15 percent level. Furthermore, the joint test from the multivariate regression model yields a p-value of .630. Finally, very few (15 of 207) univariate tests for 12 key variables are rejected at the 10 percent level across the sites, and the significant test statistics are scattered across sites and variables. We conclude that random assignment produced equivalent research groups.

TABLE D.1A
COMPARISON OF THE BASELINE CHARACTERISTICS OF ALL PROGRAM AND CONTROL GROUP MEMBERS
Variable	Program Group	Control Group	P-Value for Testing Differences
Site Characteristics
Program Approach			.813
Center-based	20.2	20.6
Home-based	46.7	45.6
Mixed	33.0	33.9
Overall Implementation Level			.957
Full and early	34.5	34.8
Full but late	35.0	35.1
Never	30.5	30.0
Family and Parent Characteristics
Age of Mother at Birth of Focus Child			.803
Younger than 20	39.0	39.5
20 to 25	33.2	32.0
25 or older	27.9	28.5
Mother Was Younger than 19 at First Birth	42.9	41.2	.336
Highest Grade Completed			.175
Less than 12	47.7	47.8
12 or earned a GED	27.3	29.8
More than 12	24.9	22.4
Race and Ethnicity			.968
White non-Hispanic	37.3	37.1
Black non-Hispanic	34.2	35.0
Hispanic	23.8	23.4
Other (Asian or Pacific Islander, American Indian, Eskimo, Aleut)	4.7	4.5
Primary Occupation			.826
Employed	22.9	23.8
In school or a training program	22.0	21.4
Other	55.0	54.7
English Language Ability			.485
Primary language is English	79.9	78.1
Primary language is not English but the applicant speaks English well	9.6	10.3
Primary language is not English and the applicant does not speak English well	10.5	11.6
Living Arrangements			.762
Living with a spouse	24.9	25.4
Living with other adults	38.3	39.1
Living with no other adults	36.8	35.5
Adult Male Present in the Household	38.1	39.1	.586
Number of Adults in the Householda			.804
1	37.8	36.6
2	49.8	50.8
3 or more	12.4	12.6
Number of Children Less than 5 Years Old in the Household Other than the Focus Child			.781
0	64.3	65.1
1	27.0	26.8
2 or more	8.7	8.1
Number of Children Between 6 and 17 in the Household			.454
0	64.3	66.4
1	23.1	21.3
2 or more	12.6	12.3
Number of Moves in the Past Year			.884
0	49.5	49.8
1	28.9	28.1
2 or more	21.6	22.1
Owns Home	11.0	11.1	.907
Household Income as a Percent of the Poverty Level (Percent)			.257
Less than 33	30.2	30.0
33 to 67	32.5	29.2
67 to 99	24.0	26.5
100 or more	13.3	14.3
Welfare Receipt
AFDC/TANFa	35.6	34.7	.627
Food Stamps	48.0	47.8	.889
Medicaid	76.6	74.7	.217
SSI	7.0	7.0	.978
WIC	87.5	85.9	.235
Public housing	9.5	8.9	.565
Has Inadequate Resources
Food	4.9	6.3	.111
Housing	12.3	13.3	.432
Money to buy necessities	20.8	21.7	.588
Medical care	14.0	14.7	.577
Transportation	20.9	22.4	.334
Child care	34.4	34.6	.913
Money for supplies	27.1	29.4	.280
Support from friends	12.9	14.0	.414
Parent information	12.5	16.3	.005*
Maternal Risk Indexc			.469
0 or 1 (low risk)	18.8	17.3
2 or 3 (moderate risk)	54.2	56.4
4 or 5 (high risk)	27.1	26.3
Random Assignment Date			.808
Before 10/96	36.0	36.5
10/96 to 6/97	30.2	30.8
After 6/97	33.8	32.7
Previously Enrolled in Head Start orAnother Childhood Development Programb	12.8	13.4	.628
Characteristics of Focus Child
Age (Months)			.330
Unborn	24.2	26.5
Less than 5	36.1	34.7
5 or more	39.7	38.7
Male	51.7	50.4	.493
First Born	62.3	62.8	.783
Birthweight Less than 2,500 Gramsb	9.9	8.4	.237
Born more than 3 Weeks Earlyb	15.8	12.0	.014*
Stayed in Hospital After Birthb	18.3	16.0	.178
People Concerned About the Child's Overall Health and Developmentb	13.0	13.3	.870
Received an Evaluation Because of Concerns About the Child's Overall Health and Development or Because of Suspected Developmental Delayb	6.0	6.9
Risk Categories
Has established risksb	11.6	10.6	.444
Has biological or medical risksb	18.3	16.8	.396
Has environmental risksb	32.5	36.4	.062*
Covered by Health Insuranceb	90.1	89.6	.723
Sample Size	1,513	1,488
SOURCE: HSFIS application and enrollment forms. aThe primary caregiver is considered to be an adult regardless of her age.(back) bThese variables pertain to families with focus children who were born at baseline. (back) cThis index was constructed by summing the number of the following risk factors that the mother faced: (1) being a teenage mother; (2) having no high school credential; (3) receiving public assistance; (4) not being employed or in school or training, and (5) being a single mother.(back) *Significantly different from zero at the .10 level, two-tailed test.

TABLE D.1B
P-VALUES FROM TESTS COMPARING THE BASELINE CHARACTERISTICS OF PROGRAM AND CONTROL GROUP MEMBERS, BY SITE
Site	Mother's Age	Mother's Education	Race and Ethnicity	Mother's Primary Occupation	Living Arrangements	Received AFDC or TANF	Received Food Stamps	Maternal Risk Index	Random Assignment Date	Child's Age	Child's Gender
1	.446	.903	.211	.976	.459	.820	.707	.809	.970	.576	.027*
2	.165	.482	.252	.948	.472	.700	.734	.820	.615	.400	.227
3	.927	.782	.795	.219	.073*	.107	.041*	.138	.981	.626	.896
4	.748	.496	.434	.722	.662	.682	.401	.131	1.00	.939	.951
5	.550	.158	.190	.559	.694	.361	.808	.840	.845	.464	.308
6	.863	.943	.505	.393	.598	.611	.757	.715	.666	.344	.952
7	.978	.084*	.840	.071*	.052*	.147	.726	.893	.924	.541	.677
8	.824	.355	.683	.499	.773	.115	.858	.879	1.00	.749	.778
9	.970	.217	.579	.533	.401	.326	.791	.286	.985	.306	.362
10	.594	.786	.507	.619	.680	.225	.331	.185	.707	.592	.951
11	.749	.534	.405	.326	.755	.402	.075*	.156	.454	.040*	.215
12	.549	.716	.739	.411	.681	.200	.095*	.083*	.990	.967	.698
13	.003*	.996	.824	.735	.367	.051*	.920	.406	.670	.751	.347
14	.381	.540	.387	.884	.993	.984	.403	.417	.948	.417	.402
15	.744	.880	.395	.343	.766	.776	.934	.469	.924	.911	.453
16	.075*	.622	.622	.464	.492	.142	.887	.244	.791	.242	.867
17	.733	.804	.367	.188	.358	.122	.895	.714	1.00	.457	.496
SOURCE: HSFIS application and enrollment forms. *Statistically different from zero at the .10 level, two-tailed test.

Table of Contents | Previous | Next