Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

CHAPTER 8 - REPORT OF INVESTIGATIONS OF THE COMPARABILITY OF GROUPS PRIOR TO OUTCOME ANALYSES

INTRODUCTION

An understanding of the characteristics and dynamics of the sample being followed within the National Transition Demonstration Study is essential to interpreting subsequent analyses. A series of investigations was undertaken to: (1) identify initial differences for which adjustments should be made; (2) determine whether or not the two cohorts could reasonably be combined for analysis without adjustment for cohort membership; (3) determine the comparability of the group of participants who were available for assessment at the end of the study versus those who were irretrievably lost to follow-up at some time during the study period; and (4) review patterns of mobility among study participants. These analyses are all closely related to one another and are fundamental to subsequent analyses of program impact.

METHODS

Data were available from a total of 7,515 former Head Start families. The participating families were enrolled in two cohorts, one entering kindergarten in the fall of 1992 and the second entering kindergarten in the fall of 1993. No other family or child characteristics were employed to determine cohort membership. Families were enrolled in the treatment (demonstration) or control (comparison) condition based on the school within which the child enrolled in kindergarten. Schools had been previously randomized to demonstration or comparison conditions based on randomization plans devised and employed within the 31 local sites. Table 17 summarizes resulting sample sizes for the two cohorts and the two treatment groups at baseline.

Families were followed longitudinally for four consecutive years, as the child progressed through kindergarten and the first three grades of elementary school. Assessments were completed in the fall and spring of kindergarten and in the spring of each year after. Because recruitment into the study occurred throughout the kindergarten year, as children and families were identified as having been Head Start participants prior to entering kindergarten, the two kindergarten assessments were collapsed into a single “baseline” period. Thus, a family was considered enrolled in the study if data were available for either or both of the two kindergarten assessments.4 Families were considered available at the end of the study (not lost to follow-up) if data were obtained for that family in either second or third grade or both.5 Key variables considered in the comparability analyses included: (1) characteristics of the child, such as (a) gender (male, female); (b) child has a health condition that interferes with school attendance (yes, no); (c) receptive language ability, as measured by the Peabody Picture Vocabulary Test (continuous); and (d) social skills, as rated by the primary caregiver using the Social Skills Rating System (standard score; continuous); (2) characteristics of the primary caregiver, such as (e) full-time employment (yes, no); (f) high school education (yes, no); (g) ethnicity (white/non-Hispanic, African American, Hispanic/Latino, other); (h) immigrant status (not born in US; yes, no); (i) chronic health condition that interferes with ability to care for the child (yes, no); (j) depression (positive screen score; yes, no); and (3) characteristics of the family, such as (k) mother present in the home (yes, no); (l) father or father figure (male stepparent or grandparent) present in home (yes, no); (m) family mobility (0, 1-2, or 2 or more moves within the past year); (n) receipt of AFDC (yes, no); (o) receipt of SSI (yes, no); (p) use of a language other than English in the home (yes, no); and (q) household income (presented as percentage of poverty; continuous). These 17 variables were chosen for inclusion in comparability analyses because they were either (1) outcome variables for primary analyses or (2) thought to be potentially related to differential program participation and/or outcomes. The follow-up status variable was computed as a binary variable for each family -- data exists/does not exist -- reflecting the existence of at least some portion of the child and/or family data at either second or third grade, or both.

Table 17.
Sample sizes by cohort and treatment condition

	Demonstration (Treatment)	Comparison (Control)	Total
Cohort 1	1889	1651	3540
Cohort 2	2039	1936	3975
Total	3928	3587	7515

COMPARABILITY OF COHORTS AT BASELINE

To the extent that the two cohorts are similar at baseline, they can be reasonably joined into a single sample without adjustment for cohort membership and resulting loss of degrees of freedom within analyses. There was no reason to believe that children entering school in one year would be systematically different from children entering school in the following year.

Results. A review of the analysesa indicates that the two cohorts were not substantially different on these key characteristics (see Table 18). They were statistically different on five of the 17 variables -- two child characteristics (receptive language and social skills), two caregiver characteristics (immigrant status and chronic health condition), and one family characteristic (family speaks a language other than English in the home). However, the difference between the two groups on, for example, receptive language and social skills was a single point -- a trivial difference that achieved statistical significance because of the large sample size.

COMPARABILITY OF TREATMENT GROUPS AT BASELINE

Background. Typically, the investigation of the comparability of the treatment groups at baseline is undertaken to validate the random assignment procedures employed. In the case of the National Transition Demonstration Study, the random assignment occurred at the level of the school or school district, not at the level of the child or family. Nonetheless, the baseline comparability analysis is considered essential to the analytical process. To the extent that the two groups of demonstration and comparison children and/or families were equivalent at baseline on key outcome and other characteristics, any differences found at the end of the study are more likely to be the result of program intervention, rather than undetected baseline differences.

Table 18.
Comparability of cohorts at baseline

	Cohort 1	Cohort 2	Test Statistic	P-value
Child Characteristics
Gender
Male	52.70%	51.30%	?2=1.333	0.248
Female	47.30%	48.70%
Health impairment	2.00%	2.00%	?2=0.001	0.977
Receptive language (PPVT Rasch-Wright score; mean, s.d.)			T=5.6931	0.0001
Score	79.9	78.6
Mean	(9.5)	(10.0)
Social skills (Social Skills Rating System; mean, s.d.)			T=-2.6174	0.009
Score	89.9	90.9
Mean	(15.2)	(15.4)
Caregiver characteristics
Ethnicity
White	46.40%	44.30%	?2=0.165	?2=0.684
African American	30.0%	32.00%
Hispanic/Latino	13.40%	14.90%
Other	10.10%	8.80%
High school graduate	67.20%	67.80%	?2=0.268	0.604
Born outside the U.S.	15.30%	18.50%	?2=12.381	0.001
Depressed	37.30%	37.70%	?2=0.083	0.774
Chronic health condition	4.20%	3.00%	?2=8.553	0.003
Employed full-time	31.90%	32.50%	?2=0.291	0.589
Family characteristics
Mother present in home	93.00%	93.80%	?2=1.824	0.177
Father/ father figure present in home	45.50%	43.40%	?2=3.051	0.081
Mobility in year prior to kindergarten
No moves in past year	74.30%	75.50%
1 move in past year	17.60%	17.50%
2 or more moves in last year	8.20%	7.00%
Receipt of AFDC income	37.30%	38.30%	?2=0.742	0.389
Receipt of SSI income	11.90%	12.60%	?2=0.658	0.417
Language other than English spoken in the home	12.80%	15.60%	?2=11.107	0.001
Relative income (Percentage of poverty (mean, s.d.)	0.79	0.82	T=-1.884	0.059
	(0.63)	(0.62)

 

The results of these analysesb indicated that the two treatment groups were not different as they entered kindergarten and the study. The demonstration and comparison groups differed statistically on only two characteristics: children’s social skills and caregiver full-time employment. In both of these instances, the difference between the demonstration and comparison group was very small (social skills: 89.9 vs. 91.1. full time employment 30% vs. 34%) and not practically significant.

Table 19.
Comparability of treatment groups at baseline

Demonstration	Comparison	Test	Statistic	P-value
Child Characteristics
Gender
Male	52.90%	51.20%	c²=2.017	0.156
Female	47.10%	48.80%
Health impairment	2.00%	2.00%	c²=0.054	0.817
Receptive language (PPVT Rasch-Wright score; mean, s.d.)	79.3 (9.5)	79.5 (9.9)	t=-0.9179	0.3507
Social skills (Social Skills Rating System; mean, s.d.)	89.9 (15.4)	91.1 (15.3)	t=-3.0645	0.0022
Caregiver Characteristics
Ethnicity
White	44.80%	45.80%	c²=1.937	0.509
African American	30.90%	31.20%
Hispanic/Latino	14.40%	14.10%
Other	9.90%	8.90%
High school graduate	66.10%	67.20%	c²=0.947	0.33
Born outside the U.S.	16.30%	17.80%	c²=2.705	0.1
Depressed	38.40%	36.60%	c²=1.790	0.181
Chronic heath condition	3.60%	3.50%	c²=0.110	0.74
Employed full-time	30.30%	33.90%	c²=10.49	0.001
Family Characteristics
Mother present in home	93.80%	93.10%	c²=1.288	0.256
Father/father figure present in home	54.40%	56.10%	c²=2.155	0.142
Mobility in year prior to Kindergarten
No moves in past year	75.3	75.8	c²=0.865	0.352
1 move in past year	17	17.4
2 or moves in past year	7.7	6.8
Receipt of AFDC income	39.1	36.3	c²=5.835	0.016
Receipt of SSI income	11.7	13	c²=2.206	0.137
Language other than English spoken spoken in home	13.8	14.8	c²=1.472	0.225
Relative income (Percentage of poverty (mean, s.d.)	0.79 (0.64)	0.82 (0.61)	t=-1.8307	0.0672

COMPARABILITY OF FOLLOW-UP GROUPS AT BASELINE

Background.

Loss to follow-up, also referred to as attrition, has implications for all research endeavors, but it may be particularly problematic within longitudinal studies, when specific participants must be located and interviewed on repeated occasions (Manheim & Rich, 1991). The threat that dropouts pose to the validity of research efforts is frequently referred to and appears to be accepted in the literature, particularly in the area of prevention research (Cook & Campbell, 1979; Jurs & Glass, 1971; Hansen, Collins, Malotte, Johnson, & Fielding, 1985; Farrington, Gallagher, Morley, St. Ledger, & West, 1990). Excessively high dropout rates decrease sample size and power available within a study, limiting the study’s ability to demonstrate a statistically significant difference between treatment groups (Fuller, 1990). More importantly, however, loss of subjects poses potential threats to a study’s internal and external validity. If high risk or less successful participants drop out of the treatment condition at a disproportionately high rate, findings may exaggerate a difference between groups, suggesting that the intervention is effective when, in fact, it is not (Cook & Campbell, 1979; Jurs & Glass, 1971; Farrington, Gallagher, Morley, St. Ledger, & West, 1990; Meinert, 1986). In contrast, high risk subjects dropping out of control groups or more successful participants dropping out of treatment groups may serve to mask program effects that are actually present (Cook & Campbell, 1979; Jurs & Glass, 1971; Farrington, Gallagher, Morley, St. Ledger, & West, 1990; Meinert, 1986).

The results of our investigation of attrition within the National Transition Demonstration Study indicate that 87% of the families enrolled in the study were available for interview at the end of the study (i.e., data collected in either second or third grade or both; see Table 20). The rate of attrition did not differ by cohort or by treatment condition.

Table 20.
Percentage of families remaining in study

	Present at End of Study	Not Present at End of Study
Cohort 1	3073 (86.8%)	467 (13.2%)
Cohort 2	3475 (87.4%)	500 (12.6%)
Demonstration	3411 (86.8%)	517 (13.2%)
Comparison	3137 (87.5%)	450 (12.5%)

 

We also compared the baseline characteristics of the group of families who were available for assessment at the end of the study with the characteristics of the group of families who were not available for interview in either second or third gradec. The two groups did not differ substantially on the majority of the key characteristics (see Table 21). Not surprisingly, the two groups did differ to a statistically significant degree in mobility. Families who remained available for interview at the end of the study period were less likely to have moved within the year prior to kindergarten than were families who had moved at least once prior to entering kindergarten.

DISCUSSION

The comparability analyses revealed few important differences between treatment groups, cohorts, or follow-up groups. Some differences between treatment groups at baseline would be reasonably expected, since treatment groups will be identical only on those characteristics controlled in randomization (Meinert, 1986); because the unit of randomization employed for this study was the school, it would not be expected that the two treatment groups would be identical on variables related to individual child, caregiver, or family characteristics. The two groups, demonstration and comparison, differ only in terms of two characteristics: Specifically, families with a caregiver who has a chronic health condition and those that are mobile (moved 1 or more times in the last year) are significantly more likely to not be included in the follow-up study.

The finding that the two cohorts are essentially equivalent at baseline in the national sample suggests that the two groups can be collapsed into a single cohort without prejudice.

Table 21.
Results of univariate analyses (logistic regression) predicting loss to follow-up
This table shows the likelihood (odds ratios) that a particular child, caregiver, or family characteristic is significantly associated with loss or dropout in the study.

	Odds Ratio	95% Confidence interval	p-value
Child Characteristics
Gender
Male	1.0	0.90-1.20	0.5852
Female
Health impairment	1.1	0.7-1.8	0.7331
Receptive language (PPVT Rasch-Wright score; mean, s.d.)	1.0	0.99-1.01	0.2733
Social skills (Social Skills Rating System; mean, s.d.)	1.0	0.99-1.01	0.4243
Caregiver Characteristics
Ethnicity
White	1.0	reference group	-
African American	1.0	0.85-1.18	0.9873
Hispanic/Latino	0.9	0.7-1.18	0.5885
Other	0.8	0.62-1.07	0.1449
High school graduate	1.1	0.91-1.25	0.4498
Born outside the U.S.	0.9	0.73-1.09	0.2665
Caregiver Depressed	0.9	0.80-1.12	0.5333
*Caregiver has chronic health condition that interferes with ability to care for child	0.6	0.37-0.96	0.0333
Caregiver Employed full-time	0.9	0.78-1.07	0.2563
Family Characteristics
Father/father figure present in home	1	0.67-1.60	0.8739
Mother present in home	1.1	0.82-1.49	0.5339
Family Mobility	1	reference group	-
No moves in past year	1.4	1.16-1.66	0.0004
*1 move in past year	1.4	1.11-1.85	0.0059
*2 or moves in past year	1	0.88-1.18	0.847
Family receives AFDC	0.9	0.75-1.17	0.5682
Family receives SSI	1.1	0.86-1.31	0.578
Family speaks language other than English at home	13.8	14.8	c²=1.472

*significant associations with drop-out

 

Taken together, the results reported above indicate that the findings from the National Transition Demonstration Study are not likely to be compromised by either the extent of attrition from the study or by differential attrition from treatment groups or other subgroups of participating families. The attrition rate of 13 percent experienced within this study is well within the expected range for a longitudinal investigation involving young families in poverty. The average attrition rate in major surveys of American populations with follow-up periods of between 4 and 10 years has been reported to be 47 percent (Farrington, Gallagher, Morley, St. Ledger, & West, 1990). The attrition rate for this 5-year longitudinal investigation is well below that average rate.

PROGRAM PARTICIPATION, CROSS-OVER, AND DROPOUT PATTERNS

It was the original intent of the program planners that children and families participating in the demonstration group of the National Transition Demonstration Project would receive four consecutive years of continuous, comprehensive, Head Start-like services, beginning when the children entered kindergarten and continuing through the first four years of their elementary school experience. Similarly, children and families enrolled in the comparison group would, by design, receive no specific Head Start-like services for a similar period of time. Rather than randomly assigning children and families to treatment groups, the National Transition Demonstration Project randomly assigned schools to the treatment (demonstration) or control (comparison) condition, and program services were offered to classrooms, families, and children within the demonstration schools. Thus, children and families received program services when they were enrolled in demonstration schools and did not receive such services when they were enrolled in comparison schools or in schools not participating in the project at all.

To the extent that children and families stayed in the same demonstration school or transferred to another demonstration school, it was possible for them to participate in the program for a full four years. However, given the natural mobility of families with young children, and especially young families in poverty, it was reasonable to expect that a number of demonstration families would receive less than the full four years of treatment, and that some families who began the study in the comparison condition might receive some benefit from some program services because the child transferred to and enrolled in a demonstration school for some period of time. It was also expected that some families would withdraw from the program by virtue of moving to a school that was not participating in the National Transition Demonstration Project at all (either as a demonstration or as a comparison school).

To the extent that children and families received the full, intended “dose” of the program, differential outcomes might be reasonably ascribed to program participation. If, however, a substantial number of comparison families in fact received as many years of program services as did demonstration families, or a substantial number of demonstration families received only a limited quantity of program services because of later enrollment in comparison schools, then differences between groups might be limited or obscured.

For each of the 7,515 former Head Start families participating in the National Transition Demonstration Project, a participation “score” was created. If the child was enrolled in a demonstration school at the time of the data collection, then one year of participation credit was given. If the child was enrolled in a comparison school or in a school outside the study, then no credit was given. Thus, the range of possible participation scores was zero to four years.

The key assumption in the calculation of the participation score is, of course, that the child was enrolled in the same school for the entire year. The data available within the National Core Data Set mandates this assumption. Reliable data are available consistently across all sites to identify the school in which the child was enrolled at the time of the data collection. Reliable and consistent data are not available for all sites for all years, however, to identify more precisely the number of different schools attended by the child during a year and/or the number of months enrolled in demonstration and/or comparison schools. Thus, the less precise assignment of years of credit is a requirement of the data.

The key concepts of program participation, crossover, and dropout from the program were defined as follows:

Program participation is defined as the number of years enrolled in a demonstration school (calculated as described above). The underlying assumption is that program services were available to child and family as long as the child was enrolled in the school. Services could be received actively or passively. For example, many of the services offered through the educational component of the program (developmentally appropriate educational curriculum and instructional practices) would be received by all children enrolled in a participating classroom, while others (e.g., tutoring services, adult educational programs) would be more specifically targeted and could be accepted or refused by participating families. Similarly, social support services and many parent involvement activities were typically offered to all families within participating classrooms but could be actively accepted or refused, based on family need and preference. Thus, the program participation score can be thought of as an indication of exposure to the program but should not be thought of as a measure of quantity, intensity, or quality of program participation at the family level.

Crossover is defined as the movement of a family from one treatment condition (either demonstration or comparison) to the other, always remaining in either a demonstration or a comparison school. Those families who moved to a school outside the project (to a school that was not participating as either a demonstration or comparison school) or who moved outside the catchment area and could not be located for interview or assessment for a period of time were defined as dropouts from the study. Study dropout, as defined here, overlaps with but is not synonymous with loss to follow-up within the National Transition Demonstration Study. Loss to follow-up is defined as those children and families for whom there is no available data for either the third or the fourth year in school. Families who are lost to follow-up are a portion of the group who dropped out of the program. That group of dropout families also includes, however, families who moved to a non-study school for one or more years but remained available for interview and families who moved outside the catchment area for one or more years but returned and/or were available for interview or assessment in either the third or the fourth year of school. The key feature of dropout families is that they moved to a school outside the study for some period of time, limiting their possible exposure to Transition or Transition-like services. Unless otherwise specified, analyses reported in Chapters 9 through 12 utilized all available data, that is, children and/or families were included regardless of their cross-over or dropout status. Those families and children who were lost to follow-up (i.e., had no data in either of the last 2 years of the study) were excluded from analysis.

PROGRAM PARTICIPATION

[D]

 

Approximately 60 percent of the families who began in the demonstration group participated in the program for a full four years, and some 87 percent of the comparison families did not participate in a demonstration school for any of the four years (see Figure 8.1). Further, it can be seen that only 6 percent of the comparison families participated in a demonstration school for two or more years. There is, however, more variability among demonstration families. Some 13 percent of the demonstration families participated in the program for a single year, and another 12 percent for two years. Thus, fully a quarter of the families who began the program in the demonstration group participated in demonstration school for no more than half of the years.

CROSSOVER PATTERNS

[D]

 

Figure 8.2 depicts crossover patterns for each treatment group. The patterns are surprisingly similar for the two groups. Overall, less than 70 percent of families remained in the same treatment condition throughout their tenure in the program.

DROPOUT PATTERNS

Figure 8.3 summarizes the movement patterns in a different manner, looking at those participants who may have dropped out for a period of one to two years but returned. Overall, approximately 80 percent of the families remained in the study for at least three to four consecutive years. Of those families, the majority (65-68%) remained in the same treatment condition throughout, while 10% changed conditions and 23% moved to an out of study school for some period of time.

[D]

SUMMARY FINDINGS

These investigations have supported a number of important assumptions that underlie the analyses that follow, including:

1. The children and families in the two treatment groups -- demonstration and comparison -- were not substantively different on a number of key characteristics as they entered the National Transition Demonstration Study. This allows us to conclude with greater confidence that any differences seen between the two groups at the end of the study are not likely to be the effect of initial differences.
2. As expected, the children and families who entered the study as Cohort 1 were not significantly different from those who entered the study as Cohort 2. This allows us to combine the two groups into a single large analysis sample, giving us greater power to detect subtle differences between groups.
3. A total of 87 percent of the children and families who began the National Transition Demonstration Study remained available for interview in at least one of the final two years. This follow-up rate is well above the average for longitudinal studies of similar duration.
4. The group of children and families who were lost to follow-up during the course of the study were not different from the group who remained in the study, at least on key characteristics that were measured when they entered the study. This finding is especially important in terms of the validity of the study and our ability to have confidence in its findings.
5. Approximately 60 percent of the families who began in the demonstration group remained in a demonstration school for the full four years, receiving the full “dose” of Transition services. Fully a quarter of the families who began the program in the demonstration group participated in the demonstration school for no more than half of the years. This suggests that the exposure of children and families to Transition demonstration services may have been somewhat diluted or limited.

---

4The collapsed baseline data included the earliest kindergarten value available for a child or family. Thus, the fall score/value was taken whenever available; the spring score/value was taken only when a fall value was not available. Analyses reported in this chapter were completed using the earliest-value collapsed baseline value. (back)

5The decision was made based on the belief that exposure to the demonstration program for 3 years should be sufficient for benefit to occur. (back)

aChi-square analyses were used when variables were categorical (nominal) (child gender, health impairment, ethnicity, caregiver education, immigrant status, caregiver depression, chronic health condition of caregiver, employment, mother present in home, father figure present in home, family mobility, AFDC receipt, SSI receipt, language) and t-tests were used when variables were continuous (receptive language score, social skills score, distance from poverty cut-off). Analyses were completed using the former Head Start analysis sample (n = 7,515) with casewise deletion because of missing data. For a discussion of missing data (extent and patterns), the reader is referred to Technical Report # 7. (back)

bAs above. (back)

cAnalyses were completed using the former Head Start analysis set (n = 7,515) and univariate logistic regression models. Loss to follow-up (defined as family enrolled at baseline but not present in either second or third grade) was included as the dependent variable; a single independent predictor variable was included in each model. Associations were not adjusted for treatment condition. The PROC LOGISTIC procedure in SAS was used for all analyses. (back)

 

Table of Contents | Previous | Next