ACF OPRE: Early Head Start Research and Evaluation Project (EHSRE)

D.6 ESTIMATING IMPACTS PER ELIGIBLE APPLICANT

In the analyses reported in Volume I, we focus on impacts per applicant for the child and family outcomes, because these impact estimates are more policy relevant and differ very little from the impacts per eligible applicant. Because it is common in randomized clinical trials to base impact conclusions on all eligible applicants for the treatment, we wanted to determine whether impacts based on participants would differ from those based on our analysis of all eligible applicants. Tables D.6A through D.6N show the impacts per eligible applicant for key outcome variables, to illustrate how similar the impact findings are to those based on participants.

TABLE D.6A

IMPACTS ON COGNITIVE AND LANGUAGE DEVELOPMENT AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicant a	Effect Size b
Bayley Mental Development Index (MDI)
MDI Standard Score	91.3	89.9	1.4**	10.6
Percent with Bayley MDI Below 85	28.1	31.8	-3.7*	-8
Peabody Picture Vocabulary Test 3 (PPVT-III)
PPVT-III Standard Score	82.8	81	1.8**	11.1
Percent with PPVT-III Below 85	51.9	57.1	-5.2**	-10.5
Sample Size
Bayley	879	779	1,658
PPVT	738	665	1,403

SOURCE: Parent interview and child assessments conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

bThe effect size was calculated by dividing the estimated impact per applicant by the standard deviation of the outcome measure for the control group times 100 (that is, it is the impact expressed as a percentage of the standard deviation). (back)

*Significantly different from zero at the .10 level, two-tailed test.
**Significantly different from zero at the .05 level, two-tailed test.
***Significantly different from zero at the .01 level, two-tailed test.

TABLE D.6B

IMPACTS ON POSITIVE SOCIAL-EMOTIONAL DEVELOPMENT AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Child Engagement of Parent During Playc	4.8	4.6	0.2***	17.7
Child Sustained Attention to Objects During Playc	5	4.8	0.1***	14
Child Engagement of Parent During Puzzle Challenge Taskd	5	4.9	0.1*	8.3
Child Persistence During Puzzle Challenge Taskd	4.6	4.5	0.1	5.7
Bayley Behavior Rating Scale (BRS): Emotional Regulation in a Cognitive Task (Average Score)e	4	4	0	0.5
BRS: Orientation/Engagement in a Cognitive Task (Average Score)e	3.9	3.8	0	3.4
Sample Size
Parent-Child Interactions	875	784	1,659
Bayley BRS	936	833	1,769

SOURCE: Child assessments, interviewer observations, and assessments of semistructured parent-child interactions conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

cBehaviors were observed during the videotaped parent-child semistructured play task and coded on a seven-point scale.(back)

dBehaviors were observed during the videotaped parent-child puzzle challenge task and coded on a seven-point scale.(back)

eBehaviors were observed during the Bayley assessment and rated on a five-point scale by the interviewer/assessor.(back)

TABLE D.6C

IMPACTS ON NEGATIVE SOCIAL-EMOTIONAL DEVELOPMENT AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Child Negativity Toward Parent During Playc	1.2	1.3	-0.1**	-12.2
Child Frustration with Challenge Taskd	2.7	2.7	0	2.9
ASEBA: Aggressive Behavior Scale (Average Score)	10.7	11.3	-0.6**	-8.7
Sample Size
Parent Interview	1,107	1,003	2,110
Parent-Child Interactions	875	784	1,659

SOURCE: Parent interviews and assessments of semistructured parent-child interactions conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members. (back)

cBehaviors were observed during the videotaped parent-child semistructured play task and coded on a seven-point scale.(back)

dBehaviors were observed during the videotaped parent-child puzzle challenge task and coded on a seven-point scale. (back)

TABLE D.6D

IMPACTS ON EMOTIONALLY SUPPORTIVE PARENTING AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Home Observation for Measurement of the Environment (HOME): Warmthc	2.6	2.5	0.1*	8.2
Parent-Child Structured Play: Supportivenessd	4	3.9	0.1***	12.7
Parent-Child Puzzle Task: Supportive Presencee	4.5	4.4	0.1	4.4
Sample Size
Parent Interview	1,107	1,003	2,110
Parent-Child Interactions	874	784	1,658

SOURCE: Parent interviews and assessments of semistructured parent-child interactions conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

cBehaviors were observed during the HOME assessment and rated on a yes/no scale by the interviewer/assessor.(back)

dBehaviors were observed during the videotaped parent-child semi-structured play task and coded on a seven-point scale. Supportiveness is a combination of Warm Sensitivity and Positive Regard.(back)

eBehaviors were observed during the videotaped parent-child puzzle challenge task and coded on a seven-point scale.(back)

TABLE D.6E

IMPACTS ON THE HOME ENVIRONMENT AND STIMULATION
OF LANGUAGE AND LEARNING AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Home Observation for Measurement of the Environment (HOME) - Total Score	27.4	27	0.5**	9.9
Structuring the Child's Day
Percentage of Parents Who Set a Regular Bedtime for Child	59.6	58.2	1.3	2.7
Percentage of Parents and Children Who Have Regular Bedtime Routines	69.3	68.6	0.8	1.7
Parent-Child Activities and Learning Support
HOME: Support of Language and Learning	10.6	10.4	0.2**	8.8
Parent-Child Activities	4.4	4.3	0.1*	7.1
Parent-Child Puzzle Task: Quality of Assistancec	3.6	3.5	0.1*	8.4
Percentage of Parents Who Read to Child Every Day	56.5	52.2	4.3**	8.7
Percentage of Parents Who Regularly Read to Child at Bedtime	32.2	29.2	3	6.6
Internal Home Environment
HOME: Internal Physical Environment	7.8	7.8	0	0.5
Sample Size
Parent Interview	1,107	1,003	2,110
Parent-Child Interactions	874	784	1,658

SOURCE: Parent interviews, interviewer observations, and assessments of semistructured parent-child interactions conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

cBehaviors were observed during the videotaped parent-child puzzle challenge task and coded on a seven-point scale.(back)

TABLE D.6F

IMPACTS ON NEGATIVE PARENTING BEHAVIOR
IN STRUCTURED PLAY AND INTERACTION AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Insensitivity
Parent-Child Structured Play: Detachmentc	1.2	1.3	-0.1*	-8.6
Parent-Child Structured Play: Intrusivenessc	1.6	1.6	0	-5.2
Parent-Child Puzzle Task: Detachmentd	1.6	1.6	0	-0.5
Parent-Child Puzzle Task: Intrusivenessd	2.7	2.7	-0.1	-5.3
Hostility and Punishment
Parent-Child Structured Play: Negative Regardc	1.3	1.3	0	-1.5
Home Observation for Measurement of the Environment (HOME): Harshnesse	0.3	0.3	0	1.5
Percentage of Parents Who Spanked the Child in the Previous Week	47.4	53.6	-6.3***	-12.6
Sample Size
Parent Interview	1,107	1,003	2,110
Parent-Child Interactions	874	784	1,658

SOURCE: Parent interviews, interviewer observations, and assessments of semistructured parent-child interactions conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

cBehaviors were observed during the videotaped parent-child semi-structured play task and coded on a seven-point scale.(back)

dBehaviors were observed during the videotaped parent-child puzzle challenge task and coded on a seven-point scale.(back)

eBehaviors were observed during the HOME assessment and rated on a yes/no scale by the interviewer/assessor.(back)

TABLE D.6G

IMPACTS ON PARENTING KNOWLEDGE: SAFETY AND DISCIPLINE STRATEGIES AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Safety Practices
Always Uses Car Seat for Child	70.3	70.7	-0.4	-0.8
Discipline Strategies
Percentage of Parents Who Suggested Responses to Hypothetical Situations with Child:
Prevent or distract	69.9	68.8	1.2	2.5
Remove child or object	80.8	81.2	-0.4	-1.2
Talk and explain	70.5	69.2	1.3	2.7
Time out	27.2	27.2	0	0
Threaten or command	10.5	13.4	-2.9**	-8.5
Shout	8.5	8.3	0.1	0.4
Physical punishment	46.9	51.2	-4.3**	-8.6
Percentage of Parents Suggesting Only Mild Responses to the Hypothetical Situationsc	44.3	40.5	3.8*	7.8
Index of Severity of Discipline Strategies Suggestedd	3.4	3.5	-0.2***	-10
Sample Size	1,107	1,003	2,110

SOURCE: Parent interviews conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

cParents were classified as suggesting only mild discipline if their responses to the three discipline situations include only the following: prevent or distract, remove child or object, talk and explain, time out, restrain child, ignore child, warn or remind, or bribe child.(back)

dThe Index of Severity of Discipline Strategies is based on a hierarchy of discipline practices, from talk and explain, remove child or object, time out, or prevent/distract (1) through physical punishment (5). The most severe approach suggested is used to code this scale.(back)

TABLE D.6H

IMPACTS ON PARENT HEALTH, MENTAL HEALTH, AND FAMILY FUNCTIONING AT AGE 3
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Parent's Physical Health
Parent's Health Status	3.4	3.5	0	-4.2
Parent's Mental Health
Parental Distress	24.8	25.5	-0.7	-7.1
Parent-Child Dysfunctional Interaction	17.8	17.7	0	0.4
CES-Depression Scale (CES-D; short form)	7.4	7.7	-0.3	-3.6
CES-D: Severe Depressive Symptoms	14.3	14.9	-0.6	-1.6
Family Functioning
Family Environment Scale-Family Conflict (Average Score)	1.7	1.7	0	-4.3
Sample Size	1,107	1,003	2,110

SOURCE: Parent interviews conducted when children were approximately 36 months old.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

TABLE D.6I

IMPACTS ON SELF-SUFFICIENCY ACTIVITIES 28 MONTHS AFTER RANDOM ASSIGNMENT
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Any Self-Sufficiency Activities
Percentage of Parents Ever Employed or in an Education or Job Training Program in First 26 Months	93.3	90.2	3.1**	10.2
Average Hours per Week Employed at All Jobs and in Any Education orTraining in First 26 Months	22	20.6	1.4**	8.7
Employment Activities
Percentage of Parents Ever Employed in First 26 Months	86.1	83.1	3.0*	8.1
Average Hours per Week Employed at All Jobs in First 26 Months	16.9	16.8	0.1	0.9
Education Activities
Percentage of Parents Who Ever Participated in an Education or Training Program in First 26 Months	59.3	51.5	7.8***	15.6
Average Hours per Week in an Education Program During First 26 Months	4.5	3.4	1.1***	16.7
Sample Size	1,139	1,097	2,236

SOURCE: Parent services follow-up interviews conducted an average of 7, 16, and 28 months after random assignment.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

TABLE D.6J

IMPACTS ON EDUCATION ACTIVITIES AND CREDENTIALS
28 MONTHS AFTER RANDOM ASSIGNMENT
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Types of Education Activities
High School	13.5	9.4	4.1***	14.3
High School or Alternative	14.3	10.8	3.5***	11.6
Adult Basic Education	4.4	3.7	0.8	4.2
English as a Second Language	3.3	2.4	0.9	6.5
GED Preparation	10.1	8.8	1.2	4.4
Any Vocational Education	19.7	17.3	2.4	6.5
Two-Year College	10.9	10.2	0.7	2.4
Four-Year College	5.9	5.7	0.3	1.1
Degrees and Credentials Received
Highest Grade Completed at Second Followup	11.6	11.6	-0.1	-3
GED Certificate	10.6	11.5	-1	-3
High School Diploma	49.2	48.4	0.8	1.6
Vocational, Business, or Secretarial Diploma	16.9	16.9	0	0.1
Associate's Degree	3.5	4.5	-1	-5.1
Bachelor's Degree	4.6	5.4	-1.4	-6.3
Sample Size	1,139	1,097	2,236

SOURCE: Parent services follow-up interviews completed an average of 7, 16, and 28 months after random assignment.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

*Significantly different from zero at the .10 level, two-tailed test
**Significantly different from zero at the .05 level, two-tailed test.
***Significantly different from zero at the .01 level, two-tailed test.

TABLE D.6K

IMPACTS ON WELFARE PROGRAM PARTICIPATION 28 MONTHS AFTER RANDOM ASSIGNMENT
Outcome	Program Group	Control Group	Estimated Impact per Applicanta	Effect Sizeb
Welfare Program Participation
Percentage of Parents Who Received Any Welfare Benefits During First 26 Months	68.4	66.8	1.6	3.4
Total Welfare Benefits Received During First 26 Months	$5,411	$5,607	($196)	-2.6
Percentage of Parents Who Received AFDC or TANF Benefits During First 26 Months	47.1	45.1	2	4
Total AFDC or TANF Benefits Received During First 26 Months	$2,171	$2,196	($25)	-0.6
Average Total Food Stamp Benefits Received During First 26 Months	$2,141	$2,099	$42	1.5
Sample Size	1,139	1,097	2,236

SOURCE: Parent services follow-up interviews conducted an average of 7, 16, and 28 months after random assignment.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

AFDC = Aid to Families with Dependent Children; TANF = Temporary Assistance for Needy Families.

TABLE D.6L

IMPACTS ON FAMILY INCOME AND RESOURCES 28 MONTHS AFTER RANDOM ASSIGNMENT
Outcome	Program Group	Control Group	Estimated Impact Per Applicanta	Effect Sizeb
Percentage of Families with Income Above the Poverty Line at Third Followup	41.7	42.5	-0.7	-1.5
Total Family Resources Scale
First Followup	149.6	148.5	1	4.9
Second Followup	152.8	151.5	1.3	6.8
Third Followup	154.5	153.5	1	5
Sample Size	1,139	1,097	2,236

SOURCE: Parent services follow-up interviews completed an average of 7, 16, and 28 months after random assignment.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

TABLE D.6M

IMPACTS ON CHILD AND FAMILY OUTCOMES AT AGE 3 OR 28 MONTHS AFTER RANDOM ASSIGNMENT,
BY PROGRAM APPROACH IN FALL 1997
Outcome	Program Group	Control Group	Impact Estimate Per Applicanta	Effect Sizeb	Program Group	Control Group	Impact Estimate Per Applicanta	Effect Sizeb	Program Group	Control Group	Impact Estimate Per Applicanta	Effect Sizeb
	Center-Based Programs				Home-Based Programs				Mixed Approach Programs
Child Cognitive and Language Development
Average Bayley Mental Development Index (MDI)	88.8	88.1	0.6	4.8	93.9	92.7	1.1	8.6	89.7	88.3	1.4	10.6
Percentage with MDI < 85***	32.7	39.4	-6.8	-14.5	21.2	22.3	-1.2	-2.5	34.2	36.5	-2.3	-5
PPVT-III Standard Score	81.5	80.6	0.9	5.7	84.2	82.9	1.2	7.6	82.6	79.2	3.5**	21.2
Percentage with PPVT-III<85***	57.3	58.4	-1.1	-2.3	45.8	48.4	-2.5	-5.1	53.8	64.9	-11.1**	-22.3
Child Social-Emotional Development
Child Behavior Checklist- Aggressive	10	10.8	-0.8	-12.1	11.2	11.7	-0.5	-7.1	10.7	11.2	-0.5	-8.5
Parent-Child Structured Play: Child Sustained Attention with Objects (Average)*	4.9	4.9	0	-1.1	5	4.9	0.1	9.8	5	4.7	0.3***	28.7
Parent-Child Structured Play: Child Negativity Toward Parent (Average)	1.2	1.4	-0.1**	-21.9	1.3	1.3	0	-6.2	1.3	1.3	-0.1	-14.4
Parent-Child Structured Play: Child Engagement (Average)	4.8	4.7	0.1	9.4	4.8	4.6	0.2**	18.5	4.8	4.5	0.3***	27.8
Puzzle Challenge Task: Child Engagement (Average)	5	4.9	0.1	5.6	5.1	5	0.1	6	5	4.9	0.1	13.3
Puzzle Challenge Task: Persistence	4.4	4.4	0.1	5.4	4.7	4.6	0.1	11.8	4.5	4.5	0	3.3
Puzzle Challenge Task: Frustration	2.5	2.7	-0.2	-12.6	2.7	2.6	0.1	6.3	2.8	2.7	0.1	9.8
Parenting
Home Observation for Measurement of the Environment (HOME) Total Score	26.8	26.1	0.8*	16.2	28.2	28.1	0.2	3.1	26.9	26.4	0.5	10.8
Parent-Child Structured Play: Parent Supportiveness	4	4	0	3.3	4	3.9	0.1**	14.5	4	3.8	0.2**	19.7
Puzzle Challenge Task: Parent Supportive Presence	4.4	4.5	-0.1	-4.8	4.6	4.5	0.1	7.4	3.7	3.4	0.3**	23.7
Puzzle Challenge Task: Quality of Assistance	3.5	3.5	0	1.7	3.6	3.5	0	2	2.7	2.9	-0.1	-10.5
Parent Reads to Child Daily***	34.5	49.1	5.4	10.7	54	55	-1	-1.9	62.6	49.5	13.1***	26.2
Parent-Child Structured Play: Parent Detachment**	1.2	1.2	0.1	14.9	1.2	1.3	-0.1	-8.9	1.2	1.3	-0.1**	-22.8
Puzzle Challenge Task: Parent Detachment	1.6	1.6	0	3.3	1.6	1.6	0	-3.4	1.6	1.8	-0.2	-15.8
Spanked Child Last Week***	51.7	59.3	-7.6	-15.2	44.5	49.7	-5.2	-10.5	45.9	56.2	-10.3**	-20.6
Parenting Stress Index (PSI) Parental Distress	24	24.7	-0.7	-7.8	25	26.2	-1.3**	-13.3	24.6	25.6	-1	-10.8
Self-Sufficiency
Ever in Education or Training**	63.2	60.7	2.5	5	64.5	51.8	12.7***	25.4	52.7	45.6	7.1**	14.3
Average Hours per Week in Education or Training	5.1	4.7	0.4	6.1	4.1	3.2	0.8*	13.1	4.4	3	1.4***	22.1
Ever Employed***	90.9	87.9	3.1	8.2	87.9	81.9	6.0**	15.9	82.3	81	1.3	3.5
Average Hours/Week Employed	21.3	20.7	0.6	3.9	16.9	15.9	1.1	7.2	14.7	15	-0.2	-1.7
Subsequent Birth by 24 Months after Random Assignment***	15.7	20.1	-4.4	-9.8	24.8	30.4	-5.6	-12.5	26.6	28.4	-1.8	-4.1
Sample Size
Parent Interview	254	216	470		350	343	693		502	448	950
Parent-Child Interactions	228	181	409		252	255	507		396	348	744

SOURCE: Parent interview and child assessments conducted when children were approximately 36 months old. Self-sufficiency information from parent service interviews completed an average of 28 months after random assignment.

NOTE: All impact estimates were calculated using regression models, where each site was weighted equally.

aThe estimated impact per eligible applicant is measured as the difference between the regression-adjusted means for all program and control group members.(back)

*Significantly different from zero at the .10 level, two-tailed [or one-tailed] test. **Significantly different from zero at the .05 level, two-tailed [or one-tailed] test. ***Significantly different from zero at the .01 l