Skip Navigation
acfbanner  
ACF
Department of Health and Human Services 		  
		  Administration for Children and Families
          
ACF Home   |   Services   |   Working with ACF   |   Policy/Planning   |   About ACF   |   ACF News   |   HHS Home

  Questions?  |  Privacy  |  Site Index  |  Contact Us  |  Download Reader™Download Reader  |  Print Print      

Office of Planning, Research & Evaluation (OPRE) skip to primary page content
Advanced
Search

Table of Contents | Previous | Next

Attachment A

Exhibit A1

Difference between Intervention Groups and Control on LAP-D, OMLIT and Arnett Scores at Baseline (Fall 2003)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
LAP-D Cognitive Total 30.43 (4.16) 30.86 (4.19) 0.43 0.10 0.53
Language Total 28.84 (4.34) 29.51 (4.26) 0.80 0.16 0.30
Fine Motor Total 38.88 (4.97) 39.68 (4.60) 0.68 0.16 0.34
OMLIT Support for Oral Language 53.26 (9.71) 53.82 (9.87) 0.57 0.06 0.72
Support for Print Knowledge 53.27 (2.89) 53.38 (2.83) 0.11 0.01 0.81
Literacy Resources 50.14 (5.62) 50.69 (4.64) 0.55 0.05 0.50
Arnett Positive Affect 51.20 (8.81) 48.76 (9.31) -2.44 -0.24 0.10
Not Punitive 47.83 (6.59) 46.49 (7.26) -1.34 -0.13 0.25
Engaged 49.78 (11.75) 46.64 (13.8) -3.14 -0.31 0.15
Sample size (centers/classrooms) 55 110      

Exhibit A2

Baseline (Fall, 2003) OMLIT Score, by Treatment Group
Measure Control Mean (SD) Treatment 1 RSL Mean (SD) Treatment 2 BELL Mean (SD) Treatment 3 (BTL) Mean (SD)
Support for Oral Language 50.14 (5.62) 50.70 (4.97) 49.81 (4.62) 51.54 (4.26)
Support for Print Knowledge 53.27 (5.62) 53.44 (4.97) 52.89 (4.62) 53.81 (4.26)
Support for Print Motivation 54.41 (9.03) 51.65 (7.24) 53.67 (8.05) 54.20 (6.48)
Literacy Resources 50.14 (5.62) 50.70 (4.97) 49.81 (4.62) 51.54 (4.26)
Sample Size (centers/classrooms) n=54 n=38 n=36 n=36

Exhibit A3

Baseline (Fall, 2003) Scores on Three LAP-D Subtests, by Treatment Group
Subtest Control Mean (SD) Treatment 1 RSL Mean (SD) Treatment 2 BELL Mean (SD) Treatment 3 (BTL) Mean (SD)
Cognitive 30.43 (4.16) 31.56 (4.52) 31.33 (3.95) 29.71 (3.91)
Fine Motor 38.88 (4.97) 39.98 (4.58) 40.33 (4.10) 38.76 (5.03)
Language 28.84 (4.34) 29.76 (4.38) 30.43 (4.17) 28.41 (4.09)
Sample Size (centers/classrooms) n=53 n=36 n=33 n=35
Sample Size (children) 580 350 319 350

Exhibit A4

Baseline (Fall, 2003) Characteristics of Teachers and Classrooms, by Treatment Group
Measure Control % RSL % BELL % BTL %
Spanish-language preference 49.1 47.2 48.5 45.7
Sample Size (teachers) n=53 n=36 n=33 n=35
Chi-square test of independence, df = 3, p = 0.99
Education High school only 21.6 30.6 30.3 37.1
Some college 13.7 13.9 15.2 11.4
AA or BA 64.7 55.6 54.6 51.4
Sample Size (teachers) n=51 n=36 n=33 n=35

Chi-square test of independence, df = 6, p = 0.84

2 Teachers had missing data.

Percent of classrooms with all English-speaking children 36 32 33 44
Percent of classrooms with all Spanish-speaking children 46 47 58 42
Percent of classrooms with mixture of language 18 21 9 14
Sample Size (centers/classrooms) n=53 n=36 n=33 n=35

Exhibit A5 summarizes results from models for Year 2004 OMLIT construct outcomes, where all three treatment-groups combined were contrasted to the control group. The data were analyzed in two-level hierarchical linear models where classrooms (level-1) were nested in randomization blocks (level-2). The models included a random intercept term for blocks. Treatment impacts (any of the three treatment groups contrasted to control) were estimated in models that controlled for year 2003 baseline OMLIT construct measures27, and year 2003 baseline value of the Arnett “positive, punitive, detached” construct. The models were specified as shown below.

Level-1 Model:

Y(2004)jk = β0k + β1k(Trtjk) + β2k(Y(2003)jk) + β3k(Arnett(2003)jk) + rjk

Level-2 Model:

β0k = γ00 + uk

β1k = γ10

β2k = γ20

β3k = γ30

rjk ˜ N(0, σ2)

uk ˜ N(0, ζ00)

where

Y(2004)jk = OMLIT construct from year 2004 observation of classroom j nested in block k.
Y(2003)jk = OMLIT construct from year 2003 observation of classroom j nested in block k.
    (This term was omitted from models for phonological awareness and literacy activities because those measures were not available from the 2003 classroom observational data.)
Trtjk = 1 if classroom j nested in block k was in Treatment Groups 1, 2, or 3;
  = 0 if control group.
Arnett(2003)jk = Arnett “positive, punitive, detached” construct from year 2003 observation of classroom j nested in block k

The parameter estimate y10 from the model above is the estimated treatment effect. The value of y10 is entered Exhibit A5 in the column labeled "Mean Difference T-C". In Exhibit A5, the values shown in the column labeled "Control Mean (SD)" were calculated as the simple mean and standard deviation of the OMLIT construct values of the n=54 classes in the control group. The value of the mean shown in the column labeled "Treatment Mean (SD)" was calculated as the sum of the treatment effect, y10, and the control group mean. The standard deviation shown in the column was calculated as the standard deviation of OMLIT construct values of the n=107 classes in the combined group of the three treatment groups. The effect size was calculated by dividing the treatment effect, y10, by the Year 2004 control group standard deviation. The p-value corresponds to a two-sided test of the null hypothesis that the treatment effect is equal to zero.

Exhibit A5

Overall Impact of the Interventions on Teacher Behavior and the Classroom Environment (OMLIT, Spring 2004)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 50 (10) 55.86 (8.60) 5.86 .59 .000
Support for Phonological Awareness 50 (10) 52.12 (9.11) 2.12 .21 .181
Support for Print Knowledge 50 (10) 55.30 (10.40) 5.30 .53 .002
Support for Print Motivation 50 (10) 55.84 (9.10) 5.84 .58 .000
Literacy Resources 50 (10) 50.85 (9.54) .85 .08 .586
Literacy Activities 50 (10) 53.90 (9.76) 3.90 .39 .018
Sample Size (centers/classrooms) n=54 n=106      
The effect sizes are standardized measures of the magnitude (size) of treatment effects. The standardization makes possible the comparison of the sizes of treatment effects, between different outcome measures. For example, if the effect sizes of a treatment on outcome measures A and B are 0.50, and 0.25, respectively, then the size of the treatment impact on A is twice the size of the impact on B. For each outcome measure, the effect size is equal to the estimated treatment impact, divided by the control group standard deviation.

Exhibit A6 summarizes results from models for Year 2004 OMLIT construct outcomes, where each of the treatment groups was contrasted to the control group. The data were analyzed in analyzed in two-level hierarchical linear models where classrooms (level-1) were nested in randomization blocks (level-2). The models included a random intercept term for blocks. Impacts of each of three treatments contrasted to control were estimated in models that controlled for year 2003 baseline OMLIT construct measures28, and year 2003 baseline value of the Arnett “positive, punitive, detached” construct. The models were specified as shown below.

Level-1 Model:

Y(2004)jk = β0k + β1k(Trtjk)β2k(Trt2jk) + β3k(Trt3jk) + β4k(Y(2003)jk) + β5k(Arnett(2003)jk) + rjk

Level-2 Model:

β0k = γ00 + uk

β1k = γ10

β2k = γ20

β3k = γ30

β4k = γ40

β5k = γ50

rjk ˜ N(0, σ2)

uk ˜ N(0, ζ00)

where

Y(2004)jk = OMLIT construct from year 2004 observation of classroom j nested in block k.
Y(2003)jk = OMLIT construct from year 2003 observation of classroom j nested in block k.
    (This term was omitted from models for phonological awareness and literacy activities because those measures were not available from the 2003 classroom observational data.)
Trt1jk = 1 if classroom j nested in block k was in Treatment Groups 1; = 0 else.
Trt2jk = 1 if classroom j nested in block k was in Treatment Groups 2; = 0 else.
Trt3jk = 1 if classroom j nested in block k was in Treatment Groups 3; = 0 else.
Arnett(2003)jk = Arnett “positive, punitive, detached” construct from year 2003 observation of classroom j nested in block k

The parameter estimates y10, y20, y30 from the model above are the estimated impacts of Treatments 1, 2, and 3, as contrasted to control, respectively. The values y10, y20, y30 are entered Exhibit A6 in the column labeled "Mean Difference T-C". In Exhibit A6, the values shown in the column labeled "Control Mean (SD)" were calculated as the simple mean and standard deviation of the OMLIT construct values of the n=54 classes in the control group. The values of the mean shown in the columns labeled "Treatment Mean (SD)" were calculated as the sum of each of the three treatment effects, y10, y20, y30, and the control group mean. The standard deviation shown in those columns were calculated as the standard deviation of OMLIT construct values of each of the three treatment groups. The effect sizes were calculated by dividing the treatment effects, y10, y20, y30, by the Year 2004 control group standard deviation. The p-values correspond to two-sided tests of the null hypothesis that the treatment effects are equal to zero.

Exhibit A6

Differential Impact of the Three Interventions on Teacher Behavior and the Classroom Environment (OMLIT, Spring 2004)
Treatment 1 (Ready, Set, Leap)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 50 (10) 57.15 (7.70) 7.15 .72 .000
Support for Phonological Awareness 50 (10) 53.23 (8.48) 3.23 .32 ns
Support for Print Knowledge 50 (10) 59.03 (8.10) 9.03 .90 .000
Support for Print Motivation 50 (10) 57.16 (9.24) 7.16 .72 .000
Literary Resources 50 (10) 52.13 (9.38) 2.13 .21 .279
Literary Activities 50 (10) 55.77 (9.37) 5.77 .58 .005
Sample Size (centers/classrooms) n=54 n=36      
Treatment 2 (B.E.L.L.)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 50 (10) 53.00 (10.30) 3.00 .30 .134
Support for Phonological Awareness 50 (10) 52.21 (8.68) 2.21 .22 .289
Support for Print Knowledge 50 (10) 48.66 (9.30) -1.34 -0.13 .517
Support for Print Motivation 50 (10) 54.45 (8.25) 4.45 .45 .0343
Literary Resources 50 (10) 51.17 (7.81) 1.17 .12 .567
Literary Activities 50 (10) 50.13 (10.20) .13 .01 .952
Sample Size (centers/classrooms) n=54 n=34      
Treatment 3 (Breakthrough to Literacy)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 50 (10) 57.12 (7.07) 7.12 .71 .000
Support for Phonological Awareness 50 (10) 50.86 (10.16) 0.86 .09 .72
Support for Print Knowledge 50 (10) 57.43 (10.50) 7.43 .74 .000
Support for Print Motivation 50 (10) 55.74 (9.75) 5.74 .57 .005
Literary Resources 50 (10) 49.13 (11.13) -0.87 -0.09 .666
Literary Activities 50 (10) 55.29 (8.84) 5.29 .53 .010
Sample Size (centers/classrooms) n=54 n=36      

The results corresponding to classes with Spanish-dominant teachers, shown in the top panel of Exhibit A7 were obtained by fitting the model described for Exhibit A5, to the subset of data consisting of classes with Spanish-dominant teachers. Similarly, the results for English-dominant teachers were obtained by fitting the model described in Exhibit A5 to the subset of classes with English-dominant teachers. Effect sizes were calculated by dividing the estimated treatment effect by the full sample Year 2004 control group standard deviation.

Exhibit A7

Overall Impact of the Interventions on Teacher Behavior and the Classroom Environment for Spanish-dominant Teachers vs. English-dominant Teachers (OMLIT, Spring 2004)
Spanish-dominant Teachers
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 48.46 (9.82) 55.47 (8.08) 7.01 .71 .001
Support for Phonological Awareness 47.79 (7.33) 52.59 (8.51) 4.80 .48 .015
Support for Print Knowledge 49.16 (9.87) 57.13 (9.98) 7.97 .80 .000
Support for Print Motivation 47.87 (10.48) 56.03 (8.37) 8.16 .81 .001
Literary Resources 50.04 (8.73) 52.34 (8.56) 2.30 .23 .2570
Literary Activities          
Sample Size (centers/classrooms) n=26 n=49      
English-dominant Teachers
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 51.43 (10.13) 56.01 (9.09) 4.59 .46 .045
Support for Phonological Awareness 52.05 (11.72) 51.73 (9.60) -0.33 -0.03 .089
Support for Print Knowledge 50.78 (10.23) 53.92 (10.71) 3.13 .31 .219
Support for Print Motivation 51.98 (9.29) 55.75 (9.72) 3.78 .38 .101
Literary Resources 49.97 (11.21) 49.65 (10.06) -0.31 -0.03 .894
Literary Activities (centers/classrooms) n=28 n=58      

Exhibit A8 summarizes results from models for Year 2005 OMLIT construct outcomes, where all three treatment groups combined were contrasted to the control group. The analytic model corresponding to Exhibit A8 is the same model as previously described from Exhibit A5. The only difference is that Exhibit A8 results are from models that were fit to the data from 2005. Therefore, the outcome variable is

Y(2005)jk = OMLIT construct from year 2005 observation of classroom j nested in block k.

and all other model terms are as specified previously. The effect size is calculated as the impact divided by the Year 2004 control group standard deviation.

Exhibit A8

Overall Impact of the Interventions on Teacher Behavior and the Classroom Environment (OMLIT, Spring 2005)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 49.81 (10.38) 55.92 (9.36) 6.11 .61 .000
Support for Phonological Awareness 48.16 (7.73) 53.05 (9.90) 4.89 .49 .001
Support for Print Knowledge 48.41 (11.18) 55.83 (9.99) 7.42 .74 .000
Support for Print Motivation 50.90 (10.85) 55.19 (9.48) 4.29 .43 .012
Literary Resources 48.91 (9.04) 51.69 (8.40) 2.77 .28 .045
Literary Activities 47.38 (11.37) 55.38 (8.50) 8.00 .80 .000
Sample Size (centers/classrooms) n=53 n=104      

Exhibit A9 summarizes results from models for Year 2005 OMLIT construct outcomes, where each of the treatment groups was contrasted to the control group. The analytic model corresponding to Exhibit A9 is the same model as previously described from Exhibit A6. The only difference is that Exhibit A9 results are from models that were fit to the data from 2005. Therefore, the outcome variable is

Y(2005)jk = OMLIT construct from year 2005 observation of classroom j nested in block k.

and all other model terms are as specified previously. The effect size is calculated as the impact divided by the Year 2004 control group standard deviation.

Exhibit A9

Differential Impact of the Three Interventions on Teacher Behavior and the Classroom Environment (OMLIT, Spring 2005)
Treatment 1 (Ready, Set, Leap)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 49.81 (10.38) 56.13 (8.95) 6.32 .63 .003
Support for Phonological Awareness 48.16 (7.73) 52.99 (11.20) 4.83 .48 .013
Support for Print Knowledge 48.41 (11.18) 57.88 (8.94) 9.46 .95 .000
Support for Print Motivation 50.90 (10.85) 56.99 (10.95) 6.09 .61 .005
Literary Resources 48.91 (9.04) 51.62 (8.80) 2.71 .27 .116
Literary Activities 47.38 (11.37) 57.12 (8.01) 9.75 .98 .000
Sample Size (centers/classrooms) n=53 n=36      
Treatment 2 (B.E.L.L)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 49.81 (10.38) 54.14 (9.97) 4.33 .43 .047
Support for Phonological Awareness 48.16 (7.73) 53.99 (9.68) 5.83 .58 .004
Support for Print Knowledge 48.41 (11.18) 51.75 (7.32) 3.34 .33 .131
Support for Print Motivation 50.90 (10.85) 53.55 (9.84) 2.65 .27 .236
Literary Resources 48.91 (9.04) 53.98 (7.97) 5.07 .51 .005
Literary Activities 47.38 (11.37) 52.40 (8.31) 5.02 .50 .000
Sample Size (centers/classrooms) n=53 n=33      
Treatment 3 (Breakthrough to Literacy)
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 49.81 (10.38) 57.39 (9.08) 7.58 .76 .001
Support for Phonological Awareness 48.16 (7.73) 52.26 (8.82) 4.11 .41 .036
Support for Print Knowledge 48.41 (11.18) 57.52 (11.81) 9.11 .91 .000
Support for Print Motivation 50.90 (10.85) 54.84 (7.24) 3.94 .39 .072
Literary Resources 48.91 (9.04) 49.62 (8.26) .71 .07 .685
Literary Activities 47.38 (11.37) 56.24 (8.57) 8.86 .89 .000
Sample Size (centers/classrooms) n=53 n=35      

The results corresponding to classes with Spanish-dominant teachers, shown in the top panel of Exhibit A10 were obtained by fitting the model described for Exhibit A8, to the subset of data consisting of classes with Spanish-dominant teachers. Similarly, the results for English-dominant teachers were obtained by fitting the model described in Exhibit A8 to the subset of classes with English-dominant teachers. Effect sizes were calculated by dividing the estimated treatment effect by the full sample Year 2004 control group standard deviation.

Exhibit A10

Impact of the Interventions on Teacher Behavior and the Classroom Environment by Language of Teacher (OMLIT, Spring 2005)
Spanish-dominant Teachers
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 49.26 (10.08) 55.59 (9.77) 6.33 .63 .009
Support for Phonological Awareness 48.07 (7.33) 52.40 (9.28) 4.33 .43 .041
Support for Print Knowledge 46.99 (10.73) 55.98 (10.23) 8.99 .90 .001
Support for Print Motivation 48.30 (9.99) 54.21 (9.84) 5.91 .59 .014
Literary Resources 49.34 (8.87) 52.79 (7.85) 3.45 .34 .075
Literary Activities 45.75 (10.33) 53.75 (8.10) 8.00 .80 .000
Sample Size (centers/classrooms) n=26 n=49      
English-dominant Teachers
Construct Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-value
Support for Oral Language 50.34 (10.83) 55.85 (9.06) 5.51 .55 .023
Support for Phonological Awareness 48.25 (8.24) 53.46 (10.48) 5.22 .52 .018
Support for Print Knowledge 49.78 (11.63) 55.18 (9.84) 5.40 .54 .032
Support for Print Motivation 53.41 (11.24) 55.64 (9.16) 2.23 .22 .358
Literary Resources 48.50 (9.36) 50.84 (8.71) 2.34 .23 .253
Literary Activities 48.94 (12.27) 56.61 (8.72) 7.67 .77 .002
Literary Activities (centers/classrooms) n=27 n=55      

Exhibit A11 summarizes results from models for Year 2005 child-level outcomes (TOPEL scores), where all three treatment-groups combined were contrasted to the control group. The data were analyzed in three-level hierarchical linear models where students (level-1) were nested in classrooms (level-2), and classes were nested in randomization blocks (level-3). The models included a random intercept terms for classes and blocks. Treatment impacts (any of the three treatment groups contrasted to control) were estimated in models that controlled for child’s age, sex, and language spoken at home, and for classroom-level mean LapD Cognitive Total scores obtained from measurements taken in the fall of 2003 or the fall of 2004. The models were specified as shown below.

Level-1 Model:

Y(2005)ijk = π0jk + π1jk(Ageijk) + π2jk(SexMaleijk) + π3jk(HomeLang1ijk) + π4jk(HomeLang2ijk) + eijk

Level-2 Model:

π0jk = β00k + β01k(Trtjk) + β02k(MeanLapD_CTjk) + rjk

π1jk = β10k

π2jk = β20k

π3jk = β30k

π4jk = β40k

Level-2 Model:

β00k = γ000 + uk

β01k = γ010

β02k = γ020

β10k = γ100

β20k = γ200

β30k = γ300

β40k = γ400

eijk ˜ N(0,ø2)

rjk ˜ N(0,σ2)

uk ˜ N(0,ζ00)

where

Y(2005)ijk = TOPEL outcome measure from spring 2005 for student i, nested in classroom j nested in block k.
Ageijk = Age at time of testing of student i, nested in classroom j nested in block k.
SexMaleijk = 1 if student i, nested in classroom j nested in block k is male;
0 if female
HomeLang1ijk = 1 if home language of student i, nested in classroom j nested in block k is English only;
0 if HomeLang2=1 or if home language is a mix of English and Spanish, a mix of English and some other language, or if some other language is the primary language in the home
HomeLang2ijk = 1 if home language of student i, nested in classroom j nested in block k is Spanish only or a mix of English and Spanish;
0 if HomeLang1=1 or if home language is a mix of English and Spanish, a mix of English and some other language, or if some other language is the primary language in the home
Trtjk = 1 if classroom j nested in block k was in Treatment Groups 1, 2, or 3;
  = 0 if control group.
MeanLapD_CTjk = Class-level mean LapD Cognitive Total Score of class j nested in block k, calculated from tests administered in the fall of 2003 and fall of 2004.

The parameter estimate y010 from the model above is the estimated treatment effect. The value of y010 is entered Exhibit A11 in the column labeled "Mean Difference T-C". In Exhibit A11, the values shown in the column labeled "Control Mean (SD)" were calculated as the simple mean and standard deviation of the TOPEL outcome measure values of the children in the control group. The value of the mean shown in the column labeled "Treatment Mean (SD)" was calculated as the sum of the treatment effect, y010, and the control group mean. The standard deviation shown in the column was calculated as the standard deviation of TOPEL outcome measure values of the children in the combined group of the three treatment groups. The effect size was calculated by dividing the treatment effect, y010, by the Year 2005 control group standard deviation. The p-value corresponds to a two-sided test of the null hypothesis that the treatment effect is equal to zero.

Exhibit A11

Overall Impact of the Interventions on Child Outcomes (TOPEL, Spring 2005)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 78.79 (16.43) 82.43 (17.77) 3.64 0.22 0.017
Phonological Awareness 88.74 (16.19) 93.28 (15.95) 4.54 0.28 0.003
Print Knowledge 95.89 (15.31) 102.82 (14.66) 6.93 0.45 0.000
Early Literacy Index 84.93 (16.32) 91.12 (16.70) 6.19 0.38 0.000
Sample Size (children) n=509 n=1014      

Exhibit A12 summarizes results from models for Year 2005 child-level outcomes (TOPEL scores), where each of the treatment groups were contrasted to the control group. The data were analyzed in same model as specified for Exhibit A11, except that three dummy variables representing the contrasts of each of the three treatment groups to the control group were entered in the level-2 model instead of the single treatment dummy (representing the contrast of all three treatments combined to control) that was utilized for the Exhibit A11 models.

Other than the differences shown below, all other model terms were identical to those used in the model for Exhibit A11:

Level-2 Model:

π0jk = β00k + β01k(Trt1jk) + β01k(Trt2jk) + β01k(Trt3jk) + β02k(MeanLapD_CTjk) + rjk

where

Trt1jk = 1 if classroom j nested in block k was in Treatment Group 1; = 0 else.

Trt2jk = 1 if classroom j nested in block k was in Treatment Group 2; = 0 else.

Trt3jk = 1 if classroom j nested in block k was in Treatment Group 3; = 0 else.

Exhibit A12

Differential Impact of the Three Interventions on Child Outcomes (TOPEL, Spring 2005)
Treatment 1 (Ready, Set, Leap)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 78.79 (16.43) 83.40 (18.11) 4.61 0.28 0.017
Phonological Awareness 88.74 (16.19) 94.36 (16.13) 5.62 0.35 0.003
Print Knowledge 95.89 (15.31) 105.83 (13.03) 9.94 0.65 0.000
Early Literacy Index 84.93 (16.32) 93.20 (15.77) 8.27 0.51 0.000
Sample Size (children) n=509 n=320      
Treatment 2 (B.E.L.L.)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 78.79 (16.43) 79.88 (17.87) 1.09 0.07 0.577
Phonological Awareness 88.74 (16.19) 89.31 (15.53) 0.57 0.04 0.767
Print Knowledge 95.89 (15.31) 97.01 (15.45) 1.11 0.07 0.565
Early Literacy Index 84.93 (16.32) 85.93 (16.93) 0.99 0.06 0.637
Sample Size (children) n=509 n=340      
Treatment 3 (Breakthrough to Literacy)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 78.79 (16.43) 83.83 (16.90) 5.04 0.31 0.009
Phonological Awareness 88.74 (16.19) 95.82 (15.63) 7.08 0.44 0.000
Print Knowledge 95.89 (15.31) 105.13 (13.63) 9.24 0.60 0.000
Early Literacy Index 84.93 (16.32) 93.81 (16.12) 8.88 0.54 0.000
Sample Size (children) n=509 n=354      

Exhibit A13 summarizes results from models for Year 2005 child-level outcomes (TOPEL scores), where treatment-groups 1 and 3 combined were contrasted to the control group. The results shown in Exhibit A13 are from the same model as specified for Exhibit A11, except that Treatment Group 2 data are omitted from the analysis. Thus, the treatment dummy becomes a contrast of the combined effect of Treatments 1 and 3, contrasted to control.

Other than the differences shown below, all other model terms were identical to those used in the model for Exhibit A11:

Level-2 Model:

π0jk = β00k + β01k(Trtjk) + β01k(Trt2jk) + β02k(MeanLapD_CTjk) + rjk

where,

Trtjk = 1 if classroom j nested in block k was in Treatment Groups 1 or 3;
  = 0 if control group.

Exhibit A13

Impact of Treatments 1 and 3 Combined on Child Outcomes (TOPEL, Spring 2005)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 78.79 (16.43) 83.72 (17.49) 4.93 0.30 0.001
Phonological Awareness 88.74 (16.19) 95.09 (15.86) 6.35 0.39 0.000
Print Knowledge 95.89 (15.31) 105.51 (13.38) 9.62 0.63 0.000
Early Literacy Index 84.93 (16.32) 93.59 (15.94) 8.66 0.53 0.000
Sample Size (children) n=509 n=674      

The results shown in the top panel of Exhibit A14 are from the same model as specified for Exhibit A13, except that only data from classes with Spanish-dominant teachers were included (data from classes with English-dominant teachers were omitted). Similarly, the results in the bottom panel of Exhibit A14 are from the same model as specified for Exhibit A13, except that only data from classes with English-dominant teachers were included (data from classes with Spanish-dominant teachers were omitted). Effect sizes were calculated by dividing the estimated treatment effect by the full sample Year 2005 control group standard deviation.

Exhibit A14

Impact of Treatments 1 and 3 Combined on Child Outcomes by Language of Teacher (TOPEL, Spring 2005)
Spanish-dominant Teachers
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 73.52 (17.13) 79.88 (18.64) 6.36 0.39 0.007
Phonological Awareness 84.64 (16.35) 93.54 (16.09) 8.90 0.55 0.000
Print Knowledge 92.69 (15.23) 105.79 (13.68) 13.10 0.86 0.000
Early Literacy Index 79.46 (16.81) 91.20 (16.64) 11.75 0.72 0.000
Sample Size (children) n=281 n=332      
English-dominant Teachers
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 83.83 (14.01) 87.52 (15.14) 3.69 0.22 0.069
Phonological Awareness 93.47 (14.67) 97.16 (15.23) 3.69 0.23 0.086
Print Knowledge 99.84 (14.49) 106.13 (12.99) 6.30 0.41 0.001
Early Literacy Index 90.16 (13.99) 95.99 (14.54) 5.84 0.36 0.010
Sample Size (children) n=228 n=342      

The results shown in Exhibit A15 are from the same model as specified for Exhibit A13, except that data from children from homes where English is the only language spoken are excluded. Effect sizes were calculated by dividing the estimated treatment effect by the full sample Year 2005 control group standard deviation.

Exhibit A15

Impact of Treatments 1 and 3 Combined on Child Outcomes for Children with Spanish or Haitian Creole as Their Home Language (TOPEL, Spring 2005)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 76.37 (16.69) 81.49 (17.90) 5.12 0.31 0.004
Phonological Awareness 87.94 (16.62) 94.56 (16.20) 6.62 0.41 0.000
Print Knowledge 95.09 (15.43) 105.57 (13.32) 10.48 0.68 0.000
Early Literacy Index 83.33 (16.82) 92.62 (16.26) 9.29 0.57 0.000
Sample Size (children) n=404 n=525      

The results shown in Exhibit A16 are also from the same model as specified for Exhibit A13, except that only data from children from homes where English is the only language spoken are included.

Exhibit A16

Impact of Treatments 1 and 3 Combined on Child Outcomes for Children with English as Their Home Language (TOPEL, Spring 2005)
Subtest Control Mean (SD) Treatment Mean (SD) Mean Difference T-C Effect Size P-Value
Definitional Vocabulary 86.97 (12.45) 91.52 (12.86) 4.55 0.28 0.034
Phonological Awareness 91.69 (14.20) 96.78 (14.47) 5.08 0.31 0.023
Print Knowledge 98.97 (14.47) 104.11 (13.66) 5.14 0.34 0.023
Early Literacy Index 90.37 (13.17) 96.20 (14.17) 5.83 0.36 0.015
Sample Size (children) n=105 n=149      

The results summarized in Exhibit A17 were obtained from the following two-level HLM model. The results in all three panels of the exhibit utilized the same model specification, but the results in the middle panel were obtained when the model was fit to data only from classes with Spanish-dominant teachers, and the results in the bottom panel correspond to the subset of data from classes with English-dominant teachers.

The two-level random intercept HLM models were of the form:

Level 1

Yij = β0j + β1 (TeacherBA) + rij

Level 2

β0j = γ00 + u0j

where Yij is a 2003 OMLIT measures on the ith class nested in the jth block, the β0j are random intercept terms for the j blocks, and TeacherBA is coded as 1 if the teacher has a bachelor degree or higher and zero otherwise.

In the Exhibit, the column labeled "Estimate of Effect" shows the parameter estimate β1, the column labeled "standard error of effect" gives the standard error of β1. The column labeled "effect size" shows the estimate, β1, divided the Year 2004 control group standard deviation of the measure, i.e., 10. The p-value is a two-sided test of the null hypothesis that β1 = 0.

Exhibit A17

Relationship of Teacher Education to Teacher Behavior and the Classroom Environment Overall and by Language of Teacher (OMLIT, Arnett, Fall 2003)
Construct Estimate of Effect Standard Error of Effect Effect Size P-value
Full Sample Support for Oral Language 0.01 1.67 .00 .997
Support for Print Knowledge 1.05 0.48 .10 .031
Support for Print Motivation -1.25 1.32 .12 .345
Literacy Resources 1.04 0.83 .10 .213
Arnett: Positive Affect 2.99 1.51 .30 .049
Arnett: Not Punitive -0.03 1.18 -.00 .981
Arnett: Engaged 3.44 2.20 .30 .121
English-Dominant Teachers Support for Oral Language 0.23 2.72 .02 .932
Support for Print Knowledge 1.02 0.77 .01 .193
Support for Print Motivation -2.17 2.26 .02 .342
Literacy Resources -0.19 1.40 .02 .893
Arnett: Positive Affect 4.27 2.32 .04 .070
Arnett: Not Punitive -1.38 1.77 .01 .438
Arnett: Engaged 2.01 3.43 .02 .560
Spanish-Dominant Teachers Support for Oral Language 0.46 2.32 .00 .841
Support for Print Knowledge 1.61 0.64 .02 .015
Support for Print Motivation 0.15 1.74 .02 .937
Literacy Resources 2.68 1.07 .03 .015
Arnett: Positive Affect 1.45 2.22 .01 .515
Arnett: Not Punitive 1.78 1.82 .02 .331
Arnett: Engaged 4.34 3.36 .04 .202
Note: Overall sample of 157 includes 82 English-dominant teachers and 75 Spanish-dominant teachers.



27 This term was omitted from models for phonological awareness and literacy activities because those measures were not available from the 2003 classroom observational data. (back to footnote 27)