Findings

A basic assumption underlying the design of the study, including the strategy for collecting and analyzing data was that child outcomes are mediated by the actions and behavior of their teachers. Therefore, significant impacts on teacher behavior and the literacy environment would be necessary precursors of improved child outcomes. Below, we present, first, the initial and later findings about the impact of the interventions on teachers and classrooms, and then findings about the impact on children’s language and emergent literacy skills. Finally, we discuss the impact of the interventions on the pattern of activities in the classroom, and the effect of teachers’ educational background on teacher and child outcomes.

Impact of the Interventions on Teacher Behavior and the Literacy Environment

We examined the effect of the interventions on teacher behavior and interactions with children through four constructs, representing support for the four building blocks of emergent literacy: a) support for oral language, b) support for phonological awareness; c) support for print knowledge; and d) support for print motivation. Each of the constructs is built from a range of observational variables, drawn from the OMLIT battery of measures.

Support for oral language incorporates the amount of read-aloud activities as well as measures of their quality in terms of: the use of open-ended questions; information about text concepts; introduction of new vocabulary; linking story elements to children’s own experiences; post-reading discussions; and the amount of teacher-child language interaction. Support for phonological awareness is a measure of the ways teachers draw children’s attention to the sounds of words through singing and rhymes, and help them blend one-syllable words into different two-syllable words (blending) and, conversely, break apart two-syllable words into their single-syllable component words (elision). Support for print knowledge incorporates the amount of time spent in teaching letters and the correspondence between letters and sounds and in helping children with writing, and extent to which the teacher encourages children to integrate print into other activities including daily routines. Support for print motivation measures the strategies teachers use to motivate children to want to read.

In addition to these teacher-focused constructs, two additional constructs were used to assess the impact of the interventions on the classroom environment: literary resources measures the amount of environmental print and text materials present in the classroom, as well as the extent to which literacy resources are integrated into various activity centers; literacy activities is a measure of all the classroom activities that incorporate literacy.

As Exhibit 2 shows, after less than six months’ implementation of the curricula, there were significant impacts on teachers’ support for oral language, print knowledge and print motivation, and on the number of activities that incorporated literacy. Teachers in treatment group classrooms were providing more opportunities for oral language development and learning about print, and they were engaging in more of the activities that foster children’s desire to read and use print. At this point, there were no significant effects on the classrooms’ literacy resources (probably because all classrooms in the study received a comprehensive package of materials to support literacy activities at the beginning of the study), or on teachers’ support for phonological awareness. Two of the three interventions delayed training on this element until spring 2004, to ensure that the other elements were in place. By Spring 2005, there were significant positive impacts on all six constructs. Teachers in the treatment group learned about and conducted many more activities to promote phonological awareness, such as singing, playing rhyming games, reading poems.

While all three interventions had significant effects on aspects of teacher behavior and the classroom environment, Exhibit 3 suggests that the three curricula had different strengths and weaknesses. Treatments 1 and 3, which had larger impacts on some aspects of teacher behavior and on the number of literacy activities, showed no significant effects on the literacy resources in the classrooms. Treatment 1, which significantly increased support for print motivation, is the only one of the three that used authentic children’s literature (trade books) rather than controlled-language books. Treatment 2, which had slightly weaker effects on most aspects of teacher behavior, had strong effects on teacher support for phonological awareness and on literacy resources. This intervention introduced the concepts of blending and elision at the initial training and continued to emphasize them. In addition, the curriculum stressed building thematic connections into the classrooms’ activity centers, increasing the richness of the print environment.

Taken together, the interventions had somewhat different effects on Spanish-dominant vs. English-dominant teachers (Exhibit 4). As we saw earlier, almost half of the teachers in the study classroom expressed their preference for training in Spanish. Initially, the presence of such a large group of Spanish-language teachers, including some who were monolingual in Spanish, was worrying for the curriculum developers. Although all three had the ability to train and provide on-going support in Spanish, and provided literacy materials in Spanish as well as English, all three curricula were intended to enhance children’s English language development, and they were concerned that their training might not be as effective for teachers whose first language was not English. These worries proved unfounded. Exhibit 4 shows that the effects on Spanish-dominant teachers were as strong as and, in some cases, stronger than the effects on English-dominant teachers.

Impact of the Interventions on Child Outcomes

The effects of the interventions on children’s language development and emergent literacy skills were assessed at the end of the four-year-old year, for children who had been in the classrooms between two and ten months. The average number of children enrolled in the four-year-old classrooms in 2004-2005 ranged from 16 to 20. The percentage of children assessed in Spring 2005 ranged from 50% to 55% of the enrollment. Children’s language and literacy skills were assessed using three subtests from the TOPEL:

• Definitional Vocabulary: This is a test of vocabulary in which the child is asked to identify a pictured item (target word) and produce an entailment (i.e., answer questions such as: What is it for? What does it do? Where is it found?) in which associated verbs, adjectives, and nouns are elicited.

• Phonological Awareness: This test of phonemic sensitivity combines blending, specifically the ability to blend sounds (put sounds together – e.g., hay +stack is -- haystack) and elision, specifically the ability to remove sounds from words (e.g., what word is left when you take stack away from haystack?). The test moves from word-level, to syllable-level, to sub-syllable level and from receptive (multiple choice, identification) to productive (free response) skills.

• Print Knowledge: This subtest measures early print knowledge (print concepts, letter discrimination, word discrimination, letter-name identification and production, letter-sound identification and production).

• Early Literacy Index: Scores from the three subtests were combined to produce an index of early literacy.

Taken together, the three curricula interventions had significant effects on all four outcome measures (Exhibit 5). However, the findings are driven by the two interventions that showed impacts on children’s language and literacy development (Exhibit 6). Treatment 1, Ready, Set, Leap and Treatment 3, Breakthrough to Literacy, had significant effects on all of the measures; Treatment 2, Building Early Language and Literacy had no significant impacts on any of the measures20. When we combine the impacts for Treatments 1 and 3 (Exhibit 7), we can see that these two curricula taken together significantly improved outcomes for children. For the remainder of the discussion, we have combined the two curricula to improve statistical power and because the impacts of each were quite similar.

Exhibit 8 shows that the impacts were different for children in classrooms with teachers whose primary language was Spanish (where the children also spoke Spanish as their home language) vs. children in classrooms with teachers whose primary language was English. Exhibit 8 shows that, for children in classrooms with Spanish-dominant teachers, there were impacts on more of the measures and that the impacts were greater than for children with English-dominant teachers. This finding reflects the earlier finding that the curricula had a larger impact on the behavior of Spanish-speaking teachers. The results are quite similar when we look at the difference in outcomes specifically for children with a home language other than English and those whose home language was English (Exhibit 9).21 Some of the English-language learners (and all of the Haitian-Creole speakers) were in classrooms with English-speaking teachers, on whom the effects of the interventions were less pronounced22.

It is important to remember that these outcomes are for tests administered in English.23 An important goal for the curricula was to help English-language learners progress in English before they entered English-only kindergarten classes, and the two interventions appear to have been quite effective in doing that.

Another way to look at the impact of the curricula on children’s outcomes is to see where they are in terms of national norms. As part of ongoing work for the Office of the Assistant Secretary for Planning and Evaluation (ASPE), we have calculated that children from low-income families are about a year behind the national norms on a test of language at the end of the four-year-old year, as they prepare to enter kindergarten (Layzer, in preparation). While the interventions had significant impacts, it seems important to ask the question, “How much of the gap was closed?” On all four measures, children in the control group scored considerably below the norms. On the overall index, the interventions succeeded in closing more than half the gap in achievement. On the individual subscales, the interventions succeeded in halving the gap for Phonological Awareness and outperforming the norming sample (a nationally-representative sample of children). On the Definitional Vocabulary subtest, although the children in the two treatment groups made significant gains, there remained a large gap in achievement (Exhibit 10). As part of the analysis, we investigated a possible age-by-treatment interaction but found none.

These gains made by children in the two treatment groups can be described in another way. The discussion above shows that, on all three subtests the gap was reduced or eliminated. How many months of growth do these impacts represent? Exhibits 11, 12 and 13 show that the impacts range from a low of almost five months for Definitional Vocabulary to nine months for Print Knowledge.24

[D]

[D]

[D]

[D]

Additional Findings

One of two questions addressed in additional analyses was whether the increased focus on language and literacy activities might come at the expense of other important developmental activities. The interventions did indeed increase time spent on language and literacy activities substantially. However, Exhibit 14 shows that, while there are some resulting differences in the proportion of time allocated to different activities, these differences were not large. Children in the treatment group spent 9% more time in language and literacy activities than children in the control group (a 64% increase), 7% less time in other developmental activities25 and 3% less time in routines, transitions and gross motor play.

Relationship Between Staff Educational Background and Teacher and Child Outcomes

Because of the national discussion about the importance of teacher educational credentials in early childhood education, which is increasingly reflected in states’ systems for improving quality, we were interested in investigating two related questions:

• What is the relationship between teacher educational background and teacher behavior and interactions in the classroom?

• Does the educational level of teachers make a difference to the impact of the interventions on teacher behavior and interactions, the classroom environment and child outcomes?

To answer the first question, we used information on teacher education from the staff background questionnaire and observational data from the baseline data collection in 2003. The analysis investigated the relationship between having a bachelor’s degree and teacher behavior and interactions with children.26

We found small but significant relationships between a bachelor’s degree and teachers’ support for print knowledge and for teacher’s positive affect toward children. The size of the effect is comparable to the effect size found by Barnett in his recent meta-analysis (Barnett and Ackerman, 2006). However, analysis of the relationships for teachers whose primary language was English vs. Spanish found significant relationships only for Spanish-dominant teachers (Attachment table A1).

Underlying the second question is the hypothesis that better-educated teachers would be better prepared to grasp and implement a new curriculum, would therefore demonstrate more of the behaviors and interactions that support language and literacy development and would produce greater impacts on children’s outcomes. To examine whether this was indeed the case, we looked first at the 2005 observational data from the OMLIT, to determine whether teachers’ educational achievement affected the impact of the treatment on teacher behavior. An interaction effect was found for one construct on the OMLIT – Literacy Opportunities (the number and type of activities and opportunities, either teacher-or child-initiated, that supported literacy), but it was not the hypothesized effect. Rather than heightening the effect of the interventions on better-educated teachers, the effect of the interaction was to eliminate the differences between less-educated teachers and their better-educated counterparts (Exhibit 15). In the treatment group, teachers at all educational levels look remarkably similar in the extent to which they provide or facilitate such opportunities, compared with quite dramatic differences in the control group teachers. As Exhibit 16 shows, the interaction effect was found in the sample of teachers for whom English was the dominant language. There were no significant interaction effects for Spanish-dominant teachers.

There were no interaction effects on child outcomes: the impacts of the treatment were similar for children, regardless of the educational background of the teacher.

[D]

[D]

[D]

Discussion

The findings show that this model of professional development, in which initial and follow-up training sessions were supported by bi-monthly mentoring over an 18-month period, was effective in changing teachers’ classroom practices and the classroom environments in ways that fostered early language and literacy development. This finding does not imply that all types of mentoring are equally effective. For all three of the interventions, mentoring activities were directly linked to research on early literacy and to teachers’ actual classroom activities.

Importantly, this focused training and ongoing support eliminated the effects of teachers’ educational background on their support for children’s literacy. As a result, impacts on children were not affected by teachers’ educational levels.

In most classrooms, the elements of each curriculum were securely in place at the end of the 18-month period. However, even after 18 months, many teachers were still not comfortable working with small groups for most of the time, as the mentors encouraged them to do. Much of the reading aloud that teachers did was with somewhat larger groups than was optimal. Mentors reported that teachers worried that some children would “miss out’ on reading time if they worked mostly with small groups.

The impacts on children are also encouraging, given the size of the achievement gap for low-income children that is revealed as they prepare to enter school. On all but one of the measures, children in the treatment group moved close to the national norm or went beyond it. It is troubling that the gaps in children’s vocabulary did not come close to being closed. A major reason for this is that Spanish-speaking children began with English-language scores well below the norms and below their English-speaking peers. Even though they made substantial progress as a result of the interventions, a large gap remained. It seems that the gap in this area may be too great to be closed in one year.

Nevertheless, the impacts on children’s outcomes are substantially larger than we are used to seeing in large-scale,”real-life” studies. There are no comparable randomized experiments in child care centers against which to compare this study; the Head Start Impact Study may provide the closest comparison. On similar measures for 4-year-olds, the Impact Study found no impact on oral comprehension and phonological awareness, and a relatively modest effect (.22 of a standard deviation) on a letter-word identification test. This effect size is identical to the overall average effect of any organized preschool experience (center-based child care, Head Start, private pre-k and public prekindergarten) reported by Magnuson and her colleagues from their analysis of ECLS-K data (Magnuson et al., 2006). On the other hand, the impacts of the Project Upgrade interventions are similar to those reported for school-based prekindergarten programs. Using a regression-discontinuity design and data from five states, Barnett et al. (2005) found an impact of preschool on print awareness of .64 of a standard deviation. The effect of the Project Upgrade interventions seems to have been to focus the attention of child care staff on aspects of children’s development that early childhood teachers in school-based programs recognize as critical elements of school readiness.

Finally, there is the finding that one of the interventions, though it had positive effects on teachers and classrooms, had no impact on children’s outcomes. There are some possible explanations for this: this intervention featured two 15-20-minute add-on sessions each day in contrast to the other two which were intended to be woven into activities throughout the day. It seems likely that, B.E.L.L. teachers, though they engaged in the behaviors and interactions that promote literacy, spent less time on them than teachers in the other two groups, and that the exposure was not sufficient to affect children’s outcomes.

In addition, the two successful interventions both used computer-based technology or electronic aids to act as a “second teacher” in the classroom; children could work by themselves in activity centers and receive feedback on what they were doing. In classrooms with Spanish-dominant teachers, these electronic aids were key elements in children’s learning English vocabulary.

In both cases, the result was greater exposure to the treatment. Since teachers liked all of the interventions, and benefited from all of them, it might be possible for the B.E.L.L. developer to modify the curriculum strategy in ways that would increase the intensity of exposure, by using electronic aids, dramatic play or fine motor materials to underscore the lessons learned in the 15-20 minute literacy activity periods.

While these findings provide the guidance that the Early Learning Coalition hoped for, the question of the longer-term meaning of these effects needs to be addressed. Did the interventions reduce the gaps in achievement sufficiently that children are better able to take advantage of the school experience? For teachers, are the effects on their behavior sustained in the absence of continued support from mentors? Do they continue to build on what they have learned? Does teacher turnover mean that later four-year-old cohorts have less exposure to the successful curricula? These questions haunt all early childhood interventions; they are especially important for interventions that have such powerful short-term effects.