CHAPTER I: INTRODUCTION

Head Start, the largest federally funded preschool program, provides comprehensive services to economically disadvantaged children and their families so that children can enter kindergarten ready to succeed in school. Performance standards for the program include requirements for the intensity and quality of a broad range of services for children and families. Head Start programs must offer education, health, and nutrition services to children, offer social services to their families, and provide opportunities for parents’ involvement in the programs. Head Start is designed to enhance children’s cognitive skills, social development, physical and mental health, and good nutrition. Programs also are expected to support the parent as the child’s economic provider, first teacher, and primary advocate for education and health services. Some tailoring of program services is expected to meet the needs of diverse communities. Education services must be appropriate to children’s linguistic backgrounds and developmental needs, and family services must be individualized to meet parents’ goals and needs.

Head Start has long emphasized the importance of continuous program improvement and, in keeping with this emphasis, has invested significant resources in strategies to enhance the quality of program services. Since its early years, Head Start has considered itself a national laboratory for developing good early childhood practice through the development of innovative approaches and the honing of “best practices” that are based on professional wisdom. In many cases, research partnerships with universities have been a part of these efforts. Particularly during the past decade, policymakers and program administrators have focused on devising strategies to enhance the quality of Head Start services that can improve children’s readiness to enter kindergarten.

A focus on program improvement and the development of innovative strategies for meeting the needs of children and families has occurred throughout the Head Start community, on both a large and a small scale. During the past several years, for example, the Head Start Bureau has initiated several large-scale quality enhancement efforts, including a series of national teacher training conferences on the subject of fostering early literacy development, a technical assistance initiative to help programs to implement mentor-coaching strategies designed to support teachers in the classroom, and regional training conferences on fostering children’s social-emotional development and on addressing difficult behaviors. Regional and state collaboration offices of the Administration for Children and Families (ACF) have launched statewide efforts to improve children’s access to health care, often through the development of partnerships with health care providers and dentists. On a smaller scale, individual Head Start programs have implemented new curricula, services to address the needs of English-language learners, and strategies for using child assessment data to improve services.

Despite this significant focus on quality improvement and the wide range of enhancement ideas that have been developed, little is known about the effectiveness of these national, state, regional, and locally initiated strategies. To more fully realize Head Start’s efforts to improve program quality and children’s readiness for school, the effectiveness of these quality enhancement strategies should be rigorously evaluated.

The Advisory Panel for the Head Start Evaluation Design Project recommended the following question as one of the central ones for research: “Which Head Start practices maximize benefits for children and families with different characteristics under what types of circumstances?” (U.S. Department of Health and Human Services 1990). This question would naturally call for a research design that compares the effectiveness of different Head Start programs that use different practices, and that have different characteristics. More recently, the Advisory Committee on Head Start Research and Evaluation considered the possibility that studying promising approaches in Head Start (by comparing some Head Start programs with others) might be a valuable complement to studying the contribution of Head Start generally, as the current Head Start Impact Study is doing by comparing children randomly assigned to participate or not participate in Head Start (Advisory Committee on Head Start Research and Evaluation 1999).

A study design that rigorously tests enhancements against current practice and against each other would provide valuable information about whether these enhancement initiatives improve children’s readiness for school beyond the benefit that may be gained from participation in typical Head Start services. The Head Start community also could learn more about how to invest resources strategically in quality enhancement activities by testing enhancement ideas that vary according to the intensity and duration of teacher training, ongoing technical assistance, or services provided to children and families. Knowledge gained from these studies could help the Head Start Bureau to decide whether to implement system-wide quality improvement efforts, determine where and how to target technical assistance, and determine which resources are required to achieve the desired results.

In recognition of the need for rigorous evaluation of Head Start quality enhancements, ACF’s Office of Planning, Research, and Evaluation contracted with Mathematica Policy Research, Inc. and its subcontractor, Xtria, LLC, to prepare design options for potential future evaluations. The purpose of the project, known as the Design Options for the Assessment of Head Start Quality Enhancements project, is to develop a framework for rigorously evaluating enhancement initiatives and program variations in Head Start. This framework is intended to provide flexible guidelines for designing research to evaluate the effectiveness of specific quality enhancements in Head Start across a range of different settings, on either a small or a large scale.1 Previous concept papers developed for this project have focused on key issues in developing research designs, implementing enhancements and measuring implementation and fidelity, and measuring intermediate and child outcomes (Ross et al. 2004; Paulsell et al. 2004; Boller et al. 2004). This final report identifies specific quality enhancement ideas and describes appropriate evaluation plans for each.

Evaluations of quality enhancement ideas in Head Start will be based on a comparison of the outcomes of children in programs that have implemented the quality enhancement with the outcomes of children in regular Head Start programs, or with those of children in programs that implemented different quality enhancements. In most cases, rigorous evaluations of quality enhancement strategies in Head Start will rely on random assignment of groups of children (classrooms, entire centers, or entire grantees), rather than on the random assignment of individual children, which has been the traditional approach in program evaluations.2 Because the outcomes of children within the same group generally are correlated, random assignment of groups of children generates design effects that considerably reduce the precision of the impact estimates relative to the random assignment of individual children. As we discuss in this report, the greater imprecision of impact estimates resulting from the random assignment of classrooms, centers, or grantees necessitates increases in the size of the samples of children for an evaluation.3 Accordingly, the evaluation may have to include more children, classrooms, and centers than are traditionally included as part of program evaluations. As a result, the costs of evaluating quality enhancement strategies relative to traditional program evaluations will generally increase as well.

1. POTENTIAL QUALITY ENHANCEMENT IDEAS

Many ideas for improving Head Start practices are currently under development by Head Start programs, by program-researcher partnerships, through federal initiatives, and through a variety of other projects. For example, some Head Start programs regularly conduct self-assessments to identify areas for improvement, and to develop strategies to make those improvements. For example, the programs might consider improved management approaches, specific teacher training, or adoption of a specialized curriculum to enhance children’s performance. Several Head Start programs are working with university or research partners to implement and test strategies to improve program practices. The Head Start-University partnerships that were funded through 2004 included five universities and their Head Start partners investigating strategies to promote language development, pro-social behavior, relationships, and school readiness. The Head Start Quality Research Center (QRC) Consortium includes eight university or private non-profit research teams and their Head Start program partners investigating the effects of interventions in the areas of early literacy, social-emotional development, and other domains of school readiness. Intervention approaches include enhancements to curriculum, teacher training and mentoring, parent involvement, and assessment practices. Federal initiatives include the Strategic Teacher Education Program (STEP) training to enhance early language and literacy practices in the classroom (provided to all Head Start teachers in 2002-2003) and Parent Mentor Training to develop parent leaders who could encourage other parents to read to their children, and to promote early language and literacy in the home (offered in 2004). Local Head Start programs and local organizations that partner with Head Start programs are using Innovation and Improvement grants, awarded by the Head Start Bureau in 2003, to develop ideas that will enhance early literacy, strengthen families and fatherhood, support children’s mental health and social-emotional development, promote youth development, promote coordinated social services within communities, and improve technical assistance to programs.

As part of the process of developing the research designs and implementation plans for this project, staff conducted telephone discussions or held in-person meetings with a range of stakeholders in the Head Start community to identify innovative practices and ideas that merit evaluation.4 The group of informants included federal Head Start Bureau staff, representatives of the 10 geographic ACF regions, local Head Start program staff, state Head Start collaboration officials, staff of the National Head Start Association, selected principal investigators in the Head Start QRC Consortium, and other researchers. These informants identified several areas in need of quality enhancement, including program management and leadership, access to child health care services, staff recruitment and retention, parents’ involvement, the use of data to improve programs, and strategies to address challenging behaviors in the classroom.

These discussions and a review of Head Start quality initiatives and research suggest that enhancement ideas in the following areas are the most prominent:

• Enhancing Children’s Language and Emergent Literacy Skills. Curricula or classroom interventions to promote children’s early literacy, including two that focus specifically on the needs of English-language learners, represented more than half of the enhancements identified.

• Program Management and Continuous Program Improvement. Several enhancements had the goal of improving program management, either through training for program directors or strategies for using data to support continuous program improvement.

• Supporting Children’s Social-Emotional Development. Some enhancements were designed to foster children’s social-emotional development and to support teachers in fostering positive classroom environments by helping to address challenging classroom behaviors.

• Increasing Access to Health Services. Two of the suggested enhancements were designed to increase families’ access to health and dental health services for their young children.

These ideas were most often mentioned as perceived quality enhancement needs, are currently under development, and, in some cases, are being evaluated in Head Start programs. Even so, as policies change and as more is learned about Head Start programs, many other quality enhancements will be considered and may even become more prominent. Accordingly, the research ideas discussed in this report are based in part on this list but also have been chosen to illustrate a range of research approaches so that they provide examples that will remain relevant even as the most important quality enhancement ideas change over time.

2. CRITERIA FOR ENHANCEMENTS

Given the large number of quality enhancement ideas that might be examined, some criteria are necessary to focus the evaluation efforts on enhancements that are the most promising in terms of both their appeal to the Head Start community and their readiness for evaluation. We propose five criteria to select a more focused set of enhancements for consideration:

1. Relevance. A theory has been developed to explain how and why the enhancement is expected to affect outcomes critical to children’s development, some evidence links the enhancement strategy to children's outcomes, and the problem that the enhancement addresses is of particular importance to Head Start staff or Head Start children.

2. Distinctiveness. The enhancement is measurably different in key program dimensions from current Head Start practice.

3. Clarity. The enhancement as it is fully implemented is clearly described, and the activities required to fully implement the quality enhancement strategy are clearly documented.

4. Replicability. The enhancement can be implemented to a high degree of fidelity in programs beyond the original sites and on a larger scale.

5. Feasibility. The enhancement can be implemented and rigorously evaluated.

Meeting the relevance and distinctiveness criteria are prerequisites for considering a quality enhancement for evaluation. If an enhancement idea meets these criteria, then we have reason to believe that child outcomes will be influenced to a meaningful degree. Relevant enhancements not only address a developmental domain or program process of central importance to the Head Start community but also are based on a strong intervention theory of change. In other words, evidence suggests that the enhancement strategy can affect the outcomes that are the target of the intervention, and that the enhancement will be delivered with sufficient intensity to produce meaningful change during the Head Start program year.

To assess distinctiveness, careful consideration must be given not only to the strength of the enhancement itself, but also to system-wide training and technical assistance efforts that are under way or in the planning stage that will affect the level of typical Head Start services at the time the evaluation is launched. For example, the Head Start Bureau has invested considerable resources in providing training to all programs about techniques to promote early literacy, and it plans to launch a new, nationwide training and technical assistance effort in the near future. Any early literacy enhancement would have to be distinct from program services that already have been strengthened through these efforts.

In addition to being relevant and distinctive, enhancements must be clear or well documented. Documentation must include clear information about how the fully implemented enhancement changes practice (documented through photographs, videos, or very precise explanations). It also must include implementation manuals, plans for classroom activities, parent-training materials, teacher-training protocols, and other materials that provide the foundation necessary to implement the quality enhancement with high fidelity to the enhancement model in a large number of Head Start programs. Documentation also should include measures of fidelity, so that the progress of implementing the quality enhancement can be monitored. Finally, documentation should include both expected interim outcomes (changes in teacher behaviors and beliefs and in observed classroom practice), and in children’s outcomes.

Enhancements that prove replicable in a small-scale study may need further refinement if they are to be replicable on a larger scale, particularly if national implementation is anticipated. Procedures, manuals, and training protocols may have to be adjusted or refined in order to ensure that an enhancement can be replicated well in a broad range of Head Start programs, and that it can be supported by the Head Start training and technical assistance system.

Feasible enhancements are those that can be implemented in a way that provide opportunities for rigorous evaluation. For example, it should be possible to implement the intervention without its effects “spilling over” to other programs or classrooms that are serving as the counterfactual, and thus are not supposed to implement the enhancement. Adjustments to implementation or evaluation designs can be made to address this issue, such as randomization carried out at the center rather than the classroom level.

3. A STAGED APPROACH TO EVALUATION

In light of both the need for larger evaluations that are potentially more expensive to deploy and the concomitant need for enhancements to meet the five evaluation criteria before large-scale evaluation is attempted, previous concept papers have described a staged evaluation approach. Such an approach first builds evidence of implementation feasibility, then of positive impacts in a best-case scenario, and finally of the strength of the idea in a broader field test. The approach thus includes the following three stages, described further in this section: (1) development (early implementation and documentation), (2) evaluation, and (3) field testing.

Development (Stage 1)

Many quality enhancement ideas could benefit from additional development during a pilot phase to ensure that implementation is well documented and well understood before the interventions are evaluated. In some cases, an enhancement may already be well developed, and lessons learned from previous implementation experiences may make this initial stage unnecessary. Nevertheless, for enhancements that are in need of further development, a pilot phase—most likely, lasting for one year—would be an important first step in the evaluation process.

During the development phase, an enhancement would be implemented in several programs. The goals of this phase would be to ensure that the theory of change underlying the intervention is well understood, implementation procedures are fine-tuned, manuals and other documentation are developed, and measures of fidelity are established. The development phase also would provide an opportunity to determine whether the enhancement can be implemented successfully with a reasonable level of resources.

Given the diversity of Head Start programs and children, the quality enhancement might have to be tailored to the specific circumstances of particular programs while maintaining the essential elements of the enhancement model. For example, the model may have to be altered due to constraints on physical space or because the training materials need modification to be more culturally appropriate. Because some amount of tailoring across Head Start settings is likely to be necessary, a critical task during the development phase is to identify the aspects of the enhancement’s implementation that are necessary to maintain a high degree of fidelity to the enhancement model. Measures of fidelity to the enhancement model should be developed, and teachers and programs should be given feedback based on classroom observations. Measures of key outcomes of the enhancement might have to be refined, or developed and pilot tested.

Evaluation (Stage 2)

After the enhancement has been fully developed and the techniques for implementing it with a high degree of fidelity are well understood, it can be implemented as part of a rigorous evaluation. The evaluation phase would measure effectiveness under favorable conditions (for example, in programs that volunteer for the study). Some enhancement strategies, such as the ones that the Head Start QRC Consortium is studying, may already have been evaluated on a small scale. Others have not been rigorously evaluated. Before investing the resources required for a full-scale field test, these enhancements should first be evaluated on a small scale to determine whether they show sufficient promise. Cost considerations and practicality suggest that an evaluation at this stage would likely focus on a few ACF regions or on other limited geographic areas, and, within those areas, on Head Start programs that volunteer to participate. However, ideally, programs selected for participation at this stage would encompass diversity in key programmatic characteristics such as part- or full-day schedules, and in key population characteristics, such as ethnicity and home language.

The evaluation schedule should allow time to implement the enhancement before children in the research sample begin their program year. For example, the enhancement could be implemented at some point during a program year, but not evaluated until the following program year. In addition to measuring impacts on children’s outcomes, an important component of the evaluation is the assessment of the quality of implementation, and of the degree of fidelity to the enhancement model programs. Knowing that the enhancement was implemented and the degree to which implementation in particular sites was true to the model will be critical for understanding impacts, and, if the enhancement has positive impacts on children, for understanding how the initiative can be replicated more broadly. If the evaluation indicates that the initiative is well implemented and has favorable impacts on children, then planners can consider either broadening the evaluation to include more programs that are willing to participate or moving to a field test of the enhancement.

Field Test (Stage 3)

Ultimately, enhancement ideas should be field tested in a representative sample of Head Start programs to determine whether the enhancement continues to be effective when implemented on a wide scale across a variety of Head Start programs. Support for implementing quality enhancements strategically (to permit evaluation) and on a wide scale could be provided through the Head Start technical assistance and training system at a level of intensity that is typical for broad, new initiatives. A field test would be most appropriate for an initiative that is under consideration for full-program implementation, regardless of whether the initiative has passed through the development and evaluation stages.

The Value of a Staged Approach to Evaluation

This staged approach to evaluation is one that allows enhancements to be well developed and well implemented prior to evaluation, and that allows for the demonstration of evidence of effectiveness before investing resources in a full-scale field test. Nevertheless, following these steps sequentially would take considerable time, which is not always necessary or feasible. For example, because potential quality enhancements are at various stages of development, the lessons learned from prior implementation experiences or from small-scale evaluation efforts may preclude the need for a development phase or even smaller-scale evaluation. Moreover, in some cases, it may not be possible to implement a staged approach to evaluation, as policy mandates may require more immediate change in program practices, teacher qualifications, or other aspects of Head Start.

Yet, despite the time and expense associated with the staged approach, the steps are often necessary for any particular enhancement idea in order to avoid implementation disasters and to gain the confidence of skeptics. In fact, as we researched enhancement ideas, we found that those achieving public recognition had gone through a development phase and often, a small-scale evaluation, although in many cases, the evaluations were not rigorous. The development and evaluation activities had been conducted over many years, and often we found that ideas were abandoned before they had fully emerged from this process.

A more systematic and planful approach to incubating new ideas and evaluating them using rigorous designs could put public research and development resources to more efficient use and in some cases, reduce the amount of elapsed time to develop and evaluate new ideas. The staged approach we have described provides a useful framework for learning about the effectiveness of enhancement ideas at different stages of development. Fully developing enhancements in a pilot phase (by setting expectations for documentation to be produced at the end of an implementation period) and launching small-scale, rigorous evaluations to identify promising practices can provide program administrators with a pool of potentially valuable ideas to be considered when broad policy mandates require a prompt response. A system for reporting and maintaining an archive of reports and other information about these ideas is critical to ensure that ideas progress through the evaluation stages, and that information is available when broad policy mandates are under consideration.

4. PLAN FOR THE REPORT

This report describes the goals and activities associated with each of the three stages of research through the use of specific examples of potential Head Start quality enhancements that are good illustrations of the particular stage of research. To provide a context for the discussion of research designs in this report, Chapter II describes the Head Start program and research. Subsequent chapters are organized according to the three stages of research described here. Chapter III discusses the development phase and the information that should emerge from early efforts to establish the essential features of a quality enhancement idea and the requirements for implementing the initiative. Chapters IV through VI describe research designs for specific quality enhancement initiatives that are ready for evaluation. Chapter VII describes field testing of broad program initiatives. Our selection of quality enhancement ideas to be discussed in this report was influenced in part by the ideas that seem to be of most interest to the Head Start community, with the addition of one area—early mathematics—that was not discussed, but that is likely to become more prominent after programs have reviewed the results of the annual child assessments conducted as part of the Head Start National Reporting System.