ACF OPRE: Early Head Start Research and Evaluation Project (EHSRE)

IV. PROGRAM IMPLEMENTATION: OVERALL LEVELS AND PATTERNS

This chapter and the three that follow report the levels and patterns of program implementation in 1999, as well as the progress in implementation that programs made over time. For these analyses, we defined the degree of implementation as the extent to which programs offered services that met the requirements of the Early Head Start grant announcement (U.S. Department of Health and Human Services 1995) and selected key elements of the revised Head Start Program Performance Standards (U.S. Department of Health and Human Services 1996). We defined “full implementation” as substantially implementing, or exceeding expectations for implementing, these key program elements.

We begin this chapter by describing our methods for measuring program implementation and then summarize the progress programs made in their overall levels of implementation between fall 1997 and fall 1999. In addition, we describe patterns in the timing by which programs reached full implementation of particular program elements. Succeeding chapters address implementation progress in broad program areas—child development and health services (Chapter V), family and community partnerships (Chapter VI), and staff development and program management systems. (Chapter VII).

A. MEASURING PROGRAM IMPLEMENTATION

To assess the extent of program implementation, we developed implementation rating scales, checklists for organizing the information needed to assign ratings to programs, and a rating process. We designed this rating system to help us reduce a large amount of information on program implementation into summary variables for testing hypotheses about how implementation relates to outcomes and to systematically analyze the research programs’ progress toward full implementation over time. This section describes our data sources, the rating scales we developed, and the rating process we followed for assessing implementation.

1. Data Sources

For these analyses, we relied primarily on information collected during site visits conducted in fall 1997 and fall 1999 and self-administered surveys completed by program staff at the time of the site visits. To facilitate the systematic assignment of implementation ratings for each program, site visitors assembled the site visit and staff survey information in checklists organized according to key program elements of the performance standards (Appendix A). In addition, site visitors wrote detailed program profiles based on information obtained during the site visits. Program directors and their local research partners reviewed the profiles and checklists for their programs, provided corrections of erroneous information, and in some cases provided additional clarifying information.

2. Implementation Rating Scales

To develop implementation rating scales, we identified specific criteria for determining the degree to which programs implemented Early Head Start’s three major program areas as defined in the performance standards: (1) early childhood development and health services, (2) family and community partnerships, and (3) program design and management. To refine our assessment, we created distinct criteria for both family and community partnerships. Likewise, within program design and management we created separate criteria for staff development and program management systems.

The criteria encompass key program requirements contained in the Early Head Start grant announcement and the performance standards. Because the purpose of the ratings was to identify and track over time the implementation of key program requirements and not to monitor compliance, we focused on key requirements needed to help us identify pathways to full implementation and to summarize and quantify a large amount of qualitative information on program implementation. We reviewed our initial criteria with representatives of the Head Start Bureau and the Early Head Start technical assistance network to ensure that the criteria included the most important subset of program requirements. We also solicited comments from members of the Early Head Start Research Consortium. Table IV.1 summarizes the 25 program elements we assessed in 1999, organized according to program area. The rating scales were slightly different in 1997, but were revised based on the initial site visit experience. In 1997, we rated 24 program elements. The only differences were that in 1997 (1) follow-up services for children with disabilities were rated as a part of developmental assessments (under child development and health), (2) “father initiatives” was a separate rating element within family development (whereas in 1999 it was included in parent involvement), and (3) in the area of management systems, communication systems was not rated.

Prior to our fall 1997 site visits, we created a rating scale for each of the 24 program elements. In 1999, we made some minor revisions to these scales to reflect clarifications in program guidance from the Head Start Bureau and our evolving understanding of the performance standards, which took effect after our fall 1997 site visits. The 1999 rating scales are shown in Appendix B.1 Each rating scale contains five levels of implementation, ranging from minimal implementation (level 1) to enhanced implementation (level 5) (Table IV.2). We considered programs rated at level 1 through 3 to have reached partial implementation and programs rated at levels 4 and 5 to have reached full implementation of the particular program element rated.

TABLE IV.1

PROGRAM ELEMENTS INCLUDED IN THE EARLY HEAD START IMPLEMENTATION RATING SCALES-FALL 1999
Program Component	Program Element
Child Development and Health	Frequency of child development services
	Developmental assessments
	Follow-up services for children with disabilities
	Health services
	Child care
	Parent involvement in child development services
	Individualization of services
	Group socializations (for home-based and mixed-approach programs)
Family Development	Individualized family partnership agreements
	Availability of services
	Frequency of regular family development services
	Parent involvement
Community Building	Collaborative relationships
	Advisory committees
	Transition plans
Staff Development	Supervision
	Training
	Turnover
	Compensation
	Morale
Management Systems	Policy council
	Communication systems
	Goals, objectives, and plans
	Self-assessment
	Community needs assessment

TABLE IV.2

EARLY HEAD START IMPLEMENTATION RATING SCALE LEVELS
Level	Definition
Partial Implementation
1	Minimal implementation	Program shows little or no evidence of effort to implement the relevant program element.
2	Low-level implementation	Program has made some effort to implement the relevant program element.
3	Moderate implementation	Program has implemented some aspects of the relevant program element.
Full Implementationa
4	Full implementation	Program has substantially implemented the relevant program element.
5	Enhanced implementation	Program has exceeded expectations for implementing the relevant program element.

aWe use the term “full implementation” throughout this report as a research term to reflect our judgment that a program had achieved a rating of 4 or 5. We recognize that programs not “fully” implemented were nevertheless often implementing many features of the performance standards. In addition, even when rated as “fully” implemented, programs may have been striving to do more and be involved in continuous improvement activities.(back)

3. Rating Process

Following each round of site visits, we used a consensus-based process to assign implementation ratings to each Early Head Start research program. We assembled a rating panel that included four national evaluation team members, a representative of the Early Head Start technical assistance network, and another outside expert. For each program, three people—the site visitor and two panel members—assigned ratings independently, based on information contained in the checklists and the program profile compiled by the site visitor. Ratings were assigned for each of the 24 (or 25 in 1999) program elements, the five program areas (as shown in Table IV.1), and for overall implementation. In completing the ratings of overall implementation, we established the following guidelines for creating the overall ratings based on the ratings of the individual program components:

Low-level Implementation: Programs that reached only a low level of implementation had achieved moderate implementation in only one or two program areas. Other programs areas were poorly or minimally implemented.
Moderate Implementation: To achieve this rating overall, programs were (1) fully implemented in a few program areas and moderately implemented in the other areas, (2) moderately implemented in all areas, (3) moderately implemented in most areas with low-level implementation in one area, or (4) fully implemented in every area except child development and health services.
Full Implementation: To be rated as fully implemented overall, programs had to be rated as fully implemented in most of the five component areas. Reflecting the Head Start Bureau’s focus on child development, panel members gave special consideration to the rating of child development and health services, and weighted it more heavily in arriving at their consensus rating of overall implementation.
Enhanced Implementation: A program demonstrating enhanced implementation was fully implemented in all areas and exceeded the standards in some of the component areas.

After these independent ratings were completed for all programs, the panel met to review the three sets of independent ratings, discuss differences in ratings across panel members, and assign consensus ratings for each program. We checked the validity of the our 1997 ratings by comparing them to independent ratings. After the Head Start Bureau completed its monitoring visits to all 17 research programs in spring 1998, we asked a member of the monitoring team to use information collected during the monitoring visits to rate programs using the rating scales we developed. We did not provide the monitoring team member with our rating results or the information we collected during site visits. The independent ratings assigned by the bureau’s monitoring team member were very similar to those assigned by our rating panel, yielding an indication that our ratings provide a valid assessment of program implementation.

B. PROGRESS IN OVERALL IMPLEMENTATION BETWEEN FALL 1997 AND FALL 1999

By fall 1999, all but one of the research programs had been serving families for three years, and the Head Start Bureau had monitored each one for compliance with the performance standards, which went into effect in January 1998. Most programs had also received technical assistance following monitoring. Consequently, ACYF expected that by fall 1999, programs would be substantially in compliance with the performance standards, or very near compliance in most areas.

Indeed, across all program areas, the research programs made great strides in implementing Early Head Start between fall 1997 and fall 1999, with the number of programs rated as “fully implemented” overall doubling from 6 to 12 over the two years (Figure IV.1).2 Of the 12 programs that achieved full implementation, two were rated as having an enhanced level of implementation overall by fall 1999 (up from one in 1997). All five programs that had not reached full implementation by fall 1999 had reached moderate implementation. In most cases, they achieved moderate levels in child development and health services and moderate or higher level in at least one other area. In 1997, in contrast, eight programs were rated as moderately implemented, and three (with low ratings in multiple areas) were rated as poorly implemented.

FIGURE IV.1

EARLY HEAD START OVERALL IMPLEMENTATION RATINGS
[D]

C. PATTERNS IN THE TIMING BY WHICH PROGRAMS REACHED OVERALL IMPLEMENTATION

The Early Head Start research programs made substantial progress in implementing key areas of the performance standards between 1997 and 1999. Altogether, nearly three-quarters of the research programs were rated as fully implemented within four years of being funded. Some accomplished this level of successful implementation relatively quickly, while others took longer. Three patterns characterize the implementation progress of the 17 research programs: those that were “early implementers,” “later implementers,” and “incomplete implementers.”

The early implementers are those programs that were rated as fully implemented in fall 1997 and remained so in fall 1999. About one-third (six programs) were in this category. Although these programs became fully implemented early in the evaluation period, they continued to develop over the two years. For example, between 1997 and 1999 three of them expanded the number of children and families they served. These early implementers demonstrated how services for infants and toddlers can be expanded within their communities.

The later implementers are the programs that had not achieved an overall rating of “fully implemented” in fall 1997 but reached that level by fall 1999. Six programs (another third) were in this group. In many cases, these were programs that were well implemented in most areas by 1997 except child development and health, but improved their implementation of child development and health services and reached full implementation overall by 1999.

Finally, five programs, which we refer to as the incomplete implementers, were not fully implemented in fall 1997 and had not reached full implementation by fall 1999. In some cases, the incomplete implementers did not meet the requirements for a rating of “fully implemented” in child development and health services or in other areas but did provide strong family development services. In every case, however, these programs had made strides in some areas, even though they still faced important challenges.

A number of factors may explain why programs achieved different levels of overall implementation at different rates. For one, experience serving infants and toddlers may have helped some programs reach full implementation of Early Head Start more quickly. Among the 11 programs that had served infants and toddlers before, 5 were early, 4 were later, and 2 were incomplete implementers. In contrast, of the six programs that were new or were Head Start programs serving infants and toddlers for the first time, only 1 was an early implementer, while 2 were later and 3 were incomplete.

Low staff turnover during the first year—including turnover in leadership positions—also appears to have been instrumental in helping programs reach full implementation more quickly. Of the six programs with a staff turnover rate of 20 percent or higher during the year prior to fall 1997, only one was an early implementer, two were later, and three were incomplete. On the other hand, among the 11 programs with staff turnover under 20 percent during the year prior to fall 1997, five were early implementers, four were later, and two were incomplete implementers. Later staff turnover does not appear to have been as important an influence on programs’ progress in becoming fully implemented.

Although the timing of reaching full implementation might be expected to vary systematically according to program approach, that does not appear to be the case. Whether or not programs became fully implemented within four years of funding, and whether they did so earlier or later, does not appear to be related to their basic approach to serving families or whether they changed their approach between fall 1997 and fall 1999. Each group of programs defined by implementation pattern includes home-based, center-based, and mixed-approach programs as well as at least one program that had changed its approach.

Some aspects of Early Head Start were easier to implement than others. Most programs were able to implement a large number of program elements well by fall 1997 and continued implementing them well in fall 1999. These “early strengths” include:3

Individualization of Child Development Services. From the beginning, most research programs were able to implement a strategy for individualizing child development services according to the needs of children. A strength of the programs was providing child development services to almost all children and families in their native languages. Many programs also individualized services according to children’s developmental assessments.
Developmental Assessments. Most research programs selected instruments for assessing children’s development and were successful in conducting assessments with most enrolled children by fall 1997.
Parent Involvement in Child Development Services. Most research programs were fully involving parents in planning for child development services by fall 1997. They did so by involving parents in their children’s developmental assessments, reviewing the results with them, and using them to plan services. In center-based programs, some parents also participated in parent committees that planned center activities.
Frequency of Parent-Child Group Socializations Offered. From the beginning, most home-based programs offered the required group socializations each month. However, although the programs offered these group socializations, attendance was often low.
Efforts to Include Fathers. Most programs made special efforts to involve fathers and father figures in program activities. However, levels of participation in special activities for fathers were often low. The involvement of fathers in Early Head Start programs is explored in depth in Father Involvement in Early Head Start Programs: Summary Report (Raikes et al. 2002).
Collaborative Relationships. Most of the Early Head Start research programs established many relationships, some based on formal written agreements, with other service providers early in their development. These programs communicated regularly with other service providers to coordinate services for families and participated in at least one coordinating group of community service providers.
Staff Supervision. Two-thirds of the research programs had fully implemented staff supervision requirements by fall 1997, and more than half were providing an enhanced level of staff supervision by fall 1997. Supervisors in these programs were conducting both group and individual supervision sessions and, partly from observation of service delivery, providing feedback on performance.
Staff Training. By fall 1997, two-thirds of the research programs were providing staff training according to a plan based on assessment of staff training needs, and all staff had received training in multiple areas. Most programs also encouraged staff members to take advantage of national, state, and local training opportunities that would equip them to provide high-quality services.
Community Needs Assessment. Nearly all the research programs had fully implemented the requirements for conducting community needs assessments by 1997 and continued to update them as required.

Many aspects of Early Head Start were more challenging to implement. Nevertheless, most programs had implemented them well by fall 1999. These “later strengths” include:

Health Services. Between 1997 and 1999, the number of research programs that had fully implemented health services for children nearly doubled, and most programs had fully implemented these services by fall 1999. All programs helped families find medical homes for their children. By 1999, most programs were also tracking receipt of health services to help ensure that children received all recommended well-child examinations, immunizations, and needed treatments.
Frequency of Child Development Services. Programs improved considerably over time in completing the required schedule of home visits. By fall 1999, most research programs with home-based services were completing an average of at least three home visits a month with enrolled families, and all center-based programs offered full-day, full-year child development services and child care.
Individualized Family Partnership Agreements. By the second rating period, most programs were creating individualized family partnership agreements with all or most of their families and updating them as needed.
Availability of Family Development Services. Over time, the number of research programs that fully implemented requirements to make a wide range of services available to families, either directly or by referral, and to follow up systematically to ensure that families receive needed services, nearly doubled.
Frequency of Family Development Services. By fall 1999, most programs were meeting regularly with all or most families to provide case management services. Many programs also provided some family development services on site and made referrals to other community service providers.
Advisory Committees. In 1997, some programs were still putting together community advisory committees in health and other areas, or the committees had formed but were not active. By 1999, most programs had established committees that met regularly and provided advice on infant and toddler issues.
Transition Planning. Early on, most research programs did not focus on planning for children’s transitions to preschool when they left Early Head Start. By 1999, however, children were beginning to transition out of the program, and most programs had procedures in place for planning with families for children’s transitions.
Staff Compensation. By 1999, more than half the Early Head Start research programs reported that staff salaries and benefits were above the average for similar community programs. Several programs were still in the process of increasing salary scales and revising them to reward staff who obtained associate’s degrees.
Staff Morale. Staff in the research programs generally reported a very positive view of their workplace. Based on site visits and staff reports, morale appeared to be very high in half the programs.
Policy Council. Initially, only half the research programs had fully implemented Policy Council requirements, but by 1999, nearly all had established Policy Councils that included parents and community members and met regularly to make key decisions about the program.
Goals, Objectives, and Plans. Initially, many programs had not formally set goals and objectives, nor had they developed written implementation plans. By 1999, however, most programs had set or updated their goals and objectives and developed written implementation plans.
Self-Assessment. In 1997, one-third of the research programs had conducted an annual assessment of their progress toward their goals and of their compliance with the Head Start Program Performance Standards. By 1999, the proportion of programs that had conducted a self-assessment in consultation with Policy Council members, parents, staff, and other community members doubled.

A third group of program elements appears to represent “ongoing challenges” for Early Head Start programs. Three elements were particularly challenging to implement, and the majority of programs had not fully implemented them by fall 1999.4 These are:

Child Care. Many Early Head Start parents were employed and needed child care services. Programs that offered center-based services were able to meet the child care needs of families more easily than were home-based programs. Home-based programs made considerable progress in developing child care options that meet the performance standards, and some even added their own center-based services. Despite progress from 1997 to 1999, however, few home-based or mixed-approach programs could ensure that the child care attended by “all or nearly all” Early Head Start children was of high quality. Helping parents arrange high-quality child care and working with child care providers to meet the quality standards in the Head Start Program Performance Standards remains a challenge.
Parent Involvement5: Although all programs offered opportunities for parents to participate in program governance, many offered opportunities for parents to volunteer, and many worked hard to involve fathers, only a few were able to involve most parents in some capacity. In part because of welfare reform, many parents were working and finding it difficult to make time for volunteering and participating in other program activities.
Staff Retention: Like child care programs in general, many of the Early Head Start research programs struggled to retain frontline staff, and in both 1997 and 1999, experienced staff turnover rates of 20 percent or more. Although most programs did not achieve low turnover rates by 1999, the number of programs that experienced very high turnover rates did decline.

The following chapters explore the levels and patterns of program implementation in more depth and describe the factors that influenced program implementation.

1The 1997 rating scales appear in Leading the Way, Volume III, Appendix B (ACYF 2000).(back)

2Implementation ratings from 1997 site visits were first described in Leading the Way: Characteristics and Early Experiences of Selected Early Head Start Programs, Volume III, Program Implementation (ACYF 2000a). 1999 ratings are described in detail in Chapters V through VII of the current report.(back)

3These program elements are defined and described in the implementation rating scales contained in Appendix B.(back)

4Although health services were among the program elements that programs implemented well by fall 1999, one aspect of these services, namely mental health services, presented an ongoing challenge. Shortages of mental health services in the community made it very difficult for programs to link all families to mental health services they needed.(back)

5This excludes involvement in child development services but includes volunteering, serving on Policy Councils, and participating in parent committees.(back)

Table of Contents | Previous | Next