Table of Contents | Previous | Next |
II. MEASURING THE EXTENT OF PROGRAM IMPLEMENTATION
The first step to measuring the extent of program implementation is establishing a clear definition of a fully implemented program. For purposes of this research, we defined the degree of implementation as the extent to which a program offers services meeting the requirements of selected key elements of the revised Head Start Program Performance Standards and the Early Head Start grant announcement. The degree of implementation across Early Head Start program components could vary within programs at any given point in time and especially during early stages of program development, reflecting variation in program emphases and levels of difficulty with implementing particular services in particular communities. Likewise, the degree of implementation of each program component could vary across programs, reflecting differences in program emphases and circumstances. The degree of implementation could also vary across programs in the early stages due to differences in programs’ understanding of the revised Head Start Program Performance Standards. Again, in fall 1997, the performance standards were not yet official, and the Head Start Bureau had not yet used the standards to monitor programs.
The degree to which programs implement Early Head Start and the quality of the services they provide are intertwined. The Early Head Start grant announcement not only specified the types of services that programs must provide, but explicitly required programs to provide high-quality services. Thus, in order to determine the extent to which programs have met the federal government’s vision for Early Head Start and have become fully implemented, we must assess both the degree to which Early Head Start research programs have implemented the required services and, to the extent we are able, the quality of the services provided. Because established measurement tools do not exist for assessing the quality of many Early Head Start services, and because of the importance of child care, we have focused our first assessment of service quality on center-based child care, drawing on the child care research literature for measuring quality. In the next implementation report, to be completed later in 2000, we will include updated quality assessments of both center-based and family child care. We will also include quality assessments of child development home visits, a key component of Early Head Start child development services in programs that serve families through the home-based or combination option.
To help us assess the extent of program implementation, we developed rating scales, checklists for organizing the information needed to assign ratings, and a process for assigning ratings to each research program. The rating scales are designed to help us reduce the large amount of implementation information into summary variables for testing hypotheses about how implementation relates to outcomes and to help us summarize the research programs’ progress toward full implementation over time.
To assess the quality of center-based child care, we used an established quality measure--the Infant/Toddler Environment Rating Scale (ITERS) (Harms, Cryer, and Clifford 1990)--and examined structural quality indicators, including group sizes and child-staff ratios. The ITERS measures were collected in observations of center-based child care provided directly by Early Head Start research programs and observations of Early Head Start children’s classrooms in community child care centers. These observations were made in connection with developmental assessments of children in the research sample at 14 and 24 months of age.
This chapter describes the process we followed for assessing the extent of program implementation in the Early Head Start research programs in fall 1997. We begin by describing the data sources we used in developing implementation ratings and then describe our methodology for developing the implementation rating scales and for assigning ratings to individual programs. A final section describes the instrument we used to conduct observations of child care centers used by Early Head Start families and the methodology we used for analyzing the preliminary observation data on child care quality presented in Chapter III.
A. DATA SOURCES FOR IMPLEMENTATION RATINGS
To assess the extent of program implementation, we relied primarily on information collected during site visits conducted in fall 1997. With one member of the site visit team visiting each program, site visitors conducted individual and group interviews with program staff, parents, community members, and local researchers; reviewed case files to learn about patterns of services provided to individual families; reviewed other program records; and observed service delivery during a home visit or in a program-operated child care center. In addition, all Early Head Start staff at the research programs completed a self-administered survey about their background, qualifications, education and training, and satisfaction with the work environment. To ensure consistency of data collection across individual programs while allowing site visitors to tailor discussion guides to the circumstances of individual programs, all six site visitors participated in a training session prior to the visits and followed discussion guides for conducting individual and group interviews while on-site.
To facilitate the assignment of implementation ratings for each program, site visitors assembled the site visit and staff survey information in checklists organized according to program components (Appendix A). In addition, site visitors wrote detailed program profiles based on information obtained during the visits. Program directors and their local research partners reviewed the profiles and checklists for their programs, provided corrections of erroneous information, and in some cases provided additional clarifying information.
B. IMPLEMENTATION RATING SCALES
To develop implementation rating scales, we identified specific criteria for determining the degree to which programs implemented Early Head Start’s three major program areas: (1) early childhood development and health services, (2) family and community partnerships, and (3) program design and management. To refine our assessment, we created distinct criteria for both family and community partnerships. Likewise, within program design and management we created separate criteria for staff development and program management systems.
The criteria encompass key program requirements in the Early Head Start grant announcement issued on March 17, 1995, and the revised Head Start Program Performance Standards issued on November 5, 1996. Because the purpose of the ratings was to identify and track over time the key elements of program implementation and not to monitor compliance, we focused on the key requirements needed to help us identify pathways to full implementation and high-quality services and to summarize and quantify a large amount of qualitative information on program implementation. We reviewed our initial criteria with representatives of the Head Start Bureau and the Early Head Start technical assistance network to ensure that our criteria focused on an appropriate subset of program requirements. We also solicited comments from members of the Early Head Start Research Consortium. After incorporating the comments and suggestions we received, we finalized the criteria and converted them into rating scales for each of the five program components we examined (Appendix B). Table II.1 summarizes the program elements we assessed under each of the five program components.
For each program element, we created a rating scale containing up to five levels of implementation, ranging from minimal implementation (level 1) to enhanced implementation (level 5). We created fewer than five implementation levels in our rating scales for a few of the program elements we examined, because our criteria were not complex enough to identify five distinct levels of implementation. For our analysis of program implementation, we considered programs rated at levels 1 through 3 to have reached partial implementation and programs rated at levels 4 and 5 to have reached full implementation of the particular program element rated. Table II.2 provides our definition for each rating level. We use the term “full implementation” as a research term to indicate that the program has substantially implemented most of the program elements.
 |
|
| Scale | Program Element |
|
|
| Early Childhood Development and Health Services | Frequency of services Developmental assessments Health services Child care Parent involvement in child development services Individualization of services Group socialization activities (for home-based and mixed-approach programs) |
| Family Partnerships | Individualized family partnership agreements Availability of services Frequency of services Parent involvement Father initiatives |
| Community Partnerships | Collaborative relationships with other service providers
Advisory committees Transition plans |
| Staff Development | Supervision Training Staff turnover Compensation Staff morale |
| Management Systems and Procedures | Policy council Goals, objectives, and plans Program self-assessment Community needs assessment |
|
|
C. IMPLEMENTATION RATING PROCESS
We designed a consensus-based approach to assigning implementation ratings to each Early Head Start research program. Following our 1997 site visits, we assembled a rating panel of four national evaluation team members and two outside experts. Each rating panel member was given responsibility for rating a subset of the research programs. For each program, the site visitor and two panel members reviewed the extensive documentation in more than 50 pages of checklists and written materials, and assigned ratings independently based on the program profile and the checklist. Once these independent ratings were completed for all programs, the rating panel met in May 1998 to review the three sets of ratings produced for each program, discuss differences in ratings across panel members, and assign consensus ratings for each program. During the course of this process, the rating panel made minor modifications to the rating scales to clarify ambiguities and create clearer distinctions between scores in some areas. The analyzes of the ratings we present in this report are based on the consensus ratings assigned in May 1998 by the rating team.
After we completed the rating process, we checked the validity of the consensus-based implementation ratings by comparing them to independent ratings. After the Head Start Bureau completed monitoring visits to all 17 research programs in spring 1998, we asked a member of the monitoring team to use information collected during the monitoring visits to rate the programs’ implementation using the rating scales we developed. We did not share our rating results or information collected during our site visits with the monitoring team. The ratings assigned by the monitoring team member were very similar to those assigned by our rating panel and confirmed that our ratings provide a good assessment of program implementation.
| Level | Definition |
|---|---|
| Partial Implementation | |
| 1 Minimal implementation | Program shows little or no evidence of effort to implement the relevant program element. |
| 2 Low-level implementation | Program has made some effort to implement the relevant program element. |
| 3 Moderate implementation | Program has implemented some aspects of the relevant program element. |
| Full implementation | |
| 4 Full implementationa | Program has substantially implemented the relevant program element. |
| 5 Enhanced implementation | Program has exceeded expectations for implementing the relevant program element. |
| aWe use the term “full implementation” throughout this report as a research term.(back) |
D. OBSERVATIONS OF CHILD CARE SETTINGS
In addition to information gathered during site visits to the Early Head Start research programs in fall 1997, we now have preliminary data from observations of the child care settings of Early Head Start children in the research sample. Child care observations are being conducted when children reach 14, 24, and 36 months of age. The preliminary data include observations completed in conjunction with child assessments conducted with children who were 14 or 24 months of age and submitted for data entry by February 1999. Thus, these preliminary data pertain primarily to the research programs’ first two years of serving families. The data include 162 classroom observations conducted in the nine Early Head Start research programs operating child care centers and 79 classroom observations in community child care centers caring for Early Head Start children from 14 research programs.
In Chapter III, we examine child care quality using the Infant/Toddler Environment Rating Scale (ITERS) (Harms, Cryer, and Clifford 1990). The ITERS consists of 35 items that assess the quality of center-based child care, including items in seven categories: (1) furnishings and display for children, (2) personal care routines, (3) listening and talking, (4) learning activities, (5) interaction, (6) program structure, and (7) adult needs.1 Each item is ranked from 1 to 7. A ranking of 1 describes care that does not even meet custodial care needs, while a ranking of 7 describes excellent, high-quality personalized care. The definitions for quality used in the ITERS are consistent with the Accreditation Criteria and Procedures of the National Association for the Education of Young Children (NAEYC 1998) and the Child Development Associate (CDA) National Credentialing Program (Council for Professional Recognition 1996).
To compute average ITERS scores for center-based child care provided directly by research programs, we began by averaging the observations over time for each classroom. Classrooms could have been observed as often as once per quarter (or more often if staff or children had changed since the last observation visit), depending on when Early Head Start children were in care. Once we calculated the average score for each classroom, we averaged the classroom scores for each center to generate a center score. If a program operated multiple centers, we then averaged the center scores to generate an average score for each research program.2
To compute average ITERS scores for child care provided in community child care centers used by Early Head Start families, we first computed an average score for each center used by a program family. As with Early Head Start centers, observations could have been conducted as often as once per quarter, depending on when children were in care. Then, we used the average scores for centers to calculate an average score for each research program.
Centers included in the scores for community child care centers represent a mix of centers to which research programs referred families and centers selected by families independently of the program. Also, in some research programs, many families chose to use family child care homes or informal child care providers such as relatives or friends. The average ITERS scores reported here are based exclusively on observations of center-based care. We are also assessing the quality of family child care homes using the Family Day Care Rating Scale (FDCRS). We will report programs’ average FDCRS scores for family child care homes used by early Head Start families in the next implementation report.
1We excluded three items from the adult needs category (opportunities for professional growth, adult meeting area, and provisions for parents) as is customary in research using the ITERS.(back)
2The average ITERS scores provided here do not reflect the average quality of child care received by individual program children. Rather, they represent the average quality of Early Head Start and community child care centers, determined at the classroom level, used by program families. Average scores for each program are not weighted to reflect the number of program children participating in each classroom or center.(back)
| Table of Contents | Previous | Next |