Table of Contents | Previous | Next |
C.5 IMPLEMENTATION MEASURES
The first step to measuring the extent of program implementation is establishing a clear definition of a fully implemented program. For purposes of this research, we defined the degree of implementation as the extent to which a program offers services meeting the requirements of selected key elements of the revised Head Start Program Performance Standards and the Early Head Start grant announcement. The degree of implementation across Early Head Start program components could vary within programs at any given point in time and especially during early stages of program development, reflecting variation in program emphases and levels of difficulty with implementing particular services in particular communities. Likewise, the degree of implementation of each program component could vary across programs, reflecting differences in program emphases and circumstances. The degree of implementation could also vary across programs in the early stages due to differences in programs' understanding of the revised Head Start Program Performance Standards. In fall 1997, the performance standards were not yet official, and the Head Start Bureau had not yet used the standards to monitor programs.
The degree to which programs implement Early Head Start and the quality of the services they provide are intertwined. The Early Head Start grant announcement not only specified the types of services that programs must provide, but explicitly required programs to provide high-quality services. Thus, in order to determine the extent to which programs have met the federal government's vision for Early Head Start and have become fully implemented, we must assess both the degree to which Early Head Start research programs have implemented the required services and, to the extent we are able, the quality of the services provided. Because established measurement tools do not exist for assessing the quality of many Early Head Start services, and because of the importance of child care, we have focused our first assessment of service quality on center-based child care, drawing on the child care research literature for measuring quality.
To help us assess the extent of program implementation, we developed rating scales, checklists for organizing the information needed to assign ratings, and a process for assigning ratings to each research program. The rating scales are designed to help us reduce the large amount of implementation information into summary variables for testing hypotheses about how implementation relates to outcomes and to help us summarize the research programs' progress toward full implementation over time.
To assess the quality of center-based child care, we used an established quality measure--the Infant/Toddler Environment Rating Scale (ITERS) (Harms, Cryer, and Clifford 1990)--and examined structural quality indicators, including group sizes and child-staff ratios. The ITERS measures were collected in observations of center-based child care provided directly by Early Head Start research programs and observations of Early Head Start children's classrooms in community child care centers. These observations were made in connection with developmental assessments of children in the research sample at 14 and 24 months of age.
The following sections describe the process we followed for assessing the extent of program implementation in the Early Head Start research programs in fall 1997. We begin by describing the data sources we used in developing implementation ratings and then describe our methodology for developing the implementation rating scales and for assigning ratings to individual programs.
a. Date Sources for Implementation Ratings
To assess the extent of program implementation, we relied primarily on information collected during site visits conducted in fall 1997. With one member of the site visit team visiting each program, site visitors conducted individual and group interviews with program staff, parents, community members, and local researchers; reviewed case files to learn about patterns of services provided to individual families; reviewed other program records; and observed service delivery during a home visit or in a program-operated child care center. In addition, all Early Head Start staff at the research programs completed a self-administered survey about their background, qualifications, education and training, and satisfaction with the work environment. To ensure consistency of data collection across individual programs while allowing site visitors to tailor discussion guides to the circumstances of individual programs, all six site visitors participated in a training session prior to the visits and followed discussion guides for conducting individual and group interviews while on-site.
To facilitate the assignment of implementation ratings for each program, site visitors assembled the site visit and staff survey information in checklists organized according to program components. In addition, site visitors wrote detailed program profiles based on information obtained during the visits. Program directors and their local research partners reviewed the profiles and checklists for their programs, provided corrections of erroneous information, and in some cases provided additional clarifying information.
b. Implementation Rating Scales
To develop implementation rating scales, we identified specific criteria for determining the degree to which programs implemented Early Head Start's three major program areas: (1) early childhood development and health services, (2) family and community partnerships, and (3) program design and management. To refine our assessment, we created distinct criteria for both family and community partnerships. Likewise, within program design and management we created separate criteria for staff development and program management systems.
The criteria encompass key program requirements in the Early Head Start grant announcement issued on March 17, 1995, and the revised Head Start Program Performance Standards first issued on November 5, 1996. Because the purpose of the ratings was to identify and track over time the key elements of program implementation and not to monitor compliance, we focused on the key requirements needed to help us identify pathways to full implementation and high-quality services and to summarize and quantify a large amount of qualitative information on program implementation. We reviewed our initial criteria with representatives of the Head Start Bureau and the Early Head Start technical assistance network to ensure that our criteria focused on an appropriate subset of program requirements. We also solicited comments from members of the Early Head Start Research Consortium. After incorporating the comments and suggestions we received, we finalized the criteria and converted them into rating scales for each of the five program components we examined. Table C.5.A summarizes the program elements we assessed under each of the five program components.
For each program element, we created a rating scale containing up to five levels of implementation, ranging from minimal implementation (level 1) to enhanced implementation (level 5). We created fewer than five implementation levels in our rating scales for a few of the program elements we examined, because our criteria were not complex enough to identify five distinct levels of implementation. For our analysis of program implementation, we considered programs rated at levels 1 through 3 to have reached partial implementation and programs rated at levels 4 and 5 to have reached full implementation of the particular program element rated. Table C.5.B provides our definition for each rating level. We use the term "full implementation" as a research term to indicate that the program has substantially implemented most of the program elements.
| Scale | Program Element |
|---|---|
| Early Childhood Development and Health Services | Frequency of services |
| Developmental assessments | |
| Health services | |
| Child care | |
| Parent involvement in child development services | |
| Individualization of services | |
| Group socialization activities (for home-based and mixed-approach programs) | |
| Family Partnerships | Individualized family partnership agreements |
| Availability of services | |
| Frequency of services | |
| Parent involvement | |
| Father initiatives | |
| Community Partnerships | Collaborative relationships with other service providers |
| Advisory committees | |
| Transition plans | |
| Staff Development | Supervision |
| Training | |
| Staff turnover | |
| Compensation | |
| Staff morale | |
| Management Systems and Procedures | Policy council |
| Goals, objectives, and plans | |
| Program self-assessment | |
| Community needs assessment |
| Level | Definition | |
|---|---|---|
| Partial Implementation | ||
| 1 | Minimal implementation | Program shows little or no evidence of effort to implement the relevant program element. |
| 2 | Low-level implementation | Program has made some effort to implement the relevant program element. |
| 3 | Moderate implementation | Program has implemented some aspects of the relevant program element. |
| Full Implementation | ||
| 4 | Full implementationa | Program has substantially implemented the relevant program element. |
| 5 | Enhanced implementation | Program has exceeded expectations for implementing the relevant program element. |
| a We use the term "full implementation" throughout this report as a research term. (back) | ||
c. Implementation Rating Process
We designed a consensus-based approach to assigning implementation ratings to each Early Head Start research program. Following our 1997 site visits, we assembled a rating panel of four national evaluation team members and two outside experts. Each rating panel member was given responsibility for rating a subset of the research programs. For each program, the site visitor and two panel members reviewed the extensive documentation in more than 50 pages of checklists and written materials, and assigned ratings independently based on the program profile and the checklist. Once these independent ratings were completed for all programs, the rating panel met in May 1998 to review the three sets of ratings produced for each program, discuss differences in ratings across panel members, and assign consensus ratings for each program. During the course of this process, the rating panel made minor modifications to the rating scales to clarify ambiguities and create clearer distinctions between scores in some areas. The analyses of the ratings we present in this report are based on the consensus ratings assigned in May 1998 by the rating team.
After we completed the rating process, we checked the validity of the consensus-based implementation ratings by comparing them to independent ratings. After the Head Start Bureau completed monitoring visits to all 17 research programs in spring 1998, we asked a member of the monitoring team to use information collected during the monitoring visits to rate the programs' implementation using the rating scales we developed. We did not share our rating results or information collected during our site visits with the monitoring team. The ratings assigned by the monitoring team member were very similar to those assigned by our rating panel and confirmed that our ratings provide a good assessment of program implementation.
REFERENCES
Brooks-Gunn, J., F.R. Liaw, R.T. Michael, and E.S. Zamsky. "Manual for Coding Freeplay Parenting Styles: From the Newark Observational Study of the Teenage Parent Demonstration." Unpublished coding scales. New York: Teachers College, Columbia University, 1992.
Harms, T., D. Cryer, and R. Clifford. Infant-Toddler Environment Rating Scale. New York: Teachers College Press, 1990.
NICHD Early Child Care Research Network. "The Effects of Infant Child Care on Infant-Mother Attachment Security: Results of the NICHD Study of Early Child Care." Child Development, vol. 68, 1997, pp. 860-879.
NICHD Early Child Care Research Network. "Chronicity of Maternal Depressive Symptoms, Maternal Sensitivity, and Child Functioning at 36 Months." Developmental Psychology, vol. 35, 1999, pp. 1297-1310.
Owen, M.T. "Qualitative Ratings of Mother-Child Interaction at 15 Months: Prepared for the NICHD Study of Early Child Care." Unpublished coding scales, 1992.
Owen, M.T., C. Norris, M. Houssan, S. Wetzel, J. Mason, and C. Ohba. "24-Month Mother-Child Interaction Rating Scales for the Three Boxes Procedure." Paper presented at the NICHD Study of Early Child Care Research Consortium, September 1993.
Ware, A., C. Brady, C. O'Brien, and L.J. Berlin. "14-Month Child-Parent Interaction Rating Scales for the Three Bag Assessment." New York: Center for Children and Families, Teachers College, Columbia University, 1998.
| Table of Contents | Previous | Next |