Table of Contents | Previous | Next |
C.5 IMPLEMENTATION MEASURES
The first step to measuring the extent of program implementation is establishing a clear definition of a fully implemented program. For the purposes of this research, we defined the degree of implementation as the extent to which programs offered services that met the requirements of the Early Head Start grant announcement (U.S. Department of Health and Human Services 1995) and selected key elements of the revised Head Start Program Performance Standards (U.S. Department of Health and Human Services 1996). We defined “full implementation” as substantially implementing, or exceeding expectations for implementing, these key program elements.
To assess the extent of program implementation, we developed implementation rating scales, checklists for organizing the information needed to assign ratings to programs, and a rating process. We designed this rating system to help us reduce a large amount of information on program implementation into summary variables for testing hypotheses about how implementation relates to outcomes and to systematically analyze the research programs’ progress toward full implementation over time. This sections describes our data sources, the rating scales we developed, and the rating process we followed for assessing implementation.1
a. Data Sources
For these analyses, we relied primarily on information collected during site visits conducted in fall 1997 and fall 1999 and self-administered surveys completed by program staff at the time of the site visits. To facilitate the systematic assignment of implementation ratings for each program, site visitors assembled the site visit and staff survey information in checklists organized according to key program elements of the performance standards. In addition, site visitors wrote detailed program profiles based on information obtained during the site visits. Program directors and their local research partners reviewed the profiles and checklists for their programs, provided corrections of erroneous information, and in some cases provided additional clarifying information.
b. Implementation Rating Scales
To develop implementation rating scales, we identified specific criteria for determining the degree to which programs implemented Early Head Start’s three major program areas as defined in the performance standards: (1) early childhood development and health services, (2) family and community partnerships, and (3) program design and management. To refine our assessment, we created distinct criteria for both family and community partnerships. Likewise, within program design and management we created separate criteria for staff development and program management systems.
The criteria encompass key program requirements contained in the Early Head Start grant announcement and the performance standards. Because the purpose of the ratings was to identify and track over time the implementation of key program requirements and not to monitor compliance, we focused on key requirements needed to help us identify pathways to full implementation and to summarize and quantify a large amount of qualitative information on program implementation. We reviewed our initial criteria with representatives of the Head Start Bureau and the Early Head Start technical assistance network to ensure that they included the most important subset of program requirements. We also solicited comments from members of the Early Head Start Research Consortium. Table C.5A summarizes the 25 program elements we assessed organized according to program area.
| Scale | Program Element |
|---|---|
| Early Childhood Development and Health Services | |
| Frequency of child development services | |
| Developmental assessments | |
| Follow-up services for children with disabilities | |
| Child health services | |
| Child care services | |
| Parent involvement in child development services | |
| Individualization of child development services | |
| Group socialization activities | |
| Family and Community Partnerships | |
| Family Partnerships | Individualized family partnership agreements |
| Availability of services | |
| Frequency of family development services | |
| Parent involvement | |
| Community Partnerships | Collaborative relationships with other service providers |
| Advisory committees | |
| Transition plans | |
| Management Systems and Procedures | |
| Staff Development | Supervision |
| Training | |
| Staff retention | |
| Compensation | |
| Staff morale | |
| Program Management | Policy council |
| Communication systems | |
| Goals, objectives, and plans | |
| Self-assessment | |
| Community needs assessment | |
Prior to our fall 1997 site visits, we created a rating scale for each of the program elements. In 1999, we made some minor revisions to these scales to reflect clarifications in program guidance from the Head Start Bureau and our evolving understanding of the performance standards, which took effect after our fall 1997 site visits. Each rating scale contains five levels of implementation, ranging from minimal implementation (level 1) to enhanced implementation (level 5) (Table C.5B). We considered programs rated at level 1 through 3 to have reached partial implementation and programs rated at levels 4 and 5 to have reached full implementation of the particular program element rated.
c. Rating Process
Following each round of site visits, we used a consensus-based process to assign implementation ratings to each Early Head Start research program. We assembled a rating panel that included four national evaluation team members, a representative of the Early Head Start technical assistance network, and another outside expert. For each program, three people—the site visitor and two panel members—assigned ratings independently, based information contained in the checklists and program profile compiled by the site visitor. Ratings were assigned for each of the 25 program elements, the five program areas, and for overall implementation. In completing the ratings of overall implementation, we established the following guidelines for creating the overall ratings based on the ratings of the individual program components:
-
Low-Level Implementation: Programs that reached only a low level of implementation had achieved moderate implementation in only one or two program areas. Other programs areas were poorly or minimally implemented.
-
Moderate Implementation: To achieve this rating, programs were (1) fully implemented in a few program areas and moderately implemented in the other areas, (2) moderately implemented in all areas, (3) moderately implemented in most areas with low-level implementation in one area, or (4) fully implemented in every area except child development and health services.
-
Full Implementation: To be rated as fully implemented overall, programs had to be rated as fully implemented in most of the five component areas. Reflecting the Head Start Bureau’s focus on child development, panel members gave special consideration to the rating of child development and health services, and weighted it more heavily in arriving at their consensus rating of overall implementation.
-
Enhanced Implementation: A program demonstrating enhanced implementation was fully implemented in all areas and exceeded the standards in some of the component areas.
| Level | Definition | |
|---|---|---|
| Partial Implementation | ||
| 1 | Minimal Implementation | Program shows little or no evidence of effort to implement the relevant program element |
| 2 | Low-Level Implementation | Program has made some effort to implement the relevant program element |
| 3 | Moderate Implementation | Program has implemented some aspects of the relevant program element |
| Full Implementation | ||
| 4 | Full Implementation | Program has substantially implemented the relevant program element |
| 5 | Enhanced Implementation | Program has exceeded expectations for implementing the relevant program element |
After these independent ratings were completed for all programs, the panel met to review the three sets of independent ratings, discuss differences in ratings across panel members, and assign consensus ratings for each program. We checked the validity of the our 1997 ratings by comparing them to independent ratings. After the Head Start Bureau completed monitoring visits to all 17 research programs in spring 1998, we asked a member of the monitoring team to use information collected during the monitoring visits to rate programs’ using the rating scales we developed. We did not share with the monitoring team our rating results or the information we collected during site visits. The independent ratings assigned by the monitoring team member were very similar to those assigned by our rating panel, providing some validation that our ratings provide a good assessment of program implementation.
1More detailed information about the implementation analysis can be found in Pathways to Quality (Administration on Children, Youth and Families 2002).(back)
| Table of Contents | Previous | Next |