Table of Contents | Previous | Next |
Appendix C: Pilot Tests of Program Rating Scales in Early Head Start Site Visits
The purpose of the site visits was twofold. They (1) provided in-depth information to Taugment survey findings, and (2) pilot-tested measures that might be useful in future research or by programs themselves as part of their self-assessments. We selected two instruments—an implementation rating scale and a work climate survey—for use in the site visits, and we describe their content and basic psychometric properties here. We stress that the sample sizes are quite small, and psychometrics should be interpreted with this in mind.
IMPLEMENTATION RATING SCALE
We adapted a checklist of program implementation ratings developed as part of the Early Head Start evaluation (ACYF 2000a). The implementation rating scale identified specific criteria for determining the degree to which programs implemented each of five program areas. These are based on the Head Start Program Performance Standards and linked to the performance measures framework. They represent cornerstones that underlie the entire Early Head Start program model; see Chapter I, Figure I.1. The cornerstones are child development, family development, staff development, community building development, and management systems and procedures (Table C.1). Site visitors asked program directors to complete the checklists, then reviewed the answers with them as part of the closing interview.
Within each of the cornerstones were listed specific items to be rated on a scale from minimal implementation (level 1) to enhanced implementation (level 5). Each point was anchored with concrete descriptions of program features necessary to achieve a given implementation level.
RATING RESULTS
We considered ratings on individual elements between levels 1 through 3 (assigned by program directors) to have reached partial implementation and elements directors rated at levels 4 or 5 to have reached full implementation for that element (Table C.2).
| Program Component | Program Element |
|---|---|
| Child Development and Health | Frequency of child development services |
| Developmental assessments | |
| Follow-up services for children with disabilities | |
| Health services | |
| Child care | |
| Parent involvement in child development services | |
| Individualization of services | |
| Group socializations (for home-based and mixed approach programs) | |
| Family Development | Individualized family partnership agreements |
| Availability of services | |
| Frequency of regular family development services | |
| Parent involvement | |
| Community Building | Collaborative relationships |
| Advisory committees | |
| Transition plans | |
| Staff Development | Supervision |
| Training | |
| Turnover | |
| Compensation | |
| Morale | |
| Management Systems | Policy council |
| Communication systems | |
| Goals, objectives, and plans | |
| Self-assessment | |
| Community needs assessment |
| Level | Definition | ||
|---|---|---|---|
| Partial Implementation | 1 | Minimal Implementation | Program shows little or no evidence of effort to implement the relevant program element. |
| 2 | Low-Level Implementation | Program has made some effort to implement the relevant program element. | |
| 3 | Moderate Implementation | Program has implemented some aspects of the relevant program element. | |
| Full Implementationa | 4 | Full Implementation | Program has substantially implemented the relevant program element. |
| 5 | Enhanced Implementation | Program has exceeded expectations for implementing the relevant program element. | |
| aThe term “full implementation” is a research term to reflect our judgment that a program had achieved a rating of 4 or 5. We recognize that programs not “fully” implemented were nevertheless often implementing many features of the performance standards. In addition, even when rated as “fully” implemented, programs may have been striving to do more and be involved in continuous improvement activities. |
We used criteria developed from the evaluation study (ACYF 2002a) to identify programs at different levels of implementation, but our main purpose was to examine how the instrument performed on this small sample of 17 programs. The rating scales were originally developed for use by site visitors on in-depth visits to programs. Here, our interest is to see what we can learn from self report. For each site, we calculated implementation ratings for each of the five cornerstones and for overall implementation. We stress that individual element ratings were based on directors’ responses; we assigned composite (cornerstone and overall) ratings using a set of rules that we define below.
Most visited programs are fully implemented in one or more of the cornerstone areas. Within the child development cornerstone, programs range from low-level implementation to enhanced implementation. Programs are most successful at fully implementing the community building cornerstone (88 percent), and 82 percent of visited programs reached full implementation for management systems and procedures. Most programs (76 percent) have fully implemented the child development cornerstone. Similarly, the majority of programs (65 percent) reached full implementation for the family development cornerstone. Just more than half of the programs (59 percent) reached full implementation for the staff development cornerstone.
To assess overall implementation, we followed these guidelines:1
-
Low-Level Implementation: Programs that reached only a low level of implementation had achieved moderate implementation in only one or two program areas. Other programs areas are poorly or minimally implemented.
-
Moderate Implementation: Programs are fully implemented in a few program areas and moderately implemented in the other areas, moderately implemented in all areas, moderately implemented in most areas with low-level implementation in one area, or fully implemented in every area except child development and health services.
-
Full Implementation: Programs that are fully implemented overall were rated as fully implemented in most of the five component areas.
-
Enhanced Implementation: Programs with full implementation in all areas and that exceed the standards in some of the component areas demonstrate enhanced implementation.
Visited programs range from low to enhanced implementation. We assigned programs overall ratings on a case-by-case basis by assessing the ratings for each of the five categories. Just over half (nine programs, or 53 percent) of the visited programs reached full implementation, and, of the fully implemented programs, three have enhanced overall implementation. We rated only one program as having low-level implementation. This program reached moderate to full implementation on all of the five categories except the child development cornerstone. Less than half of visited programs (41 percent) were moderately implemented. Although these programs were not fully implemented, they all have moderate to full implementation in the child development category. Figure C.1 illustrates the range of implementation among the 17 sites.
PROGRAM DIRECTOR IMPRESSIONS
In our reviews of implementation ratings that program directors gave their own programs, many expressed positive opinions about the implementation rating scale. Positive comments included finding most of the descriptors for each rating anchor to be clear and easy to understand. More critical comments about the scale included some confusion about how to rate particular areas when they met some but not all of the requirements for a given rating. A few directors requested clarification about “family development services” and examples of these services. Some program directors were displeased that they met the standards but still could not meet the highest implementation rating (enhanced implementation often goes beyond the performance standards). Most directors, despite finding the instrument interesting and thought provoking, did not indicate that they would use it for program assessment or improvement activities (or, if so, only as part of a much larger process). By and large, directors are satisfied with the instruments they currently use and are not interested in making changes to them.
[D] |
RELATIONSHIP OF IMPLEMENTATION RATINGS TO SURVEY DATA
We examined survey data between fully implemented and partially implemented programs, based on their self rating. This analysis is very preliminary and based on only a small number of programs. In general, we found no differences in the type of community served, but we did find a few differences in program and family features (not shown). Specifically, fully implemented programs are more likely to have a combination model than partially implemented programs. They are also less likely to have management turnover, although they tend to have high enrollee turnover and to serve enrollees with lower levels of risk than partially implemented programs.
WORK CLIMATE SURVEYS
Work climate surveys were adapted from the Support Subscale of the Policy and Program Management Inventory (Lambert et al. 2001) and were intended to be a quick and easy way to identify areas of concern for staff in the workplace. We created a version of the survey for teachers and a parallel one for home visitors. The primary difference between the instruments is wording specific to “classroom” or “home visit.” During discussion groups with teachers and/or home visitors, site visitors asked the group to complete the relevant instrument. Our aim here is to describe the internal consistency reliability of the scales and to provide basic descriptive summary scores for each program and overall. We conducted analyses separately for each group (teachers and home visitors). The same caution about small sample sizes applies here as well, because each site had multiple teachers or home visitors, the total number of completed surveys was 56 teacher surveys and 36 home visitor surveys among the 17 programs.
The surveys were brief (17 and 18 items for teachers and home visitors, respectively) and asked for ratings of agreement with statements on a five-point scale from 1 (strongly disagree) to 5 (strongly agree). All items were worded so that higher ratings indicate more positive impressions of the work climate.
ANALYTIC PROCESS
Before calculating total scores for individuals and then for sites, we ensured that each item contributed to the quality of the overall rating. We assessed internal consistency by calculating an overall alpha for all survey items. The standardized alphas for both teacher and home visitor surveys were very high (.97 for all 17 items on the teacher survey and .96 for 18 items on the home visitor survey). We created total scores by summing all items in the scale, then calculating total scores for teachers and for home visitors and averages within programs.
DESCRIPTIVE STATISTICS
Next we present overall means and standard deviations for all teacher and home visitor survey items, on the individual level, and the average site score (Tables C.3 through C.6).
STAFF IMPRESSIONS OF THE WORK CLIMATE SURVEYS
Staff generally found the surveys easy to use and took little time to complete them. Many seemed to think the surveys could be useful for program self-assessment. Specifically, home visitors were most enthusiastic about the instrument and found it useful because it easily illustrated problem areas. On a few occasions, however, staff said the questions were too specific and did not capture their overall feelings about the program, and that honest individual ratings could give the mistaken overall impression that they were unhappy in their jobs. We subsequently added an open-ended question to ask about overall job satisfaction and whether the ratings accurately reflected their feelings about their job. In general, staff who liked the work climate survey liked it because it provided an opportunity to think about a variety of situations.
Because of the low variability in work climate ratings across sites, we are unable to examine survey responses in light of these ratings.
| Survey Item | Mean | Standard Deviation |
|---|---|---|
| Helps teachers feel good about their jobs | 3.7 | 1.3 |
| Promotes teamwork among teachers | 3.8 | 1.2 |
| Helps teachers feel that they are part of a team | 3.6 | 1.3 |
| Ensures that teachers do not feel isolated | 3.5 | 1.2 |
| Provides enough assistance to teachers in the classroom | 3.4 | 1.2 |
| Provides orientation to new teachers | 4.0 | 1.1 |
| Helps new teachers adjust to the classroom | 3.7 | 1.2 |
| Knows what teachers deal with in the classroom | 3.2 | 1.2 |
| Has timely delivery of materials for use in classrooms | 3.4 | 1.3 |
| Provides opportunities for teachers to identify their strengths and weakness | 3.5 | 1.1 |
| Provides an atmosphere that is free from destructive gossip | 3.3 | 1.4 |
| Provides freedom for teachers to create their own unique classrooms | 4.0 | 1.1 |
| Has clear guidelines for ordering classroom materials efficiently | 3.4 | 1.3 |
| Provides appropriate and supportive supervision | 3.6 | 1.2 |
| Allows teachers input into planning curriculum | 3.9 | 1.1 |
| Helps teachers to work effectively with children with disabilities | 3.9 | 1.3 |
| Provides useful professional development training | 3.9 | 1.3 |
| Average Site Score | 3.6 | 0.9 |
| Sample size = 56 teachers. |
| Site | Average Site Score |
|---|---|
| S01 | 4.3 |
| S02 | 4.6 |
| S04 | 3.8 |
| S05 | 4.2 |
| S06 | 2.4 |
| S09 | 1.7 |
| S10 | 3.8 |
| S11 | 3.4 |
| S12 | 4.8 |
| S13 | 4.5 |
| S16 | 3.4 |
| S17 | 3.1 |
| Note: Not all sites employ teachers. |
| Survey Item | Mean | Standard Deviation |
|---|---|---|
| Helps home visitors feel good about their jobs | 4.3 | 0.8 |
| Promotes teamwork among home visitors | 4.3 | 1.0 |
| Helps home visitors feel that they are part of a team | 4.2 | 1.0 |
| Ensures that home visitors do not feel isolated | 4.0 | 1.1 |
| Provides enough assistance to home visitors | 4.0 | 1.1 |
| Provides orientation to new home visitors | 4.1 | 1.1 |
| Helps new home visitors adjust to their jobs | 4.0 | 1.1 |
| Knows what home visitors deal with in working with families | 4.0 | 1.1 |
| Has timely delivery of materials for use in home visits | 4.1 | 1.0 |
| Provides opportunities for home visitors to identify their strengths and weaknesses | 4.2 | 0.9 |
| Provides an atmosphere that is free from destructive gossip | 3.8 | 1.1 |
| Provides freedom for home visitors to create their own approaches to working with families | 4.5 | 0.8 |
| Has clear guidelines for ordering home visiting materials efficiently | 3.8 | 1.2 |
| Provides appropriate and supportive supervision | 4.1 | 1.1 |
| Allows home visitors input into planning home visit activities | 4.6 | 0.8 |
| Helps home visitors to work effectively with children with disabilities | 4.2 | 0.8 |
| Provides useful professional development training | 4.1 | 1.0 |
| Enables home visitors to access services outside of Early Head Start for families who need them | 4.5 | 0.6 |
| Average Site Score | 4.1 | 0.7 |
| Sample size = 36 home visitors. |
| Site | Average Site |
|---|---|
| S03 | 4.3 |
| S05 | 3.3 |
| S06 | 2.7 |
| S07 | 4.8 |
| S08 | 4.0 |
| S12 | 4.6 |
| S14 | 4.2 |
| S15 | 4.3 |
| S17 | 2.9 |
| Note: Not all sites employ home visitors. |
1 These guidelines were developed by the December 2002 Pathways to Quality and Full Implementation in Early Head Start Programs study. (back to footnote 1)
| Table of Contents | Previous | Next |