Table of Contents | Previous |
APPENDIX B
PROCEDURES FOR TRAINING AND ESTABLISHING RELIABILITY ON THE CLASSROOM OBSERVATION QUALITY MEASURES
Mathematica Policy Research (MPR) trained more than 80 observers to collect child care quality data in preparation for the first set of observations (conducted when the children were 14 months old). Prior to attending centralized training sessions, observers reviewed detailed training manuals that described the instruments and study procedures. The first day of training included a combination of lecture, interactive discussion, and viewing and discussion of the Infant-Toddler Environment Rating Scale (ITERS) or Family Day Care Rating Scale (FDCRS) videotaped training materials. On the following two days of training, MPR group leaders (who had established inter-rater reliability with two of the instruments’ developers, Thelma Harms and Debby Cryer, prior to the training session) accompanied small groups of observers into the local community to conduct one center and one family child care observation. After the observation, the group leader answered questions about any information needed to score items that could not be observed. All observers independently scored each item. The group leader then facilitated a discussion of each item, spending time discussing items on which there was disagreement about the score. The group came to a consensus score for each item and the group leader computed the percent agreement for each observer against the consensus score. To be certified to collect Early Head Start child care observation data, observers had to have rated items within one scale point of the consensus score on 80 percent of the ITERS, FDCRS, and Arnett Caregiver Interaction Scale (CIS) items. After returning to their sites, observers were also required to conduct one center and one family child care within-site reliability practice visit with another observer from their site and again meet the certification requirements. MPR reviewed the reliability data from those visits. Almost all observers met the certification criteria in their first attempt. We worked with those who did not to further explain items and indicators and asked the observers to conduct additional practice visits until they met the certification requirements of establishing reliability on two post-training observations.
Given that the main child care observation measures did not change from 14 to 24 months, we focused the 24-month child care data collection training on preparing observers to conduct the Child-Caregiver Observation System (C-COS). Following one day of classroom instruction on the C-COS, five MPR group leaders led groups of head trainers/lead observers from each site in conducting visits to community child care settings to establish reliability on the ITERS, FDCRS, and Arnett CIS and to practice the C-COS in the context of a full child care quality observation. After training and any local training of additional observers, we also asked them to conduct two within-site reliability visits. Again, very few observers did not meet the reliability criteria. For the C-COS, we developed test videotapes and compared observers’ codes with the codes developed by a team of gold standard coders to establish reliability. For the 36-month observations, we adopted a site-based training model and distributed detailed training manuals and videotapes to the sites, where experienced local site coordinators conducted the training. Training in conducting the Early Childhood Environment Rating Scale-Revised (ECERS-R) followed the same pattern described for the ITERS and FDCRS. The criteria for certification were the same as described above and almost all observers met them on the first attempt. As before, if an observer did not meet the criteria, we worked with him or her to answer any questions and required that he or she conduct additional practice reliability visits until they met the criteria.
| Table of Contents | Previous |