Table of Contents | Previous | Next |
B.1 DATA COLLECTION
a. National and Local Research Roles
The national contractor team (MPR and Columbia) was responsible for all aspects of preparation for data collection, tracking of interview status, data entry, quality control, coding of interview responses, coding of parent-child interaction videotapes, and data analysis. Preparation for data collection included nominating evaluation measures, creating and distributing interviews, writing operations and training manuals, conducting centralized training sessions for staff from all 16 sites (2 programs were located in one city, so one research team conducted the data collection for both), certifying that data collectors met the quality and reliability standards set for each measure, providing assessment materials, and notifying local data collection teams when families were to be interviewed. MPR’s tracking of interview status included requiring the local teams to send biweekly updates on the data collection status of families with open interview “windows,” working with the sites to assist in locating hard-to-reach families, and conducting regular telephone meetings with the sites to review the biweekly reports.
In addition to conducting their own research, the local research teams were responsible for hiring a site coordinator as the key person to work with MPR on the cross-site data collection, hiring data collectors, locally supervising the data collection team, conducting all interviews and assessments, tracking interview status, and sending the data to MPR for processing. Sites decided how they staffed the data collection, and data collection team personnel varied, with some staff members working full-time and some part-time. We began with two data collection roles at each site: (1) interviewer/assessors (IAs) were hired with the primary responsibility of conducting the birthday-related parent interviews, child assessments, and parent-child videotaped assessments; (2) community/family coordinators (CFCs) were designated to conduct the follow-up parent services interviews using the Computer-Assisted Personal Interviewing (CAPI) technique. Individuals with a variety of experiences assumed data collector roles, including graduate students, professional interviewing staff, and members of the local community. In some sites the site coordinators collected data themselves, and in other sites they did not.
b. Interviewer Training, Certification, and Reliability
Interviewer Training. The national team conducted group training for local research staff members (site coordinators, CFCs, and IAs) who conducted the Parent Services Interviews (PSI), Parent Interviews (PI), and Child and Family Assessments. Training sessions for the 6-month PSI, the 14-month PI, and the 14-month Child and Family Assessments were conducted in August 1996 and during several smaller sessions throughout the first year of data collection to accommodate different data collection schedules at the sites, as well as to respond to staff turnover. Training sessions were approximately 3 days long for CFCs conducting the 6-month PSI, and 5 days long for IAs conducting the 14-month PI and the Child and Family Assessments. Site coordinators conducted all the 15- and 26-month PSI training locally. In July 1997, we conducted a four-day training session for the 24-month PI and Child and Family Assessments. Representatives from each site were required to attend. The site coordinators conducted all subsequent 24-month training locally. For all centralized training sessions, we asked CFCs and IAs to review the training manual prior to training and prepare to participate in group lectures and discussions, hands-on practice, and taping of practice administrations. All 36-month PI and Child and Family Assessment training was conducted at the local research sites by the site coordinators. MPR prepared training materials and videotapes and site coordinators worked with IAs to train staff and prepare them for certification.
Interviewer Certification and Reliability. After training, we required CFCs and IAs to conduct practice interviews and assessments and submit audiotapes or videotapes to the national team for certification. The mode of administration, initial certification requirements, and ongoing reliability procedures for each type of interview are described in this section.
- Parent Services Interview. CFCs conducted the PSIs by CAPI. Most of the interviews were conducted by telephone, but CFCs visited families in their homes if a telephone interview was not possible. CFCs were required to practice using CAPI with nonrespondents and conduct a mock interview with their site coordinator. The site coordinator reviewed the completed interview on the computer and sent an audiotape of the practice interview and the diskette containing the interview data to MPR for review. CFCs were certified to collect data from respondent families if the mock interview was administered correctly. If a CFC was not certified on their first attempt, we asked them to practice and conduct another mock interview until they met the certification requirements. After a CFC was certified, site coordinators monitored every fifth interview until the CFC reached her/his 25th. Beyond the 25th interview, site coordinators monitored one audiotaped interview every month and one live interview every 6 months.
- Birthday-Related Measures. IAs conducted the 14-, 24-, and
36-month PI and the family and child assessments (including the Bayley
II, the parent-child videotaped assessments, the MacArthur CDI, PPVT-III,
TVIP, and a modified version of the HOME) in the families’ homes.
Most of the birthday-related interviews and assessments were conducted
in the homes, but if the parent was unable to conduct the interview
and assessments in her/his home, the IA conducted the PI by telephone
and tried to complete the assessments at a different time. The interviews
and assessments were conducted using paper-and-pencil questionnaires.
Bayley Scales. After the 14- and 24-month central training sessions and the 36-month local training, IAs were required to critique and score a videotaped Bayley administration and score a second administration to practice what they learned during training. A team of Bayley trainers and reviewers (expert consultants from New York University) provided feedback on the practice exercises. IAs were asked to practice the Bayley and the videotaped parent-child protocol with families who were not part of the evaluation. - Father Study Interview. Twelve of the 17 research sites participated
in the father study. Eleven of the sites conducted the 24- and 36-month
father interview and one site conducted an abbreviated interview. The
father interview was administered after the PI was completed with the
child’s primary caregiver. The primary caregiver (the mother in
over 96 percent of the families) identified whether the biological father
lived with the child or saw the child regularly. If the biological father
did not live with the child, the IA determined whether there was a father
figure. If the mother identified both an involved nonresident biological
father and a father figure, the IA asked the mother which man was more
involved with the child. If the mother did not object to having the
father contacted, the IA reported to the site coordinator that there
was an identified father and MPR began tracking the father as a respondent
for the father study. In some sites, the same team of IAs conducted
the father interviews and other sites hired new IAs. The site coordinator
and certified IAs in each site conducted father interview training.
Father study IAs were required to submit audiotapes of the father interview
for review by the national team. Father study IAs had to meet the same
certification and reliability standards as the IAs in the main study.
Father-Child Videotaped Assessments. After training and practice with at least two fathers who were not part of the evaluation, IAs were required to submit one videotape to MPR for review. A team of experts from MPR and Columbia reviewed the tapes and scored the interviewer on the administration of the protocol instructions, timing of the activities, and videography. IAs were certified to collect data with study fathers if they met the certification criteria established by the review team. If an IA did not meet the criteria, he/she was asked to submit another practice tape and self-critique for review. The review team provided feedback to IAs about the video protocol for approximately every 15th administration.
After a minimum of two practice administrations, IAs submitted a videotaped Bayley administration, a self-critique, the score sheet, and the completed behavior rating scale for review. The Bayley trainers and reviewers provided written feedback for two administrations per IA and determined whether the IA met our certification criteria of 85 percent reliability on administration and scoring. If an IA did not meet the certification criteria, he/she was asked to practice and resubmit. All IAs were required to meet the certification requirements before they collected data with study children. To ensure reliability of administration, IAs were required to videotape every 15th Bayley and submit it and a self-critique to MPR for review. Our Bayley trainers and reviewers found that most IAs met the certification criteria throughout data collection. If an IA did not, he/she was asked to review the feedback from the reviewer and conduct another Bayley with a child who was not part of the study. Usually the IA did not require more than one practice administration to reestablish reliability for the Bayley administration and scoring.
Parent-Child Videotaped Assessments. After training and practice with at least two families who were not part of the evaluation, IAs were required to submit one videotape to MPR for review. A team of experts from MPR and Columbia reviewed the tapes and scored the interviewer on the administration of the protocol instructions, timing of the activities, and videography. IAs were certified to collect data with study families if they met the certification criteria established by the review team. If an IA did not meet the criteria, he/she was asked to submit another practice tape and self-critique for review. The review team provided feedback to IAs about the video protocol for approximately every 15th administration.
PPVT-III/TVIP. As part of the local 36-month training, IAs studied the PPVT-III and the TVIP. They completed practice scoring exercises and were asked to conduct practice administrations with adults and with children who were not part of the research. Site coordinators were asked to monitor practice administrations and determine whether the IA met the criteria for certification. MPR staff members reviewed the scoring for the first two administrations each IA completed and provided feedback as necessary.
Other Measures. As part of the field monitoring of the practice administrations of the PI, Bayley, and videotaped assessments, the site coordinators determined whether the IAs were certified on the PI, which included the MacArthur CDI (completed at 14 and 24 months by the parent as a self-administered questionnaire or administered by the interviewer according to the parent’s preference) and the modified version of the HOME. To determine whether IAs were ready to conduct the interviews and assessments with study families, site coordinators were asked to assess the flow of the interview, transitions between components of the PI and the assessments, rapport with family and child, and completeness and accuracy of the interview and assessment documents.
Data collectors were not informed of families’ program status; however, if families shared information that revealed their program status or kept Early Head Start materials in their homes, data collectors may have learned of some families’ status by the time of the final assessments.
c. Data Collection Windows, Tracking, and Receipt Control
Data Collection Windows. Site coordinators were required to monitor the data collection window for each family for all the interviews and assessments. MPR generated contact sheets and advance letters for every family and sent them to the sites. The contact sheet included contact information for the family, the dates between which the interview was to be completed (the “window”), space to code the status of the interview, and space to record attempts to reach the family. All windows opened 4 weeks before the target date of the interview (targeted for 6, 15, and 26 months after random assignment for the PSIs, and the date of the child’s 14-, 24-, and 36-month “birthday” for the birthday-related interviews and assessments). See Table B.1 for the target length of the windows by type of interview.
Timing of Interviews/Assessments by Child’s Age and Months Since Random Assignment. Table B.2 gives a summary of the distribution of months between the target date and the completion of the 26-month PSI and the 36-month PI by research status. On average, the 26-month PSI was conducted about 28 months after random assignment, and the 36-month PI was conducted when the children were 37 months old (overall there were no differences by research status).
| Data Collection Instrument | Window |
| 6-Month PSI (Parent Services Interview) | 5 months to 11 months and 30 days |
| 14-Month PI (Birthday Related Parent Interview) | 13 months to 19 months and 30 days |
| 14-Month Parent-Child Videotaped Assessments and Bayley | 13 months to 16 months and 30 days |
| 15-Month PSI | 14 months to 22 months and 30 days |
| 24-Month PI/Parent-Child Videotaped Assessments and Bayley | 23 months to 28 months and 15 days |
| 24-Month Father Interview/Father-Child Videotaped Assessments | 23 months to 31 months and 30 days |
| 26-Month PSI | 25 months to 33 months and 30 days |
| 36-Month Parent-Child Videotaped Assessments, Bayley, and PPVT-III | 35 months to 38 months and 30 days |
| 36-Month Father Interview/ Father-Child Videotaped Assessments | 35 months to 43 months and 30 days |
| Number of Months | 26-Month Parent Service Interviews | 36-Month Parent Interviews | ||||
|---|---|---|---|---|---|---|
| Program Group |
Control Group |
Combined Sample |
Program Group |
Control Group |
Combined Sample |
|
| -3 to -1 | 2.3 | 1 | 1.6 | 2.2 | 2.6 | 2.4 |
| -1 to -.5 | 9.1 | 7.4 | 8.2 | 10.5 | 11.9 | 11.1 |
| -.5 to 0 | 8.6 | 11.2 | 9.9 | 12.1 | 11.3 | 11.7 |
| 0 to .5 | 12.5 | 11.3 | 11.9 | 13.3 | 13.3 | 13.3 |
| .5 to 1 | 9.6 | 9.9 | 9.7 | 10.2 | 11.7 | 10.9 |
| 1 to 2 | 16.4 | 16.3 | 16.3 | 19.8 | 16.6 | 18.3 |
| 2 to 3 | 9.3 | 12.1 | 10.7 | 15.6 | 15.6 | 15.6 |
| 3 to 4 | 6.9 | 8.2 | 7.6 | 8.1 | 7.7 | 7.9 |
| 4 or Greater | 25.3 | 22.7 | 24 | 8.2 | 9.6 | 8.9 |
| Average Number of Months | 2.3 | 2.5 | 2.4 | 1.4 | 1.4 | 1.4 |
Tracking of Interview Cooperation Rates. When the interview window was open, MPR and the site coordinators worked together to develop strategies to increase interview completion rates. Site coordinators reported interview status to MPR and participated in phone meetings with MPR staff members to review data collection issues and update tracking information. For interviews that were difficult to complete or families that were hard to locate, the site coordinator requested assistance from MPR that included the search of locating data bases and telephone or, in some sites, field support from a trained MPR specialist in locating families.
Receipt Control. Completed birthday-related interviews and assessments were reviewed by site coordinators and any data edits were conducted at the site as necessary before the materials were sent to MPR. Site coordinators sent regular shipments to MPR of CAPI diskettes containing the PSIs, originals of the PI, and videotapes. MPR staff logged the materials into the tracking database and prepared the interview and assessment materials for data entry.
d. Data Processing, Data Entry, and Quality Control
Data Processing. MPR staff copied the parent-child videotapes and sent them to the Columbia University team for coding. MPR and the site coordinator compared logs of materials sent by the sites and received by MPR to ensure that all the data had been received. CAPI diskettes were downloaded and included in a database organized by a unique family identification number. To protect families, any documents that included both the family identification number and the family contact information were kept in locked files.
Data Entry and Quality Control. Prior to data entry, all paper-and-pencil instruments were reviewed by quality control staff for any problems with the skip logic and other interview administration errors. All paper-and-pencil instruments were data entered with 100 percent verification into data entry programs with prescribed ranges for each item. For the PSIs, automatic range checks and skip patterns were part of the CAPI programming to reduce data collection and data entry errors. For questions that required or provided an option for the parent to specify her/his response, we recoded responses or developed codes to classify responses and included them as additional values if 10 or more respondents gave the same answer.
B.2 SOURCES OF NONRESPONSE
All multisite evaluations of the size and complexity of Early Head Start face a variety of data collection and analytic challenges that affect the overall and site-level response rates. This study is no different. Overall response rates, response rates by site and by data source, and response rates by evaluation subgroups are presented and discussed in Chapter II. Here we describe the nature of the nonresponse.
The primary sources of nonresponse were refusals to participate and inability to locate the families. Overall for the 15-month PSI, 45 percent of the families who did not respond refused to participate, and 49 percent moved or could not be located (the remaining 6 percent included families for whom the interview window closed before the interview was completed. For the 24-month PI, 51 percent of the families who did not respond refused to participate, and 44 percent moved or could not be located (the remaining 5 percent included families for whom the interview window closed before the interview was completed). Overall for the 26-month PSI, 41 percent of the families who did not respond refused to participate, and 52 percent moved or could not be located (the remaining 7 percent included families for whom the interview window closed before the interview was completed). For the 36-month PI, 46 percent of the families who did not respond refused to participate, and 51 percent moved or could not be located (the remaining 3 percent included families for whom the interview window closed before the interview was completed).
In addition to these more typical sources of nonresponse, unfortunately 21 children died during the course of the study (12 children in the control group, and 9 in the Early Head Start group). Over half of the deaths were miscarriages or stillbirths and we do not have complete data on age and cause of death for the remaining children. Three children were adopted after random assignment. No further data collection was attempted with families of deceased or adopted children.
Site coordinators reported that the data collection was very challenging. From the beginning of the project, some site coordinators reported that some families had not understood what they were signing up for (related to the program, the research activities, or both), and some site coordinators reported that control group families refused participation in the study after they learned that they were not going to receive Early Head Start services.
Analysis of the categories of nonresponse by site showed that the center-based sites were more successful in completing interviews and assessments with Early Head Start families than they were with the control group families. One explanation for this is that the Early Head Start families were using center-based services and may have been easier for research and program staff members to contact. To some degree, the same pattern might have been expected across all the programs—if the local research team used all available leads, they may have been able to contact and successfully complete interviews with a larger proportion of the Early Head Start group than the control group. This was not true across all sites, and in a number of sites research teams completed a larger proportion of the interviews with control group families.
In general, the PI response rate establishes the maximum for the Bayley, PPVT-III, TVIP, and parent-child videotaped assessment response rates. This is because if an interview was not done, it was generally the case that the other assessments also were not done. In some sites, IAs completed the PI by telephone if the interview window was about to close or if the family moved away, rather than lose the entire data collection wave for the family. In those cases it was impossible to conduct the Bayley, PPVT-III, TVIP, and the parent-child videotaped assessments. Sites reported other data collection-related reasons for nonresponse on the Bayley, PPVT-III, TVIP, and the parent-child videotaped assessment, including child illness on the interview date, child refusal to participate in the Bayley or PPVT-III, TVIP, assessment or the videotaped assessments, parental refusal to participate in the videotaped assessments, and insufficient time during the visit to complete the assessments.
Some of the data that were collected could not be used because of technical problems or errors in administration of the assessment. Between 3 and 8 percent of the 1,854 24-month videotapes and between 2 and 3 percent of the 1,701 36-month videotapes sent to Columbia for coding could not be coded because of incorrect administration of the parent-child assessments, lack of video or sound, or other technical problems. Nine percent of the 1,950 24-month Bayley assessments and 7 percent of the 1,793 36-month assessments conducted could not be scored because of errors in administration of the test or the lack of a basal.
B.3 FATHER STUDY RESPONSE RATES
The father study data in this report are from interviews conducted with fathers or father figures of children in the program and control groups. As described above, the 12 father study sites recruited the men after the mothers identified them either as a resident biological, an involved nonresident biological, or a father figure. Here we report updated response rates using the complete sample of 24-month interviews as well as those for 36 months. Response rates at 24-months are slightly lower than were reported previously, because originally we reported only completed interviews that had been received from the sites. After the sites sent in the final cases, we were able to compute final response rates. Across the sites at 24 months, approximately 76 percent of interviewed mothers identified a father or father figure. Of those who were identified, we were able to interview 727, or 62 percent of them. At the 36-month interview, we also asked mothers to identify a father or father figure and for permission to interview him. Across sites, 73 percent of interviewed mothers identified a father or father figure, of whom 698, or 64 percent were interviewed. Father study sample sizes and response rates at 24 and 36 months, by site are included in Table B.3.
Father Interview Response Bias. We examined baseline characteristics of families that had a father or father figure interviewed at either 24 or 36 months, and those that did not. We examined the following characteristics (unless noted, all were in reference to the mothers’ characteristics at baseline): teenage mother, race/ethnicity, education, living arrangement, primary occupation, and child’s sex. In most cases, t-tests of the proportions of fathers interviewed and not interviewed at each point showed significant differences in baseline characteristics between families with and without interviewed fathers. At 24 months, there were no differences in the proportions of families with a teen mother or with a male child, but there were differences in race, education, living arrangement, and primary occupation. The families with fathers or father figures interviewed at 24 months were generally more advantaged compared to families without an interviewed fathers. Families with interviewed fathers were composed of higher proportions of whites and lower proportions of African Americans, lower proportions with less than a high school education, higher proportions who lived with a spouse and correspondingly lower proportions living alone, and higher proportions in the “other” occupational category (unemployed or out of the labor force by choice). Findings at 36 months were similar, with the families who had interviewed fathers having an even larger proportion of whites and fewer African Americans, lower proportions with less than a high school education and a greater percentage with some college, higher proportions living with spouses, and significantly more who were employed or in the “other” occupational categories at baseline.
| Site | Father Interviews | |||
|---|---|---|---|---|
| 24-Month Sample Size |
Response Rate (Percentage) |
36-Month Sample Size |
Response Rate (Percentage) |
|
| 1 | 51 | 45 | 44 | 37 |
| 3 | 54 | 45 | 30 | 25 |
| 4 | 63 | 57 | 52 | 46 |
| 6 | 36 | 24 | 55 | 40 |
| 8 | 83 | 55 | 96 | 66 |
| 10 | 47 | 51 | 44 | 52 |
| 11 | 30 | 25 | 37 | 27 |
| 13 | 102 | 69 | 101 | 69 |
| 14 | 48 | 44 | 44 | 44 |
| 15 | 71 | 54 | 53 | 43 |
| 16 | 74 | 50 | 82 | 59 |
| 17 | 68 | 46 | 60 | 44 |
| Total | 727 | 62 | 698 | 64 |
aThe response rate was calculated by using the number of fathers identified by mothers during the 24- or 36-month parent interviews as the denominator. |
In addition, compared to their proportion at baseline, there were significantly fewer fathers interviewed at 36 months from families with a teenage mother at baseline. It is necessary to be mindful of the systematic ways that families with interviewed fathers differ from the overall sample of program and control families. Therefore, findings about the interviewed group may not generalize to the larger group of fathers and father figures in families in the entire sample, nor to the population of families eligible for Early Head Start.
We examined baseline characteristics of families with interviewed fathers at 24 and 36 months, to assess the similarity of the Early Head Start and the control groups. We compared proportions of teenage mothers, race/ethnicity, primary occupation, education, living arrangements, and child's gender between program and control families with interviewed fathers at each period. At 24 months, there were a few differences in baseline characteristics between program and control groups. Specifically, the program group had lower proportions of teenage mothers, whites, living arrangements with other adults, and higher proportions living alone compared to the control group. By 36 months, among families with an interviewed father or father figure, the only statistically significant difference was for living arrangements, with program families more likely to have mothers who lived alone at baseline rather than with a spouse or other adults compared to the control group.
| Table of Contents | Previous | Next |