Table of Contents | Previous | Next |
B.1 DATA COLLECTION
a. National and Local Research Roles
The national contractor team (MPR and Columbia) was responsible for all aspects of preparation for data collection, tracking of interview status, data entry, quality control, coding of interview responses, coding of parent-child interaction videotapes, and data analysis. Preparation for data collection included nominating evaluation measures, creating and distributing interviews, writing operations and training manuals, conducting centralized training sessions for staff from all 16 sites (2 programs were located in one city, so one research team conducted the data collection for both), certifying that data collectors met the quality and reliability standards set for each measure, providing assessment materials, and notifying local data collection teams when families were to be interviewed. MPR's tracking of interview status included requiring the local teams to send biweekly updates on the data collection status of families with open interview "windows," working with the sites to assist in locating hard-to-reach families, and conducting regular telephone meetings with the sites to review the biweekly reports.
In addition to conducting their own research, the local research teams were responsible for hiring a site coordinator as the key person to work with MPR on the cross-site data collection, hiring data collectors, locally supervising the data collection team, conducting all interviews and assessments, tracking interview status, and sending the data to MPR for processing. Sites decided how they staffed the data collection, and data collection team personnel varied, with some staff members working full-time and some part-time. We began with two data collection roles at each site: (1) interviewer/assessors (IAs) were hired with the primary responsibility of conducting the birthday-related parent interviews, child assessments, and parent-child interaction videotaping; (2) community/family coordinators (CFCs) were designated to conduct the follow-up parent services interviews using the Computer-Assisted Personal Interviewing (CAPI) technique. Individuals with a variety of experiences assumed data collector roles, including graduate students, professional interviewing staff, and members of the local community. In some sites the site coordinators collected data themselves, and in other sites they did not.
b. Interviewer Training, Certification, and Reliability
Interviewer Training. The national team conducted group training for local research staff members (site coordinators, CFCs, and IAs) who conducted the Parent Services Interviews (PSI), Parent Interviews (PI), and Child and Family Assessments. Training sessions for the 6-month PSI, the 14-month PI, and the 14-month Child and Family Assessments were conducted in August 1996 and during several smaller sessions throughout the first year of data collection to accommodate different data collection schedules at the sites, as well as to respond to staff turnover. Training sessions were approximately 3 days long for CFCs conducting the 6-month PSI, and 5 days long for IAs conducting the 14-month PI and the Child and Family Assessments. Site coordinators conducted all the 15-month PSI training locally. In July 1997, we conducted a four-day training session for the 24-month PI and Child and Family Assessments. Representatives from each site were required to attend. The site coordinators conducted all subsequent 24-month training locally. For all training sessions, we asked CFCs and IAs to review the training manual prior to training and prepare to participate in group lectures and discussions, hands-on practice, and taping of practice administrations.
Interviewer Certification and Reliability. After training, we required CFCs and IAs to conduct practice interviews and assessments and submit audiotapes or videotapes to the national team for certification. The mode of administration, initial certification requirements, and ongoing reliability procedures for each type of interview are described in this section.
- Parent Services Interview. CFCs conducted the PSIs by
CAPI. Most of the interviews were conducted by telephone, but CFCs
visited families in their homes if a telephone interview was not
possible. CFCs were required to practice using CAPI with nonrespondents
and conduct a mock interview with their site coordinator. The site
coordinator reviewed the completed interview on the computer and
sent an audiotape of the practice interview and the diskette containing
the interview data to MPR for review. CFCs were certified to collect
data from respondent families if the mock interview was administered
correctly. If a CFC was not certified on their first attempt, we
asked them to practice and conduct another mock interview until
they met the certification requirements. After a CFC was certified,
site coordinators monitored every fifth interview until the CFC
reached her/his 25th. Beyond the 25th interview, site coordinators
monitored one audiotaped interview every month and one live interview
every 6 months.
- Birthday-Related Measures. As conducted the 14- and 24-month
PI and the family and child assessments (including the Bayley II,
the parent-child videotaped structured play assessment, the MacArthur
CDI, and a modified version of the HOME) in the families' homes.
Most of the birthday-related interviews and assessments were conducted
in the homes, but if the parent was unable to conduct the interview
and assessments in her/his home, the IA conducted the PI by telephone
and tried to complete the assessments at a different time. The interviews
and assessments were conducted using paper-and-pencil questionnaires.
Bayley Scales. After the 14- and 24-month training sessions, IAs were required to critique and score a videotaped Bayley administration and score a second administration to practice what they learned during training. A team of Bayley trainers and reviewers (expert consultants from New York University) provided feedback on the practice exercises. IAs were asked to practice the Bayley and the videotaped parent-child protocol with families who were not part of the evaluation.
After a minimum of two practice administrations, IAs submitted a videotaped Bayley administration, a self-critique, the score sheet, and the completed behavior rating scale for review. The Bayley trainers and reviewers provided written feedback for two administrations per IA and determined whether the IA met our certification criteria of 85 percent reliability on administration and scoring. If an IA did not meet the certification criteria, he/she was asked to practice and resubmit. All IAs were required to meet the certification requirements before they collected data with study children. To ensure reliability of administration, IAs were required to videotape every 15th Bayley and submit it and a self-critique to MPR for review. Our Bayley trainers and reviewers found that most IAs met the certification criteria throughout data collection. If an IA did not, he/she was asked to review the feedback from the reviewer and conduct another Bayley with a child who was not part of the study. Usually the IA did not require more than one practice administration to reestablish reliability for the Bayley administration and scoring.
Parent-Child Structured Play Assessment. After training and practice with at least two families who were not part of the evaluation, IAs were required to submit one videotape to MPR for review. A team of experts from MPR and Columbia reviewed the tapes and scored the interviewer on the administration of the protocol instructions, timing of the activities, and videography. IAs were certified to collect data with study families if they met the certification criteria established by the review team. If an IA did not meet the criteria, he/she was asked to submit another practice tape and self-critique for review. The review team provided feedback to IAs about the video protocol for approximately every 15th administration.
As part of the field monitoring of the practice administrations of the PI, Bayley, and structured play assessment, the site coordinators determined whether the IAs were certified on the PI, which included the MacArthur CDI (completed by the parent as a self-administered questionnaire or administered by the interviewer according to the parent's preference) and the modified version of the HOME. To determine whether IAs were ready to conduct the interviews and assessments with study families, site coordinators were asked to assess the flow of the interview, transitions between components of the PI and the assessments, rapport with family and child, and completeness and accuracy of the interview and assessment documents.
-
Father Study Interview. Twelve of the 17 research sites participated in the father study. Eleven of the sites conducted the 24-month father interview and one site conducted an abbreviated interview. The father interview was administered after the 24-month PI was completed with the child's primary caregiver. The primary caregiver (the mother in over 96 percent of the families) identified whether the biological father lived with the child or saw the child regularly. If the biological father did not live with the child, the IA determined whether there was a father figure. If the mother identified both an involved nonresident biological father and a father figure, the IA asked the mother which man was more involved with the child. If the mother did not object to having the father contacted, the IA reported to the site coordinator that there was an identified father and MPR began tracking the father as a respondent for the father study. In some sites, the same team of IAs conducted the father interviews and other sites hired new IAs. The site coordinator and certified IAs in each site conducted father interview training. Father study IAs were required to submit audiotapes of the father interview for review by the national team. Father study IAs had to meet the same certification and reliability standards as the IAs in the main study.
c. Data Collection Windows, Tracking, and Receipt Control
Data Collection Windows. Site coordinators were required to monitor the data collection window for each family for all the interviews and assessments. MPR generated contact sheets and advance letters for every family and sent them to the sites. The contact sheet included contact information for the family, the dates between which the interview was to be completed (the "window"), space to code the status of the interview, and space to record attempts to reach the family. All windows opened 4 weeks before the target date of the interview (targeted for 6 months after random assignment and 15 months after random assignment for the PSIs, and the date of the child's 14- and 24-month "birthday" for the birthday-related interviews and assessments). See Table B.1 for the target length of the windows by type of interview.
Timing of Interviews/Assessments by Child's Age and Months Since Random Assignment. Table B.2 gives a summary of the distribution of months between the target date and the completion of the 15-month PSI and the 24-month PI by research status. On average, the 15-month PSI was conducted about 16.5 months after random assignment, and the 24-month PI was conducted when the children were 25 months old (overall there were no differences by research status).
Tracking of Interview Cooperation Rates. When the interview window was open, MPR and the site coordinators worked together to develop strategies to increase interview completion rates. At the start of the study, site coordinators reported interview status to MPR every two weeks and participated in phone meetings with MPR staff members to review data collection issues and update tracking information. For interviews that were difficult to complete or families that were hard to locate, the site coordinator requested assistance from MPR that included search data bases and in some sites telephone or field support from a trained MPR specialist in locating families.
Receipt Control. Completed birthday-related interviews and assessments were reviewed by site coordinators and any data edits were conducted at the site as necessary before the materials were sent to MPR. Site coordinators sent regular shipments to MPR of CAPI diskettes containing the PSIs, originals of the PI, and videotapes. MPR staff logged the materials into the tracking database and prepared the interview and assessment materials for data entry.
d. Data Processing, Data Entry, and Quality Control
Data Processing. All interview materials were logged into a data tracking system upon their arrival at MPR from the sites. MPR staff copied the parent-child videotapes and sent them to the Columbia University team for coding. As the data collection effort ended at each site, MPR and the site coordinator compared logs of materials sent by the sites and received by MPR to ensure that all the data had been received. CAPI diskettes were downloaded and included in a database organized by a unique family identification number. To protect families, all data items were organized using the identification number, and any documents that included both the family identification number and the family contact information were kept in locked files.
Data Entry and Quality Control. Prior to data entry, all paper-and-pencil instruments were reviewed by quality control staff for any problems with the skip logic and other interview administration errors. All paper-and-pencil instruments were data entered with 100 percent verification into data entry programs with prescribed ranges for each item. For the PSIs, automatic range checks and skip patterns were part of the CAPI programming to reduce data collection and data entry errors. For questions that required the parent to specify her/his response, we developed codes to classify responses and included them as additional values if 10 or more respondents gave the same answer.
| Data Collection Instrument | Window |
|---|---|
| 6-Month PSI (Parent Services Interview) | 5 months - 11 months and 30 days |
| 14-Month PI (Birthday Related Parent Interview) | 13 months - 19 months and 30 days |
| 14-Month Parent-Child Structured Play Assessment and Bayley | 13 months - 16 months and 30 days |
| 15-Month PSI | 14 months - 22 months and 30 days |
| 24-Month PI/Parent-Child Structured Play Assessment and Bayley | 23 months - 28 months and 15 days |
| 24-Month Father Interview | 23 months - 31 months and 30 days |
| 15-Month Parent Service Interviews | 24-Month Parent Interviews | |||||
|---|---|---|---|---|---|---|
| Number of Months | Program Group |
Control Group |
Combined Sample |
Program Group |
Control Group |
Combined Sample |
| -3 to -1 | 3.8 | 3.8 | 3.8 | 1.6 | 1.8 | 1.7 |
| -1 to -.5 | 9.1 | 8.3 | 8.7 | 9.9 | 10.5 | 10.2 |
| -.5 to 0 | 11.3 | 13.3 | 12.3 | 14.4 | 12.6 | 13.5 |
| 0 to .5 | 12.8 | 10.9 | 11.9 | 14.2 | 15.5 | 14.8 |
| .5 to 1 | 13.8 | 14 | 13.9 | 14.7 | 16.5 | 15.6 |
| 1 to 2 | 18.7 | 19.4 | 19.1 | 21.3 | 21 | 21.2 |
| 2 to 3 | 11 | 10.5 | 10.7 | 11.9 | 10 | 11 |
| 3 to 4 | 7.5 | 5.7 | 6.6 | 5.9 | 6.4 | 6.1 |
| 4 or Greater | 12 | 14 | 13 | 6.2 | 5.8 | 6 |
| Average Number of Months | 1.6 | 1.6 | 1.6 | 1.1 | 1.1 | 1.1 |
| Table of Contents | Previous | Next |