Table of Contents | Previous | Next |
Appendix B: Expanded Description of Survey Data Collection Methods
Chapter II presents a streamlined description of the survey data collection process. C Because the process was complex, lengthy, and, ultimately, very successful in achieving high response rates, here we present a much more detailed description of the procedures we employed. We describe specific components of the survey work, including (1) development of item content, (2) web survey development, and (3) data collection procedures. We conclude with implications for practice.
DEVELOPMENT OF ITEM CONTENT
The survey instrument was designed to accomplish the research objectives described in Chapter I. Working from the Early Head Start performance measures pyramid, we developed survey items that would address as many areas as possible within the conceptual model (Figure I.1). We also replicated or modified some items from the primary administrative data source for Early Head Start programs, the Program Information Report (PIR).1 We vetted earlier versions of the instrument with staff from MPR, our project officer, and outside consultants and Technical Working Group (TWG) members who represented broad expertise in the Early Head Start program. This group worked together to develop the survey instrument, a process that took more than a year due to the complexity of the domains to be measured.
After we had a working survey document, we gave it to program directors and Early Head Start technical assistance providers for their reactions. This took place in focus groups held at the 2004 Birth To Three Institute, the primary annual training conference for Early Head Start program staff. With the assistance of ZERO TO THREE, the conference host organization, we were able to access lists of conference registrants and their job titles before the conference. We invited 15 program directors and 12 training and technical assistance (T/TA) staff to participate in one of four focus groups during a scheduled break in the conference. We asked program directors and T/TA staff to complete a paper-and-pencil copy of the survey and then to discuss the clarity of the questions, whether programs were likely to have access to requested information, preferred mode of data collection, and any other comments and impressions they had. Focus group participants were forthright with their comments and were unanimous that a web-based survey was strongly preferred, followed by mail. A telephone survey was least desirable. Two themes that emerged from these groups were that (1) they appreciated the opportunity to be able to showcase the services Early Head Start provides to families, and (2) they were concerned that the survey would duplicate effort by too closely resembling PIR information that programs already had to provide.
We revised the survey based on focus group suggestions and pretested the revised version with nine Early Head Start program directors. They completed a hard-copy version of the survey, then participated in a debriefing telephone call. In the end, we created a survey with five sections that would take approximately 70 minutes to complete.
As we discuss in detail in the body of this report, the survey collected information on program characteristics, including program service approaches, family characteristics and involvement, staff characteristics, community partnerships, and program improvement activities.
WEB SURVEY DEVELOPMENT
After the items were finalized, we began constructing the web survey. The programming language we used (BLAISE) is attractive and user-friendly, containing easily understood instructions. The web version of the survey replicated, to the extent possible, the paper version. Its primary limitation was that it was not possible to keep some item structures that were in the paper copy (for example, item tables and grids could not be represented).
We maximized responses to the web survey by limiting use of survey features that require completed answers to move to the next item and providing as much clarifying information as possible. Because respondents were unlikely to complete the survey in one session, we allowed them to log in and out at will, saved data automatically, and skipped reentering respondents to the next unanswered question. We also made every accommodation we could to decrease the time needed to complete the instrument online. Those accommodations included using a minimum number of “data check” pop-up screens, allowing respondents to “click through” the entire instrument without entering any data (many opted to preview the items in this way), and giving respondents the ability to proceed through a data check screen even if the data were not changed as prompted.2 To ensure the cleanest data possible, we provided assistance and clarification by including a link for frequently asked questions (FAQs) on each page of the web survey, as well as an email link to a help desk. Some of the FAQs were constant through the survey, and some were specific to items asked in a particular section. Finally, we ordered the survey sections from most to least important, being mindful of the time required to complete all items. Our hope was that even incomplete survey responses would cover the most critical questions before stopping. After extensive pretesting of skip patterns, item wording, and content, we put the instrument online in early February 2005.
Data Collection Procedures
The survey was introduced in an advance email announcing the study. This email, written by the then Associate Commissioner of the Head Start Bureau, explained the study’s objectives and encouraged all programs to participate. At the same time, a packet was sent via Federal Express to each program director at the 748 Early Head Start sites providing services to children and families (our sample frame). The packet included a cover letter from the MPR project and survey directors, an endorsement letter from the ACF project officer, a hard-copy survey, a colored flyer with individual login and password information, and a business reply envelope. The cover letter explained the purpose of the study, provided an offer of assistance from our toll-free helpline, and reiterated the voluntary nature of the study—and that identities and responses of all participants would be kept confidential. The business reply envelope gave programs the option to complete the paper survey and return it by mail. Programs received a $20 Barnes & Noble gift card and a certificate of completion for participation in the survey.
With assistance from ZERO TO THREE as host, we held two question-and-answer conference calls approximately four weeks into data collection. Nearly 100 program directors, regional office staff, and T/TA providers participated in the two calls. MPR project and survey directors, the ACF project officer, and ZERO TO THREE staff participated in the calls. In the calls, the project officer described the background of the survey questions and research objectives. The project director explained the process of survey development and explained particularly complex survey items and answered questions about them. The survey director provided helpful hints about negotiating the web survey. Many good questions were raised in these calls, some of which were subsequently addressed in the FAQ sections of the survey.
Throughout the field period, we kept close watch on responses and prepared many reports that would be generated from the survey database automatically either daily or weekly. During the eighth week of data collection, the MPR Survey Operations Center (SOC) began making reminder calls to all program directors who had not yet responded by mail or web. The MPR survey director trained approximately six telephone interviewers to cover the calls Monday through Friday, 9:00 A.M. to 8:00 P.M. E.S.T. and six hours each day on Saturday and Sunday. During this time, we also sent periodic reminder emails to programs that had not responded or that had incomplete surveys.
We developed a tracking database to manage reminder mailings, keep track of the mode in which the survey was completed, and monitor the progress of cases being conducted from the SOC. Throughout the data collection period, we used the database to monitor the progress of cases and generate reports for managing data collection. The database allowed us to issue the Barnes & Noble cards to programs that completed the survey. We also designed and mailed a frame-ready certificate of participation personalized for each Early Head Start program.
We developed a call-back plan to recover incomplete or missing data. Recognizing that the survey was too long to complete by telephone, we identified a series of “critical questions” for a case to be considered partially complete. Programs that had responded at all or that had omitted answers to these critical questions were targeted for reminder telephone followup. Approximately 10 weeks into data collection, MPR created a preliminary file to identify any missing information on critical questions. Trained quality control clerks contacted program directors by telephone and email and faxed them abbreviated surveys to directly collect responses to these critical questions. The quality control clerks also contacted any programs returning hard-copy surveys with inconsistent or missing data.
Data Cleaning Procedures
We took steps to ensure that data collected via the web and other modes were equivalent. One key limitation of BLAISE for the web is that it cannot support tables—therefore, some items (particularly the B6 series—see Appendix A) presented on the web looked different than those in the paper copy. On the web, each item had to be asked separately, with subquestions embedded as separate screens. For example, a screener question would ask if there were any children with emotional or behavioral disorders. If the respondent answered yes, then subsequent questions would ask about referrals and eligibility status. Presenting the item in this way, rather than as a grid (as in the paper copy), may have elicited different responses. We assessed whether responses varied by mode by creating a series of flags to categorize types of problematic responses to these items (such as skipped questions, illogical answers, and so forth) and noted that problems were somewhat more prevalent in web respondents. To correct any possible unintended bias toward error among web respondents, we called back any program that was flagged to have any of several types of problematic answers. Any differences in the information retrieved through the callback were corrected in the database by quality control clerks. There were no other items that presented challenges beyond those normally associated with preparing data. The final data file was prepared for analysis (variable names and labels applied, values formatted, logical skips coded, and so on).
Results and Implications
This combination of careful survey development and proactive, continuing support and assistance with responses resulted in a very high response rate, not only for surveys of this type, but for any survey: 89 percent of eligible programs responded. This high response was in spite of a variety of factors, including length and complexity of the survey, that would ordinarily work against good response. In addition to the overall high response rate, we found that nearly two-thirds of programs responded via the web-based option, much more than is typical for web-based surveys. In addition to the implications for good survey practices and consistent followup, it appears that, at least in programs like Early Head Start, a web-based approach may be a successful way to reach programs.
1 The PIR is an annual survey that all Early Head Start and Head Start programs complete. It is the primary national administrative data source for the Early Head Start and Head Start programs. (back to footnote 1)
2 We discuss our data checking and cleaning procedures in the “Data Cleaning Procedures” section. (back to footnote 2)
| Table of Contents | Previous | Next |