ACF OPRE: Survey of Early Head Start Programs

Methodology

Questionnaire Development and Administration

The MPR research team designed a self-administered questionnaire (see Appendix A) to be completed by the individual staff member designated to lead the implementation of IM/IL, hereafter referred to as the program’s “IM/IL coordinator.” The questionnaire had three sections:

IM/IL Training-of-Trainers Event: the type and number of program staff who participated, as well as the content, format, and quality of the training that was received
IM/IL Implementation to Date: IM/IL goals selected; training of local staff; types of enhancements tried; target audiences (parents, staff, and/or community partners); perceptions about success; challenges faced; and factors affecting sustainability
Program and Respondent Characteristics: IM/IL enhancements extended to home visits and Early Head Start (if the programs were combined Head Start/Early Head Start programs); respondents’ professional backgrounds; perceptions about the importance of obesity as a health problem for children, parents, and staff

The questionnaire went through many stages of revision. First, the draft questionnaire was reviewed by staff at OPRE. Second, it was reviewed by two consultants: Dr. Mary Story, Professor of Epidemiology at the University of Minnesota; and Dr. Russell Pate, Professor of Exercise Science at the University of South Carolina. These two experts have considerable research experience in diet and physical activity behaviors, respectively, and in preschool children, including those enrolled in Head Start. Third, a former Head Start program director was asked to read the survey and to help determine how long it would take to complete each section. The research team conducted a one-hour interview with her, which focused on confirming that the phrasing of items was clear and understandable, identifying any unclear phrasing, and determining whether any items asked for information that would be difficult for respondents to provide. The questionnaire underwent further modifications following each of these stages of review. The final instrument contained 124 questions and could be completed in about 20 minutes. The survey design and final questionnaire were approved by the Office of Management and Budget (OMB no.: 0970‑0318).

In early March 2007, about a year after the Region III TOT event was held, the questionnaire was mailed to the 53 Head Start programs in Region III that had sent staff to the TOT event.3 Prior to the questionnaire mailing, programs received a letter of support from the Director of the Office of Head Start, that endorsed the survey, explained its purpose and importance, and encouraged programs to complete it. Each questionnaire packet was sent by express mail and included a cover letter from the MPR survey director, an endorsement letter from Nancy Elmore, the Office of Head Start Region III Program Manager, the questionnaire, and a prepaid return envelope. Data collection occurred over an eight-week period (from March 8th through May 4th, 2007) and email reminders and calls were made to nonrespondents. One program director called MPR to complete the survey by telephone. Fifty completed questionnaires were received, for a response rate of 94 percent. Twenty-two of the 50 respondents completed 100 percent of the items. Of those with missing items, 12 programs had about 2 percent of the questionnaire items missing.

Each returned questionnaire was reviewed for accuracy and completeness and then the data were entered into a computer file. For each survey, the research team conducted 100 percent verification, in which all of the data was reentered and any discrepancies between the first and second data entry were identified. The final data file was then exported into SAS for further analysis.

The data from this questionnaire has important limitations that should be noted. First, the survey was completed by the program director, who may not have had a clear understanding about how IM/IL is being implemented at the ground level. Second, the survey was completed one year after the TOT event, so participants may not have been able to accurately recall the training. Third, the questionnaire items assessed programs’ participation in nutrition and physical activity in the context of IM/IL, but many programs may have carried out these types of activities prior to IM/IL and respondents may have had difficulty making that distinction. Despite these limitations, these survey results offer valuable insights into the implementation of IM/IL by programs who participated in the spring 2006 TOT.

Data Analysis

The data analysis focused on descriptive statistics to characterize programs’ assessment of the IM/IL TOT event and their efforts to implement IM/IL enhancement activities. Frequencies of categorical variables and distributions of continuous variables were examined, as well as frequencies for combinations of responses items. Five variables were constructed in order to distinguish the implementation status across the 50 programs. One of these variables was used to categorize programs based on their rating of their success with IM/IL implementation. Programs were asked to rate their implementation on a scale that ranged from 1 (not at all successful) to 5 (extremely successful) (see survey item C9d, Appendix A). Those that gave their programs ratings of 4 or 5 were categorized as high implementers. Those that gave their programs ratings of 2 or 3 (no programs rated their implementation as a 1) were classified as being programs that were not high implementers.4 In addition, four new variables were derived by collapsing items that assessed IM/IL enhancements or supports in items C34 and C38; these new variables were: (1) purchased equipment and/or used vocabulary for teaching structured movement, (2) purchased new equipment for free play, (3) reconfigured or enhanced space to facilitate physical activity, and (4) changed policies or practices related to foods served to children.

In addition to the descriptive analyses, a limited number of bivariate analyses were also conducted. To compare characteristics of programs that participated in IM/IL to those Region III programs that did not participate,5 the questionnaire data were linked to the 2005‑2006 Head Start Program Information Report (PIR).6 Data on center, staff, and child characteristics were extracted from the PIR. The Prozip database was used to code programs’ metropolitan status based on their zip code.7 T-tests were used to identify significant differences by participation in the IM/IL training event. Factors associated with high implementation also were examined. These analyses examined different variables hypothesized to be related to programs’ success with implementing IM/IL enhancement activities, such as the successes and challenges that they experienced, the type and intensity of training that they received, and the types of IM/IL activities and supports that they implemented. T-tests were used to identify significant differences across levels of implementation.

Finally, two programs that reported not having tried to implement IM/IL activities (item C1) were asked to skip the remaining items in Section C. Therefore, most of the analyses of items from this section were completed with 48 programs rather than 50.

3 The questionnaire was mailed to 54 Head Start programs, but upon further investigation, we found that one program was a Head Start grantee that did not provide direct services to children and their families. We excluded this program from the sample. (back)

4 No variable was derived to classify programs as being “low implementers.” (back)

5 Participating programs only include the programs that participated in the TOT in spring 2006. The 17 programs that participated in the pilot effort in FY2005 were classified as programs that did not participate in the IM/IL training event. (back)

6 For more information about the PIR, see [www.acf.hhs.gov/programs/hsb/programs/pir/index.html]. (back)

7 For more information about Prozip, see [www.emory.com/progress/prozip.htm]. (back)

Table of Contents | Previous | Next