ACF OPRE: Survey of Early Head Start Programs

Chapter II: Methods

For this study, we aimed to collect data from the complete universe of Early Head Start programs funded at the time of the survey and to supplement our quantitative survey data with qualitative information collected during in-depth site visits to a subset of programs. The high response rate we achieved on the survey allows us to generalize our survey findings to all Early Head Start programs. The site visits to a subset of programs provided rich data that helped us interpret the survey findings and develop hypotheses about why programs follow particular management or staffing patterns. In this chapter, we describe our approach to both survey and site visit data collection and analysis.

DATA COLLECTION

In this section, we describe our approaches to data collection, beginning with designing and administering the survey, identifying a sample frame, and calculating response rates. Next we discuss developing site visit protocols, selecting a subset of sites to visit, and conducting the site visits.

Survey Design

The Survey of Early Head Start Programs is intended to supplement and extend the primary administrative data source for Early Head Start programs, the annual Program Information Report (PIR).1 Thus, we replicate or modify some basic PIR items about program characteristics and families. Items for the survey can also serve as pilots for future revisions of the PIR. One important divergence from the PIR is our use of a common reference period (program status as of January 1, 2005) to capture a “snapshot in time” of program enrollment and service delivery activities. In addition, the survey is the first instrument designed specifically to collect data from programs serving infants and toddlers, so it goes beyond the PIR in asking detailed questions about program management, services, and staffing specific to these programs. The survey questions focus primarily on community and family characteristics, partnership activities, assessment strategies, and children with disabilities.

In addition to collecting basic information about the characteristics of grantees and enrolled families, we designed our survey to address performance measures contained in the two bottom layers of the performance measures framework described in Chapter I—management systems and services (Figure I.1). We developed survey items to measure implementation of as many of the performance measures as possible in the management systems and services layer. Table II.1 displays the specific performance measures, organized by performance objectives in the conceptual framework (Figure I.1) along with the data sources we used to collect information on the implementation of each measure. (The complete survey is in Appendix A.)

During the survey development process, we consulted with an internal project team consisting of MPR staff, federal staff, and consultants. This team and the Technical Work Group (TWG), whose members represented diverse areas of expertise regarding Early Head Start programs and early childhood education, provided feedback on early versions of the instrument. This group collaborated to develop the survey instrument—a process that, because of the complexity of the domains to be measured, took one year to complete. Appendix B gives a complete description of the survey development process.

Survey Administration

To maximize response rates, we fielded the survey through multiple modes. Because in focus groups Early Head Start staff indicated preference for a web-based interface, we developed both a web and a paper-and-pencil version and gave respondents their choice. We mailed them an individualized login and password for accessing the web version, as well as a copy of the paper-and-pencil version. Data collection ran from February 2005 to early July 2005, with web and paper modes occurring simultaneously.

We used several strategies to encourage high response: a comprehensive series of advance mailings, endorsement letters from federal Head Start officials, informational conference calls hosted by the Early Head Start National Resource Center at Zero To Three, a seven-day-a-week help desk (available by email and toll-free telephone), and periodic reminder emails. We contacted programs that did not complete either self-administered form and invited them to complete a partial interview by telephone.2

Table II.1. Data Sources for Early Head Start Program and Family Characteristics and Performance Measures, by Performance Objective
Data Element			Data Source
Data Element			Survey	Site Visit
A. Grantee Characteristics	Program approach		X
	Program size		X
	Program sites		X
	Years in operation			X
B. Characteristics of Families Served	Enrollment criteria			X
	Recruiting approaches			X
	Characteristics of families and children served		X	X
	Families' needs			X
	Enrollment turnover		X	X
C. Management Systems: Processes for Improvement	Programs comply with Head Start regulations. (Not assessed)
	Programs are well managed operationally and financially.	- Use of MIS	X	X
		- Leadership		X
		- Communications	X	X
	Programs design and implement services to be responsive to the needs of families in the community.			X
	Programs conduct self-assessments that are used for continuous improvement.			X
D. Management Systems: Support Staff	Programs employ qualified staff with the skills necessary to provide high-quality services.		X
	Programs support ongoing staff development, training, and mentoring.		X	X
	Programs support staff activities through ongoing reflective supervision.		X	X
	Programs promote staff retention and continuity.			X
E. Management Systems: Relationships with Community Partners	Programs form partnerships with other community programs and organizations to support an integrated community-wide response to the needs of families with young children.		X	X
	Programs form partnerships and coordinate services with local Part C agencies.		X	X
	Programs form partnerships and coordinate services with community child care providers to meet the needs of families and enhance the quality of local child care services through the sharing of resources, training and knowledge.		X	X
	Programs form partnerships and coordinate services with local health agencies and health care providers to meet the health-related needs of families.		X	X
F. Management Systems: Involving Parents	Parents are actively involved in program planning and decision making.		X	X
F. Management Systems: Involving Parents	Programs encourage and support fathers' involvement in program planning, decision making, and activities.		X	X
G. Program Services: Linking to Community Services	Programs work collaboratively with families to identify their goals, strengths, and needed services, and offer them opportunities to develop and implement individualized family partnership agreements that take into account other family plans.			X
	Programs link parents with social services agencies to obtain needed services.		X	X
	Programs link parents with educational and employment agencies to obtain needed services.		X
	Programs link parents with physical and mental health care prevention and treatment services to obtain needed care.		X
	Programs link parents with needed prenatal care and education services.		X
	Programs help parents secure high-quality child care in order to work, attend school, or gain employment training.			X
	Programs help parents and children make a smooth transition to Head Start or other preschool programs.		X	X
H. Program Services: Responsive and Caring Relationships with Parents and Children	Staff form respectful and supportive relationships with parents through all aspects of service delivery.			X
	Staff form nurturing relationships with children in group-care settings or during home visits.			X
	Programs support and honor the home cultures and languages of families.			X
I. Program Services: Provide Children with Individualized Services	Programs provide developmentally enriching educational environments in group-care settings and developmentally enriching parenting and child development services during home visits and group socializations. (Not assessed)
	Programs link children with needed medical, dental, and mental health services.		X	X
	Programs link pregnant women with comprehensive prenatal health care and education.		X	X
	Programs provide children in group-care settings with meals and snacks that meet their daily nutritional needs, and parents receiving home-based services are given information about meeting their children's nutritional needs. (Not assessed)
	Programs provide individualized services for parents and children, including children with disabilities.		X	X

Sample Frame and Response Rates

Our intention was to take a census of the universe of Early Head Start programs operating in late 2004/early 2005. We developed a sample frame from the most recently available PIR data by extracting grantee and delegate identification numbers, selected program characteristics, and contact information. Using these PIR data, we selected Early Head Start programs (not Head Start) and only those providing services directly, excluding grantees that provided services only through delegates.3 We refer to all respondents to the survey as “programs.” This process resulted in a sample frame of 748 programs.

We obtained a response rate of 89 percent, receiving survey responses from 660 programs.4 This rate includes partially completed surveys. Respondents clearly preferred the web interface, with nearly two-thirds responding in that way. Twenty percent returned paper copies, and the rest completed the survey by telephone (Table II.2). Appendix B presents a detailed description of our methods for fielding the survey and creating final data files.

Table II.2. Response Rates by Survey Method
		Number of Programs	Percentage of Responding Programs	Percentage of All Programs
Survey Mode	Web	479	72.6	64.0
	Paper	148	22.4	19.8
	Telephone	33	5.0	4.4
All Responding Programs		660	100	88.2
Nonrespondents	Ineligible a	7	-	0.9
Nonrespondents	No contact/refused	81	-	10.8
Total Sample Size (Programs)		748		100

Source: Survey of Early Head Start Programs and 2004-2005 Program Information Report.

a These programs lost their federal Early Head Start grants and discontinued service provision during the field period.

Site Visit Protocol Development

We designed our site visits to provide in-depth, rich information about program organization, services, and staff through semistructured interviews with staff and two focus group discussions—one with staff and one with parents. We developed the site visit interview guides using an approach similar to the one we used for the survey. Working from the Early Head Start performance measures framework, we identified areas of interest for in-depth exploration beyond or in addition to what we could learn from the survey alone. We created protocols for specific types of program staff—the director and staff members in five different specialty areas (community partnerships, early childhood education, family involvement, health and disabilities, and home visiting). We discussed a core set of questions with multiple staff members, but each protocol also included specific items relevant to particular specialty areas. We also created separate focus group discussion guides for Early Head Start center teachers, home visitors, and parents. During the development process, we consulted with other MPR staff, our ACF project officer, outside consultants, and TWG members.

Interview topics covered the five major program areas described in the survey but in much greater depth and detail (Table II.1). In addition, we gave special attention to staff views on program leadership, the extent of Early Head Start’s integration with Head Start, strategies for serving high-risk families, the use of management information systems and data, and partnership arrangements, especially child care partners.

Site Visitor Qualifications and Training

A team of MPR staff conducted the site visits between October 2005 and early February 2006. These people had extensive experience in conducting site visits, and many had visited Early Head Start programs for other studies. To ensure that all visitors would be well prepared to carry out the interviews and focus groups, MPR conducted a day-long training in September 2005. The session included briefings on the study’s background and objectives, a review of the interview protocols, and extensive discussion about the research questions and the intent of specific items. We provided all site visitors with training materials that included descriptions of procedures, suggested talking points to use when contacting programs to set up the visits, copies of the interviews and group discussion guides, and supplementary information on Early Head Start. We also held weekly team meetings to discuss the visits and resolve issues and questions that had come up in the field.

Site Selection and Procedures During the Visits

To ensure that we could address questions of interest, we purposively selected sites to visit based on the following eight characteristics: (1) ACF region, (2) program size, (3) whether the programs served pregnant women, (4) whether they served children through partnerships with community child care providers, (5) urbanicity, (6) whether the programs had experienced recent changes in demographics, (7) whether they had experienced recent turnover in the program director, and (8) program approach. We used data from survey responses to categorize the programs according to these characteristics, then we selected programs to ensure the greatest diversity possible along each dimension and across combinations of dimensions (Table II.3).5

Table II.3. Characteristics of Programs Selected for Site Visits
Characteristic		Number of Programs
Region 1		1
Region 2		2
Region 3		2
Region 4		2
Region 5		2
Region 6		1
Region 7		2
Region 8		2
Region 9		1
Region 10		1
Region 11		1
Serves Fewer than 50 Children and Pregnant Women		6
Serves 50 or More Children and Pregnant Women		11
Serves Pregnant Women		14
Does Not Serve Pregnant Women		3
Has Child Care Partner		11
Does Not Have Child Care Partner		6
Serves Children Through Child Care Partners		5
Does Not Serve Children Through Child Care Partners		12
Urban Area		9
Non-urban Area		8
Change in Enrollee Race/Ethnicity	Increase in Hispanic enrollment	4
Change in Enrollee Race/Ethnicity	No change/do not have Hispanic enrollment	13
Director Turnover During Past Year		1
Program Approach	Home-based	4
	Center-based	4
	Both home- and center-based	9
Sample Size (Programs)		17

Source: Survey of Early Head Start Programs.

We contacted the selected programs and asked the director or a staff person to help us set up interviews with appropriate staff members and recruit teachers, home visitors, and parents to participate in focus groups. Site visits took two days to complete. To increase participation, we gave each program a $20 Barnes and Noble gift card; parents received a $20 incentive for participating in the focus groups.

ANALYTIC APPROACHES

This section provides a brief description of our data preparation procedures and the analysis of survey data. We follow with a description of our approach to coding and analysis of the qualitative data we obtained in site visits.

Analysis of Survey Data

Before beginning analysis, we thoroughly prepared the data obtained from the survey. An advantage of the web survey was that there were some built-in data checks, including restricting the range of responses, and screens that alerted respondents to inconsistent answers.6^,7 We applied variable names and labels to all items and created a program that checked all skip patterns (inserting a logical skip code when items were missing for that reason).

To analyze the data, we computed descriptive statistics such as frequencies and means. We created some key constructs for purposes of aggregating programs (for example, categories to describe program service approaches and demographics of enrollees—described in later chapters). We answer the research questions posed in Chapter I that center on the first two layers of the conceptual framework. These include characteristics of programs, families served, services provided, and program management using descriptive statistics of overall program characteristics. We answer questions about differences among program subgroups by creating categories of programs to compare characteristics across programs.

We intended this survey to represent all Early Head Start programs so that descriptions would be generalizable. Because we contacted all Early Head Start programs in operation and obtained a response rate of nearly 90 percent, we are able to generalize our findings to the universe of Early Head Start programs. For the same reason, we are also able to detect small differences between subgroups. To reduce any potential for nonresponse bias, we computed an adjustment factor based on information we knew about all programs from the PIR. To calculate the weights, we used program size, location in a metropolitan statistical area (MSA), agency size, and agency type. Respondents did not differ from nonrespondents, except that the latter were more likely to be from an MSA (78 percent versus 65 percent, p < .05). We then used this factor as a weight in our analysis to allow projections to the full population versus only the sample that responded. Weighted and unweighted data are nearly identical, and therefore we report unweighted data in the body of the report and include tables of weighted data in Appendix D. We use weighted data for subgroup analyses to avoid the possibility that response rates differ across subgroups (something we cannot assess for nonrespondents, because for those programs we do not have the survey data that we used to create the subgroups). We did however compare responding and nonresponding programs on other characteristics from the PIR, including number of children, pregnant women, and children with disabilities, and found no significant differences between groups.

Analysis of Site Visit Data

To ensure the quality of site visit data, we developed a structured site visit report template and subjected all reports to a quality control review before coding and analysis. We used a qualitative analysis software package, Atlas.ti (Scientific Software Development 2004), to organize and synthesize the large amount of data collected during the site visits. This software enabled research team members to use a structured coding scheme for organizing and categorizing data that are linked to the primary research questions. After the site reports were coded, we used Atlas.ti to conduct searches and retrieve data on our research questions and subtopics. Findings from the qualitative data are presented mainly as general descriptions of common program features; in some cases, we identify particularly interesting or unique features to use as examples. In contrast to the survey findings, our site visit findings are not generalizable to all programs. They do, however, provide rich information on a subset of programs and may provide clues to help us understand better why programs organize themselves differently and how various systems operate.

1 The PIR is a web-based survey that all Head Start and Early Head Start programs must complete annually. It includes basic administrative data, such as enrollment counts, numbers of children served through service options, and staff education and credentials. Programs may submit their reports at any time between May 15 and August 31. Programs do not use a common date for enrollment counts; rather, they provide end-of-month enrollment for three months (November, February, and April). If a program is not operating in one or more of these months, it reports enrollment for an alternate month. (back to footnote 1)

2 For the telephone version, we selected the most important questions and asked programs to respond only to this abbreviated set of items in hopes of collecting at least partial data from as many programs as possible. Telephone surveys were completed by trained telephone interviewers who entered data directly into the web survey. (back to footnote 2)

3 A grantee agency is one that receives federal funding to operate an Early Head Start program. A delegate is a program that provides Early Head Start services. Some grantees provide Early Head Start services directly; others provide services only indirectly, through delegates. (back to footnote 3)

4 This figure excludes seven programs that were deemed “ineligible” because they lost their Early Head Start funding at some point during the field period. If we include these seven programs in the denominator, the response rate falls to 88 percent. (back to footnote 4)

5 Before contacting programs, our project officer verified that they were in good standing and notified regions of our intent to contact (unnamed) programs in their area. We guaranteed anonymity to participating programs and therefore offer only very general information about this sample. (back to footnote 5)

6 For example, one data check was that applicable responses using percentages should total 100. (back to footnote 6)

7 An example is in Survey Section A (see Appendix A). Total child enrollment is reported in item A4C, and if the number of children reported in item A5D did not match, an alert screen would appear and ask respondents to check their answers. These alert screens were placed strategically and only at key places so as to minimize time and burden on respondents. (back to footnote 7)

Table of Contents | Previous | Next