Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

Chapter 3: Data Collection Procedures, Measures, and Data Sources

This chapter begins with an overview of the study’s overall data collection plan and then describes the Field Test that was conducted prior to implementing the full-scale study. The chapter next describes the measures and data sources being used to obtain the necessary information about children, parents, and programs.

Overview of Planned Data Collection

Data collection for the main study began in fall 2002 and will continue through spring 2006, following children from entry into Head Start (either 3 or 4 years old) through the end of their preschool years, end of kindergarten, and end of 1(st) grade (see Exhibit 4). The study is collecting comparable data for both Head Start and non-Head Start group children, including interviews with parents, direct assessments of children’s development, surveys of Head Start and non-Head-Start teachers, interviews with center directors and other care providers, direct observations of the quality of various care settings, and teacher/care provider ratings of individual study children.

The Field Test

The Field Test, conducted from fall 2001 through spring 2002, tested the feasibility of all study procedures, including the recruitment of programs to conduct random assignment, the random assignment procedures themselves, and the data collection instruments and procedures.

Exhibit 4: Schedule of Planned Data Collection

School Year 3-Year-Old Cohort (C-1) 4-Year-Old Cohort (C-2)		2002-2003 3-year-old Preschool 4-year-old Preschool		2003-2004 4-year-old Preschool Kindergarten		2004-2005 Kindergarten Grade 1		2005-2006 Grade 1
Data source	Cohort	Fall	Spring	Fall	Spring	Fall	Spring	Fall	Spring
Child Assessment	C-1
	C-2
Parents/Primary Caregiver Interview	C-1
	C-2
Administrative Records	C-1
	C-2
Staff Interviews	C-1
	C-2
Observed Quality of Care Settings	C-1
	C-2

 

The sample for the Field Test included eight grantee/delegate agency groups in geographic clusters that were not part of the full-scale study. The clusters were selected to vary in terms of the availability and comprehensiveness of program or child care options (in addition to Head Start) within the community. The grantee/delegate agencies and/or centers varied across auspices, number of unserved children, local service richness, grantee size, ethnicity of population served, and urbanicity.

Within the selected grantee/delegate agencies, the Field Test involved approximately 400 3- and 4-year-old new applicants to Head Start spread across a total of 21 Head Start centers. This early feasibility assessment served several purposes. First, the Field Test was an important way to understand the recruitment and application process in Head Start programs and essential to testing and refining random assignment procedures. Second, it allowed for the testing of data collection measures and procedures and provided an opportunity to learn more about the best strategies to deal with the complexities of both Head Start and other available programs for low-income children. Valuable lessons learned and used to inform the design of the full-scale study included the following:

• It is critical to build and maintain study team, regional office, and grantee/delegate agency partnerships. For the full-scale study, Recruitment Teams, led by senior study staff and including a local study site coordinator, communicated with both federal regional office and grantee/delegate agency staff throughout the recruitment and random assignment phases of the project. Recruitment teams made site visits to each grantee/delegate agency and maintained frequent communication with key staff.
• It is important to gain a thorough understanding of the recruitment and application process of each grantee/delegate agency and to “tailor” random assignment and data collection procedures to fit local processes. Recruitment teams for the full-scale study gathered information on local program operations and worked collaboratively with staff to design study procedures that maintained the necessary scientific rigor while, to the extent possible, minimizing disruption to usual recruitment and enrollment procedures.
• It is necessary to understand that Head Start enrollment does not always happen as expected or when expected, and this can affect whether sites can implement random assignment and provide the desired sample size. Consequently, random assignment for the full-scale study was conducted in “waves” from spring through fall 2002 (e.g., using monthly batches of new applicants). In addition, a small number of centers initially recruited for the study were eliminated due to saturation.
• It is important to address program concerns about serving the neediest families, monitoring compliance, notifying parents of enrollment decisions, and staff responsibilities and burden. Recruitment teams met with program administrators, teaching and social service staff, and parents to provide an orientation to the study and address concerns. Study Fact Sheets were also developed for each group to answer frequently asked questions.
• High response rates could be achieved from both the Head Start and non-Head Start control groups. An 80 percent response rate (parent and child combined) was attained from the fall 2001 field test data collection and maintained in the spring data collection.

Data Sources and Measures

To address the Congressional mandate, and respond to the key research questions, a wide variety of data sources and measures are being used to assess the effects of Head Start on fostering and enhancing child development, including direct child assessments, parent/primary caregiver interviews, interviews with providers of early care and program services to participating study children, observations of early care settings, and the collection of secondary data. The following types of data are being collected:

• Child outcomes through: 1) direct assessments of children’s development, 2) parent/primary caregiver reports of children’s development, 3) teacher/care provider report of children’s development, and 4) administrative school records. Child outcomes will be measured in the key domains of approaches to learning, language usage and emergent literacy, math, cognition and general knowledge, physical well-being, motor development, social and emotional development, and eventually, school success.
• Characteristics and quality of children’s home environments through: parent/primary caregiver interviews including such topics as parenting practices, household composition, parents’ health and mental health status, safety of environment, child stress/risk, and receipt of health and other services.
• Characteristics and quality of the primary preschool and child care arrangement through: (1) interviews with center-based directors, (2) surveys of teachers or interviews with care providers, and (3) observations of these settings.

Each type of data is described below in more detail. It is first important to understand how children’s care settings have been defined for the present study because this had influenced the choice of instruments and what data have been collected for individual children.

Defining and Assessing the Child’s Care Setting

Head Start children may attend other programs during the hours they are not in Head Start and will be exposed to numerous other experiences at home and elsewhere that shape their development. Similarly, children assigned to the control group will also have a variety of experiences that affect their development, such as care provided in their own home, as well as time spent in other settings. Although all care experiences are important, practical considerations and differences among the settings limit what data can be collected and in what depth across these various settings. Consequently, criteria were developed to help identify, categorize, and prioritize the range of settings in which children spend time. These criteria were then used to select the one setting that will later serve as the most appropriate comparison to Head Start for each child in the control group. The characteristics and quality of this setting were assessed through caregiver interviews and interviewer observations. The setting chosen was based on a hierarchical decision tree, prioritizing out of home experiences with center-based care over family day care. For those children who did not have any out-of-home program or care experiences, a small number of additional items were used to assess the home setting as the primary daytime care setting. The information used to define each child’s care setting is derived from the parent interview. Parents are asked to identify all of the places where sampled children spend time between the hours of 8am and 6pm weekdays during the school year. This information is used to categorize each child’s experiences into the following Monday through Friday timeframes:

• The care setting (including parental care) where they spend the most time between 9am and 3pm—the portion of the day when most children are in Head Start.
• The non-parental child care arrangement where they spend the most time between 8am and 6pm. This arrangement is termed the Primary Non-parental Care Setting and serves as the data collection site for comparative data on the quality of care setting.
• Any additional non-parental child care arrangements where they spend time between 8am and 6pm.

Exhibit 5 summarizes the data sources for each of the different settings in which children may be found. These data sources are discussed in greater detail below.

Exhibit 5: Assessment of Setting Characteristic by Type of Care Setting

	Parent Report23		Center Director		Teacher/Care Provider Report	Observation	Care Provider Interview	Teacher Survey
Primary Non-Parental Care Setting24 Hierarchical Selection			(Monday-Friday, 8am-6pm)
1. Head Start	X		X		X	-ECERS-R		X
2. Center-Based (most time, at least 5 hours per week)	X		X		X	-Arnett		X
						-Teacher directed activities
						-5 question observation
3. Non-Center-based (most time, at least 5 hours per week)
-relative	X				X	-FDCRS	X
-non-relative	X				X	-Arnett
						-5 question observation	X
4. Own home
-with relative	X				X	5 question observation	X
-with non-relative	X				X		X
Primary Care Setting is Child's Parent (Monday-Friday, 9am-3pm)
1. Own Home with parent	X					5 question observation
Additional Child Care Settings (Monday-Friday, 8am-6pm)
1. Center-based	X
2. Non-Center-based
-relative	X
-non-relative	X
3. Own Home

 

Direct Child Assessments

Child assessments provide direct measures of how well Head Start and non-Head-Start preschool programs, or other child care services, are achieving the goal of assisting children to be physically, socially, and educationally ready for success in kindergarten. These measures, which represent the key developmental outcomes for the Head Start Impact Study, are being assessed using a 35- to 45-minute battery of standardized tasks administered by specially selected and trained child assessors. The assessments are being administered at the location where the child spends the most time, Monday through Friday from 9am to 3pm.

The assessment battery (see Exhibit 6) is composed of a short series of tasks that are feasible and interesting for preschoolers to carry out and that have been shown to be predictive of later school success and achievement or academic difficulties (test citations are provided in Appendix E). There is an emphasis on tasks that relate to the acquisition of reading skills because reading is so central to success in school and to later functioning in society. For non-English speaking children, the Interviewers used a Language Decision Form at the time of the assessment to determine the appropriate language for conducting the assessment (See Appendix F).

For children requiring assessment in Spanish, a bilingual Interviewer/Assessor administered the assessment battery in Spanish in fall 2002, and also administered two subtests in English- the PPVT-IIIA and the Woodcock-Johnson III Letter-Word Identification. In spring 2003, the children assessed in Spanish in fall 2002 were assessed primarily in English, along with the continued administration of two Spanish language measures: the TVIP and the Batería Woodcock-Muñoz Identificación de letras y palabras. One exception is Puerto Rico where, because instruction is in Spanish, all children were assessed only with the complete Spanish battery in spring 2003.

For children who could not be assessed in English or Spanish, either a bilingual Interviewer/Assessor or an interpreter for the child’s language were used. The Interviewer/Assessor or interpreter used the English version of the booklet, translated the instructions into the child’s language and administered four subtests: McCarthy Draw-A-Design, Color Names & Counting, Leiter-R-Adapted, and Story & Print Concepts. For spring 2003, these children were assessed in English.

Exhibit 6: Child Assessment Battery by Study Year and Child Language: Preschool Years

Domain	Measure	Child's Language at Beginning of Head Start Year
		English		Spanish		Other
		Fall 2002	Spring 2003	Fall 2002	Spring 2003	Fall 2002	Spring 2003
Language Development And Literacy	PPVT III (adapted)	E	E	E	E	-	E
	TVIP (adapted)	-	-	S	S	-	-
	CTOPPP Print Awareness	E	E	S	E	-	E
	CTOPPP Elision	E	E	S	E	-	E
	Woodcock-Johnson III Letter Word Identification	E	E	E	E	-	E
	Woodcock-Johnson III Spelling	E	E	-	E	-	E
	Woodcock-Johnson III Oral Comprehension	E	E	-	E	-	E
	Woodcock-Munoz Letter Word Identification	-	-	S	S	-	-
	Woodcock-Munoz Dictation	-	-	S	-	-	-
	Story and Print Concepts	E	E	S	E	O	E
	Letter Naming Task	-	E	-	E	-	E
Mathematics	Counting Bears	E	E	S	E	O	E
	Woodcock-Johnson III Applied Problems	E	E	-	E	-	E
	Woodcock-Munoz Applied Problems	-	-	S	-	-	-
Perceptual Motor	McCarthy Draw-a-Design	E	E	S	E	O	E
	Woodcock-Johnson III Spelling (H1-H7)	E	E	-	E	-	E
	Woodcock-Munoz Dictation (G1-G6)	-	-	S	-	-	-
Other Cognitive Ability	Color Names	E	E	S	E	O	E
Sustained Attention	Leiter-R (adapted)	E	E	S	E	O	E
Assessor Ratings	Task persistence	X	X	X	X	X	X
	Attention span	X	X	X	X	X	X
	Body movement	X	X	X	X	X	X
	Attention to directions	X	X	X	X	X	X
	Comprehension of directions	X	X	X	X	X	X
	Verbalization	X	X	X	X	X	X
	Ease of Relationship	X	X	X	X	X	X
	Confidence	X	X	X	X	X	X

Notes: “-“ =subtest not administered during the data collection period; “E”=subtest administered in English; “S”=subtest administered in Spanish; “O”=subtest administered in language of child other than Spanish or English; “X”= an observed assessor rating not a direct Child Assessment.

 

Parent/Primary Caregiver Interviews

The parent/primary caregiver interview, conducted concurrently with the child assessments in both fall 2002 and spring 2003, collects information in the following areas: (1) parental beliefs and attitudes toward their child’s learning and satisfaction with their child’s child care experience; (2) family household and demographic information, including parent-child relationships and the characteristics of the child’s home environment; (3) parent ratings of their child’s behavior problems, social skills, and competencies; (4) parent's perception of their child's accomplishments; (5) parent’s perception of their relationship with their child; and (6) child and family receipt of a variety of comprehensive services.

The parent interview also includes items that address key parenting behaviors that may be affected by participation in Head Start, such as attitudes about how parental authority is conveyed, encouragement of the child's exploration and independence, and parental activities, particularly reading. In spring 2003, parents were also asked about services they received and the help they received in coordinating the services. Other topics included the child’s transition from preschool to kindergarten and any information or services the family received to assist with this transition. These data are being collected because it is hypothesized that such behaviors may be affected by Head Start, and changes in parenting behavior may have a subsequent impact on children’s development.

Staff Surveys and Interviews

Data are also being collected through surveys and/or interviews with staff in the primary non-parental care setting. In center-based programs, interviews are being conducted with the center director and paper-pencil surveys are being completed by the teacher in the classroom where the child spends the most time. For children whose primary non-parental setting is a family daycare home, data comparable to that collected through the center-based director interview and teacher survey are collected through a single combined care provider interview with the child care provider.

In spring 2003, data were collected from each child’s teacher/care provider in the child’s primary non-parental care setting. This included information on: their demographic characteristics; education, training, and experience with young children; the nature of the setting in which they work; and the types of services they provide to the selected study children. In particular, items on literacy-promoting activities are included in three questionnaires: the Teacher Survey, Care Provider Interview, and Center Director Interview. During the kindergarten year, the Teacher Survey will collect information about the kindergarten program, provisions that are made for the child’s transition to kindergarten, and whether the teacher obtained any information from the Head Start program or other care settings about the child’s development or special needs.

Teachers and care providers are also asked to rate each of the study children who are in their classroom or care in the following areas: relationships with adults; classroom behavior and conduct, problem solving and initiative, social relationships, creativity, musical and movement ability, language ability, and mathematical ability. Although not as objective as direct assessments or observations by impartial observers, parent/primary caregiver and teacher/care provider ratings of children’s accomplishments and behaviors are an important source of information because these individuals see children over extended periods of time and in a variety of situations. These ratings, when combined with the direct child assessments, provide for fairer and more robust appraisals of children’s skills and competence and such assessments have been found to predict later child outcomes.

The Center Director Interview provides information on the operation and quality of the Head Start and non-Head Start center-based programs. Issues addressed in this interview include: child recruitment and enrollment; staffing; teacher professional development; parent involvement; curriculum; classroom organization and activities; child assessment; home visits; and kindergarten transition. The Care Provider Interview collects comparable information regarding child care for non-Head Start children in non-center-based settings or those who are at home with a relative or non-relative other than the parent/primary caregiver. The interview includes questions on the number of children in the child care setting, staffing, typical child activities, and beliefs on how children should be taught and managed.

Care Setting Observations to Assess Quality

In addition to interviews and surveys, direct observations are being used to assess the quality of each child’s primary non-parental care setting. These tools provide direct measures of the extent to which Head Start centers and other child care arrangements employ skilled teachers/care providers and provide developmentally appropriate environments and curricula for the children. Trained observers spend enough time in each setting to ensure observation of a major portion of the daily schedule and a variety of center, classroom, and family daycare activities.

The following standardized observational instruments and coding schemes that have been widely tested in child development research are being used in the Head Start Impact Study:

• The Early Childhood Environment Rating Scale (revised) (ECERS-R), provides a global rating of classroom quality based on structural features of the classroom including personal care, furnishings, language and reasoning activities, gross and fine motor activities, social activities, and provisions for adults and teachers. For observations in non-center-based settings outside the child’s own home the related
• Family Day Care Rating Scale (FDCRS) is being used.
  The Classroom Observation of Teacher-Directed Activities Checklist is designed to collect more definitive information about the kinds of teacher-guided instructional activities that take place in the classroom.
• The Arnett Scale of Teacher/Provider Behavior is a rating scale consisting of 26 items organized under five areas: sensitivity, punitiveness, detachment, permissiveness, and promotion of independence.

In the interest of having some observational measure of quality across all types of settings, a five-question observational instrument designed for use in the child’s home as well as in center-based and family daycare homes is being used. These supplemental items are completed by interviewers for all primary care settings regardless of whether the setting is Head Start, another center-based program, the child’s home, or someone else’s home and are completed in addition to the standardized observational instrument appropriate to each type of setting, as noted above. Interviewers rate each child’s primary care setting in four areas: overall safety, basic hygiene standards, availability of educational materials, and overall positive and negative interaction between provider and child.

At the kindergarten and first grade level, the study will rely primarily on existing indicators of school quality that have been found to be important determinants of student performance on tests of academic achievement and other indicators of school success. Examples include: school grade composition and enrollment size; economic level of the student population (free and reduced-price lunch eligibility); racial and ethnic composition of the students; average pupil/teacher ratios; average per pupil expenditures; the availability of federal funds (e.g., Title I) and other special programs (e.g., after school programs, and programs for Limited English Proficiency students); teacher and other staff characteristics; measures of school safety; measures of average student achievement by subject area; and, attendance and dropout statistics.

Specifically, the schools attended by the study children will be linked to annual data collected from every public school in the U.S. by the Department of Education’s National Center for Education Statistics (NCES). Two primary data sources will be used: The Common Core of Data (CCD) for Public and Private Elementary Schools, and the Schools and Staffing Survey – Data for Public and Private Elementary Schools. The NCES data will be augmented by linking to state- and district-level data that are publicly available as school “report cards” under state accountability systems.

---

23Based on the winter cataloging of care settings.(back)

24If child has more than one setting between 8am-6pm, only one will be chosen for measuring quality through observation and provider report. To choose the setting, the hierarchy (as listed in the table with #1 Head Start being highest in priority) will be followed, and if child has more than one setting of a certain type, the setting with the most hours will be chosen.(back)

 

Table of Contents | Previous | Next