Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

MSHS Design Development Project Executive Summary

Project Goals

The unique characteristics of migrant and seasonal families are important concerns for the Office of Head Start, yet the research and evaluation efforts that address this segment of the Head Start program have been limited. This study was designed to ascertain the state of research knowledge about Migrant and Seasonal Head Start (MSHS) programs, assess the feasibility of a range of measures and research methods, and select and pilot test methods, instruments, and procedures appropriate for this population. The Migrant and Seasonal Head Start Research Development Design Project was the result of the collaboration between MSHS programs and the Administration for Children and Families (ACF) to determine viable methods for assessing the unique characteristics of programs while also examining common programmatic components. The goals of the study were as follows:

1. To “try out” methods and measures of key constructs for measuring both program implementation and child/family outcomes to determine whether methods and measures currently used in mainstream Head Start programs are feasible, and to identify potential new measurement approaches that may be warranted, particularly those focusing on child multilingualism and family environment;

2. To identify the feasibility of tracking a sample of migrant families across sites, determining where they went, the programs in which they participated, and the degree to which continuity of services occurred within and between streams; and

3. To develop methods to describe the issues and challenges faced by migrant and seasonal farmworker families and how MSHS programs operate to serve their unique needs.

This report is meant to provide an overview of the issues in conducting research in these programs. Caution should be used in interpreting any specific results from this project because they are limited to the programs that participated in this project and thus should not be taken as being representative of all MSHS program; therefore very few specific details regarding responses to measures are presented in this executive summary. The information that is provided should improve future research efforts attempting to accurately describe programs, families and children of the MSHS community.

Background

The Indian and Migrant Programs Division was established in 1969, and in 1984 this division was further subdivided into two branches: the Migrant Programs Branch and the American Indian Programs Branch. Amendments to the Head Start Act, P.L. 105-285, sections 640(a)(2)(A) and 640(1)(2) established the eligibility of seasonal farm workers to receive services through the Migrant Programs Branch, and the name of the programs was changed to Migrant and Seasonal Head Start. There have been few large-scale efforts to study Migrant and Seasonal Head Start (MSHS) families and programs.

It would be difficult to accurately assess these programs within the methodological frameworks used in evaluation of Head Start mainstream programs (e.g., Head Start Impact Study, Family and Child Experiences Survey (FACES). There are several challenges to studying MSHS programs: they are organized differently and with more variations than mainstream programs. In addition, their program participants primarily speak Spanish and are highly mobile. 

MSHS shares the child and family goals of all Head Start programs. They strive to enhance children's growth and development; strengthen families as the primary nurturers of their children; provide high quality educational, health, and nutritional services; link children and families to needed community services; and have well-managed programs that involve parents in decision-making. MSHS endeavors to ready low-income children for successful transitions to elementary school.

However, MSHS programs are flexibly structured to serve their diverse families. Not only are programs designed around the work schedules and migratory patterns of agricultural workers but the services provided are also tailored to the particular competencies and needs of their families. Potentially, this results in an increased focus on child health, family employment and educational support, and the parent-MSHS partnership. MSHS serves as a community resource for migrant families and can be of assistance as the families adapt to communities in different parts of the United States.

Designing a comprehensive evaluation of MSHS programs, with a focus on child and family outcomes and program implementation, requires an initial concerted effort to begin to identify potential research methods and measures. The MSHS Research Design Development Project reviewed and piloted selected measures (i.e., interviews, questionnaires, direct assessments) with appropriate populations (i.e., staff, program directors, parents, children), and assessed process features and feedback, and then further adapted the measures. Assessments of the process focused on the interaction between research and the cultural, theoretical, logistical, and demographic issues pertaining to migrant and seasonal farmworker families. Overall, the Design Development Project represents one step towards identifying useful evaluation methods and measures for future consistent study of these programs.

Preliminary Identification of the Challenges of Researching MSHS Families

The characteristics of MSHS programs produce significant practical problems for implementing research initiatives that will regularly inform the Office of Head Start and the MSHS community about program strengths and needs. This section presents the issues that emerged as a result of preliminary information-gathering activities that began when the project started.  The information presented here is based on reading the literature regarding migrant and seasonal farmworkers, analyses of available datasets, and conversations and discussions with research experts, program providers, and policymakers.

Language. Simply put, more than 95% of the MSHS families speak Spanish. In order for programs to serve as effective cultural and community resources for the families, MSHS makes a strong effort to hire staff and teachers with similar cultural and language backgrounds. Sometimes, this effort is strengthened through programs involving parents themselves as staff or teachers.

MSHS age groups. Migrant and Seasonal Head Start programs serve children from birth to five years of age; approximately half of the children served are under the age of three. Researchers considering evaluation of the programs should therefore vary their measures of program structure and teacher skills, depending on the age of the child in question. Further, direct assessment of these young at-risk children must be done with developmentally-appropriate measures and careful interpretations.

MSHS eligibility variations. In order to qualify for MSHS program, all MSHS families must earn their income primarily from agricultural work that involves the production and harvest of tree and field crops. The Head Start Act further mandates that “migrant” families participating in MSHS must have changed their residence from one geographical area to another within the last two years; “seasonal” farmworking families have not changed their place of residence from one geographical area to another in the last two years. In addition, the client population must meet the Head Start annual poverty guidelines. There are variations in programs’ application of MSHS eligibility requirements (e.g., serving only migrant workers) which should influence the approaches used for assessing the programs as a whole.

U.S. born versus immigrant families. Previous studies indicated that MSHS participants consist not only of migrant and seasonal farmworkers, but also of at least two distinct subpopulations of migrants.  There are U.S.-born migrants from communities with a long history of migration and there are also recently arrived immigrant families who are seeking employment opportunities (Aguirre International, 1997).  In 1997, two-thirds of MSHS parents were identified as being born in the U.S., while one-third were born in other countries (Aguirre International, 1997). Further, the National Agricultural Workers Survey (2001-2002) reports that the proportion of migrant workers who were newly arrived in the U.S. increased by 69% between 1997 and 2002. Measurement and methods of a MSHS program survey should consider the possible language, culture and service variations of these subgroups of migration families.

Geographic mobility. Since the late 19th century, migrant farmworkers have followed “streams” of changes in locality, depending on the harvest and production seasons. Downstream locations are those that have harvest and production seasons in the spring-summer seasons, while upstream locations tend to have harvest and production seasons in the late summer to late fall. According to tradition, there are three migrant streams, from south to north in the United States—Eastern, Midwest and Western. In recent decades, however, these streams seem to have become less organized and distinct, perhaps reflecting inaccuracy in the early streamlined interpretation, or an increasing tendency for workers to either pursue cross-stream migrations or to stay in one location (i.e., settle out) while shuttling to various jobs relatively close to home. Some families also periodically return to their countries of origin during the non-growing seasons. This range of mobility indicates that accurate evaluations of the MSHS programs will have additional challenges if attempting to track families and measure long-term outcomes. Those challenges could include tracking within stream, across stream, across states, across programs, and even across countries.

Continuity of MSHS services. Continuity of services refers to the sharing of information from one MSHS center to another, regarding a migrating family. This can ease transitions and increase the supportive relationship between MSHS and the participating families. Theoretically, continuity efforts can occur within one program (if families change centers within a program) or between programs across the nation. However, there is currently no consistent national continuity effort across MSHS programs. Regional differences in the migratory patterns of MSHS families, as well as the degree to which different Head Start program grantees are organized, appear to influence the continuity of services. Programs with greater continuity structure (e.g, shared databases), more within-program mobility (e.g., family moves from one center to another within a program), or with more “settled out” families, will have more success with continuity efforts. For researchers, continuity is of particular interest if the families are going to be tracked over time in order to consider long-term MSHS outcomes and migratory patterns.

This is only a brief overview of some of the characteristics of MSHS and migrant and seasonal families that would shape studies examining the programs. Numerous other features might also need to be considered in future research. Of high importance are the cultural considerations that could influence the families’ perceptions of MSHS, education, and research. The assessment activities of the MSHS Research Design Development Project were designed to incorporate direct feedback from the families, staff and teachers to increase the validity and utility of the research measures and methods.

Design

The project designers piloted and adapted interviews, direct assessments and questionnaires and rating scales with the staff, teachers, children and families of MSHS. The project consisted of a preliminary phase, a pilot study including focus groups, and on-site visits. The on-site visits usually involved a preliminary visit to introduce the project and on-site data collection visits during which the full range of assessments and interviews were completed.

Selection and development of measures continued throughout the project, as interview formats and protocols were adjusted and adapted, incorporating the feedback from interviewers, parents, teachers, center directors and staff. Of primary importance throughout the project was evaluation of the methods and measures themselves, in order to fully inform a future, larger-scale study of the MSHS program. This research development was one goal of the project. However, since the measures changed over time, any results describing the families and centers as a whole are necessarily limited. The evaluation included specific gathering of feedback through focus groups and debriefing questions, consideration of process and implementation features and, to the extent possible, careful analyses of the variability and reliability of the assessment tools.

Preliminary Phase

The preliminary phase included a literature review, discussions with key constituencies serving migrant and seasonal farmworker families, meetings with the Technical Work Group (TWG; See Appendix A), and focus groups with migrant families and MSHS teachers/staff (MSHS Conferences, 2003 and 2004). These activities served to highlight the major issues to be addressed by the project, to shape the selection of measures and the development of new instruments, and to inform the field work procedures.

Pilot Study

A pilot study was designed to determine whether 1) the selected measures were understood by the respondents, 2) constructs could be translated for bilingual (Spanish-English) speakers, 3) scores on the children’s assessments demonstrated levels of variation equivalent to those found with other Head Start groups, and 4) the data collection fit within reasonable time and burden limits. Four MSHS sites, selected on the basis of convenience, previewed the instruments, with approximately four parents and four teachers from each site participating in individual interviews and focus groups. Preschool-aged children of these participating parents completed pilot testing of the child assessments. These families and teachers were very cooperative and informative regarding their perceptions of the interviews and questionnaires, and their input resulted in several important adjustments to the measures and methodology.

Site Visit Component

The evaluators asked program sites to participate based on characteristics that were of particular relevance to the goals of the study;  it is important to remember that this was not a representative sample of programs. For geographical and seasonal diversity, the project collected data from all three migrant streams—Eastern, Midwest and Western—as well as data from both downstream and upstream sites.  A total of seven sites participated in the on-site data collection (Florida, Texas, California, North Carolina, Tennessee (Follow-up Interviews Only), North Dakota, and Oregon). See Figure 1 for a graphic representation of the study design.

Preliminary visit.  A preliminary visit to each center proved invaluable to increasing participation, staff support and parent consent. This was an opportunity to share project goals and details, and answer participants’ and staff’s questions.

Onsite data collection visits. For on-site data collection visits, researchers conducted child and family assessments, interviews with teachers, administrators and program staff, and monitored the migration plans of families. Downstream data collection was done in April and July, while upstream data collection was done during September and November. Figure 2 indicates that the expectation was to collect data from 40 preschool-aged children per site for the preschool-aged child assessments and 40 families for the parent interviews. However, this plan assumed a one-to-one correspondence between families and preschoolers (i.e. one family=one child in the center), and did not take into account the fact that a large number of families had more than one child enrolled in the program. Given the sizes of the classrooms involved, it was not possible to select the requisite number of unrelated preschoolers and infants/toddlers and arrive at the target number of families. Only 134 preschool children were assessed. Since sibling participants appear to frequently occur in MSHS centers, it may work for future research projects to collect information about all participating family members. The unit of analyses in such a design could be at the family level (not child level) within MSHS centers, thus allowing the inclusion of siblings.  

Assessments and Interviews

See Figure 2 for evaluations planned for each site.

Child assessments. The direct child assessment for preschoolers was a battery of tests similar to those used in FACES, measuring different aspects of emergent literacy, language, and pre-academic skills.1 The direct assessment for infants and toddlers consisted of the Preschool Language Scale Fourth Edition (PLS:4; Zimmerman, Steiner & Pond, 2002).

Teachers of preschoolers rated the children primarily in terms of social and classroom behavior (Teacher Child Report), whereas teachers of infants and toddlers were asked to rate the children across a number of developmental domains, using items from several well-known developmental checklists (Ages and Stages Questionnaire (Squires, Bricker & Potter, 1997); MacArthur Communication Developmental Inventory (Ariaga, Fenson, et al, 1998); Minnesota Developmental Inventory (Saylor & Brandt, 1986). Parents also completed questionnaires considering the early development of their children.

Dual language development (and, in some cases, multilingual language development) presents specific challenges for both children and program evaluation researchers. In order to identify preschoolers who were developing multiple language skills, language screeners were used as initial assessment tools. All preschoolers completed the Spanish language version of the screener tool and, if they passed, went on to complete additional assessments given in Spanish. Then the children were given the English language screener and those who passed completed additional assessments given in English.

Parent interviews. Individual in-person interviews were conducted with the parents of the study children. Topics included family background, demographics (socioeconomic circumstances; household structure; languages spoken in the home), parent-child interactions, use of social services, experience with MSHS programs, detailed questions about the parents’ past work and migration patterns and use of community services (within the past year), and their future work and migration plans (within the 6 month period following the interview). The final section of the initial parent interview consisted of a formal debriefing that asked parents to provide their reactions to the overall interview and to particular items.

As noted in the introduction, development of the measures was a primary goal of the project. Perhaps more than the other measures in the Research Design Development Project, the parent interview changed frequently during the project. Adjustments were based on parent feedback and interviewer observations. Over the course of the preliminary phase pilot tests, the interview was shortened substantially from an average of 1.75 hours at the first site to one hour at the final site.  (Refer to Interviews from the Migrant and Seasonal Head Start Design Development Project, Spanish Version and English Version).

Center director and coordinator interview protocols.  This component focused on how MSHS programs organized to offer useful support to migrant and seasonal farmworker families and their children. At six MSHS programs, a series of structured interviews were conducted with center directors/coordinators/staff and program directors. For simplification and accuracy, the original protocol changed from conducting two interviews for each of the MSHS centers (one for the program director and a second center staff survey), to a combined protocol that obtained information from the appropriate members of each given Head Start program community (e.g., education coordinator, health coordinator, family service workers, program and center directors). (Refer to Interviews from the Migrant and Seasonal Head Start Design Development Project, Spanish Version and English Version).

Teacher interviews. The 19 teacher interviews were approximately 30 minutes long and focused on classroom activities, curriculum, and teacher background and experience. During the preliminary phase, minor modifications were made to the teacher interview to eliminate redundancy and clarify concepts. The teacher interviews concluded with a 5-minute debriefing, asking the teachers to report regarding their interview experience. (Refer to Interviews from the Migrant and Seasonal Head Start Design Development Project, Spanish Version and English Version).

Classroom observations.  Additionally, structured classroom observations were completed in one preschool classroom in each center visited, using standardized observational methods widely used in child development research (Early Childhood Environment Rating Scale-R; Harms, Clifford, & Cryer, 1998; Arnett Scale of Caregiver Behavior; Arnett, 1989).

Tracking and Locating Component

A subcomponent of the study assessed methods of finding families once they had moved from the program in which they were originally assessed. This was to inform future research efforts regarding the feasibility of longitudinal assessment of program outcomes. 80 families from the downstream Midwest (Texas) and Eastern (Florida) streams formed the “tracking group” of study participants. Researchers identified the location of these families at two time points over a gap of 4 to 6 months, to ascertain whether migrants can be found as they followed the harvest upstream. 

The “tracking” group of families was divided into two sub-groups for consideration of two tracking methods. The multi-source method (N=18) for tracking families involved contact with program staff from both the sending and receiving MSHS programs, and telephone contact with family members and the parents. The 18 families tracked with the multi-source methods were selected because of their stated migration plans: these families planned to migrate to 2 geographically clustered locations which simplified follow-up efforts.2 The single source method (N=61)3 of tracking families involved only contact with program staff from both the sending and receiving MSHS programs.

[D]

 

[D]

 

Learning from the Research Process: Obstacles and Adaptations

The preceding section listed the measurements used for various members of the MSHS community (i.e., parents, children, teachers, staff). Beyond the initial implementation of the measures, a second important component of this project was assessing and developing the measures in the context of the MSHS programs. Were the types of questions posed well-received by the parents and program personnel? How could the interview process be facilitated? In addition, the researchers, ACF , and the MSHS programs wanted to make sure that the questions resulted in important and meaningful information.

What follows is a description of the themes and challenges perceived by the participants, derived from the debriefing portion of the interviews and from focus group feedback. Often, the process information obtained from the respondents was applied directly to the interview protocols, resulting in adjustments to the interview questions over time. The time and energy contributed by the families and staff to this process was extensive, and their input significantly and positively influenced the research development procedure.

Consent Rates

There were significant differences in consent rate across sites (range 39% to 100%). Several factors could have contributed to this variation. These include:

Age of children: Consent rates appeared to vary by age of children, as the consent rate was somewhat higher for infants and toddlers (70%) than for preschoolers (63%).

Data collection timing: The first two sites had lower consent rates compared to the last three sites, perhaps a result of researcher experience or families’ migratory plans.

Families’ degree of contact with the center. It seemed that the more established the parent-center relationships, the higher the consent rate for participation in the research.

Wording of the consent form: Some individual families were hesitant to consent, due to the concerns regarding mandatory reporting of potential child abuse to authorities. Discussion with MSHS staff eased these concerns for some parents.

Regional cultural differences. Potentially, regional and program differences in families’ culture, acculturation and immigration status may also influence consent rates.

Table 1 shows the number of cases completed for each measure.

 

Table 1. Summary of Data Collection by Source and Instrument

		Target (see Figure 2)	Cases Selected	Cases Completed	Response Rate
Preschoolers	Direct Assessment	150	142	134	94%
	Teacher-Child Ratings	150	142	137	96%
Infants-Toddlers	Direct Assessment	15	15	15	100%
	Teacher Ratings	60	59	58	98%
Parents	Full Initial Interview	195	194	194	100%
	Second-Child Interview	30-50		30
	Follow-up Interview	15-20	18	17	94%
Teacher Interview		18	19	19	100%
Classroom Observation		8	8	8	100%
Directors/Coordinators Interview		9	9	9	100%
Grantee Director Interview		6	6	6	100%

 

Learning from Parent Interviews/Focus Groups

The parent interviews successfully gathered descriptive information from 194 parents. The majority of the parent respondents were the target child’s birth parents (97 %) followed by a grandparent and/or a godparent (3%). Mothers, in particular, were the most frequent respondent (66%), although a substantial number of interview sessions included family members in addition to the mother (33%).

When developing the parent interview, researchers continually counterbalanced the need to gather comprehensive information with the need to limit the length of the interview. Adaptations to the interview and themes gathered from parent feedback and focus groups are discussed below.

Attitude towards research. Parents’ level of trust in the center seemed to be associated with parents’ approval of the research interviews and assessments. As one parent mentioned, “whatever the center says, we will follow because we trust them.”  Thus, in some cases, the research team received a “transfer” of confianza (trust) from the MSHS center staff. However, a few parents suggested that the term for research in Spanish (investigaciónes) has a negative connotation that may need to be clarified to MSHS parents in order to increase participation.

Interview length.  Although the interview was progressively shortened throughout the research project, some parents continued to state that the interview was too long. Several parents felt that the tone and content of the parent interview were too similar to a psychological exam or a school test and at times they were hesitant to respond to personal questions for fear of giving ‘incorrect’ answers. As the interviewers developed familiarity with the questions over the course of the project, they tended to increase the conversational nature of the interview process and thereby may have reduced the “testing” tone.

Translation limitations. The parents reviewed the translation of the questions, and inaccurate or awkward translations were identified and corrected. Some questionnaires were particularly repetitive when considered in translation (e.g., depression scale).

Likert-type response scales. One major challenge to the methodology did arise during the original interview process. The original interview included a number of sections that involved the use of a Likert-scale, in which the parent was asked to consider a set of declarative statements and select their response to the statement from a 5-point scale. Descriptive phrases for each point on the scale indicated a level of quality (e.g., “Not at all like me”) or frequency (e.g., “very often”).  Such Likert-type responses were used for sets of items regarding parenting beliefs, depression, and locus of control. MSHS parents were often reluctant to limit their answers with Likert-type scale or categorical responses. Questions using these methods often elicited complex answers that were prefaced by the words “it depends.” Some parents apparently responded with “it depends” because they wanted to provide more exact and contextualized information than would be gathered by the fixed-choice format.

Two strategies were attempted to make the Likert-type scales more effective. The first strategy was to limit the number of response options (e.g., three-point instead of five-point), while the second approach was to convert the Likert-type items into open-ended or narrative questions. Reducing the number of response choices (from 5-points to 3-points) was somewhat helpful when combined with the standard practice of using reminder cards to represent the 3-point scale. The alternative strategy, leaving questions open-ended in a narrative manner (in which parents could respond as they wish) was problematic because, without the additional structure, some parents felt that there must be a “correct” response to give. However, throughout the interview, whenever parents were asked to think of additional responses to fixed-choice questions (i.e., Thank you for choosing from our categories; Is there any other way that you might respond?), many gave an informative variety of alternative responses.

Parenting beliefs. There were no culturally relevant or appropriately standardized measures of parenting attitudes available at the time this project was initiated; The Parenting Belief Scale (previously used in FACES) was used in an exploratory attempt to consider these constructs with the MSHS families. This scale asks parents to report their level of endorsement of a series of statements regarding parenting attitudes.

The families did appear to have some consistent difficulties with this questionnaire. First, statements for the Parenting Beliefs Scale used the Likert-response method, which was often problematic. Many parents wanted to be particularly informative regarding variations in their beliefs and attitudes, depending on the context. In addition, at least for this sample, some parents of very young children felt that one or more of the Parenting Beliefs (i.e, “I encourage my child to be independent”) were not developmentally appropriate.

It might be particularly useful to ask for specific feedback regarding any measures of beliefs and attitudes, as they will be strongly influenced by culture and specific feedback will aid in interpretation.

Access to services. For this section, interviewers first asked parents to identify which services they were receiving, using a list of community services. For each service that they were not receiving, the parents were then asked to choose a reason to explain why. The list of potential reasons worked well for some types of community services, but not for others. In order to accurately assess the service gap (between families’ need for services and actually receipt of services), it appears that the questions may need to be asked more directly.

Activities done with child by household members. Interviewers asked parents about interactive home activities for the child; these questions regarding the parent’s or household members’ involvement in activities with the study child appeared to work well.  Overall, parents reported an average of 7 activities per week with an informative amount of variation in their responses. It could be very informative to collect additional information about which household members performed which activities, in order to identify the host of socialization and developmental resources available to the MSHS children.

Questions about the amount of time parents read to their child, either during 1) the past week or 2) in one sitting, produced inconsistent data. From the results, it appears that some parents read to their children for zero minutes, but also read to their child at least once a week. There may be cultural differences in the perceived importance of literacy activities, and it is possible that misinterpretation or social desirability bias might be raised by the repeated questions on reading activities.

Discipline. Prior to the implementation of the interview, there had been questions from focus groups and the technical work group about whether parents would feel comfortable reporting about the full range of discipline activities. Of the parents responding to the discipline questions, 85% reported sending the child to a corner and 58% reported spanking (among other responses), suggesting that parents were relatively comfortable reporting about all discipline types. This might be further explored with specific feedback questions. In addition, discussion regarding such sensitive issues reminds researchers regarding the importance of clearly and accurately explaining research rights (e.g., confidentiality, right to refuse to participate) and limitations to those rights (e.g., abuse reporting laws), particularly with newly immigrant families.

MSHS satisfaction. A number of parents chose to not respond to some of the questions regarding satisfaction with MSHS services. Future work should pursue feedback and clarification, as accurate measure of parents’ satisfaction and suggestions will only serve to improve MSHS services.

Suggestions for child assessment. Parents also provided information about the child assessment methods.  Parents indicated that, in order for the assessments to be successful:

• the evaluator would need to speak the child’s language,

• the evaluator should be skilled at adjusting the conversation to the child’s vocabulary level,

• sessions with children should be done over several days,

• care should be taken to maximize familiarity with the evaluator, and

• teacher presence would make the process smoother. 

Learning from the Child Assessment Process

A number of important specific logistics issues were identified regarding the child assessments. These issues were identified from the teacher, parent and evaluator perceptions of the assessments of the MSHS children:

• presence of a teacher and a parent facilitated the child assessment;

• assessments were perceived as more valid if conducted after the child was settled in program (for at least three weeks);

• simplification of some assessment instructions appeared to reduce the effect of language on performance;

• development of standardized additional probes regarding instructions also appeared to increase the accuracy of the results;

• the Australian numeracy assessment (using concrete items to count) appeared to be less language-dependent and was therefore adopted as the measure of numeracy; and

• screening the preschool children for both English and Spanish language skills was very important for assessment of these bilingual children, in order to accurately identify skills in both domains.

The assessment process was an extended trial of methods and measures, testing the applicability and acceptability of the procedures and gathering preliminary information about reliability and validity with these MSHS children. In other words, the MSHS Research Design Development Project addressed the feasibility of the measures and not the abilities of the children. Extensive analyses, considering the variability of the measures, sensitivity to development, and overall reliability, resulted in the condensed information presented below.

There were some measures that did not appear to differentiate among children with lower level abilities (i.e., a ‘floor’ effect). These included, for example, Spanish Letter-Word Identification and the English and Spanish Story and Print Concepts tests. Future work with larger samples should be closely considered to further evaluate the viability of these measures with these populations. The measures did demonstrate an appropriate tendency to become easier across the age groups assessed (i.e., the older the child, the easier each of the tests tended to be). Table 2 reflects interpretation of the Standard Error of Measurement (SEM) identified for each assessment tool; standard errors of measurement provide a measure of test precision. No measure should be rejected out of hand based on results from such a small sample size (N=134), but the interpretations suggest areas where there is a potential need for further study and alternative measures.

 

Table 2. Assessments of Child Assessment Measures: Usefulness for Assessment of the MSHS Sample

	Spanish Tests		English Tests
Child Assessment Measure	Standard Error1	Reliability2	Standard Error1	Reliability2
Language Screener (Pre-LAS)	Excellent	Excellent	Excellent	Excellent
Vocabulary (TVIP/PPVT)	Excellent	Excellent	Excellent	Good
Letter Naming	Excellent	Good	Excellent	Excellent
Letter-Word Identification (Woodock- Muñoz/Woodcock-Johnson)	Marginal	Poor	Good	Good
Number Knowledge Test	Marginal	Marginal	Excellent	Good
Story and Print Concepts	Marginal	Poor	Marginal	Marginal

1 Excellent = SEM’s ≤ .37, Good = SEM’s ≤ .49, Marginal = SEM’s ≤ .59, and Poor = SEM’s > .59. 2 Excellent = reliability coefficients ≥ .85, Good = coefficients ≥ .75, Marginal = coefficients ≥ .65, Poor = coefficients < .65.

 

Preschool assessment. One hundred and thirty four assessments were completed; only 7% of the preschoolers were either too shy or too restless to complete the procedures. Preschool children assessed for this project tended to be at the younger end of the preschool age range, as 41% of the sample was 3 years old. Each child attempted the Spanish and English language screeners, in order to measure their abilities to try the additional assessments in each language. The largest proportion of the children (40%) were administered only the full Spanish assessment (which means they passed the Spanish screener but not the English screener) while 27% of the children passed enough items on the English screener but not the Spanish Screener, and therefore attempted only the full English assessment. One-quarter of the children (26%) passed both screeners and thus attempted both the Spanish and English full assessments. These findings suggest, in general, that the separate language screener approach is potentially viable with this population.

Infant/toddler assessment. With regards to the infant and toddler assessment (the Preschool Language Scale 4th Edition), this small exploratory portion of the study was only able to demonstrate that direct assessment of MSHS infants and toddlers was possible (N=15). Assessments with these young children occurred only when the parent was present, in order to increase the child’s comfort level. Some attending parents appeared unsure of their role in the assessment process, and further standardized coaching for parents might reduce parent and child anxiety and increase parental assistance during the assessment.

Ratings. The teacher responses to ratings of preschoolers’ social skills, early literacy skills and engagement in learning appeared to result in an appropriate range of variability.

For the infants and toddlers, their teachers completed developmental checklists. Although they may have some value when considered in a larger sample, many items on the developmental scale demonstrated either a ceiling or floor effect. In other words, teachers tended to think that some emergent literacy and language items would generally be too difficult for their classroom’s children, while fine and gross motor items reflected activities that were too easy. Adjustments and refinements of the infant/toddler developmental rating scales, or identification of alternative established scales, should be considered for future projects.

Learning from the Teacher Interviews

Separate interviews were developed for teachers of infant/toddlers and teachers of preschoolers. About three-fourths of the teacher interviews were conducted in Spanish (by teacher choice). Interviews were conducted with 19 teachers in 5 of the 6 MSHS project sites: 7 teachers in infant/toddler rooms and 12 teachers in preschool classrooms. Clearly, this was a very small and non-representative sample, and allowed for only very preliminary exploration of the range and variety of responses to the interview questions.

Topics included classroom population, typical daily activities, classroom environmental features, developmental knowledge, languages and curriculum used, and administrative supports (such as mentoring and coaching). Teachers were also asked about the frequency and format of communications and interactions with parents. Finally, teachers were asked about their perceptions of the programs, satisfaction with their work, and their plans to return to MSHS the following year.

In the focus groups, done as part of the pilot study for the project, teachers discussed the kinds of assessments used with children, and what they felt were the most critical outcomes of MSHS programs. Teachers hoped to make differences in the lives of the families and they discussed numerous outcomes that could reflect the impact of MSHS, including:

• parents knowledge of the importance of a healthy child and family;

• parents’ attitudes towards education;

• family awareness of community resources, particularly literacy and ESL supports;

• parent awareness of adult educational opportunities;

• father involvement;

• parent participation in MSHS activities;

• increased socialization; and

• increased communication.

The teachers urged the evaluators to “spend time in their classrooms,” in order to observe the children within the educational settings. Teachers listed a number of changes in children’s skills and development that could reflect the impact of MSHS programs.  These included:

• improved self-esteem,

• improvements in motor skills,

• earlier identification of developmental delays or special needs,

• increased appropriate social play,

• language development, and

• school readiness.

These MSHS teachers generally felt that it was important that outsiders know that the MSHS programs make a difference for both parents and children. 

Suggestions for researchers. Although these teachers tended to be supportive of MSHS research, they emphasized the need for confidentiality and a sensitive understanding of the impoverished conditions of the community that they served. Teachers also cautioned evaluators not to make the mistake of ‘labeling’ the MSHS children when using standard evaluative tools.

Likert-scale. Teachers tended to find the Likert-scale responses to be too limited, much as did the MSHS parents: Likert-style questions regarding child development often produced an “it depends” answer. Teachers frequently wanted to emphasize the range of student behavior and not focus on the average. They noted that the frequency of behaviors would depend on the age, the amount of prior exposure to MSHS classrooms, and even the time of the year that the teacher is asked the question. 

Age and salary. Teachers were consistently very hesitant to give their age and salary, reportedly due to discomfort with sharing such information, and sometimes due to concern that the numbers would be used to assess their relative abilities and cost to the programs.

Uniform questions. Some questions appeared to address issues that were either very uniform across programs or had a heightened sociability bias. Teachers reported very uniform answers for:

• all three job satisfaction questions,

• presence of educational toys in the classroom,

• scheduling and conducting parent conferences/home visits, and

• curricula in use in classroom.

Questions that do not provide useful information regarding variation across programs should be dropped from future research.

Learning from Center Director/Coordinator Interviews

The final version of the Center Director/Coordinator Interview was designed as one protocol, combining information from multiple sources in order to reduce redundancies and effort required from the program. Copies of the interviews were sent to the sites ahead of time so that center directors could complete a preliminary summary of some of the more “factual’ data about their centers and programs, such as the number and ages of children, the number of children from migrant versus seasonal families, and the numbers and types of staff working at the center.  Staff and the center director were asked to address the following issues in the direct interview: agreements with community resources; enrollment, waiting lists and program expansion efforts; health services and health concerns for client population; teacher training and role of the education coordinator; teacher turnover; Family Needs Assessments; assessments used to measure children’s progress; kindergarten transition activities; parent involvement activities; and staff’s perceptions of MSHS.

Center directors identified five parent-oriented goals as ‘most important’ for MSHS programs:

• informing parents about their own child’s development,

• helping parents identify their personal goals and ways to achieve them,

• informing parents about the support services in their community and help them to use them,

• helping parents become economically self-sufficient, and

• helping parents improve their literacy skills.

When center directors were asked to report on two activities that the MSHS programs performed well for children and families, the following responses were given:

• “Provide the services for children and for parents, for them to feel that we are not just here for a paycheck but they can come to us if they have a problem.”

• “We provide the best quality child care services to the children.”

• “We support the values of the family and value each as unique individuals.”

• Communication and information to parent through conferences, home visits, screening, parent meetings;

• Provide services in physical health and speech to parents and children and working with other agencies in making sure they get these services; and

• Provide a safe and pleasant environment.

These themes were generally reiterated by the health coordinators, education coordinators, and family service workers.

When asked for areas for improvement, the most common issues raised were adjustment of MSHS qualification and the income eligibility requirement for families, as the guidelines limit the families that could be served. Additional issues mentioned were staff salaries, staff turnover, maintaining health and education coordinators as separate positions, and the length of times that programs were open each year.  Leadership and administration training were also discussed.

Overall, the interviews with the center directors and coordinators went well and respondents gave thoughtful answers, though a few procedural issues were observed. As with teachers, the center directors and staff were reluctant to discuss age and salary. There were again several questions that received consistently uniform responses across sites, suggesting that the questions will not provide differentially useful information (e.g., problems faced in expansion, parent participation efforts, parent communication efforts, etc.). This could partly be caused by social desirability bias (e.g., directors wanting to endorse a full set of positive activities for their center) and partly because the behaviors may actually be universally present in MSHS programs. Given a long interview, questions where little variation in answers was found should be either eliminated or adjusted to increase the amount of new information that can be obtained.

Learning from the Tracking and Locating Component

To measure long-term outcomes of program participation, families and children must be followed as they change locations so that data collection can take place at least two time points. The MSHS Research Design Development Project hoped to assess the extent to which the families sampled in downstream locations could be located at upstream MSHS program locations. As discussed above, the tracking sample for this project consisted of 80 participating families from two downstream sites.

The two upstream areas where migrant families in the project would be geographically clustered were identified during the initial interview, one in Tennessee and another in North Dakota. For these geographically clustered families (N=18), multi-source tracking methods were used (i.e., contact with program staff, family members, and parents). Although one family withdrew from the study, 61 other families were tracked using a single source method (i.e., contact with programs’ sending and receiving staff). Parents from both groups were also asked to complete periodic postcards, notifying project staff of moves. Respondents were paid $5 for each returned postcard.

Tracking and locating success rate. Overall the tracking and locating efforts were successful, albeit with some differences in methods and success rates for the multi vs. single source samples (see Table 3). On average, the multi-source tracking sample involved more than twice as many contact attempts and took more than three times longer than the single source methods. The success rate for the multi-source tracking sample was slightly higher than that for the single source tracking sample (94% and 89% respectively), although the researchers suggest that the relatively small differences may not justify the significantly larger effort. Only seven families sent the postcards in over the course of the project, so this intervention did not appear to increase tracking effectiveness.

 

Table 3. Tracking of Families from Downstream to Upstream

	Multi-Source (n = 18)		Single-Source (n = 61)		Total (N = 79)
	Mean		Mean		Mean
Total No. of Attempts to Contact Families	6		2		3
Days Spent Contacting	57		16		25
	N	Percent	N	Percent	N	Percent
Families Found	17	94%	54	89%	71	90%
Winter Location Matched Parent Report in Spring	14	82%	21	39%	35	49%
Study Children with Family in New Location	26	100%	67	85%	93	89%

 

It is important to note that this locating and tracking effort was to assess the possibility of finding families over time in order to allow for longitudinal analyses of MSHS outcomes and family migratory patterns in future research efforts. It was not planned to address the continuity possibilities between MSHS programs.

Conclusion

The invaluable feedback and cooperation offered by the MSHS programs, parents and children allowed for the initial development of measures and methods appropriate for MSHS populations. This project could not have been completed without substantial input from the Technical Working Group as well as from a variety of administrators and participants of the MSHS programs. The researchers also gained additional perspectives from attending and  presenting the project to the national MSHS conferences.

Consistent and detailed descriptive assessments of MSHS programs, staff, parents and children would provide invaluable information to Office of Head Start. Although this project’s exploratory nature did not allow for any firm conclusions about the evaluation methods that should be applied to MSHS programs, multiple lessons were learned regarding methodology and implementation. These lessons were derived from development and direct implementation of measures with the appropriate populations; incorporation of the resulting feedback and observation will lead to even more effective and valid measures and methods in future MSHS studies.

---

1 For Spanish: Screener Spanish Pre-LAS, Test de Vocabulario en Imagenes Peabody, Spanish Letter-Naming, Spanish Woodcock-Johnson Letter-Word Identification, Spanish “Australian” Number Knowledge Test, Spanish Story and Print Concepts; For English: Screener English Pre-LAS, English Peabody Picture Vocabulary Test, English Letter-Naming, English Woodcock-Johnson Letter-Word Identification, English “Australian” Number Knowledge Test, English Story and Print Concepts Test (back)

2 8 of these families reported that they would be in Tennessee within the next six months, while 10 reported that they would be in South Dakota. (back)

3 One family withdrew from the program just after giving consent, and thus was not tracked. (back)

 

Table of Contents | Previous | Next