ACF OPRE: Evaluation of the Head Start Family Service Center Demonstration Projects (FSC)

Executive Summary

This executive summary highlights findings from the evaluations of the Head Start Family Service Center (FSC) Demonstration Projects. The final report of the evaluation includes two volumes. Volume I contains the report of the national evaluation, which describes program services, participants and impacts across 25 FSC projects. Volume II contains a summary of the local evaluation reports conducted by third-party evaluators in each FSC project.

Family Service Center Demonstration Projects

The FSC demonstration projects were initiated in 1990 to enable Head Start programs to provide a more comprehensive set of services and enhance Head Start’s capacity as a “two-generational program” that offers services to both parents and children. Two key features of an FSC project were (a) collaborative efforts with community organizations, and (b) intensive case management that included a needs assessment and integrated services for families.

The design for the FSCs rested on a set of four assumptions:

Head Start families have important yet unmet needs in three areas: literacy, employment, and substance abuse.
Head Start, as currently constituted, is unable to address those needs adequately because of the large caseloads carried by social work staff, which make it difficult for them to provide the focused attention many families need.
FSCs will help meet family needs by reducing caseloads which will increase the likelihood of families receiving needed services.
These services will result in improved family economic and psychological well-being.

The FSCs were three-year demonstration projects funded by grants from the Administration on Children, Youth and Families (ACYF) within the U.S. Department of Health and Human Services. All Head Start grantees were eligible to apply for the funds. A total of 65 FSC projects were funded by ACYF in three cohorts or waves over three fiscal years (12 from Wave I, 28 from Wave II and 25 from Wave III), with the average grant totaling $250,000 a year. Projects were located in 36 states throughout the country, including projects associated with Migrant Head Start and Head Start programs on Indian Reservations.

National Evaluation Design

On September 30, 1991, Abt Associates Inc. was awarded a contract to conduct a national evaluation of the FSC projects, each of which was also participating in a site-specific study conducted by a local evaluator. The national evaluation addressed three main questions. The first question focuses on program processes, while the other two address short-term and long-term outcomes.

Questions Addressed by the National Evaluation

How was the program implemented?

Were there effects on service utilization?

Were there any effects on families?

All of the 25 Wave III projects were required to implement a design in which interested families were randomly assigned to the FSC or to a control group that received regular Head Start services. Random assignment was carried out by Abt Associates in collaboration with the local evaluators at each site. These are the projects that were included in the national evaluation.

Findings of the National Evaluation

The Extent of Participants' Unmet Needs

Figure 1
Percentage of FSC Participants
with Prior Work Experience
[D]

Low literacy skills were not a major problem for the participants. Contrary to expectations, a majority of the FSC participants had high school diplomas or the equivalent, and most scored in the highest category (high school) on a test of functional literacy administered at entry into the program.

Employment, the second area targeted by the program, was a problem for many participants. In spite of their higher than expected educational and literacy levels, more than half of the participants had not worked during the year before they enrolled in the program, and about 15 percent had never worked.

At baseline, only a small proportion of adults reported current or prior problems with alcohol or drugs. Based on self-reported data, approximately 10 percent of target adults and 25 percent of spouses or partners were reported to have drunk five or more drinks in one sitting on more than one occasion in the month before they entered the program. Smaller percentages of target adults and their partners were reported to have used an illegal drug, usually marijuana, in the same period.

Thus, the majority of adults only reported needs in one of the three areas of focus. The only area where a substantial impact would be expected is in employment.

The Need for Additional Case Management

The assumption underlying the FSC was that intensive case management was essential to meeting families' needs. In the regular Head Start programs, caseloads averaged 75 families and a quarter of social service staff had caseloads of more than 100 families. Thus, if there were a large unmet need, Head Start social service staff would not have time to address it.

Case Management Services Provided by the Program

The program increased access to social workers or case managers. FSC participants were much more likely to have met with a social worker or case manager than were families in the regular Head Start program (78 percent versus 28 percent).

Figure 2
Percentage of Target Adults Who
Met with a Head Start/FSC Case Manager
[D]

Figure 3
Average Caseload Size for
FSC Case Managers
[D]

Caseloads tended to be small. The average caseload size in the Wave III FSC projects was 23 families. Only three percent of case managers had caseloads of more than 40 families.

Contact with families was frequent and often face to face.

Over a third of the FSC families had in-person contact with their case managers on at least a weekly basis.

Case managers spent as much time on families' basic needs and personal issues as they spent on literacy and employment needs. Case managers most often rated families’ basic needs as the primary topic on which they spent time. Literacy, employment, and personal issues were all among the top five topics discussed with families. Half of the case managers indicated that transportation and child care issues required their attention as well.

Effects of the FSC on Participants' Use of Services

More FSC adults participated in educational programs or employment services than did adults in regular Head Start. FSC adults participated more in: General Education Development (GED) preparation, Adult Basic Education (ABE) classes, computer instruction, employability classes, job training, and assisted job search.

Adults in FSCs were more likely than those in regular Head Start to report that they were working toward a diploma or degree. However, there were no differences between the groups in actual diplomas or degrees attained during the time frame of this evaluation.

A greater proportion of FSC adults than adults in regular Head Start participated in some type of drug program. In general, participation in drug programs was low across all FSC projects, which could either reflect a lower incidence of substance abuse problems than initially hypothesized or a greater difficulty in identifying or acknowledging these problems.

Figure 4
Percentage of Target Adults Working
Toward Degree or Certificate
[D]

Effects on Participants' Literacy, Employment, and Substance Abuse

FSC families, compared with families in regular Head Start, received more attention from case managers and participated more in educational and employment services that could help them move toward self-sufficiency in the future. However, these activities did not translate into measurable impacts in the areas of literacy, employment, or substance abuse during the time of the evaluation. There are several possible explanations for this absence of long-term program impacts.

Impact was difficult to show in literacy or substance abuse where need was reportedly low.
Differences between the FSC and Head Start adults in participation rates in services may not have been sufficient to effect changes in longer term outcomes.
Economic self-sufficiency is difficult to achieve, particularly in a short time period (i.e., 19 months).
When programs broker services from other community agencies, there is little control over these services, and the quality of the services will vary.

Findings of the Integration Study: Reported Effects of the FSCs on Head Start Programs

The national evaluation included a study of the extent to which the FSCs had become integrated into regular Head Start programs after the demonstration had ended. The integration study examined how the FSC case manager, as well as services in literacy, employment, and substance abuse, were incorporated into Head Start at the end of the three-year demonstration period.

Results from discussions with FSC and Head Start staff indicated that:

The staff and services of the FSC were successfully integrated into local Head Start programs after the three-year demonstration ended.
Regardless of the particular integration approach used, caseloads in regular Head Start programs that had an FSC have been reduced.
Most programs still focus on literacy, employment, and substance abuse.
The FSC demonstration has increased the visibility of Head Start in the community.

Figure 5
Average Caseload Sizes of Head Start
Programs During and After FSC Demonstration
[D]

Figure 6
Percentage of Head Start Programs Keeping
Same or Greater Level of Focus in FSC
[D]

There were a number of positive organizational changes resulting from the FSC, including stronger family focus, increased coordination among Head Start components and staff, and increased parent involvement.

Description of Local Evaluations

Evaluation Context

The local evaluations were the responsibility of the individual FSC projects who hired an independent evaluator to conduct an evaluation responsive to the specific demonstration project. Wave I grantees were given considerable freedom in designing their local evaluations, and many focused on formative evaluations and collaborative feedback to program staff. The FSC grant announcements for Wave II projects listed a number of required components of their evaluation plan, including that the evaluation contain both formative and summative information about program activities and participant outcomes, and that the evaluation design should allow for a comparison group and repeated measurement of child and family outcomes. The evaluation guidelines for the Wave III FSC projects were even more prescriptive, with the specific requirement that they be able to recruit 80 families to be randomly assigned to the FSC or regular Head Start. FSC project directors were required to submit quarterly, annual, and final evaluation reports to their project officer at the Head Start Bureau within ACYF.

Most of the local evaluators were affiliated with a local college or university; the rest were independent consultants or affiliated with a local consulting or research firm. Most of the local evaluators had advanced degrees (e.g., Ph.D., Ed.D.) in a variety of disciplines such as education, social work, public health, and psychology. Many had prior experience in evaluation work pertaining to community development, human services, and mental health.

Summary of Local Evaluation Reports

In the fall of 1997, Abt Associates asked the local evaluators of all Wave I, II and III FSCs to send copies of their final reports and any other pertinent reports from their evaluation of the FSC project. As a result of these letters and further follow-up efforts, 58 local evaluation reports (89 percent) were received, which were fairly evenly distributed across the three waves of projects.

In order to summarize the content of local evaluation reports, a list of categories was developed to capture the full range of possible evaluation domains. The percentage of local evaluations that included each component appears in the exhibit, “Summary of Local Evaluation Components”.

**Summary of FSC Local Evaluation Components**
Categories	Percent of Local Evaluation Reports (N=58)
Planning and Development	47
Community Context	38
Program Operations	67
Description of Services	62
Participant Case Studies	36
Participant Characteristics/Needs	90
Participant Perceptions	66
Staff and Community Perceptions	21
Participation Patterns	88
Participant Outcomes/Goal Attainment	95
Local Evaluation Design to Measure	67
Lessons Learned/Recommendations	76
Other	17

The local evaluation reports displayed a great deal of variability in terms of both approach and content. For example, some reports presented details about the methodology of local evaluation activities and included multiple tables of participation and impact data but did not mention the type of neighborhood or community in which the FSC was located or the type of staff in the project. Other reports provided rich and detailed case studies and descriptions of community collaborations, services, and staffing but did not present results of any impact analyses. One-third of the local evaluators presented information in nine to eleven of our evaluation categories; 43 percent covered six to eight categories. None of the evaluators included information in all twelve categories (excluding “other”).

Lessons Learned and Recommendations Included in Local Evaluations

Three-fourths (76 percent) of the local evaluators either discussed lessons learned in the FSC demonstration or presented at least one recommendation to improve the FSC project or overcome barriers to service implementation. These included changes and modifications in program services or operations implemented during the course of the demonstration as well as recommendations or suggestions made at the end of the demonstration to improve or enhance future projects. The recommendations reported by local evaluators came from multiple sources; some were made by the local evaluators, while others were obtained through interviews, surveys, and focus groups with FSC project staff, participants, and community service providers.

The recommendations and lessons learned center around four common themes that cut across the various programmatic categories.

**Recommendations in Local Evaluation Reports**
Grantee Characteristics Local evaluators discussed the role and resources of the FSC grantee and its effect on the areas of project administration and staffing. Three out of the four factors that were seen as facilitating project implementation focused on aspects of the grantee, including: being well connected to community service providers; providing some direct services independently of other community agencies; and being available to provide support and supervision to the project. Integrating permanent employees from the grantee agency into the FSC to reduce staff turnover among temporary demonstration staff was also a suggested function for the grantee. Documentation Local evaluators recommended increased or improved documentation in the areas of project administration, service delivery, and community collaboration. Specific areas for improving or implementing documentation policies were in: program procedures; participant files; staff roles and responsibilities; and community contact names and information. Accessibility Local evaluators recommended that the FSC staff and services be readily accessible to FSC families through: physically locating the FSC in an area convenient to FSC families; providing on-site services at the FSC; and providing transportation to bring participants to services. Communication The necessity for clear and adequate communication among staff, between staff and FSC participants, and between staff and community collaborators was also identified by local evaluators as a factor in the areas of project administration, staffing, and community collaboration.

Locally Reported Program Impacts

Designs Used in Local Evaluations

Fifteen percent of the evaluators analyzed program impacts over time in a pretest-posttest design; these were most likely to be evaluators from Wave I projects. Thirty-six percent of evaluators (most often from Wave II projects) had a comparison group design. Forty-nine percent of evaluators (primarily from Wave III) reported impacts based on a randomized design, as part of either their local evaluation or for the national evaluation. This shift from pretest-posttest to a comparison group and then a randomized design mirrors the change in instructions and requirements in the FSC grant announcements from ACYF.

Summary of Locally Reported Impacts

Of all the areas measured, participation in education services and employability skills showed the highest proportion of positive findings. In both areas, more than half of the local evaluations found increased activity either between pretest and posttest or for the FSC participants relative to a comparison or control group from Head Start.

**Summary of Impacts inthe FSCLocalEvaluations**
Area of Impact Reported in Local Evaluations	Percent of Loca l Evaluation Reports
Area of Impact Reported in Local Evaluations	Positive Impact	No Impact	Negative Impact
Participation in	57	43	0
Education Classes (n=21)
Employment
Employability (n=20)	55	40	5
Employment (n=23)	9	74	17
Public Assistance or	21	67	12
Income (n=24)
Literacy
Education Level	32	63	5
Literacy Skills	40	56	4
Substance Abuse (n=26)	28	54	20

The local evaluations point to modest positive impacts of the FSCs on education and literacy levels. In particular, FSC participants were more likely than adults in Head Start to obtain a GED or other educational certificate. Positive program effects on literacy skills were also reported by local evaluations, although these results were most often based on self-ratings of reading ability or progress towards personal goals rather than on standardized tests.

There were very few local evaluations that reported positive program impacts on employment, income, or receipt of public assistance. In the area of substance abuse, the results were mixed, with most local studies reporting no impact. There were few local evaluations covering areas beyond these primary focal areas of the FSC; although some local evaluations did report on psychological well-being, there were few positive impacts in this area.

Recommendations for Future Local Evaluations

Given the central role accorded the national evaluation in reporting program impacts, what role was there to play for the FSC local evaluators both at the national and local level? ACYF correctly anticipated that it was beyond the scope of the national evaluator to be able to capture the unique qualities of each individual FSC program. However, judging from the wide range in quality and content of the local evaluation reports, there did not seem to be a clear consensus as to the purpose or use of these studies.

What steps could be taken to ensure that local evaluation reports are better utilized
in future evaluations of federal programs?
Require an Evaluation Plan from each local evaluator containing a list of research questions; description of proposed sample and research methodology; data collection plan; and analysis plan. Specify an Evaluation Report Structure for each local evaluator to cover in their reports: context; program services, operations, and staff; program participants; study design and methodology; outcomes/impacts; and lessons learned/recommendations. Develop Common Set of Research Questions and Expectations that ACYF is interested in addressing, such as “What barriers do parents report that prevent them from fully utilizing Head Start services?” and “Are programs successful in identifying families’ needs and goals?” Promote Communication Among Evaluators by organizing a series of meetings around common themes to enable local evaluators to gain some perspective on the issues they face in their individual sites. Build Local Capacity to strengthen local evaluations through activities such as: promoting public dissemination of local evaluations; establishing internet-based information exchange among local evaluators; commission papers dealing with research and evaluation issues; providing assistance in obtaining statistical software programs for data analysis; and involving local projects in evaluation process to facilitate using evaluation findings for program improvement.

These are some recommendations intended to enhance the utility of local evaluations in national demonstration initiatives. There will be benefits to ACYF and other government agencies who carefully plan how local evaluation information is to be used in order to guide the evaluation process in a direction that will satisfy both client and program needs.

Return to Previous page