Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

III. PROGRAMS’ THEORIES OF CHANGE AND THEIR EVOLUTION OVER TIME

A. INTRODUCTION

“Theories of change” are increasingly important in program evaluations (Birckmayer and Weiss 2000; Connell and Kubisch 1998; and Weiss 1995). They provide a way for programs to identify the specific outcomes they expect to achieve and to describe the programmatic strategies and activities that they have designed. Theories of change (sometimes called “logic models”) also make it possible for program evaluators, working with program staff, to identify the outcomes that programs expect their services to influence in the various areas they focus on, select ways of measuring these expected outcomes, and plan analyses that will focus on the outcomes that the programs believe to be important. In the Early Head Start evaluation, theories of change contribute both to our descriptions of the program intentions and processes and planning the analyses of program impacts.

From its very beginning, the Early Head Start evaluation has emphasized the importance of understanding the expected outcomes of the 17 research programs. In 1996, shortly after most of the programs were funded, the national evaluation team began to engage both local researchers and program directors from the research sites in discussions of theories of change. In many sites, research-program discussions continued. We first reported on the programs’ expected outcomes in Volume I of Leading the Way: Characteristics and Early Experiences of Selected Early Head Start Programs (ACYF 1999a). That report was based on information from 1997 site visits and 1998 discussions with the 17 program directors. It presented three perspectives on the programs’ expected outcomes: (1) all the “important” expected outcomes that programs articulated, as obtained from interviews conducted during fall 1997 sites visits; (2) the program directors’ reports of the three “highest-priority” outcomes for their programs, obtained in a spring 1998 meeting; and (3) directors’ descriptions of a “success story” that exemplified outcomes they had achieved with their children and families.

The theory-of-change discussions presented in Leading the Way described expected outcomes in five areas: (1) parent-child relationships, (2) child development, (3) family development, (4) staff development, and (5) community building. Since spring 1998, theory-of-change discussions continued among the research-program partners across the sites, and in the summer/fall 1999 site visits, the national team explored any changes in the programs’ expected outcomes in these five areas. We acknowledge that the approach taken to describing and understanding programs’ expected outcomes contained variability. Participating staff represented different roles across sites and spent varying amounts of time on this activity, both during and between site visits. Site visitors were balancing competing demands and devoted differential attention to obtaining details on their programs’ expected outcomes. Finally, the process conducted in 1999 differed somewhat from the way it was conducted in 1997, so the two sets of information are not entirely comparable. In spite of these caveats, however, the Early Head Start evaluation was successful in obtaining extensive information on the expected outcomes of all 17 research programs at different points in time. The information is useful for describing the focus and change in expected outcomes over time, as we do in this chapter.

Table III.1 presents the expected outcomes for each program as described to us in 1997 and 1998 and in 1999.1  All important program outcomes are listed in the table, with the ones identified by the programs as priority outcomes at each time point shown in italics. The rest of this chapter discusses and summarizes these expected outcomes, the ways they have changed over time, and the implications they have for understanding program development and impacts.

TABLE III.1

OVERVIEW OF KEY OUTCOMES IDENTIFIED BY PROGRAMS IN 1997 AND 1999

Programs	Parent-Child Relationships	Child Development	Family Development	Staff Developmenta	Community Buildingb
A    1997	Parental knowledge of child development Attachment, knowledge of child development, and understanding the parent-child relationship	Cognitive development Cognitive, language, social-emotional, physical, approaches toward learning, and school readiness	Physical health, mental health and healthy family functioning, self-sufficiency, literacy and education, and home environment	Improved staff competencies Staff competencies and community involvement	Quality of community child care, quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Enhanced parental competencies Stronger attachment, enhanced knowledge of child development, more understanding of the parent-child relationship	Enhanced cognitive and language development Enhanced social-emotional development, greater school readiness, better physical development, better approaches toward learning	Better mental health, physical health, healthier family functioning, greater self-sufficiency, increased literacy and education, and enhanced home environment	Improved staff competencies More community involvement	Increased quality of community child care, increased quality of other community services, greater coordination of services and collaboration, and more involvement of parents in the community
B    1997	Parent-child relationships Attachment and knowledge of child development,	Cognitive, social-emotional, physical, and school readiness	Mental health Physical health, mental health and healthy family functioning, self-sufficiency, and home environment	Staff self-esteem Staff competencies	Quality of community child care and involvement of parents in the community
1999	Parent-child relationships Secure attachment, parenting efficacy	Age-appropriate levels of cognitive, social-emotional, physical, and language development	Mental health Physical health, self-sufficiency, physical and emotional quality of the home environment (stable, nurturing)	Staff self-esteem Greater competence and teamwork	Increased availability and better quality of community child care, greater sophistication of parents as consumers of health, social, and educational services
C    1997	Parent-child relationships Attachment and knowledge of child development	Cognitive, social-emotional, physical, approaches toward learning, and readiness for Head Start	Self-efficacy mental health and healthy family functioning, self-sufficiency, and literacy and education	Improved staff competencies Staff competencies and career development	Involvement of parents in the community
1999	Enhanced parent-child relationships Increased knowledge of child development	Cognitive development Social-emotional development Healthy physical development and readiness for Head Start	Parent self-efficacy Improved mental health and healthy family functioning, improved literacy and education, and healthier lifestyles	Improved staff competencies Career development	Increased involvement of parents in the community
D   1997	Parent-child relationships Knowledge of child development	Cognitive, social-emotional, approaches toward learning, and school readiness	Economic self-sufficiency/employment Self-sufficiency and home environment	Improved staff competencies Staff competencies and teamwork and morale	Involvement of parents in the community
1999	Knowledge of child development and of how to stimulate young children	Cognitive development Social-emotional development (social skills, willingness to share, self-esteem) Physical development Approaches toward learning (independence and self-help skills)	Parent self-sufficiency (skills necessary for employment, access services on own)	Improved staff competencies (successfully transitioning from Head Start to EHS, increased training and education), increased supportive supervision	Increased collaboration and partnerships with community services providers
E   1997	Parental knowledge of child development Attachment, knowledge of child development, and understanding the parent-child relationship	Cognitive development Social-emotional and approaches toward learning	Family goal setting Mental health and healthy family functioning, self-sufficiency, and home environment	Staff development not discussed during site visit	Community cornerstone not discussed during site visit
1999	Enhanced parental knowledge of child development and children's needs Stronger attachment; better understanding of the parent-child relationship	Cognitive development Social-emotional; approaches toward learning; emergent literacy skills	Enhanced family goal setting Healthier family functioning, greater self-sufficiency; enhanced home environment	Increased staff professionalism (awareness and assessment of family needs, ability to make appropriate referrals, staff have goal of improving in this area) Greater staff skills and knowledge about child development and child care	Greater awareness of community child care needs and importance of early education issues; increased supply and quality of child care; more coordination of services and collaboration with community partners; greater community knowledge about low-income families
F   1997	Understanding the parent-child relationship	Language development Cognitive, social-emotional, and physical	Literacy/education Mental health and healthy family functioning, self-sufficiency, literacy and education, and home environment	Staff competencies and teamwork and morale	Improved quality of community child care Involvement of parents in the community
1999	Improved understanding of the parent-child relationship	Improved language, cognitive, social-emotional, and physical development	Parental mental health Family education and literacy Healthy family functioning (stable home environment) Increased self-sufficiency, better quality home environment	Greater competencies, teamwork, and morale	Improved quality of community child care, greater involvement of parents in the community
G   1997	Parent-child relationships Parenting stress Attachment, knowledge of child development, and understanding the parent-child relationship	Cognitive, language, social-emotional, and approaches toward learning	Mental health and healthy family functioning, self-sufficiency, and father involvement	Improved staff competencies Staff competencies, team work and morale, career development, and community involvement	Quality of community child care, and coordination of services and collaboration
1999	Stronger and secure parent-child attachment Parents understand and promote child development (identify developmental milestones, monitor and support development)	Demonstrate gains in language and social development, be ready to learn	Decreased family stress Parents advocate for their children, and act on anticipatory guidance and education related to their own and child's health, fewer life crises and respond to crises and stress with constructive decision making, live in affordable safe homes free of substance abuse and illegal activities, have extended social support system, purchase and prepare meals meeting family's nutritional needs, employment that meets basic economic needs and provides opportunities for advancement, if no GED will complete adult basic education or advance 2 grade levels, have employable skills and means of transportation, be financially stable and able to financially plan for future	Better prepared, trained staff	Children and parents will have access to developmentally appropriate child care, family members will volunteer in the community
H   1997	Parent-child relationships Parental knowledge of child development Attachment, knowledge of child development, and parenting	Language, social-emotional, physical, approaches toward learning, and school readiness	Self-sufficiency, home environment, and father involvement	Improved staff competencies Staff competencies, team work and morale, and career development	Quality of community childcare, quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Enhanced parent-child relationships (attachment parenting, increased nurturing, increased responsiveness to child) Parental knowledge of child development (what is developmentally appropriate) Infant-parent play interaction	Enhanced functioning in domains of language, social-emotional development (secure attachment, positive peer play interactions atage 3), physical development and health, approaches toward learning, and school readiness	Greater family self-sufficiency, improved home environment, greater male involvement and social networking	Improved staff competencies, teamwork, and morale; career development	Enhanced quality of community child care, quality of other community services, coordination of services and collaboration, and parent involvement in the community
I   1997	Attachment, knowledge of child development, understanding the parent-child relationship, and parenting	Cognitive development Language development Social development Social-emotional and physical	Physical health, mental health and healthy family functioning, self-sufficiency, and home environment	Staff development not discussed during site visit	Quality of community child care, quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Increased security of parent-child attachment Increased parental availability to the child Parent is available for the child (emotionally and physically); increase in parents ability to read cues (communication needs to be reciprocal and parent needs to learn to speak for the child); child has a secure base to return to (explore and grow); empathic listening, holding interactions; parent expresses pleasure of child/ acceptance of child	Achievement of appropriate developmental milestones Ability to emotionally connect with parent and others (this encompasses confidence and self-esteem, emerging sense of self, and having a secure base); for delayed/disabled children, promote maximum development and growth; achieve developmental milestones (language/ motor skills/ cognitive); increased self-regulation and ability to withstand delayed gratification	Increased family self-sufficiency Increased family access of appropriate community resources Decreased number of unsafe home environments Parent has increased self-regulation and ability to delay gratification; increased income, education, and satisfaction with life; become financially independent; parents establish and maintain healthy relationships; use a healthy support system, give a voice, reduce isolation; understand consequences of choices and actions; increased safety	Advocate for and with families	Link agencies and service providers
J   1997	Parent-child relationships Knowledge of child development, understanding the parent-child relationship, and parenting	Cognitive, social-emotional, physical, and approaches toward learning	Literacy/education Physical health, mental health and healthy family functioning, self-sufficiency, literacy and education, and home environment	Staff competencies and career development	Quality of community child care Quality of community child care, quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Stronger parent-child relationships Parents will understand rationale of CD activities and continue them after specialist leaves; parent feels better about self and more available to child; understand where child is developmentally and recognize changes, understand link between child’s language and communication and reduced violence later; increased understanding of why CD is important; actively teach children and read to them more; more activities conducive to CD; reduced abuse and neglect; increased parent-child interactions	More social, initiate play; verbalize feelings better; ready for school academically; ready for school in terms of temperament; increased social competency; improved health (including immunization rates)	Greater parental literacy/education Parents attain better sense of family’s needs; greater confidence in parenting; better environment for children; more stable homes; parents empowered to know and ask for what they need; think of solutions to own dilemmas, higher self-esteem; greater confidence in achieving goals; greater family self-sufficiency; better family health (including prenatal care, knowledge of own bodies, sexuality and STDs); more assertive in advocating for own children; more-positive outlook on life; more positive approach to own and child’s well-being; have plan of action regarding achieving goals; increased social competency; sufficient literacy to seek solutions and help from agencies; greater knowledge of community resources/ learning opportunities; fathers involved	Increased knowledge of child development, increased knowledge of community resources, attitude consistent with philosophy of family strengths rather than deficits; take advantage of opportunities in the community	Quality of community child care Parents understand importance of continuity and quality and can evaluate quality of child care and make informed choices; develop relationship with their child care provider; systems affecting children will be more sensitive to child and family needs; more streamlined services; parents are listened to and heard in relation to community building; parents positive role model for peers in the community
K   1997	Parenting confidence and competence Parent-child relationships Knowledge of child development and parenting	Social-emotional development Cognitive, language, social-emotional, physical, and approaches toward learning	Self-sufficiency and home environment	Staff competencies	Quality of community child care and involvement of parents in the community
1999	Enhanced parent-child relationships (age-appropriate play with child, positive intra-family relationships) Greater parenting confidence and competence Greater knowledge of child development and parenting (age-appropriate expectations; good parenting skills)	Better social-emotional development Cognitive development (fewer developmental delays, holistic cognitive development), better health, approaches toward learning (increased curiosity, able to conquer new challenges, able to remember prior experiences and relate to current tasks)	Enhanced ability of parents to meet the family’s social and economic needs (self-sufficiency) (able to obtain needed resources, make informed decisions, articulate and reach goals, advocate for the family, achieve economic self-sufficiency) More supportive home environment	Stronger staff competencies (obtain CDAs); enhanced staff supervision and support	Higher quality of community child care; more involvement of parents in community (advocating for selves, involved in policy council); More peer support among parents
L   1997	Parent-child relationships Parental knowledge of child development Attachment, knowledge of child development, understanding the parent-child relationship, and parenting	Physical development/health Cognitive, social-emotional, physical, and school readiness	Physical health, mental health and healthy family functioning, self-sufficiency, literacy and education, and father involvement	Staff competencies, teamwork and morale, and career development	Quality of community child care, quality of other community services, and involvement of parents in the community
1999	Parent-child relationships Parental knowledge of child development Attachment, parenting	Physical development/health Cognitive development, social-emotional development, school readiness	Physical health, mental health and healthy family functioning, self-sufficiency, literacy and education, and father involvement	Staff competencies, teamwork and morale, and career development	Quality of community child care, quality of other community services, and involvement of parents in the community
M   1997	Parent-child relationships Attachment, knowledge of child development, and understanding the parent-child relationship	Social-emotional and approaches toward learning	Economic self-sufficiency/employment Mental health and healthy family functioning	Staff development not discussed during site visit	Quality of community child care Involvement of parents in the community
1999	Stronger parent-child relationships Stronger attachment, enhanced knowledge of child development, better understanding of the parent-child relationship	Enhanced child health and physical development Enhanced language development (overarching), enhanced social-emotional development, stronger approaches toward learning, enhanced cognitive development	Greater economic self-sufficiency and more employment and education Healthier family functioning, and better physical and mental health	Better knowledge about and implementation of Head Start Program Performance Standards, High-quality performance and ability to reflect on program goals	Higher quality of community child care Greater involvement of parents in the community,, more community service provider collaboration
N   1997	Knowledge of child development and parenting	Language development Language, social-emotional, physical, approaches toward learning, and knowledge of their community and diversity	Economic self-sufficiency/employment Mental health and healthy family functioning, self-sufficiency, home environment, and father involvement	Teamwork and morale and career development	Coordination of services Quality of community child care
1999	Increased knowledge and practice of positive parenting strategies (especially discipline, setting firm limits) Increase parent-child bond and responsiveness to children	Babies are healthier and display developmentally appropriate growth (in all areas—cognitive, self-help, language, motor, social-emotional, intellectual development) Ability to express needs and wants positively by gestures and words	Increased awareness and use of community resources Improved self-esteem; improved ability to articulate feelings and appropriately deal with conflict; greater knowledge of resources and make progress toward own goals; Greater motivation to improve standard of living; higher educational attainment; greater knowledge of community and cultural diversity (develop sense of pride, recognize roots and share with children and community, more involved in community, better understanding of all cultures in the community)	Obtain advanced degrees; receive salaries comparable to other child development programs and schools	Higher child care quality (age appropriate activities, nurturing staff), stronger support for EHS in community
O   1997	Parenting stress Knowledge of child development and parenting	Physical development and health Cognitive, language, social-emotional, physical, and approaches toward learning	Physical health, mental health and healthy family functioning, self-sufficiency, and home environment	Staff competencies and career development	Collaboration Quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Enhanced parent-child relationships Increased knowledge of child development and parenting (realistic expectations, reduced child abuse, read to children more often, increased confidence in parenting, use appropriate discipline techniques, follow routines with children)	Social-emotional development (self-control, social skills) Language development (communication skills, self-expression), cognitive development (prepared for reading)	Self-sufficiency (improved life skills, social skills, and advocacy for self and children) (progress toward employability, improved housing, increased planning skills, better financial management skills) Improved physical health (reduced substance abuse and smoking, better nutrition); improved mental health and healthier family functioning (healthier lifestyle, reduced social isolation); safe home environment	Increased staff competencies (better trained); career development (better educated)	Increased quality of community child care; enhanced coordination of services and collaboration (increased collaborative work style when staff move to other agencies); increased awareness about importance of early child development
P   1997	Attachment, knowledge of child development, and parenting	Language development Social development Cognitive, language, social-emotional, and physical	Physical health, mental health and healthy family functioning, self-sufficiency, and home environment	Staff competencies, teamwork and morale, and career development	Quality of community child care Quality of community child care, coordination of services and collaboration, and involvement of parents in the community
1999	Stronger attachment ; enhanced knowledge of child development; better parenting	Enhanced cognitive development Enhanced language development Enhanced social-emotional development (empathy, social skills) Enhanced physical development (reduced severity of injuries and illnesses)	Enhanced physical health; better mental health; healthier family functioning, greater self-sufficiency; enhanced home environment, greater independence/ self-determination/ self-confidence	Enhance staff competencies (getting CDAs, relationship building, cultural sensitivity); more teamwork and better morale; stronger career development	Higher quality of community child care More coordination of services and collaboration; greater involvement of parents in the community
Q   1997	Parent-child relationships Parental knowledge of child development Attachment, knowledge of child development, and parenting	Social-emotional and physical	Mental health and healthy family functioning and self-sufficiency	Teamwork and morale	Quality of community child care Quality of other community services, coordination of services and collaboration, and involvement of parents in the community
1999	Parent-child relationships (secure attachment) Parental knowledge of child development (especially realistic expectations)	Age-appropriate levels of social-emotional and physical development	Families’ abilities to set goals Mental health and coping skills; self-sufficiency; healthy family functioning (goal-setting, focus on change); social support (especially for parenting)	Professional development and advancement	Service coordination and collaboration (especially for transitions); involvement of parents in the community

NOTE: In 1997, programs were limited to identifying three priority outcomes; in 1999, several programs named more than three. aThe entries under each cornerstone indicate the key areas in which each program indicated important outcomes in the theories of change discussions during the fall 1997 and fall 1999 site visits. The outcomes highlighted in italics are the programs’ “priority” outcomes.(back) bDue to time constraints, this cornerstone was not discussed during some 1997 site visits.(back)

B. EVOLUTION IN PROGRAMS’ EXPECTED OUTCOMES

We describe programs’ priority outcomes in two ways. First, we consider the extent to which the programs, as a group, were focusing on particular areas. To do this, we report the priority outcomes that fell into each area as a percentage of all priority outcomes. This is shown in part A of Table III.2. Next, we look at the number and percentage of programs that focused on particular types of outcomes. These are shown in part B of Table III.2.

1. Specific Changes That Occurred in Programs’ Focus on Priority Outcomes in Particular Areas

While a small number of programs did not change the priority outcomes identified in May 1998, the focus of most programs became refined and/or modified in important ways over time, reflecting changing views of the important outcomes they wanted to achieve. As shown in part A of Table III.2, the proportion of priority outcomes that were in the areas of parent-child relationships and child development did not change: in 1998, 59 percent of the priority outcomes were in the combined area of parent-child relationships and child development, and this combined area comprised 60 percent of the outcomes in 1999. It is important to consider child and parent-child relationships together, for, as we learned in discussions with program staff, programs often stress parent-child relationship goals because of the expected effects they will have indirectly on children’s development.

Family development outcomes became a larger proportion of all the priority-expected outcomes in 1999 than they were in 1998, rising from 16 to 27 percent. At the same time, a substantially smaller proportion of the total expected priority outcomes were in staff development and community building—together these areas constituted about a quarter of all the priority outcomes (26 percent) in 1998. However, staff and community development became even less likely to be priority outcomes in 1999, constituting only 13 percent of all priority outcomes that programs reported to us.

TABLE III.2

EARLY HEAD START PROGRAMS’ PRIORITY OUTCOMES

A. Percentage of Priority Outcomes in Each Area, 1998 and 1999.2 

Area	1998	1999
Parent-Child Relationships	37	34
Child Development	22	26
Family Development	16	27
Staff Development	12	8
Community Building	14	5

Note: When child development and parent-child relationships are considered together, they account for 59 percent of all priority-expected outcomes in 1998 and 60 percent in 1999.

 

B. Number (and Percentage) of Programs with Priority Outcomes in Each Area, 1998 and 1999

Area	1998	1999
Parent-Child Relationships	13 (76)	14 (82)
Child Development	9 (53)	11 (65)
Family Development	8 (47)	13 (76)
Staff Development	6 (35)	5 (29)
Community Building	7 (41)	3 (18)

Note: When child development and parent-child relationships are considered together, five programs (29 percent) identified outcomes in both areas in 1998 and nine (53 percent) did so in 1999.

 

Looking at the percentage of programs with priority outcomes in each area (part B of Table III.2), it is clear that an increasing number of programs were working toward outcomes in the parent-child, child development, and family development areas in 1999, compared with 1998. At the same time, fewer programs in 1999 than in 1997 considered staff development and community building to be among their priority outcomes. We should point out, however, that the lowered priority for outcomes in these areas did not mean that programs were neglecting staff and community development. We continued to see strong programmatic efforts in these areas, as noted in chapters V and VI. Rather, programs were undoubtedly responding to guidance from the Head Start Bureau and articulating the choices they made when it was not possible to have every area be high priority.

We examined, from the program perspective, the nature of these changes. First, two programs that did not identify parent-child relationships as a priority outcome in 1998 added that focus in 1999. One program dropped its parent-child priority focus, which yielded a net increase to 14 programs with priority outcomes in that area. A similar change occurred in child development. Three programs added it as a priority focus, while one program dropped it, which resulted in a net increase from 9 to 11 programs that placed child outcomes among their top priorities.

Another pattern of change was that, over time, programs with priority outcomes in staff development and community building shifted focus to outcomes in the family development area. Five programs added priority outcomes in that area, and no programs that identified family development outcomes in 1998 dropped them, which resulted in an increase from 8 to 13 programs identifying such outcomes.

One program added expected outcomes in staff development, and two no longer identified staff outcomes in 1999, which resulted in a net decrease from six to five programs that were giving priority to that area. Substantial change among priority outcomes occurred in the community area, however. Four programs that had identified priority outcomes in this area in 1998 no longer did so in 1999, and no program added this as a priority focus. Thus, in 1999 three programs had community building as a priority focus (compared with seven in 1998). Three of the programs that no longer identified community outcomes as priority added family outcomes.

The evolution of expected outcomes also involved changes in program thinking within each of the five areas. For example, in the child development area, programs identified specific aspects that they focused on, as shown in Table III.3. Among the 11 programs identifying child development priority outcomes in 1999, subsets of programs focused on the following specific outcomes:

• Five programs specified social-emotional development

• Five programs specified cognitive development (or both cognitive and language development)

• Two programs specified language development

• Three specified health and physical development

• Two named generic child development outcomes (for example, “achieving appropriate developmental milestones”)

TABLE III.3

EVOLVING PRIORITIES WITHIN THE CHILD DEVELOPMENT AREA: NUMBER (AND PERCENT) OF PROGRAMS IDENTIFYING EACH ASPECT OF CHILD DEVELOPMENT AS A PRIORITY OUTCOME

Child Development Outcome	1998	1999
Social or social-emotionaldevelopment	3 (33)	5 (45)
Cognitive development	3 (33)	5 (45)
Language development	4 (44)	2 (18)
Health and physical development	2 (22)	3 (27)
Generic child development	0 (0)	2 (18)
Total programs with child development outcomes	9	11

 

Thus, a somewhat greater proportion of programs had a priority to achieve social-emotional and cognitive outcomes in 1999 (compared with 1998), and a smaller percentage identified language as a priority child development outcome.

2. Changes Across All Expected Outcomes Between 1997 and 1999

In addition to considering the priority outcomes, we also documented all outcomes that programs deemed “important.” These are shown in Table III.1, along with the 1997 and 1998 outcomes. One of the first things to note is that every program identified outcomes in all areas. This was an important first step for programs as they attempted to implement all four program areas as specified in the original program grant announcement.

A number of programs reported more-detailed outcomes in 1999 than in 1997; several programs have become more detailed in their identification of outcomes in parent-child relationships, child development, and family development. Both in 1997 and 1999, all programs identified social-emotional outcomes as ones they expected to achieve. Thirteen programs identified cognitive outcomes (a slight increase from 12 in 1997), and 11 expected language outcomes (increased from 9 in 1997). The largest increase occurred in the area of health and physical development, where 15 of the 17 programs mentioned these outcomes in 1999, in contrast to 11 in 1997.

3. Summarizing Programs’ Expected Child and Family Outcomes

One complication of our variable approach to discussing expected outcomes is the variation in terminology. Programs reported both “important” and “priority” outcomes in 1997, 1998, and 1999. We have also shown the changes in programs’ expected outcomes over time, combining priority and other outcomes, and combining information across years. Because no single approach or point in time yields an exact picture of programs’ expected outcomes, we created a composite index derived from (1) 1997 expected outcomes; (2) 1998 priority outcomes; (3) all expected outcomes programs described in the fall 1999 site visits; and (4) priority expected outcomes from 1999, as confirmed by local researchers.

If an outcome area was identified in at least three of these four analyses, we considered there to be a “consensus” that it was a legitimate expected outcome of the program and could be the basis for targeted subgroup impact analysis.(2)The resulting clustering of programs is shown in Table III.4. The largest number or programs (12) expected parent-child relationship outcomes. Within child development, the most common expected outcome was social-emotional development. Looking across the four child development areas, 10 programs indicated expected outcomes in at least one child development area, 7 identified two of the four areas, and 4 reported that they expected to achieve outcomes in three or all four of the areas.

4. The Relationship Among Expected Outcomes, Program Approaches, and Program Impacts

The programs’ expected outcomes shown in Table III.4 are generally consistent with the types of services they offered at the time of the 1997 site visits. In general, as shown in Figures III.1 and III.2, center-based programs were more likely to emphasize child development outcomes, while home-based programs were more likely to invest their efforts in enhancing parent-child relationships and parenting/home environment outcomes (which they expected to lead to impacts on children’s development later). Among programs that gave priority to parent-child relationship or parenting outcomes, mixed-approach programs were most likely to emphasize enhancing parent-child relationships (Figure III.1). Many home-based programs also explicitly emphasized parent-child relationships, while others focused on aspects of parenting and the home environment, such as increasing parents’ knowledge of child development or encouraging parents to spend more time with their children.

We also examined the expected outcomes within child development (Figure III.3). Among programs that gave priority to child development outcomes, the percentage of center-based programs emphasizing cognitive and social-emotional development was equal (50 percent), and, mixed approach and home-based programs were more likely to emphasize social-emotional development.

Interim findings of program impacts through the children’s second birthday were generally—but not completely—consistent with the program approaches and expected outcomes (ACYF 2001). All program approaches resulted in positive benefits for children, but the types of impacts differed across approaches. Center-based programs were the only ones to enhance children’s cognitive development significantly, while home-based programs improved children’s language development and mixed-approach programs improved both language and social-emotional development. Early Head Start impacts on parenting and the home environment were concentrated in home-based and mixed-approach programs (with a few exceptions).

TABLE III.4

CLUSTERS OF PROGRAMS WITH PRIORITY OUTCOMES IN EACH ASPECT OF CHILD AND FAMILY DEVELOPMENT

Area	Specific Outcome	Programs Number	in Cluster Percent
Parent-child relationships	Parent-child relationships	12	71
	Knowledge of child development	6	35
Child development	Social-emotional development	7	41
	Cognitive development	5	29
	Language development	4	24
	Physical development and Health	3	18
Family development
	Family self-sufficiency	11	65
	Family mental health	6	35

 

FIGURE III.1

VARIATION IN PROGRAM APPROACH AMONG PROGRAMS WITH DIFFERENT PRIORITY OUTCOMES

[D]

 

FIGURE III.2

PRIORITY EXPECTED OUTCOMES BY PROGRAM APPROACH

[D]

 

FIGURE III.3

PRIORITY EXPECTED CHILD DEVELOPMENT OUTCOMES, BY PROGRAM APPROACH

[D]

C. PERSPECTIVES FROM THEORY-OF-CHANGE DISCUSSIONS AMONG RESEARCHERS AND PRACTITIONERS

A special feature of the Early Head Start Research and Evaluation project has been the presence of local research teams to work with 16 of the 17 programs. As noted earlier, researchers to varying degrees in different locations engaged their program partners in discussions of expected outcomes and theories of change. In this section, we highlight the theory-of-change work within the Early Head Start programs and between the program and research staffs in the local partnerships.

1. The Value of Research-Program Partnership in Developing Theories of Change

The experience of the research-program partnership at the Bear River Early Head Start program in Logan, Utah, illustrates how this process can occur and what the benefits may be for both the programs and the researchers and, ultimately, for the children and families. Lori Roggman, the local researcher at Utah State University, who has worked with the Bear River staff from the beginning of Early Head Start, noted that even though program staff members often do not articulate a “theory of change,” they develop strategies for working with families based on a general philosophy or “theory” about how to make changes in the lives of families and children. Dr. Roggman has served as the continuous program improvement partner with the Bear River staff. This program, serving rural and semirural areas in and around Logan, Utah, emphasized home visits as a critical element in their theory of change. The process and the outcomes of the theory-of-change discussions in Utah reveal the importance of an active, interactive process. Although there may be many ways in which theory-of-change discussions between researchers and program staff might unfold, this provides an example of how the process developed in one site.

2. Voices of the Staff: Home Visitors Describe Their “Theories of Change”

Frontline staff members in Early Head Start programs are dedicated to their jobs and to their families (see discussion of Early Head Start staffing in Chapter V). Sometimes, even when staff members are not explicitly discussing a “theory of change,” as they did in Utah, they often reveal an implicit theory of change when they talk about their families and the successes their families have achieved. An example appears in the next box, taken from the words of a home visitor with the Community Action Agency Early Head Start program in Jackson, Michigan.

3. Local Variations in the Development of Program Theories of Change3 

Susan Pickrel, a local researcher with the Sumter, South Carolina, Early Head Start program, led a cross-site effort to learn about the ways in which program staff think about and articulate their theories of change. Local researchers in nine of the research sites held discussions with their program partners in 1999-2000, following a standard set of questions. Questions asked about program successes and outcome areas in which the program was less than successful. They audiotaped the discussions and transcribed the tapes, and Pickrel’s South Carolina team coded the discussions. The coding identified the key concepts that program staff used in describing barriers to and facilitators of success in working with their families. Through this process, researchers gained greater understanding of the programs’ theories as to how changes in families come about. These elements related to (1) the characteristics of the mothers; (2) the features of the program; and (3) the program process characteristics (operational features, staff behavior, and staff-family interactions) that might relate to the outcomes expected within the particular theory of change.

Creating a Theory of Change at Bear River Early Head Start, Logan, Utah Lori A. Roggman Utah State University The first time I, as the local researcher, talked about a “theory of change” with the staff at Bear River Early Head Start, I asked two questions: “How will families who are in your program end up different from those who are not in your program?” and “How exactly will this program make that happen?” In response to the first, staff had a long list of outcomes they believed would be changed by their program. They believed the families in the program would be happier parents with happier babies. They believed parents would be more knowledgeable and less stressed and feel better about themselves as parents. They believed the babies would be healthier, happier, more secure, and smarter. The second question was more difficult. After a long pause, someone said, with conviction, “Because we believe in this program and we believe in these families.” Through weeks of training, staff who were about to begin making home visits to families learned about child development, the Head Start Program Performance Standards, infant and family health, social services, and how to do all the necessary documentation. They had learned how to use the lesson plan forms and how to fill out forms for mileage reimbursement. They understood the research design and believed that the children and families in Early Head Start would end up better off in many ways. But they lacked a clear idea of the actual mechanisms of change. They knew they were supposed to make home visits to parents, and they knew how parents and infants were supposed to be affected by the program, but they did not seem to have a clear idea of how exactly one connected to the other. The authors of the program’s grant proposal had a clear vision of the program, but those who would have the responsibility for working directly with families weren’t seeing it as clearly. Since then, the Early Head Start staff members have worked together to write (and regularly review and revise) a “theory of change.” By clearly specifying “how families will end up different” and “how exactly this program will make that happen,” staff described a “vision” that then guided their decision making. For example, for their primary goal, “to increase positive parent-infant play interactions, nurturant and responsive parenting, and parents’ knowledge about child development,” staff identified a specific strategy: that three-fourths of home visit time will be spent in “direct play interactions to enhance the parent-child relationship.” The vision that guides program activities also guided the researchers to focus their evaluation on staff-parent relationships. Bear River Early Head Start staff members have described home visits and the role of home visitors with increasing clarity over the years. From Year 1 to Year 2, descriptions of home visits shifted toward a more-active intervention process that emphasized direct interactions between parents and infants (instead of interactions that were primarily discussions with parents). From Year 2 to Year 3, the descriptions shifted toward a greater emphasis on father involvement and family independence that involved helping both mothers and fathers plan their own activities with infants, both during home visits and between home visits. By writing a description of the connections between staff activities and what happens to families, the program was able to get off to a good start serving families with infants and toddlers. Beyond their self-confidence, staff members had specific ideas about what strategies to use. By regularly reviewing and revising this written “vision,” the program is able to continue improving and fine-tuning its efforts.

 

A Home Visitor’s View of Her Family’s Successes Christina Katka Community Action Agency Early Head Start Jackson, Michigan Carol (not her real name) called to request early intervention services for her 27-month-old son, “Jack,” who had been born prematurely, at just under 5 pounds. The toddler was receiving speech therapy from the local children’s hospital and participating in Part C services. Jack lives with Carol, his father, Peter, and a 14-year-old brother. This family’s situation is unusual—entering Early Head Start with less than a year of services possible—but enrollment was considered important, given the needs of the child and the family.1 Carol is herself disabled, from burns suffered as a child; Peter works full-time at a local university and part-time as a sheriff. Jack appeared small, shy, and guarded during our first meeting. He was easily frustrated, experiencing difficulty in expressing his wants and needs. As he became more familiar with me, his energy level increased. He actively engaged me in his play. And Jack often gave me a sense of “invitation” to “join” him in his world, a special place for a sensitive, loving child. During our home visits, both parents talked openly about their concerns and worries about Jack’s development. I realized I needed to begin with an alliance that offered Carol a strong and consistent relationship. I attempted to nurture and respect the family and be sensitive to their needs, providing a weekly presence in their home. I also felt they needed information, so I provided some on child development and age-appropriate toys, and offered help with guided activities that would enhance Jack’s large and small muscles. I introduced information about self-help skills, as well as cognitive development and the opportunity to use weekly play that would facilitate positive parent-child interactions. I was encouraged that the family also joined in on biweekly socialization groups, where Jack began to interact with other children—first in individual play, then in parallel play, and finally in cooperative play. I eventually began to see the results of these interactions. Carol’s confidence improved, and Jack’s language and communication developed. As Carol found the courage to face her fears, Jack found his own courage, supported by his ever-present drive toward independence. His play became more organized as he used appropriate exploration. Jack is affectionate, expressive, and interactive, while demonstrating a strong capacity for attachment and trust. Carol, in addition to taking great pleasure in her son’s growth, is caring and compassionate, and provides a safe, nurturing environment for Jack’s continuing development. Peter provides a strong male influence, providing an active role model in Jack’s life. As Jack enters the Head Start preschool program in the fall, he is being placed in the half-day inclusion classroom, where his new caregivers expect the progress we’ve seen in Early Head Start to continue. 1For participation in the research, programs enrolled families when children were 12 months of age or younger.(back)

 

Six of the nine programs identified characteristics of the mothers as key to Early Head Start program outcomes, and three considered the program or program process characteristics as key. In other words, one-third were oriented toward taking responsibility for the success of Early Head Start, independent of the participant characteristics. In the first set of programs, responsibility for change was articulated to be such characteristics of the mother as (1) desire or willingness to participate in the Early Head Start program, (2) focus on being a good parent, (3) ability to see positive developmental changes quickly in her child, (4) readiness to receive program information, (5) desire to make her and her child’s life better, and (6) enjoyment in being with her baby.

When staff members mentioned program characteristics as the factors producing the change, they tended to focus on generic features. Those programs features mentioned in more than one site included (1) case management (six sites), (2) home visits (four sites), (3) center-based child development services (three sites), (4) other child development services (two sites), and (5) on-site medical/pediatric and dental assessments and information (two sites). Although, for coding purposes, program characteristics were defined as static characteristics of a program (in contrast to the process characteristics, which reflect activities that occur between two persons or organizations), there was some overlap between the static and process characteristics. For example, case management, home visits, and child development services mentioned by multiple sites all involve interchanges between Early Head Start staff and program participants. Characteristics were coded as process, however, only if the discussions directly described personal process features rather than labels for program elements. The process elements listed next make this distinction clearer.

The programs that identified program process characteristics indicated a “theory of change” based on what program staff did to meet participant needs rather than on parent characteristics. The program or process characteristics included such factors as (1) staff skills in mental health interventions, (2) accepting and managing difficult behaviors in participants, (3) adapting to parent and family circumstances, and (4) persistence in trying to establish a relationship with the family in spite of obstacles.

Whether or not the dominant factors in the programs’ implicit theories of change were characteristics of participants or of the program/program process, staff members at all nine program sites mentioned process elements in their discussions. Those mentioned by staff at three or more sites were:4 

• Building a relationship of trust with the mother (mentioned in all but one of the sites)

• Providing support for mother or family (all but one site)

• Educating (six sites)

• Focusing on strengths (five sites)

• Modeling (four sites)

• Teaching and problem solving (four sites)

• Working as a team (for Early Head Start staff) (three sites)

These discussions indirectly yielded a qualitative sense as to how well developed the staffs’ theories of change were. The emerging “theories” could be assessed in terms of the coherence of the stories that Early Head Start staff generated and the manner in which staff used terms to describe program success and nonsuccess. Coherence was judged by how clearly staff articulated what their program activities were, why they conducted these activities, and how they defined program success (the families’ responses to the Early Head Start intervention). Just as the researchers evaluated staff discussion of barriers and facilitators in terms of characteristics of the mother, the program, or the program process, the terms program staff used to describe program successes could be categorized along the same dimensions. A program’s theory of change was considered to be less well developed if the program described success only in terms of characteristics of the mothers. Theories of change were considered better developed when success was described in terms of both program and process characteristics.

Two of the nine programs were considered to have well-developed theories of change, two had moderately well-developed theories, and four were judged to have underdeveloped theories. In the two programs that had the best-developed theories of change, staff members went into greater detail in describing the change process. One program detailed the relationship between Early Head Start staff and the mothers, and then described how that resulted in specific child development outcomes. The other program articulated a step-by-step process by which each family achieved its success. When theories of change were judged to be less well-developed, they failed to link important process factors (such as the staff-mother relationship) to the program’s expected outcomes (such as child development) or failed to articulate the outcomes clearly, or staff were inconsistent in describing the outcomes and process elements.

D. SUMMARY

Programs that wish to understand and communicate their goals and their strategies for achieving them increasingly use theories of change. At the same time, researchers who desire to understand better the programs they are evaluating adopt a theory-of-change approach so they can target their analyses on the outcomes that are most important to the programs, and then be better positioned to explain the results. In the Early Head Start evaluation, we have assessed programs’ theories of change using a variety of methods across various points in the programs’ implementation. The Early Head Start research programs have been working toward outcomes primarily in the areas of parent-child relationships, child development, and family development. Within child development, the greatest priorities lie in the areas of social-emotional and cognitive development, yet considerable variation exists across programs. Programs that are center based tended to emphasize child development outcomes while those that are home based were more likely to emphasize parent-child relationships and parenting outcomes. Mixed-approach programs tended to emphasize parent-child relationship outcomes. This chapter has illustrated the variety of perspectives that contribute to understanding programs’ theories of change, based on discussions among research and program partners at various sites participating in the national evaluation.

---

1This table adds the 1999 information to the expected outcomes reported in Table II.6 of Leading the Way, Volume I.(back)

2The reason the two analyses shown under A and B appear somewhat different is that each program could (and often did) identify multiple outcomes in one area. Since programs were limited to naming three priority outcomes, the total number of priority outcomes is fixed and the percentage of outcomes in each area must equal 100 percent. In contrast, because programs could name priority outcomes in multiple areas, the percentage of programs that named priority outcomes in each area can sum to more than 100 percent across the five areas. Note that for the purpose of these analyses, we focus on the child and family outcomes, as the study design does not allow for impact analyses of staff and community outcomes.(back)

3This section was contributed by Susan G. Pickrel, a local researcher working with the Sumter, South Carolina, Early Head Start program, who is currently with the Mercy Medical Center in Roseburg, Oregon.(back)

4Twenty-four other process features were mentioned by just one or two programs each.(back)

 

Table of Contents | Previous | Next