Screening and assessment results for individual children, along with other information from parents and caregivers, are needed to help staff tailor services for those children. When the same instruments are used for all children in a program, aggregating data across families can provide a picture of how children and families in the program are doing overall. When this aggregate information on child and family outcomes is linked to information on services and other program characteristics, it can provide insights that are useful to staff members in their continuous program improvement efforts. Aggregating data provides a picture of how children and families and the program are doing (that is, it measures the program’s performance). Over time, aggregated data can be used to track changes in child and family functioning, which along with information about changes in program services or characteristics, can help staff learn about what works best for particular types of children and families. The aggregated information on how children and families in the program are doing can also be useful for meeting other funders’ reporting requirements.

Screening. Screening is a generic term referring to activities designed to identify individuals who have a high probability of exhibiting delayed, abnormal, or problematic development. The screening is intended to identify problems at an early stage and identify individuals for whom further, in-depth assessment activities are needed.

Assessment. Assessment is a generic term referring to procedures for obtaining systematic information on a child’s, parent’s, family’s, or program’s strengths or needs. As noted in Chapter I, the Head Start Program Performance Standards focus on the child and family assessment purposes of identifying “(i) the child’s unique strengths and needs and the services appropriate to meet those needs; and (ii) the resources, priorities, and concerns of the family and the supports and services necessary to enhance the family’s capacity to meet the developmental needs of their child.”

[See the Early Head Start National Resource Center’s Technical Assistance Paper #4 for more detail on screening and assessment activities.]

Performance measurement includes data collection and aggregation activities that give staff members the opportunity to look at how their program is doing, that is, to determine whether they are providing the services they intend to provide and to learn how children and families are faring. Program staff can use this information in planning for continuous program improvement activities, and it can be shared with stakeholders such as parents and funders.

Screening results for individual children and families are useful for deciding whether further assessment is needed. Screening results can also be aggregated to provide information on the extent of potential problems in the population and the need for in-depth assessments among children and families overall. Aggregated screening results and information on referrals can inform a program as to whether these in-depth assessments are happening if the program’s tracking system does not provide this information.

Assessment results for individual children can be used for planning services; in addition, they can often be aggregated to provide broader information on child outcomes. Some approaches to assessment are valuable for individualizing services, but cannot be aggregated unless they are translated into another form. For example, portfolio assessment can be extremely useful for individualizing services for children, but unless a systematic way of coding the information is developed, the results cannot be aggregated.

Programs face both a great opportunity and a significant challenge as they consider performance measurement. They have an opportunity to select instruments and collect data that best meet the needs of their families and their program, and they face the challenge of figuring out just what those instruments should be, how they should be administered, and how the information collected using those instruments should be analyzed. Meeting that challenge effectively requires programs to develop a plan that considers the purpose(s) for which the data will be used, what data are already being collected and additional data that need to be collected, and how data will be aggregated and analyzed for continuous program improvement. Taking the time to develop a comprehensive plan will help ensure that the program’s resources for the required ongoing screening and assessment of individual children and families, as well as the new performance measurement and data aggregation activities, are used well and that the activities are as useful as possible.

To formulate a comprehensive plan, programs need a clear understanding of why performance measurement is needed. What does the staff need to know to determine whether the program is achieving its intended outcomes and to individualize or optimize services for children and families? What information is essential? What, in addition, would be good to know? Some questions programs might consider to help reach this understanding are:

To make performance measurement activities as useful as possible, programs need a theory of change. This model, or theory of change, provides a framework for guiding the selection of individual instruments and for integrating information obtained from the selected instruments and other sources of information about a child and family. Such a model or theory of change simply specifies explicitly what child and family development outcomes the program is trying to improve and how the program’s services will influence those outcomes (comprehensive answers to the first two questions in the list above constitute a theory of change).2 With such a model or theory of change, the staff can select instruments that will focus on measuring targeted outcomes and the key services designed to improve them. The program’s theory of change can also guide the interpretation of data at both the individual and program levels and decisions about what to do in response to the information.

The performance measures pyramid, along with a theory of change, provides a framework for developing a program-specific comprehensive plan for performance measurement. We do not recommend that programs collect data related to every element of the pyramid (at least to start with). We recommend that programs collect some information related to each level of the pyramid (management systems, services, outcomes, and the ultimate outcome), because having such information allows staff to link information on program characteristics and services with outcomes for children and families and learn about both how well children and families are doing and how services might be improved to promote better outcomes. All plans should include measurement of child outcomes.

The following example illustrates how the performance measures framework and an explicit theory of change can help program staff develop a comprehensive plan for gathering and analyzing information. A program that provides home-based services may have a theory of change that indicates that it is trying to improve child development outcomes indirectly by improving parenting and parent-child relationships. To accomplish this, it provides home visits in which staff members work with parents on specific parenting skills. The program may emphasize improving children’s language development and train home visitors to work with parents on activities they can do with their child to promote learning and language development, such as reading to them regularly and providing a home environment that supports learning and language development. Based on its theory of change and emphasis on language development, this program might give priority to selecting instruments that measure children’s language development (outcome), the frequency of parents’ reading to their child (outcome), support for language and learning in the home environment (outcome), the frequency and quality of home visitors’ interactions with families focused on children’s language development (services), and supervisory practices to support home visitors’ interactions with families (management system). Ideally, the data obtained with these instruments will indicate how children’s language development is progressing, provide insights into how parents’ progress in parenting skills may be promoting their children’s language development, illuminate how program services may be contributing to parents’ progress (and indirectly to children’s language development), and indicate how home visitor supervision is supporting home visitors’ activities related to language development. The data may also provide insights into ways that home visitors’ activities with parents can be improved to enhance children’s language development further.

A comprehensive plan for gathering and using data includes several important elements. The following questions indicate important elements that a comprehensive plan should include:

• Does the plan include clearly stated purposes for gathering and using data?
  
  
• What instruments will be used to gather information for individual service planning and aggregate performance measurement?
  
  
• With which children or families will each instrument be used?
  
  
• When and how often will each instrument be administered?
  
  
• Who will administer each instrument, and what training will they receive?
  
  
• How will the results of administering each instrument be recorded?
  
  
• How will the results of each instrument be kept confidential?
  
  
• How will the results be shared with parents?
  
  
• How will results be used to plan services for individual children and families? Will they be used for referrals to other community service providers, such as Part C agencies?
  
  
• Will the individual results be aggregated across children and families and analyzed for program planning? If so, how?
  
  
• How will the results feed into continuous program improvement?
  
  
• How will the results be reported to other stakeholders?
  
  
• How will the plan be modified as you gain experience with it?

Appendix A contains a worksheet that provides one way to summarize some of this information and that may be helpful as a component of a comprehensive plan.

In developing a plan that is appropriate and feasible for its program, the staff needs to consider its priorities among information needs. Programs may not have the resources to gather and analyze all useful data, and staff members may not have experience with aggregating results from screening and assessment instruments. Although it is important for programs to create a comprehensive plan, it may be necessary, for guiding program activities, to identify priorities that are feasible with current resources and staff experience. As staff gain more experience with collecting and aggregating data, additional elements of the plan can be implemented (if resources are available).

The kinds of resources that should be considered in developing and implementing a comprehensive plan for gathering and analyzing information include staff resources, technical resources, and financial resources. Staff resources include the time that could be devoted to administering instruments and analyzing the results, as well as activities to build skills and obtain training. Technical resources include training and technical assistance, consultation, and computer hardware and software—as well as qualified personnel—to manage and analyze data. Financial resources include money to purchase needed materials, pay for additional training and technical assistance, and purchase computer hardware or software.

In determining measurement priorities, programs should consider giving priority to instruments that together represent a balance across elements of the pyramid for Head Start programs serving pregnant women and families with infants and toddlers. As noted earlier, lower entries in the pyramid (program management and services) support the outcomes above them. Thus, information on program management and services along with information on child and family outcomes will provide greater insights into ways in which program services can be improved to promote better outcomes at both the individual and the program levels. First priority, however, should be given to measuring child outcomes.

A comprehensive plan for gathering and analyzing data will be useful only if the staff is willing and able to implement it. In developing such a plan, program managers need to build support among staff members and prepare them to use the selected instruments. Involving key staff members who will implement the plan in its development may promote their “buy-in” to the new activities. Involving staff will also help to illuminate the questions that the program most needs to answer. In addition, providing enough training and time to administer the selected instruments may also help staff members embrace the new activities. Finally, involving staff members in interpreting and using the results will help them see how they are useful, motivate them to administer the instruments well, and enable them to help identify needed modifications to the plan.

A key part of the development of a comprehensive plan is the selection of specific instruments that will be used. The checklist in Box 2 lists some questions to help the program staff identify instruments that will best meet its needs. In addition to these questions for considering individual instruments, it is important that, together, the selected instruments draw on multiple sources of information. To be useful for aggregating across families, they also need to be administered to all children or families (or the group of children or families for whom aggregated information is needed) in a consistent manner.

In developing a comprehensive plan, care should be taken to prevent misuses of instruments. It is not appropriate to select a subset of items from an instrument, combine items from multiple instruments, or change the wording or response categories for items in an instrument, because the abbreviated or changed instrument may not be reliable or valid. Some instruments, however, include official subscales or subtests that may be used alone. Only if the directions for using an instrument indicate that using just a sub-scale or subtest is appropriate should staff select and use parts of instruments. To prevent misuse, it is also important that staff members who will be administering an instrument and interpreting the results have sufficient knowledge and training to enable them to do so accurately and appropriately. Lack of knowledge and understanding of an instrument can lead to its misuse.

While implementing a plan for gathering and analyzing data, a program may see the need for changes to the plan. The instruments selected initially may not work well (for example, requirements for administering them may be too difficult to meet or scoring may be too difficult), and different instruments may meet program needs better. The staff may also find that instruments selected initially do not provide all the information needed and that alternative or additional instruments may better meet the program’s needs.

The development and implementation of a plan for gathering and analyzing data takes time. Figure 2 illustrates the development of a continuous improvement model in the Clayton Family Futures program in Denver, Colorado. It summarizes the steps that the program has taken to develop its model, the resources required, the timeframe for each step, and the implications of each step for the program. Over time, the program’s continuous improvement activities have grown as the staff has experienced the value of the information and asked more questions about program services and how children and families are doing.

A plan for measuring outcomes can also be implemented at a broader level. Box 3 illustrates a statewide plan for collecting and analyzing data on a common set of performance measures. State Early Head Start Programs in Kansas have agreed to collect data on a common set of measures to support outcomes-based community planning.