ACF Compassion Capital Fund

Increasing the Capacity of Individuals, Families and Communities

Outcomes Measurement

Measuring Outcomes Overview | Logic Models and Program Theory

Getting Started

Page: 1 | 2

Your outcome measurement efforts will benefit greatly from some planning at the front end. Before you jump in, bring together the staff that will be involved in the outcome measurement process and discuss the questions outlined below:

Where Should We Focus?
In starting out, it’s particularly important not to try to measure too much. Start slowly and learn from your experience. Don’t try to perform outcome measurement at the same time for everything you do—pick one program or service as a beginning. Questions that will help you figure out where to begin include: Is a funder, board member or staff person requesting that you look at a particular program? Do you have a new program with an unproven track record? Do you have an existing service with shaky performance?

What Do We Want to Accomplish?
Once you have decided what program to evaluate, explore the purpose for the outcome measurement processes. There are often a variety of reasons underlying the decision to carry out outcome measurement related to your programs. Which of the options below apply most directly to the program you intend to examine?

To identify what constitutes success for a program and how it will achieve that success (often as part of a program design process prior to funding and implementation)
To determine whether the program theory underlying the program is correct (often as part of a program redesign when new research indicates that the assumptions underlying the program cannot be substantiated)
To describe how you intend to measure a program or service’s impacts (often as part of a funding request)
To report on a program’s impact (often as part of a staff or board planning process)
To determine whether the program has been effective in achieving its intended outcomes (often as part of a funder’s request for evidence that the program merits continued funding)
To determine whether to continue the program and allocate ongoing funding (often as part of internal budget-related priority setting)
To make improvements in how the program is delivered (often as part of program-wide or organization-wide quality improvement effort)

Your Outcome Measurement Team
Once you have decided what you want to learn, it’s time to make it operational. You can start by putting together a small team of people to manage the process. As you think about the roles of the team, consider who on your staff possesses the following skills:

Project coordination, including laying out tasks in a sequence, informing other staff of their roles and assignments, providing assistance to people as they complete their parts of the evaluation process and ensuring that the work is being done
Service or program knowledge, including the ability to identify the relationship between the activities being provided and the intended impacts and an understanding of the types of outcomes your program could achieve
Computer skills, including expertise in formatting surveys and other data collection instruments, creating spreadsheets or databases and entering data

Don’t take on outcome measurement work alone. Create a group within your organization to share the decision-making and responsibilities.

Like any organizational activity, someone must be in charge of outcome measurement to make sure it proceeds smoothly and according to a timeline. Among the team members, who should be designated as the team leader? Outcome measurement is more difficult to implement successfully when it is an unbudgeted add-on to a staff person’s full-time job.

Your outcome measurement work will go much more smoothly if you carve out the time for the lead staff person to manage these efforts. If you start with an outcome measurement focus on one program, estimate that the team leader managing your evaluation will require about one day per week to accomplish the work.

Do We Need to Get Help?
Often smaller organizations feel they do not have the staff capacity to implement outcome measurement activities. It is possible to contract out for help. You can hire an individual experienced in evaluation to design and/or implement your outcome measurement efforts. You may have someone on your board with the expertise you need. You may be able to seek assistance from a local university or community college or advertise for a volunteer with evaluation skills. You may not need them to take on the entire evaluation; perhaps you need an outside person just to help your staff with some of the more technical aspects of the work (like survey design, data analysis, etc.).

Checklist for Getting and Using Outside Help

	Do we need specialized expertise to carry out the outcome measurement tasks, and what specific expertise do we need?
	How much control should this individual have as compared to our outcome measurement team? How flexible is this person going to be?
	Do we know any other organizations who have obtained outside help? Can they provide any recommendations?
	What exactly do we need help to do? What specific tasks should we contract out?

How Can We Make This a Participatory Process?
Once you have clearly identified why you are carrying out your outcome measurement process and put a team in charge of carrying it out, you can decide on involving others in the process. Frontline staff may find outcome measurement threatening. Many people have experienced a punitive approach to evaluation and may feel their programs are under attack. You can allay people’s concerns about outcome measurement by involving them in the process. Outcome measurement results have the greatest likelihood of being used (for the intended purpose(s) you have identified above) if the people most directly affected have meaningful participation in the design and implementation of the process. When deciding whom to invite into the evaluation process, consider the various tasks and roles needed and the following options:

Program director
Frontline staff directly involved in providing the service
Participants or clients who use the program
Staff from other programs or organizations working with you
Volunteer researchers or students from a university, if available

You might want to include local funders. They can be very helpful in identifying the programs to focus on and in articulating the key questions to answer.

What Resources Will We Need?
Outcome measurement does require resources. The categories below will give you some idea of what to consider when planning your budget:

Staff time
Consultants
Communication (e.g., postage and telephone)
Supplies and equipment
Printing

A standard many organizations use for estimating the cost of evaluation is five percent of a program’s budget. However, once an outcome measurement system is put in place, it costs less to maintain over time.

Most organizations include the cost of outcome measurement as part of their administrative overhead, but funding to support the work may be available from other sources. Program funders often are willing to support a portion of the outcome measurement cost, particularly if you have built these costs into the funding request at the front end or the funder has placed a high priority on outcome measurement as part of its funding cycle. Some funders specify an amount or percentage of their grants that must be used for evaluation. In addition, organizations wishing to evaluate a program more deeply may seek funding specifically for that purpose. (For more information, see the Acquiring Public Grants guidebook, part of the National Resource Center’s Intermediary Development Series.)

Timing
Developing an outcome measurement plan likely will take you a number of months. For organizations first learning outcome measurement, it may take as long as three years to develop a comprehensive, smoothly-running outcome measurement system. The first year may be a learning year; the second year may be devoted to testing your systems. It may not be until the third year of data collection that staff sense the results are valid and meaningful. It may take you much less time, but be patient! United Way of America’s Measuring Program Outcomes manual (1996)⁵ includes a section on timing and a template you can use to lay out a timeline for your outcome measurement work.

Checklist for Getting Started

	What program will you start with?
	Who will be involved in the outcome measurement team for planning and implementation?
	Who will manage the process?
	Will you need outside help with your work?
	What additional costs do you anticipate and how can you budget for them?
	What resources are available to conduct outcome measurement?
	What is the timeline for your outcome measurement process?

Timing also is important to funders. Make sure you consider when your funders’ reporting cycles are so that you are producing outcome measurement results at a time that aligns with their requests for information about your programs’ impacts.

There are times when conducting outcome measurement may not be a good idea. If your organization is in crisis, people cannot give their attention to evaluation tasks. If the organization is in severe financial trouble, people must focus on solving that problem. If this is true for you, and if you have any leeway in terms of reporting requirements, consider delaying your outcome measurement work until it can get the attention it deserves.

Next, we will look at the key steps in the outcome measurement process and provide information to help you prepare for and conduct outcome measurement. The steps include the following:

Identify meaningful, relevant and realistic outcomes
Create a logic model for your program
Identify indicators to measure success in achieving outcomes
Select data collection methods and create data collection instruments
Design a practical and sustainable data collection plan

Page: 1 | 2

Measuring Outcomes Overview | Logic Models and Program Theory