< Back to Search

CCF/SCF Tools Developing a Plan for Outcome Measurement

Published: September 14, 2012
Audience:
Strengthening Communities Fund (SCF), Compassion Capital Fund (CCF)
Category:
Guidance, Policies, Procedures, Tools

 

Overview



Outcomes communicate value.

Outcomes are the reason nonprofit organizations strive to deliver programs and build their capacity. Most importantly, however, proper evaluation techniques provide your organization with proof of their value to existing funders, potential funders, and the larger community. Whether this value is communicated in dollars or the number of individuals served, quantifiable performance measures are becoming important in the increasingly competitive social service industry.
Successful outcome measurement hinges on a cohesive plan.

Preparation and planning is a key part of the outcome measurement process.  The better the planning, the more impact the outcome measurement will have on your organization and your organization's bottom line.  By the conclusion of this lesson you will be able to:

  • Understand how to identify measurable outcomes and outcome chains
  • Develop a logic model
  • Identify performance indicators and performance targets to support the outcome measurement process

 

1. Getting Started



Narrow your scope and clarify your goals.

In starting out, it's important not to try to measure too much. Start slowly and learn from your experience. Don't try to perform outcome measurement at the same time for everything you do. Pick one program or service as a starting point. Questions that will help you brainstorm where to begin include:

  • Is a funder, board member, or staff person requesting that you look at a particular program?
  • Do you have a new program with an unproven track record?
  • Do you have an existing service with shaky performance?
  • Do you have an effective program that you want to document as being so?

Once you have decided what program to evaluate, explore the purpose for the outcome measurement processes. There are a variety of reasons that could motivate outcome measurement, including:

  • To support new program design and identify success in a new program area
  • To support program redesign by determining whether the program's underlying theories and assumptions are correct
  • To support funding requests and describe how you will measure a program or service's impacts
  • To support staff or board planning processes and report on a program's impact
  • To support a funder's request for evidence of the program's effectiveness
  • To support internal reviews and determine whether to continue to allocate funding to a program
  • To support organization-wide quality improvement and refine program delivery

Assemble a well-rounded team.

The outcome measurement team will require a well-rounded group with varying skill sets and a devoted leader.  As you begin to assemble your outcome measurement team, consider who on your staff possesses the following skills:

  • Project coordination, including laying out tasks in a sequence, informing other staff of their roles and assignments, providing assistance to the team as they complete their parts of the evaluation process, and ensuring that the work is being done
  • Service or program knowledge, including the ability to identify the relationship between the activities being provided and the intended impacts and an understanding of the types of outcomes your program could achieve
  • Computer skills, including expertise in formatting surveys and other data collection instruments, creating spreadsheets or databases, and entering data

The outcome measurement process can often feel intimidating or threatening to program staff, as outcome measurement is sometimes a response to poor performance or part of a corrective action plan. To allay your staff's concerns about outcome measurement, involve them in the process whenever possible.

Click below to open interactivity Analyze your staffing needs.

Outcome measurement is an investment.

Developing an outcome measurement plan will require an investment of time and resources. For organizations first learning outcome measurement, it may take as long as three years to develop a comprehensive outcome measurement system. Timing is also important to funders. Make sure to consider when your funders' reporting cycles are so that you are producing outcome measurement results at a time that aligns with their requests for information about your programs' impacts.  Keep in mind that there are times when conducting outcome measurement may not be a good idea. If your organization is in crisis or severe financial trouble, outcome measurement cannot be a priority.

In addition to investing time, outcome measurement may require an investment of other resources, including:

  • Staff labor
  • Consultants
  • Communication (e.g., postage and telephone)
  • Supplies and equipment
  • Printing

 

2. Understanding Outcomes



Outcome statements capture the "who," "what," and "how."

Formulaic, straightforward outcome statements can help you develop your outcomes. When crafting an outcome statement, consider the details of the initiative you are providing, the recipient of that service or program, and the intended impact of that service on your target recipients.   For instance, a youth-serving nonprofit might identify the following outcome statements in relation to their afterschool programming:

  • Increased school engagement for participating students
  • Reduced high-risk behavior for participating students
  • Increased high school graduation rates for participating students

Outcome chains document correlative relationships.

Outcome chains can help organize your thinking about what you hope to achieve. They require you to put your program theory to work and articulate how your activities will bring about the impacts in the organizations with whom you are working. Outcome chains create a logical progression of the short-term, intermediate, and long-term outcomes that lead to your goals. Consider the example of a large nonprofit that provides training to smaller nonprofit organizations. The provision of training can create the following chain, linking reactions, learning, behavior, and results:

Not all organizations can and will measure all the different outcomes noted in an outcome chain. Consider the example of the outcome chain above. Your organization may not have the tools and resources to evaluate outcomes as they relate to reaction, learning, behavior, and results. If your circumstances do not allow you to evaluate all areas, focus on the earlier outcomes noted in the outcome chain. There is no point in measuring for results if you cannot point to the series of outcomes that impacted those results.
Click below to open interactivity Use if/then statements to identify outcomes.

Develop realistic, informed outcomes.

When crafting outcomes, consider the following:

  • Are the outcomes related to the “core business” of your organization or program?
  • Is it within your control to influence these outcomes?
  • Are your outcomes realistic and attainable? Are your outcomes achievable within funding and reporting periods?
  • Are your outcomes written as change statements—will things increase, decrease, or stay the same?
  • Have you moved beyond client satisfaction in your outcomes?
  • Is there a logical sequence among your short-term, intermediate, and long-term outcomes?
  • Are there any big “leaps” in your outcomes, i.e., gaps in the progression of impacts?

 

3. Logic Models and Outcome Measurement


Ask the right questions.

When looking to identify the elements of your organization's logic model, consider the following:

  • Inputs/Resources: What inputs or ingredients do you need to operate your program? How many staff? How large a budget? How many clients do you have/need?
  • Activities: What will you do? What methods will you use to deliver your program or activities? What content areas will you cover? What will you provide?
  • Outputs: What will be the tangible products or units of your program or activity? How many clients will you serve?
  • Outcomes: What impact will your program or activities have on your clients? What is reasonable to expect in the way of change?

Logic models capture underlying assumptions and program theory.

This connection between the activities you provide and the outcomes you hope to accomplish is known as the program theory; it articulates the assumptions about the ability of certain activities to drive particular changes. Many nonprofit providers use logic models as a tool to reflect the program theory underlying their programs. One contribution that outcome measurement can make in your own organization is to demonstrate whether or not the program theory underlying your organization's activities and initiatives is valid.

Logic models document relationships.

While not all logic models look the same, they all serve the same purpose: to graphically capture the assumptions and cause and effect relationships that drive your organization's work on a project.

Download a sample logic model template.

 

4. Developing Performance Indicators

Indicators are specific and observable.

In order to serve effectively as a bridge to data collection, indicators must be specific items of information that describe observable and measurable characteristics or changes in corresponding outcomes. Indicators must be measures that can be seen, heard, counted, reported, or enumerated using some type of data collection method.

Performance indicators answer questions like, "How you will know when changes have occurred?" and "How you will know when you have achieved the outcomes?" Thinking ahead to possible data collection methods will tell you if your indicators are specific enough. Ask questions like these to determine whether your indicators will work:

  •     How can I see the change? (Through what kind of observation?)
  •     How can I hear the change?  (Through interviews? Focus groups?)
  •     How can I read the change?  (Through surveys? In records?)

Click below to open interactivity Indicators support larger outcomes.

Be thorough, yet realistic when identifying indicators.

Each outcome you identify will generally have one to three supporting indicators depending on the complexity of the outcome. When selecting indicators, keep things simple and be sure to collect indicators that are useful, meaningful, and informative.  Keep in mind that you will be responsible for measuring these indicators through your data collection process. Consider the following checklist:

  • Do your indicators make sense in relation to the outcomes they are intended to measure?
  • Are your indicators directly related to the outcome?  Do they define the outcome?
  • Are your indicators specific?
  • Are your indicators measurable or observable?  Can they be seen (i.e., observed behavior), heard (i.e., participant interview), or read (i.e., client records)?
  • Is it reasonable that you can collect data on the indicators?
  • Is it within your resources to collect data?

Targets, comparative standards, and baselines give meaning to indicators.

When developing indicators, it is often helpful to identify targets, baselines, and comparative standards.

  • Targets are the specific levels of achievement that you hope to achieve for an indicator or outcome.  Targets are the quantifiable goals you hope to achieve.
  • Comparative standards are the data points that are used as a comparison or standard of achievement for a particular indicator or outcome. Comparative standards help to provide insight into how your targets relate to or improve on past performance.
  • Baselines are the data points that are gathered to provide a comparison for assessing program changes or impact. A baseline will allow you to set reasonable targets and measure the progress against these targets.

 

Summary
Outcome measurement proves correlation, not causation.

Keep in mind that outcome measurement explores what your program provides, what its intended impacts are, and whether or not it achieves them. It will not prove that the changes that take place are a result of your program. In order to prove direct causation, an organization will need to take part in experimental research and a controlled study to link your specific programs and activities to results. Naturally, this requires considerable time, effort, and resources.
Learn more about outcome measurement at the links below.

    Center for Program Evaluation and Performance Measurement
    This website from the Office of Justice Programs' Bureau of Justice Assistance provides basic information about initiating and planning evaluation processes and planning and links to other government resources on the topic, as well.
    Outcome Measurement Resource Network
    This website, maintained by the United Way of America, includes key definitions and concepts around outcome measurement, resource lists, sample outcome measurement initiatives, and a full-text resource library.
    Demystifying Outcome Measurement in Community Development
    Developed for Neighbor Works America by the Joint Center for Housing Studies of Harvard University, this 2007 report by Renu Madan provides valuable information on how and why community development organizations measure outcomes.

Download