Outcomes Measurement
Performance
Indicators |
Creating a Plan for Your Outcome Measurement System
Data Collection Methods
Page: 1 | 2 | 3Your data collection will include attention to all the elements of your logic model: what resources you had available, what activities you actually provided, how many of each output you delivered and to what degree you accomplished your outcomes. For all of these elements, except the outcome and indicator data, your organization’s records should provide the information you need. However, in collecting indicator data, you have other options. Intermediaries are likely to use one or more of four methods for collecting data: surveys, interviews, observation and record or document review. In selecting the best method for data collection, you will need to consider the advantages and disadvantages regarding:
- Type of information needed: some outcomes and indicators lend themselves to one particular method; in other cases, data could be collected in more than one way
- Validity and reliability: some methods generate more accurate and consistent information than others
- Resources available: including staff availability and expertise, time and money
- Cultural appropriateness: using the broadest definition of culture to ensure the methods fit language, norms and values of the individuals and groups from whom you are collecting data
Surveys
Surveys are standardized written instruments that can
be administered by mail, email or in person. The primary
advantage of surveys is their cost in relation to the
amount of data you can collect. Surveying generally
is considered efficient because you can include large
numbers of people at a relatively low cost. There are
two key disadvantages. If the survey is conducted by
mail, response rates can be very low, jeopardizing the
validity of the data collected. There are mechanisms
to increase response rates, but they will add to the
cost of the survey. Written surveys also provide no
opportunity for respondents to clarify a confusing question,
again creating a problem with the validity of their
responses. Good pre-testing of a survey instrument can
reduce the likelihood that problems will arise.
Examples:
- Survey to track grassroots organizations’ use of and satisfaction with technical assistance services you provide
- Survey of all organizations receiving technical assistance to learn about changes in their fundraising tactics and the results of their efforts to raise more money
Interviews and focus groups
Interviews also use standardized instruments but are
conducted person-to-person either in person or over
the telephone. In fact, an interview may use the same
instrument created for a written survey, although interviewing
generally provides the advantage of having the opportunity
to explore questions more deeply than does a written
survey. You can ask more complex questions in an interview,
since you have the opportunity to clarify any confusion.
You also can ask the respondents to elaborate on their
answers, eliciting more in-depth information than a
survey provides. The primary disadvantage of interviews
is their cost. It takes considerably more time and,
therefore, more money to conduct telephone and in-person
interviews. This usually means you are able to collect
information from many fewer people. The reliability
of interviews also can be problematic if interviewers
are not well-trained, since they can ask questions in
different ways or otherwise bias the responses.
Examples:
- Interviews at different grassroots organizations to learn about the way in which they are applying new knowledge of partnership development
- Interviews with individuals within an organization to explore their perceptions of changes in capacity and ability to deliver services
A focus group is a particular type of interview conducted with a small group of people to obtain information in a defined area of interest related to your outcomes and indicators. While interviews with individuals are meant to solicit data without any influence or bias from the interviewer or other individual, focus groups are designed to allow the participants to discuss the questions and share their opinions. This means that people can influence one another in the process, stimulating memory or debate on an issue. The advantage of focus groups lies in the richness of the information generated. The disadvantage is that you can rarely generalize or apply the findings to your entire population of participants or clients. Focus groups often are used prior to creating a survey to test concepts and wording of questions. Following a written survey, they are used to explore specific questions or issues more thoroughly.
Examples:
- Structured meetings with staff in a faith or community-based organization to learn more about their grants management practices, what worked during the year and what did not
- Structured meetings with staff from several organizations to explore their use of computer technology for tracking financial data
Observation
Observations can be conducted of individual behaviors
or interactions among individuals, of events or of physical
conditions within a site or facility. They require well-trained
observers and detailed guidelines about whom or what
to observe, when and for how long and by what method
of recording. The primary advantage of observation is
its validity. When done well, observation is considered
a strong data collection method because it generates
first-hand, unbiased information by individuals who
have been trained on what to look for and how to record
it. Observation does require time—for development
of the observation tool, training of the observers and
the data collection—making it a more costly data
collection method than some of the others.
Examples:
- Observation of individuals participating in training to track the development of their skill in the training topic
- Observation of community meetings sponsored by grassroots organizations to learn about their partnership-building techniques and collaborative behavior
Record Review
Record or document review involves the systematic collection
of needed data from internal, organizational records
or official records collected by other groups or institutions.
Internal records available to an intermediary might
include financial documents, monthly reports, activity
logs, purchase orders, etc. The advantage of using records
from your organization is the ease of data collection.
The data are there and no additional effort needs to
be made to collect it—depending on if the specific
data you need are actually available and up-to-date.
If the answer is yes, record review is a very economical and efficient data collection method. If the answer is no, it is likely well worth the time to make improvements to your data management system so that you can rely on internal record review for your outcome measurement work. Just a few changes to an existing form can make it into a useful data collection tool. Just a small amount of staff training can increase the validity and reliability of internally-generated data.
Examples:
- Sign-in logs from a series of workshops to track attendance in training, measuring consistency of attendance as an indicator of organizational commitment to learning.
- Feedback forms completed by workshop participants to learn about satisfaction with training provided
Official records can include federal, state or local government sources such as the U.S. Census, health departments, law enforcement, school records, assessor data, etc. Record review from institutional sources may be of little value to intermediaries in the outcome measurement work related to the Compassion Capital Fund (CCF). It might, however, be useful for other types of evaluations you conduct. If the data are relevant to your outcomes and indicators and are accessible, then official record review is very low cost. But the questions about relevance and accessibility are major ones.
Performance
Indicators |
Creating a Plan for Your Outcome Measurement System