Increasing the Capacity of Individuals, Families and Communities

Outcomes Measurement

Performance Indicators | Creating a Plan for Your Outcome Measurement System

Data Collection Methods

Page:  1 | 2 | 3

Validity and Reliability
Validity and reliability are two critical concepts in implementing effective outcome measurement systems. Validity is the accuracy of the information generated. The validity of a data collection instrument is how well it measures what it is supposed to measure. Putting in the time to create good instruments, carefully considering what is being measured and pre-testing the instruments will help increase their validity.

Reliability refers to consistency, or the extent to which data are reproducible. Do items or questions on a survey, for example, repeatedly produce the same response regardless of when the survey is administered or whether the respondents are men or women? Bias in the data collection instrument is a primary threat to reliability and can be reduced by repeated testing and revision of the instrument.

You cannot have a valid instrument if it is not reliable. However, you can have a reliable instrument that is not valid. Think of shooting arrows at a target. Reliability is getting the arrows to land in about the same place each time you shoot. You can do this without hitting the bull’s-eye. Validity is getting the arrow to land on the bull’s-eye. Lots of arrows landing in the bull’s-eye means you have both reliability and validity.

This checklist can help you decide which data collection methods are most appropriate for your outcome measurement.

Checklist for Selecting Data Collection Methods8 Surveys             Do I need data from the perspective of the participant, client, beneficiary or customer? Yes No   Do I have a way to get it from these individuals in a systematic way? Yes No   Do I need data that are standardized so that statistical comparisons can be made? (For example, will I need to report percents or other statistics?) Yes No   Will participants be able to understand the survey questions? (Consider age, cultural backgrounds, etc.) Yes No   Do participants have the necessary knowledge or awareness to accurately answer questions about the outcomes? Yes No           If you have answered YES to questions #1 through 5, surveys may be appropriate for collecting data on your outcomes and indicators.       Interviews               Are more in-depth answers necessary to adequately measure the indicators or to get information on what is needed or what should change? Yes No   Will it be necessary for someone to personally ask participants questions (either on the phone or in person) in order to collect the information related to this outcome? (Consider age, cultural backgrounds, as well as state of mind or receptivity of participants.) Yes No           If you have answered YES to questions #6 and 7, interviews may be appropriate for collecting data on your outcomes and indicators.         Observations                   Is it difficult to accurately measure the indicators by asking people questions about opinions or perceptions? Yes No     Can this outcome or indicator be assessed accurately by someone trained to observe it in action — can something actually be observed? Yes No     Do you have the staff resources for someone to observe events, conditions, interactions or behaviors? Yes No               If you have answered YES to questions #8, 9 and 10, observation may be appropriate for collecting data on your outcomes and indicators.         Internal Record Review                       Do you have individualized records, reports, logs or other systematic ways that you track things in your program or services? Yes No     If an information system exists, are the data consistently entered into it in a timely way? Yes No     If a system exists, can information be extracted from it easily? Yes No               If you have answered YES to questions #11, 12 and 13, internal record review may be appropriate for collecting data on your outcomes and indicators.         Official Record Review       Do official records exist which track the data you need on your outcomes and indicators?   Yes   No     Are the data accessible to you — will it be possible to get the cooperation of outside agencies or institutions in order to get access to official records? Yes No               If you have answered YES to questions #14 and 15, official record review may be appropriate for collecting data on your outcomes and indicators.

Page:  1 | 2 | 3

Performance Indicators | Creating a Plan for Your Outcome Measurement System