3J. |
Designing an AFI Project: Evaluation |
||
PDF Version (661KB)As a Federal demonstration program, AFI focuses on learning what works in asset building so that promising practices can be replicated. In addition to giving grants to organizations willing to implement AFI Projects, AFI supports a national evaluation to study the long-term effects of AFI Projects on participants and to determine the extent to which asset-building strategies in general and IDAs in particular are useful for addressing poverty. AFI also encourages each program to use its own evaluation and quality improvement mechanisms to help improve their programs.
The National Evaluation
Each AFI Project must collect data for the national evaluation by providing annual data reports to OCS , usually from the AFI 2 System. Some Projects may also be asked to coordinate a day-long site visit from OCS researchers and to identify Project participants who will fill-out surveys for evaluation purposes. AFI Projects are required to set aside a small portion of their Reserve Account to pay for activities related to the national program evaluation. This money is to cover expenses such as staff time for entering data or completing required reports, to pay for AFI Project staff time in working with the national evaluator, or to work with participants who take part in the national evaluation.
Project-Level Evaluation and Overall Program Management
In addition to the national evaluation, AFI encourages Projects to use evaluation techniques and to build data collection and analysis into their program design. Project-level evaluation and program management or quality control activities help Projects track progress in achieving their goals and objectives and determine when to consider changes in program design or delivery.
At its most basic level, evaluation looks at effort (what you did) and effect (what happened as a result of that effort). Evaluation information may be used to keep track of Project activities, to find out the effect of the Project on clients, to make sure high quality services are delivered, to identify areas for needed change, and to identify models for service delivery.
Outputs and Outcomes
Two important evaluation measures are outputs and outcomes.
- Outputs refer to quantifiable activities. They measure your program's efforts and activities. Output measures represent the goods and services a Project produces in the course of implementing their AFI Project.
- Outcomes are changes that your Project participants, or clients, experience as a result of their participation in the AFI Project. Outcomes, ideally, are also what happen when the needs of your target population are met. They are ways your program will enhance your IDA holders' strengths and provide them with resources for lasting change. Outcomes are often expressed in time frames (short-, medium-, and long-term increments) and are related to changes in behavior, knowledge, skills, attitude, values, or condition of participants and/or AFI agency (see Table 3.5).
People |
Areas for Change |
Time Range |
|---|---|---|
• Individual participants • Participants' families • Participants' communities • Agency |
• Knowledge and skills • Behavior • Attitude • Values • Asset development • Self sufficiency |
• Short term • Medium term • Long term |
AFI encourages projects to develop outputs and outcomes specific to their AFI work. Because outputs and outcomes must be measurable, Projects must also identify how they will collect the necessary data for the outcomes and outputs.
Tip: A good data management system is fundamental to an effective evaluation process. Not only are data collection and reporting critical to any AFI Project's success, but they also provide the central collection point and tracking guidance for all data needs of the program. OCS has developed the Assets for Independence Information System (AFI 2 ) to help AFI Projects maintain data. AFI 2 is a web-based, user friendly, and secure online system to manage AFI IDA data. The system is a central repository for AFI Projects. AFI 2 is a single point of entry in that it allows for data input from OCS, AFI grantees, and AFI sub-grantees. This improves the accuracy of data and analytical reports to better support analysis and decision making. To learn more, go to the OCS Asset Building Web site at http://www.acf.hhs.gov/assetbuilding .
Typical outcomes of AFI Projects might include increased percentage of participants who are homeowners, increased percentage of participants who acquire secondary education, and increased percentage of participants who create or expand small businesses. The specific outputs associated with these outcomes, therefore, might be the number enrolled, the number who are working toward the outcome of purchasing each asset, and the number of asset-specific counseling sessions, which facilitate the outcome of successful asset purchase.
AFI Projects can formulate outputs and outcomes for their agency as well as for the AFI participants. Agency outcomes might include leadership enhancement or better relations with the community.
Tip: For more information on outcomes, outputs, and ways to develop these, check out the resources on outcome evaluations developed by the United Way of America at its Web site, http://national.unitedway.org/outcomes /. Additional information on outcomes evaluation can be found at http://www.mapfornonprofits.org/ , sponsored by Management Assistance Programs for Nonprofits. Finally, another important performance management framework, used by the Community Action Agency network, is Results Oriented Management and Accountability, or ROMA. To learn more about this Federal initiative, see http://www.roma1.org/ .
Quality Improvement
Aside from evaluation measures such as outputs and outcomes, many projects routinely use other quality improvement tools to assess the effectiveness of their program. Projects use these tools to solicit input from participants or partners to ensure that the communication, services, training, and other program components are meeting participants' needs. If they are not, then the Project can make adjustments to improve the program.
Projects may gather quality improvement information through surveys, interviews, and focus groups. Some projects administer surveys at the end of training or other workshop sessions, upon program entrance and exit, and/or after six months or longer in the program. Along with using this data to make project improvements, innovative projects also can use the information for fundraising and marketing. For example, this information can be used to develop testimonials that can be used in a press release or program brochure.