Assets for Independence Act Evaluation:
Design Phase, Concept Paper
February 16, 2000
5. |
Program and Participant Tracking and Monitoring |
||
| References | |||
Tracking IDA programs, IDA participants, savings patterns, and uses of IDAs will yield basic information about success or failure of the IDA demonstration. According to Section 412 of the Assets for Independence Act (AFIA), reporting is separate and distinct from evaluation.
The purpose of reporting on IDA program progress and goal attainment is to answer the following questions:
- How much do participants save?
- What is the pattern of savings?
- How does savings vary by program and participant characteristics?
- For what purposes do participants use their savings?
In order to collect the basic information necessary to answer these questions, the Center for Social Development (CSD) spent several years developing and refining a tracking and reporting instrument. In 1996, a committee was formed to assist CSD in determining what data from IDA programs and participants should be tracked. Initially in paper form, the tracking tool was to become a comprehensive program management tool. With significant time and investment, the Management Information System for Individual Development Accounts (MIS IDA) was developed, revised, and finally released in 1998. MIS IDA was designed not only to assist IDA program staff in keeping track of program and participant progress but also to provide account statements to participants, reports to funding sources, data integration with financial institutions, and centralized data management in multi-site settings. A database from MIS IDA can then be sent to a central location and merged with other databases from other IDA programs for analysis.
Strengths of this reporting method include:
- a standardized data collection method;
- data comparison across a variety of settings; and
- an integrated tool used for tracking and in daily program functioning.
Where possible, MIS IDA uses standard lists from which staff can select to describe program and participant characteristics. Although programs may differ somewhat in design and in participant population characteristics, MIS IDA provides a general framework for reporting. This framework allows data from various programs to be aggregated and compared.
The strategy in creating MIS IDA has been to make a management information system so practical that IDA programs will want to use it. Research shows that field personnel are more likely to use an "evaluative" tool when it assists them in their daily functioning (Austin, et al., 1982; Clark, 1985; Freel & Epstein, 1993). The tension between reporting and daily program functioning is lessened when processes are fully integrated (Grasso and Epstein, 1993). The expansion of MIS IDA from a paper "tracking and reporting instrument" to a robust centralized information system increases the likelihood of its use and therefore its usefulness in data tracking. Version 2.02 of MIS IDA is the current standard, and is now being used in IDA programs around the country.
Data to be collected in MIS IDA and reported include: program characteristics, program costs, numbers of participants, characteristics of participants, longevity of participation, patterns of savings; amounts saved, and uses of IDAs (typically for home ownership, education, or business capitalization). Specific program information includes program design features, administrative costs, and funding partner contributions. Specific participant information includes: demographics, financial assets/liabilities, account information, IDA uses, participation in economic education, and program exit status. Analysis will generate information on savings patterns and how program and participant characteristics are related to savings. "Savings" may be measured in various ways, including:
- total participant savings (participant closing balance + matched withdrawals);
- total IDA savings (participant closing balance + matched withdrawals + match dollars);
- average monthly (net) savings (total participant deposits + interest - unmatched dollars, averaged by the number of months the participant is in the program); and
- deposit regularity (number of months of positive deposits divided by the number of months the participant is in the program).
There are several important considerations in using a reporting instrument such as MIS IDA. The first is that MIS IDA should be distributed to the field prior to program startup. A corollary is the need for training and technical assistance in getting the program properly installed and set up. Many programs already use MIS IDA but will require instruction in how to separate their current IDA database from the AFIA database. And some will have multi-site configurations that differ from single-site IDA programs. Second is data validity. MIS IDA is based on self-report. Program staff will be entering program design characteristics of their own program as well as data collected from participants. Training and technical assistance on the use of MIS IDA will increase understanding of the content of questions and will increase awareness of the importance of accuracy. Third is ensuring data quality. MIS IDA is coded with multiple data edit features, but data entry errors can and do occur. It is essential that data are reviewed for accuracy and "cleaned" prior to reporting. These issues point to the need for immediate technical assistance and training on the use of MIS IDA to enhance the level of accuracy in later reporting.
MIS IDA will require some revisions based on the implementation of
AFIA. However, in its current form, it is usable for AFIA sites to
begin the data collection process; in fact, many already have. As
noted at the beginning of this section, reporting is clearly outlined
in Section 412 of the AFIA legislation and the evaluation is separately
delineated in Section 414. Given this distinction and the immediate
deployment of MIS IDA in the field, it is recommended that the implementation
of tracking and reporting occur as soon as possible and under separate
contract from the evaluation, which will not begin until after the
year-long evaluation design phase.
References
Austin, M.J., et al. (1982). Evaluating your agency's programs. Beverly Hills, CA: Sage Human Services.
Clark, D.L. (1985). Emerging paradigms in organizational theory and research. In Y.S. Lincoln (Ed.). Organizational theory and inquiry: The paradigm shift. Beverly Hills, CA: Sage Publications, Inc.
Freel, C. & Epstein, I. (1993). Principles for using management information data for programmatic decision making. Child & Youth Services, 16(1), 77-93.
Grasso, A.J., & Epstein, I. (Eds.). (1993). Information systems
in child, youth, and family agencies: Planning, implementation, and
service enhancement. Binghamton, NY: The Haworth Press, Inc.
