Assets for Independence Act Evaluation:
Design Phase, Concept Paper
February 16, 2000

3.	Process Analysis
		Rationale
		Approach

 

The process analysis will provide a comprehensive picture of the development, planning, start-up, and on-going operations of AFIA programs. It will help HHS staff understand how the programs work and the factors influencing effective operations. In describing how clients interact with program staff and receive program services, the process analysis also will help us interpret the findings of the impact analysis (to be described in Section 6).

Rationale

The contribution of the process analysis will be:

• to describe the goals of the AFIA legislation and the program features prescribed by the Act.
• to assess the implementation of AFIA-funded programs among grantees, as viewed in relation to the provisions of both the statute and grantee applications;
• to compare the experiences of different grantees in developing and implementing their IDA programs; and
• to help interpret the findings of the impact analysis by providing programmatic context.

In short, the process analysis can both explain and expand upon the impact analysis. More specifically, the process analysis should speak to several of the "factors to evaluate" indicated in the Act. As noted previously in Exhibit 1-1, the process analysis will provide insight regarding:

• the effects of incentives and organizational or institutional support on savings behavior-It will do so particularly at the experimental site, where process-related findings will provide the programmatic context for interpreting the impact findings.
• the effects of IDAs on savings rates, homeownership, level of post-secondary education attained, and self-employment, and how such effects vary among different populations or communities-Again, the findings of the process analysis would be particularly insightful at the experimental site.
• the potential financial returns to the Federal Government and to other public and private sector investors in IDAs over a 5-year and 10-year period of time; and
• the lessons learned from the demonstration projects, and whether a permanent program of IDAs should be established.

We view the process and impact analyses as closely complementary. The impact analysis will provide estimates of the effects of IDA incentives. Although the impact analysis can indicate whether IDAs affect participant savings and asset accumulation, it is of limited use to explain why and how those effects accrue. The process analysis will indicate the dynamics of program-client interactions and suggest the mechanics by which behavioral changes occur. It is therefore crucial that the process analysis be conducted at the experimental site, among others.

The value of the process analysis goes beyond it tells us about the dynamics of change at any one site. It can also illustrate the variety of program models that evolve under AFIA. Existing research into IDA programs--for example, from Abt Associates' study of asset accumulation initiatives (sponsored by the USDA) and CSD's evaluation of the American Dream Demonstration—indicates that current IDA program models are quite diverse. For example, programs may vary significantly in the strictness with which staff monitor and enforce the requirements of program participation (e.g., minimum deposit amounts, frequency of deposits, attendance at counseling and training sessions). Programs may also vary philosophically in their approaches to the use of funds for family emergencies. Some programs adopt a laissez faire approach to emergency withdrawals, on the premise that the accountholder should have ultimate discretion over the use of funds. Others adopt a more interventionist approach, seeking to help the accountholder avoid an emergency withdrawal if at all possible. To the extent that programs vary on these and other important features, it is important to ask whether these differences appear to influence participant outcomes. Of course, without an experimental design in each site, we cannot definitively attribute causality to the program. What we observe in the process analysis, however, can serve to narrow and sharpen our focus on those aspects of the program that appear to offer the most plausible explanation of effects.

Second, a process analysis that traces the development of an IDA program over time, can provide valuable lessons for other programs. It may identify issues that were found problematic across all sites or only under certain conditions. For example, establishing relationships with financial institutions, or devising procedures for efficient verification of account use, may prove to be more difficult than sites anticipated. The lessons learned about how sites overcame these challenges (or the implications of not overcoming them) would be extremely useful to both current and future sites and may have policy implications, to the extent some policy elements appear to promote or impede success.

 top of page

---

Approach

Perhaps the greatest challenge of the process analysis will be to make sense of the great variety of program features observed. We seek to avoid a research design that produces cacophony of descriptive findings, but stops short of deriving meaningful, salient, generalizable insights from them.

Accordingly, it is useful to think of the process analysis as composed of two complementary activities: a description of the program models that emerge under AFIA and an analysis of what these program models imply about program outcomes, replicability, and long-term sustainability.

A variety of data sources, collection techniques, and analytic approaches will be needed to meet the multiple objectives of the process analysis. The following data collection activities will be necessary to support the process analysis:

• a program survey to all grantees;
• multiple site visits to selected programs;
• a treatment-group module in the follow-up surveys planned primarily for the impact analysis.

These three data collection efforts are described below.

Program Survey

The descriptive task can be effectively met by the use of a brief program survey sent to each year's cohort of grantees. The survey will supplement findings from the site visits, which will necessarily be restricted to a small group of grantees. Our experience is that short program surveys can be very effective to capture straightforward descriptive information about program features. Having this information available for the universe of grantees can be useful in several ways. First, it can identify the degree to which IDA programs are implemented consistently with what was intended (as articulated in grantees' applications). Second, it can place the program features identified in the process analysis in a broader context. (For example, how common is a particular program feature that was observed during visits to several sites?) Third, it can identify any differences in cohorts over time. It may be expected that, as each successive cohort's experiences become known, IDA programs "mature" over time. That is, newer sites will take into account their predecessors' successes and challenges. Cohort differences may also occur if certain AFIA requirements, over time, tend to encourage or discourage certain types of program models.

The program survey could collect the following types of information:

• funding levels (federal, state, and local);
• eligibility requirements;
• program requirements (minimum deposits, counseling requirements, etc.);
• number of account holders currently anticipated;
• length of time the program has been operational;
• financial institutions involved in the program; and
• terms of the savings accounts.

Our experience is that program surveys can be most effective when the information sought is straightforward and descriptive. Such surveys, however, have limited ability to indicate the dynamics of program operations. Accordingly, the process analysis would certainly require site visits to a limited number of sites. Important considerations with regard to the site visits are described below.

Site Visits

Site visits to selected AFIA programs are a major component of the process analysis. These visits will allow us to view first-hand how programs operate. The visits will also allow us to obtain a comprehensive description of each program, including its planning phase, organizational aspects, service provisions, and program operations. In addition, we will document key contextual factors, such as the social, economic, and political aspects that may have a bearing on program performance.

The programs to be visited will need to be selected carefully. For obvious reasons, the experimental site must be included. Other programs selected should encompass to some meaningful extent the variation in program features and operational settings found among funded sites. We recommend developing a typology of salient program features that are hypothesized to affect program "success" (however defined). These parameters might include, for example:

• program age;
• organization size;
• urban/rural location; and
• program cohort (Year 1 grantee, Year 2 grantee, etc.)

Site selection for the process analysis might even include one or several of these features as "stratification factors." These factors can be collected in the program survey and validated in the course of the three site visits conducted during the research design phase.

Even if not used for site selection per se, the typology forms the conceptual framework that should be used to guide the research. Site visits should be conducted with that typological framework in mind, while still remaining open to the importance of unexpected factors. This can be done by developing research protocols that probe certain pre-identified program features very carefully, while also including open-ended queries that allow for unanticipated issues.

We recommend multiple site visits to capture the evolution of programs over time. Analyzing the trajectory of programs' development over time can shed light on the common challenges and successes, and can suggest areas in which technical assistance might be targeted. The evolution of the early sites in particular can provide important lessons for subsequent AFIA applicants trying to determine how to configure their program.

Specific topics that might be examined during site visits include:

• local context (community, economic, social, and physical features);
• "closeness of fit" of IDA goals with organization's overall mission;
• resources available (material resources as well as expertise and links with other groups);
• eligibility criteria;
• degree of monitoring/verification;
• financial counseling;
• relationships with financial institutions;
• magnitude/type of other funding and any associated constraints;
• program strengths and weaknesses.

Treatment-Group Module in the Follow-up Surveys

A special module for treatment-group respondents will be included in the follow-up surveys used for the impact analysis. The module will contain a series of questions related to the services that participants receive through the program. The module will help us understand the services provided by the program from the participants' point of view. More intensive and selective input from participants will be provided from the in-depth participant interviews, as described in detail in Section 4. The combination of the treatment-group module in the follow-up surveys and the in-depth participant interviews are the vehicles used to collect the participants' perspectives on the program.

Some topics that could be covered in the treatment-group module include the following:

• how did they hear about the program?
• what motivated them to apply?
• what was involved in enrolling in the program?
• what, in their minds, are the objectives of the program?
• how frequently did they attend counseling or training sessions?
• what types of counseling did they receive?
• what were the burdens of participating in the program, in time and expense?
• how easy/difficult was it to meet the requirements of the program?

In brief, the process analysis described in this section will contribute importantly to the evaluation in three ways. The first is to describe and document the process of planning, implementing, and operating IDA programs. The second is to evaluate the extent to which the programs were implemented as planned. The third is to provide useful information for interpreting the impact findings.

 top of page