Assets for Independence Act Evaluation:
Design Phase Final Report
August 9, 2000

3.	Process Analysis
	3.1	Purpose
	3.2	Data Collection Plan
	3.3	Analysis Plan
	3.4	Cost Estimante

 

The process analysis will provide a comprehensive picture of the development, planning, start-up, and on-going operations of AFIA programs. It will help HHS staff understand how the programs work and the factors influencing effective operations. In describing how clients interact with program staff and receive program services, the process analysis also will help interpret the findings of the impact analysis (to be described in Chapters 4 and 5).

3.1 Purpose

At its core, process analysis examines how policies are implemented. The primary intent is to understand how AFIA program sites[9] are structured, designed, and operated and what factors influence these aspects of the site. The secondary intent is to shed light on the effects of IDA program structure, design, and operations on program results and outcomes. Whereas the impact analysis will try to assess more precisely whether IDA participants are better off, the process evaluation will help shed light on what exactly is provided by different AFIA sites and why certain outcomes are ultimately observed.

There are four basic objectives of the process analysis for the evaluation of AFIA:

• to describe the goals of the AFIA legislation and the program features it requires;
• to document and assess the implementation of the AFIA by grantees;
• to compare and contrast the experiences of grantees in establishing IDA policies and operating program sites; and
• to provide a programmatic context for findings from the impact analysis.

More specifically, the process analysis, speaking to several of the "factors to evaluate" indicated in the Act, will provide insight regarding:

• the effects of incentives and organizational or institutional support on savings behavior—It will do so particularly at the experimental site, where process-related findings will provide the programmatic context for interpreting the impact findings.
• the effects of IDAs on savings rates, homeownership, vehicle ownership, level of post-secondary education attained, and self-employment, and how such effects vary among different populations or communities—again, the findings of the process analysis will be particularly insightful at the experimental site.
• the potential financial returns to the Federal Government and to other public and private sector investors in IDAs over a 5-year and 10-year period of time; and
• the lessons learned from the demonstration projects, and particularly whether a permanent program of IDAs should be established.

We view the process and impact analyses as closely complementary. The impact analysis will provide estimates of the effects of IDA incentives. Although the impact analysis can indicate whether IDAs affect participant savings and asset accumulation, it is of limited use to explain why and how those effects accrue. The process analysis will indicate the dynamics of program-client interactions and suggest the mechanics by which behavioral changes occur. It is therefore crucial that the process analysis be conducted at the experimental site, among others.

The value of the process analysis goes beyond what it tells us about the dynamics of change at any one site. It can also illustrate the variety of program models that evolve under AFIA. Existing research into IDA programs—for example, from Abt Associates' study of asset accumulation initiatives (sponsored by the USDA) and CSD's evaluation of the American Dream Demonstration—indicates that current IDA program models are quite diverse. For example, programs may vary significantly in the strictness with which staff monitor and enforce the requirements of program participation (e.g., minimum deposit amounts, frequency of deposits, attendance at counseling and training sessions). To the extent that programs vary on these and other important features, it is important to ask whether these differences appear to influence participant outcomes. Of course, without an experimental design in each site, we cannot definitively attribute causality to the program. What we observe in the process analysis, however, can serve to narrow and sharpen our focus on those aspects of the program that appear to offer the most plausible explanation of effects.

Second, a process analysis that traces the development of an IDA program over time, can provide valuable lessons for other programs. It may identify issues that were found problematic across all sites or only under certain conditions. For example, establishing relationships with financial institutions, or devising procedures for efficient verification of account use, may prove to be more difficult than sites anticipated. The lessons learned about how sites overcame these challenges (or the implications of not overcoming them) would be extremely useful to both current and future sites and may have policy implications, to the extent some policy elements appear to promote or impede success.

To meet these varied objectives, the process analysis needs to provide a comprehensive, detailed analysis of the development, start-up, and ongoing operations of AFIA-funded IDA programs. Further analyses will compare implementation activities to AFIA regulations and draw cross-site comparisons.

The process analysis will describe and analyze three key aspects of the AFIA, as detailed below: (1) the context of AFIA implementation, (2) the implementation itself, and (3) the perceived effects.

3.1.1 Context

Understanding the context within which the AFIA was implemented is important because it identifies major contextual factors that might influence the results and outcomes observed and it points out other issues external to the AFIA that might need to be addressed by policy makers or practitioners attempting to replicate the IDA policy.

To the extent possible, the process analysis should examine both the internal and external context of the AFIA. The external context includes factors such as the economic environment (e.g., labor market conditions) and community characteristics, which often have a large role in determining outcomes. At the very least, the evaluation should identify such factors and speculate on the relative size and direction of their influence.

The internal context of the AFIA merits particular exploration. This includes the AFIA legislation itself, how it is envisioned, and how it is delivered. Analysis will emphasize these issues at varying levels of policy and practice, including the Federal, AFIA grantee, AFIA sub-grantee, program office, and individual practitioner level.

3.1.2 Implementation

The process analysis will also explore how AFIA was implemented by its grantees, including the planning, development, and ongoing operations of local IDA sites. In terms of program planning, care will be taken to assess the organizational structure of the local IDA site, the partners involved, the resources secured, and how these influence the site. Elements of the process analysis concerning site development will focus on the start-up activities, how long startup required, and the challenges of establishing the program at the site. Questions concerning site operations will address ongoing activity levels, how activities are conducted, and the experience of participants in the IDA program. As such, it will capture a description of the intervention that IDA participants undergo.

In documenting program implementation, the process analysis will describe the intervention received by the treatment group in the impact evaluation and whether the intervention was implemented as planned. It will also provide the ability to contrast the policy and planning processes from the actual implementation of the IDA program. Often, one policy implemented in two different places can succeed or fail based on the different methods, issues, and individuals that shape a local site.

3.1.3 Perceived effects

Finally, the process evaluation will begin to measure the perceived effects of AFIA-supported programs on their participants, according to site-level IDA practitioners. "Perceived effects" are different than "outcomes" in that perceived effects are more preliminary and more subjective than formal outcomes. Perceived effects topics will include issues such as savings behavior, asset purchases, and social-psychological changes. In addition, this inquiry will seek to determine not only whether such effects are perceived, but also why practitioners believe they occur.

 top of page

---

3.2 Data Collection Plan

The process analysis has two components: (1) periodic site visits, and (2) a treatment group module in the follow-up survey. These components are discussed below.

3.2.1 Site visits / interviews

Although the review of a grantee's funding application will provide a descriptive, quantitative snap-shot of site operations, it will not provide any detailed or qualitative information. Such detail needs to be collected through in-person interviews with key staff during site visits. These visits and interviews will provide the bulk of information collected in the process analysis.

Individual questions in the draft interview guides for the site visits are constructed to provide direction to respondents, but not to restrict responses. Many questions have open-ended probes to encourage further discussion of the topic. Despite the highly-structured design of the instrument, the interview itself will be conducted in an informal and relaxed manner. Interviewers will be made sufficiently familiar with the interview protocol as to be comfortable addressing topics in an alternative order that the interviewee might prefer.

Two main groups will be targeted for site visit interviews: IDA program coordinators (or directors) and IDA program associates (or front-line staff). These groups are expected to provide relevant descriptions of the IDA program from different perspectives. In addition, certain topics will be covered only with one group or the other, depending on whether the topic is more policy or implementation oriented. These divisions are indicated in Exhibit 3-1. Separate interview guides for both groups are included in Appendix C.

Process analysis sites will be selected from the FY 1999 through FY 2002 cohorts of AFIA grantees as described in Chapter 1. Interviews will be held at selected site offices in two rounds. The initial round will focus on establishing a baseline understanding of the site and its activities; the subsequent round will seek to document changes that have occurred to baseline conditions in the interim period. However, in a few areas concerning longer term effects of IDA participation, the initial visit may yield little dependable information simply because the site may not have been in existence long enough to reach conclusions about long-range effects. The subsequent visit is more likely to yield more robust information.

In larger IDA sites, multiple coordinator and associate representatives will be interviewed as appropriate. By interviewing multiple representatives from each group, breadth of opinion and depth of detail will be maximized. Coordinator interviews are expected to take a maximum of two hours to conduct, with associate interviews likely to be somewhat shorter.

3.2.2 Treatment group module in follow-up surveys

As discussed in detail in Chapters 4 and 5, a follow-up survey will be conducted for IDA participants in the treatment group. This survey will include a module that addresses the specific experiences of individual participants in IDA programs. Specific questions in this module will include (but will not be limited to) those presented in Exhibit 3-2.

The survey information will help to describe what IDA participants receive in the course of their participation. (In addition, this information will be used in conjunction with impact-related information to determine which IDA-related services are associated with substantial impacts, as discussed in Chapter 4 and 5.)

 top of page

---

3.3 Analysis Plan

The data collected through application and other document reviews, site visit interviews, and follow-up surveys will ultimately be quite voluminous. These data will need to be compiled and analyzed systematically to extract findings and lessons most effectively.

Initially, information concerning individual IDA sites will be collected and condensed to form a clear picture of each individual site. This will result in a series of site-specific case studies containing both quantitative and qualitative information.

Thereafter, this site-specific information will be synthesized to determine cross-site patterns and trends. Matrices of like information from different IDA sites will be constructed. Typologies of major policy classifications, or groupings at a lesser policy-level, will be investigated. Observations will be made concerning the structure and resources of different sites and the levels of activities and results they generate.

 top of page

---

3.4 Cost Estimate

This section presents the costs associated with the process analysis as a component of the AFIA evaluation.

The estimated costs, as shown in Exhibit 3-3, are based on the following assumptions:

• The process analysis will include multi-round site visits to occur in two phases. The first phase will take place during 2001 and 2002; the second phase will occur during 2003 and 2004. The visits are scheduled for April-June of each year.
• During the first phase, two rounds of visits will be conducted to the experimental site and to five other selected grantees. During the second phase, another two visits will be made to the experimental site; two visits will also be made to a second set of five grantees.
• In total, eleven grantees—the experimental site and 10 other grantees—will thus be visited. Among the 10 other grantees, 5 will be selected from the FY 1999 and FY 2000 cohorts, and 5 will be selected from the FY 2001 and FY 2002 cohorts. (To date, only the FY 1999 cohort has been selected.)[10]
• Site selection will be made in consultation with HHS, through a series of project memoranda. For the first phase, these site selection memoranda are scheduled for December 2000 (draft) and January 2001(revised), as shown in Exhibit 1-1. Similarly, for the second phase, memoranda will be provided in December 2002 (draft) and January 2003 (revised). Sites will be selected on a purposive basis, with the aim of including grantees that span a range of program characteristics and operational environments. We assume that state-level grantees (Indiana and Pennsylvania) will not be selected for the process analysis.
• The site visits will be conducted using interview guides that have been developed and pretested. Separate guides will be used for program coordinators and program associates.
• The interview guides were pretested on April 27, 2000 with staff at the Allston-Brighton Community Development Corporation (Allston, Massachusetts). Based on this pretest, we have assumed an interview length of 75 minutes for program coordinators and 60 minutes for program associates.
• For those grantees who operate at multiple locations through subgrantees, visits will take place at up to five subgrantee locations. Consistent with the pattern observed in the FY 1999 cohort, in which there are more than 120 program locations associated with the 38 non-state grantees, we assume that an average of three separate locations will be visited for each selected grantee. Of the 38 non-state grantees in the FY 1999 cohort, 16 are multi-site programs.
• Each site visit will be conducted by a single interviewer, at the Associate (Class II) level. Two interviewers will conduct visits, with each interviewer visiting three sites at each round.
• For each selected grantee in each round, the data collection cost is based on an average time budget of 58 hours for the interviewer: 4 hours for scheduling, 16 hours for preparation (e.g., review of documents), 20 hours on-site for interviewing and travel (to the site and among subgrantee locations), and 16 hours for writing up notes and preparing a site visit report.
• Following the completion of all site visits at each round, the site visit reports will form the basis of a cross-site process analysis. Each Interim Report (September of 2001 through 2004) will include a process analysis chapter and appendix materials that provide descriptive information on each of the studied sites.
  

Exhibit 3-3 Process Analysis -
Estimated Costs Per Grantee Per Year

Item			Rate	Year 1		Year 2		Year 3		Year 4		Total
				Units	Cost	Units	Cost	Units	Cost	Units	Cost	Units	Cost
Staff Labor
		Class I - Senior		208	$9,539	184	$8,438	172	$7,888	228	$10,456	792	$36,321
		Class II - Associate		664	$18,837	616	$17,489	600	$17,027	700	$19,841	2850	$73,192
		Class III - Intermediate		0	$0	0	$0	0	$0	0	$0	0	$0
		Class IV - Junior		192	$3,047	160	$2,539	160	$2,539	360	$5,713	872	$13,839
		Class V - Clerical		132	$2,412	96	$1,754	72	$1,315	132	$2,412	432	$7,893
	Labor Inflation Adjustment		4%		$1,353		$2,466		$3,592		$6,526		$13,938
	Subtotal Staff Labor				$35,187		$32,686		$32,361		$44,948		$145,183
	Fringe and Overhead				$38,763		$36,007		$35,649		$49,514		$159,933
Total Staff Labor				1196	$73,950	1056	$68,693	1004	$68,011	1420	$94,462	4676	$305,116
Other Direct Costs
	Survey Direct Costs				$0		$0		$0		$0		$0
	Travel				$9,966		$9,966		$9,966		$9,966		$39,864
	Telephone and Computer				$13,619		$13,148		$12,973		$4,771		$44,511
	Duplicating and Delivery				$2,512		$2,372		$2,512		$0		$7,396
	Payments to Respondents				$0		$0		$0		$0		$0
	ODC Inflation Adjustment		3%		$783		$1,552		$2,360		$1,850		$6,545
Total Other Direct Costs					$26,879		$27,038		$27,811		$16,587		$98,316
G&A and Fee					$26,357		$25,024		$25,048		$29,028		$98,316
Total Estimated Costs					$127,186		$120,755		$120,870		$140,077		$508,889

 top of page

---

Notes

[9] "Grantees" are organizations that applied for and received AFIA funds. "Sub-grantees" are established or funded by grantees to oversee or operate specific IDA programs. Grantees or sub-grantees may deliver IDA services through single or multiple "offices." A "site" is the most dis-aggregated level at which a single IDA program is administered—that is, the lowest level at which the same IDA policies are implemented (i.e., the same program, eligibility, and participation rules). As such, a site will be defined as a sub-grantee in most cases. [Return to Text]

[10] If no experimental site is selected, each phase will consist of two visits to a set of six selected grantees. [Return to Text]