Assets for Independence Act Evaluation:
Design Phase Final Report
August 9, 2000

6.	In-depth Participant Interviews
	6.1	Purpose
	6.2	Data Collection Plan
	6.3	Data Analysis Plan
	6.4	Cost Estimate

 

This chapter describes the activities to be undertaken for the AFIA evaluation in regard to in-depth participant interviews. The chapter includes a discussion on the data collection plan, and the data analysis plan, as well as a section on the need for conducting a pretest on the instruments developed for the in-depth interviews.

6.1 Purpose

In-depth interviews were designed to respond to Section 414 of AFIA. Specifically, Section 414(c), item (3) states that the research organization shall:

"develop a qualitative assessment, derived from sources such as in-depth interviews, of how asset accumulation affects individuals and families."

In-depth interviews are a principal research tool for social scientists to learn how people respond to complicated and often understudied issues. This method utilizes guided, but open-ended interviews, that reflect on events and ways that respondents understand their world and how and why they do certain things. In these interviews, participants are "conversation partners," not respondents as they are in survey interviews (Rubin & Rubin, 1995).

The purpose of the IDA in-depth interviews is to learn about the circumstances that brought participants into an IDA program, their personal experiences with saving, how they view the successes and failures of the program, and the effects of IDAs. The interviews explore details of how participants manage their IDAs in the context of their everyday lives. In-depth interviews are aimed at understanding the following relationships from the respondents' point of view:

• the impact of earlier life experiences (e.g., family, education, neighborhood) on respondents' savings behavior;
• the impact of organizational and institutional support on savings behavior;
• the effects of matching savings on respondents' saving behavior; and
• the economic, psychological, and social effects of savings on respondents.
  

 top of page

---

6.2 Data Collection Plan

This section describes the data collection plan to be used to guide the data collection activities undertaken to conduct the in-depth interviews for this evaluation. Specifically, this section describes the site selection, sample size and selection, training, data collection protocols, and techniques.

6.2.1 Site selection

In an experimental design, in-depth interviews would likely occur at the experimental site. Because it is not assured that an experiment will occur, this design outlines a plan for interviews to be conducted across three different AFIA grantee sites, from among the grantees selected for the process study.

Conducting interviews at multiple sites will provide more information on effects of variation in program characteristics. Three sites will be chosen based on the following factors: (1) ease of entry and smooth program operations (e.g., record keeping is adequate, participants are accessible, good staff-participant relationships); (2) willingness of participants to engage in evaluation, (the sample size will include an additional three persons in each subgroup to allow for those refusing to participate); and (3) presence of key program components including:

• Economic education
• Number of program participants (approximately 75 participants)

Availability of economic education will provide an opportunity to examine the importance of this key feature of IDA programs.

6.2.2 Sample

The best design for a qualitative assessment would include interviews with participants in the IDA program and a comparison group. In an experiment, participant interviews would include those from both treatment and control conditions. Absent an experimental design, the comparison group might be comprised of clients who received services (other than the IDA) from the organization sponsoring the IDA program. However, given the uncertainty of conducting the experimental design as well as cost constraints, this design includes only those participating in the IDA program.

A total of 45 participants will be interviewed over two time periods for a total of 90 interviews. This sample size was deemed necessary in order to: 1) exhaust variations of responses within a group, and 2) do simple statistical test of difference. All participants should have been in the respective conditions for at least one year. (It may be necessary to consider participants who have been in the program for less than one year if evaluation implementation is delayed.) There will be three separate groups of respondents at each of the three sites. The selection will be based on the average monthly deposit, with three categories:

• median savings or above (5 per site x 3 sites, n=15)
• below median savings (5 per site x 3 sites, n=15).
• drop-outs or very low savers (5 per site x 3 sites, n=15).

Note: Conducting interviews with 20 participants from each category is a preferable design, but cost considerations suggest reducing numbers to 15 participants per category.

Once all the participants who meet the above categories are segmented by group, additional selection will occur based on variation in participant demographics and other potential issues identified by site program staff. The purpose of this additional selection process is to learn more about the influences of unique participant and program characteristics on IDA participation.

Each group will be over-sampled (but not necessarily all interviewed). Two additional people will be selected for each group at each site, as some in the original sample may refuse to participate in the interviews. However, no more than 45 participants will be interviewed.

Interviews will be conducted in years 2 and 4 of the demonstration. Beginning interviews in year 2 will provide time for participants to have had experience in the IDA program. A second wave of interviews is important to explore themes and issues that have arisen from the first wave of interviews and from other evaluation methods being employed.

6.2.3 In-depth interview staff and training

A qualitative researcher will be the in-depth interview coordinator (referred to as the "coordinator") and will conduct interviews with two additional in-depth interviewers. The research organization conducting the in-depth interviews will hire and train the interviewers.

In-depth interviews require highly skilled interviewers. The interviewer has responsibility for guiding wide-ranging and potentially unpredictable discussions with respondents. This lack of routine requires that the interviewer be able to engage the respondent, ask good follow-up questions, be a good listener, be able to interpret answers, be flexible, have a thorough understanding of the research questions, be sensitive and responsive to issues identified by respondents, and keep the interview on track. Interviewers will be selected on the basis of their ability to (1) communicate effectively, (2) understand the goals and objectives of the evaluation, (3) work with a team, and (4) complete all tasks related to the interview process. Interviewer training will include evaluation objectives, evaluation protocol, confidentiality and consent for interviewing and tape recording, respondent contact procedures, interviewing skills (question asking, clarification, probes, techniques for dealing with digression, feedback), record-keeping, and pilot interviewing.

6.2.4 Evaluation protocol

The coordinator will contact the IDA program to begin the process of respondent selection. A procedure will be established to select respondents for the in-depth interviews (see 6.2.2). The IDA program contact person will forward respondent contact information to the coordinator.

Interviews will be set up in blocks where a team of three interviewers (including the coordinator) will visit a site for seven days at a time. The first day will include an orientation with a program staff person about the site, its operations, and the social and economic context. The site visit will include a weekend to facilitate interviewing people who work every weekday. Interviewers will contact respondents by telephone to set up interview appointments. Interviewers will conduct at least one interview per day. Time permitting, a second interview may either be started or an interview that was started a previous day will be finished. It is estimated that each interview will require up to seven hours: 2 to 3 hours for the interview; 2 to 3 hours to review tape and write up a case profile; and up to one hour for travel. The seventh day is reserved for travel.

The goal for each of these visits will be 15 completed interviews in five days. Forty-five interviews should take a total of three weeks. However, if an interview must be rescheduled outside of the scheduled week, the interviewer will have to return to the site at another time to complete the interview.

6.2.5 Data collection techniques

Interviewing

The central data collection tool in this study will be in-depth semi-structured interviews (Merton et al. 1990, Rubin & Rubin 1995). In-depth interviews will explore competing hypotheses about savings including the importance of early experiences, income surplus, savings structures, and savings education. Interview topics and open-ended questions are carefully derived from the study questions. Questions will be constructed in such a way as to provide direction to respondents, but not to restrict responses. Each question will have several open-ended probes that may be used to encourage further discussion of the topic. Despite the explicit design of the instrument, the interview itself will be informal and relaxed. The interview will be memorized to facilitate the conversation flow. Respondents must feel that they are in control: free to talk about topics in the order they prefer, and comfortable bringing up other issues. Respondents will be given a choice of locations for the interview, although typically, they will be conducted in respondents' homes in one session, totaling two to three hours. The interviewer will also request permission from the respondent to tape record the interview and will provide assurances regarding the confidentiality of the information.

Because qualitative interviews are designed to build trust and collaboration, they tend to elicit forthright and clear responses about sensitive topics such as family influence, non-traditional income sources and savings mechanisms. An in-depth interviewer is an interested and sympathetic listener, and respondents typically respond by trying to help the interviewer understand life from their perspective. In-depth interviews also allow the interviewer to help the respondent clarify recollections. For example, apparent contradictions can be gently probed, permitting the respondent to clarify a narrative. Clarification is particularly important in this study of low income/low wealth families, whose savings attitudes and behaviors have not been studied.

Interviews will follow a chronological approach, emphasizing topics that shed light on savings and program experiences. (See Appendix F - Interview Guide.) Some closed-ended questions will be asked at the end of the interview in order to confirm very specific content, such as income and expenses. For the treatment group, additional information about income, expenses, and savings will be acquired from a printout of each respondent's MIS IDA report. This will provide the interviewer with a "snapshot" of the person's savings record. Obtaining this record will require a signed consent by the respondent and will be obtained through the IDA program contact person.

Specific topics include respondents' (and their families') economic well-being, education, financial management (and banking experiences), and savings attitudes and behaviors beginning in childhood and extending to the present. The second half of the interview delves into their experiences with the IDA program, including access, savings patterns, sources of savings, planned uses, personnel, structures, expectations, goals, evaluation, and perceived outcomes.

Respondents will be paid $35. In addition, interviewers will bring a token gift to each family (for the children if they have them). Respondents will be offered a summary of research results to be mailed after initial analysis of the interviews. Interviews will be undertaken in a manner consistent with an approved human subjects protocol.

Managing Data

Notes, tapes, and transcriptions will be coded and filed without any identifying information. All identifying and coded information will be stored separately in a locked cabinet. Identifying information will be destroyed upon completion of interviewing.

 top of page

---

6.3 Data Analysis Plan

Qualitative interviewing generates large amounts of data that must be systematically reduced during the analysis phase. This process of data reduction occurs after the interview, instead of before the data collection phase as in survey interviews. After a tape-recorded in-depth interview is transcribed, it is entered into a qualitative software program to facilitate organization of the data and development of concepts and theory. Atlas/Ti or NUDIST are two qualitative analysis software programs appropriate for the task. This method maintains the integrity of the original data and keeps it readily accessible (Lewis, 1998). Additionally, some qualitative data will be enumerated, entered into a statistical program (SPSS), and assessed for descriptive and analytic purposes.

6.3.1 Analysis procedures

Systematic and reproducible techniques will be utilized throughout analysis of the data in order to guard against bias and validate findings, including attention to descriptive, interpretive, theoretical, and evaluative validity, as well as generalizability (Maxwell 1992, Castro & Bronfman 1997).

1. Data collection and analysis procedures will be carefully documented, including development of a system that records creation, handling, and transformation of data.
2. No more than three people will be involved in analysis of the same data followed by inter-rater reliability checks (MacQueen et al. 1998), until codes and coding patterns are substantially similar. After that point, coding can be done separately, with occasional checks on coding reliability.
3. Analysis will begin as soon as the first set of interviews is completed. Alterations to the interview guide, additional training, or changes in procedures can be made at this point.
4. Data analysis will use a rigorous and reproducible four-stage qualitative coding process. First, a list of potential codes will be generated based on hypotheses, results from the surveys, and interview content. Second, interviews will be line-by-line coded, examining intently each sentence and phrase within the interviews in order to develop a list of "codes" or "themes." This process has been called open coding (Strauss & Corbin 1990) or initial coding (Charmaz 1988). Care will be taken to base these codes on actual data, rather than common usage or accepted definitions. In this way, the original code list will be transformed. Coding schemes will be compared among coders until a common list emerges. Third, based upon the results of this process, a sub-set of analytic categories will be defined. Each interview will then be reviewed a second time, systematically examining the prevalence and variation of these categories. Fourth, relationships between different coded categories will be examined (sometimes referred to as families) and inferences will be generated.
5. In addition to the coding process described above, a profile of each respondent's interview will be created. The content of the profiles will be organized according to the basic research questions and key demographic variables. By combining the profiles with close coding of the interviews, analysis can reveal disaggregated themes and patterns that occur across different interviews, while at the same time checking emerging codes and categories against the coherence of respondent's actual life stories and experiences. In this way, emerging patterns will help build theory, as the analysis compares and contrasts empirical data and developing explanations.
6. Researchers will conduct a systematic examination of variations and divergent cases that challenge interpretations.

6.3.2 Management plan and time line

Data collection, transcribing, and analysis are overlapping processes in qualitative studies, as researchers refine the interview protocol and procedures, coordinate activities across study sites, and build coding schemes. However, transcribing and analysis continue well after interviews are concluded. Data collection should take three weeks if all interviews are completed within the scheduled week set up for each site. Transcriptions and initial analysis designed to refine open coding schemes and develop reliable coding schemes will also take approximately three weeks. These are interspersed with weeks of interviewing, as shown in Exhibit 6-1. Analysis is a lengthy process, especially in the beginning when interview transcriptions are coded and the coding scheme is developed. Once interviewing is completed, open coding is anticipated to take approximately 5 ½ additional weeks (assuming 10 interviews coded per week with the first week being a review of the process). Advanced analysis and a preliminary report will take another five weeks.

 top of page

---

6.4 Cost Estimate

This section provides the estimated costs associated with in-depth participant interviewing as a component of the AFIA evaluation.

The cost estimates, as shown in Exhibit 6-2, are based on the following assumptions:

• Interviews will be conducted with the treatment group participants in the experimental site.[23] If there is no experimental site, in-depth interviews will take place at three sites.
• Forty-five interviews will be conducted in each of two rounds, scheduled for April-June of 2002 and 2004. A total of 90 interviews will thus be conducted.
• An Associate-level program coordinator and a Senior-level principal investigator will plan the interviewing. The planning activities will include selecting and training the interviewers, organizing and coordinating their work activities, developing the interview plan, and making arrangements with the experimental site.
• The interviewing visits will involve collecting data from participants in the general geographic location where the experimental grantee is running its IDA program. Interviews will last approximately 3 hours, with 4 hours of post-interview write-up by the interviewer. Each interviewee will receive $35 for participating.
• Interviewers will use Dragon7 Naturally Speaking transcriber software to transcribe the interviews as part of the post-interview write-up. The cost estimate includes the cost of three laptop computers with a copy of the Dragon software on each laptop, assuming that interviewers will share these computers.
• The program coordinator will have a conference call with each site prior to the site visit by the interviewers. The program coordinator will prepare and coordinate the data collection process with the staff at the organization. The visit is expected to last for six days.
• All site visits will involve expenses for airfare, lodging, ground transport, and meals
• Two Junior-level interviewers and the program coordinator will conduct the interviews. The three interviewers will be at each of the three sites for seven consecutive days. A total of forty-five interviews will be conducted. (Although not included in this cost estimate, additional visits may be needed in case of interview cancellations.)
• Analysis will involve coding and analyzing the interviews using the Atlas/Ti qualitative software. All 45 interviews will be coded and analyzed.
• The findings from the first-round interviews will be presented in the September 2002 Interim Report. The findings from both rounds of interviews will be presented in the September 2004 Interim Report. Each report is assumed to require 100 hours of Senior-level staff effort and 100 hours of Associate-level staff effort.

Exhibit 6-2 In-Depth Participant Interviewing -
Estimated Costs by Year

Item			Rate	Year 1		Year 2		Year 3		Year 4		Total
				Units	Cost	Units	Cost	Units	Cost	Units	Cost	Units	Cost
Staff Labor
		Class I - Senior		0	$0	162	$16,759	0	$0	162	$16,759	324	$33,518
		Class II - Associate		0	$0	443	$22,084	0	$0	443	$22,084	886	$44,167
		Class III - Intermediate		0	$0	0	$0	0	$0	0	$0	0	$0
		Class IV - Junior		0	$0	494	$11,367	0	$0	494	$11,367	998	$22,734
		Class V - Clerical		0	$0	0	$0	0	$0	0	$0	0	$0
	Labor Inflation Adjustment		4%		$0		$4,097		$0		$8,528		$12,626
	Subtotal Staff Labor				$0		$54,306		$0		$58,738		$113,044
	Fringe and Overhead				$0		$59,824		$0		$64,706		$124,530
Total Staff Labor				0	$0	1099	$114,130	0	$0	1099	$123,444	2198	$237,574
Other Direct Costs
	Survey Direct Costs				$0		$0		$0		$0		$0
	Travel				$0		$7,623		$0		$7,623		$15,246
	Telephone and Computer				$0		$14,553		$0		$4,413		$18,965
	Duplicating and Delivery				$0		$758		$0		$758		$1,516
	Payments to Respondents				$0		$1,575		$0		$1,575		$3,150
	ODC Inflation Adjustment		3%		$0		$1,493		$0		$1,803		$3,296
Total Other Direct Costs					$0		$26,001		$0		$36,496		$73,126
G&A and Fee					$0		$36,630		$0		$36,496		$73,126
Total Estimated Costs					$0		$176,762		$0		$176,111		$352,873

 top of page

---

Notes

[23] The Center for Social Development (CSD) staff have recommended that interviews be conducted for both treatment and control group members (if the experimental evaluation component is implemented), to optimally perform qualitative analyses. The CSD staff feel that having both quantitative and qualitative data in an experimental design would be highly desirable, for the following reason. Comparisons with multiple data sources between those receiving the intervention and those not receiving it would enable greater explanation of differences that may be caused by IDAs and also how those differences occur. In response to comments received from HHS staff on the Evaluation Design Plan (February 17, 2000), this strategy was not adopted here. [Return to Text]

[24] For [Return to Text]