Assets for Independence Act Evaluation:
Design Phase Final Report
August 9, 2000

4.	Experimental Impact Analysis
	4.1	Purpose
	4.2	Data Collection Plan
	4.3	Data Analysis Plan
	4.4	Cost Estimate

 

This chapter explores the strategy for estimating AFIA program impacts through an experimental impact evaluation. We discuss the proposed experimental design, the data collection and analysis plans, as well as the challenges of experimental research.

4.1 Purpose

This section presents our proposed approach to implementing the mandated experimental design component of the evaluation. Our approach seeks to satisfy two objectives: to create procedures that meet the needs of a rigorous experimental evaluation, but at the same time fit practically into ongoing AFIA program operations and minimize the burden on AFIA program staff.

4.1.1 Mandated experimental design

AFIA specifies that the research organization shall "for at least one site, use control groups to compare participants with nonparticipants." In the experimental site(s), individuals will be randomly assigned to either a treatment group, which is allowed to participate in the program, or a control group, which is not. In addressing the research questions through an experimental design, Congress has properly sought to establish the strongest empirical foundation for drawing policy implications from the demonstration.

Experimental impact analyses are used to estimate the effects of a program as measured against the outcomes that would have happened in its absence. Measures of this sort provide the best indication possible of the effectiveness of a program in achieving its desired outcomes. For policy makers, the experimental evaluation provides the best policy counterfactual: a control group whose experiences can be interpreted as representing what would have happened to the treatment group in the absence of the demonstration. Any observed differences between the treatment and control groups can be attributed to the program.

Properly implemented, an experimental design through random assignment assures that the control group does not differ from the treatment group in any systematic way other than the receipt of program services. Thus, any subsequent differences in outcomes between the two groups that exceed the bounds of statistical fluctuation can be confidently attributed to the intervention. Non-random comparison groups carry the risk that differences in outcomes are the result of pre-existing differences between the two groups, rather than the program itself.

An experimental impact analysis will strive to answer the key research questions posed by the evaluation by collecting data from the research sample over a period of time, initially at baseline (i.e., immediately prior to random assignment) and then at one or more prescribed follow-up interval(s). Experimental impact studies typically consist of four elements: baseline data collection; random assignment of program applicants to treatment and control groups; follow-up data collection; and impact estimation.

4.1.2 Research questions

In general, the experimental component of the evaluation will seek to quantify program impacts, or the influence of IDA programs on participating individuals. As a result, many of the research questions concern the difference between participants' pre-program baseline status and their status after participating in an IDA program.

Most fundamentally, AFIA programs-and IDA programs more generally-are intended to increase the savings rates and assets of program participants. The experimental research questions will address whether these effects occur, and whether they have longer-term implications for individual well-being. Three major categories of program effects have been identified from the "factors to evaluate" in the AFIA legislation. These categories include effects on savings and asset accumulation, on employment and income, and on the personal well-being of IDA program recipients.

 top of page

---

4.2 Data Collection Plan

In this section, we describe the approach to be used for the experimental impact analysis in determining sample, random assignment methodology, baseline and follow-up data collection procedures and instruments, and procedures for tracking members of the research sample.

4.2.1 Sample size determination

A key issue in designing the experimental data collection is the size of the research sample to be enrolled at the experimental site. The sample must be large enough to make it very likely that, if indeed the treatment causes an effect, one will detect that effect as statistically significant. The larger the sample, the greater the likelihood—or "power"—of detecting the treatment effect.

One's judgment about the necessary sample size depends importantly on the size of the effect that one expects the treatment to cause, plus the degree of likelihood that one seeks in detecting such an effect. The latter assumption, the level of statistical power, is normally set at 80 percent, so that the specified sample provides an 8 out of 10 chance of detecting the effect as statistically significant. The higher the specified level of power, the larger the required sample size. The former assumption, the size of the effect in question, is of course unknown, which is why the research is undertaken. The larger the assumed treatment effect, the smaller the required sample size.

Given the inherent uncertainty regarding the treatment effect, a standard approach to determining sample size is to consider the "minimum detectable effect" associated with alternative sample sizes. Under this approach, one specifies the required level of statistical power (along with other assumptions[11]), and then proceeds to answer the question "how large must the treatment effect be?" to enable a sample of given size to meet one's statistical requirements.

Exhibit 4-1 shows the minimum detectable effect for samples of 200, 250, 300, and 500 per group. (These represent the size of the treatment and control groups each, assuming two equal-sized groups.) For these sample sizes, we have computed the minimum detectable effect for an unspecified outcome measured as a proportion. (This could be, for instance, the proportion of individuals who achieve a threshold level of annual savings, or the proportion of individuals who purchase an asset of particular type during a specified time interval.) Such computations require that one assume the control-group value for this outcome. We have used alternative control-group values ranging from 0.10 to 0.40. The minimum detectable effects represent differences between the treatment group value and the assumed control-group value. To illustrate, a sample size of 300 yields a minimum detectable effect of 0.100, under an assumed control-group value of 0.300. This implies that the treatment-group value would need to be 0.400 (or more) for the sample of 300 per group to provide an 80 percent chance of detecting the treatment effect as significant. The larger the sample size, the smaller the minimum detectable effect.

Given that treatment effects of 0.100 or more are quite large for policy interventions of this type and for outcome measures of the kind that this impact analysis will address, it was prudent to adopt a sample size of 500 per group for the experimental site in the American Dream Demonstration. As shown in the exhibit, a sample size of 500 yields a minimum detectable effect of less than 0.100 at all assumed control-group values. It is certainly desirable to have samples of such size, if feasible.

At the other extreme, a sample size of 200 per group yields a minimum detectable effect of less than 0.100 only if the assumed control-group value is also in the range of 0.100—i.e., only if the treatment leads to a near doubling of the outcome measure. Samples as small as this pose a risk of failing to detect effects of a magnitude even larger than one might reasonably expect to occur. Sample sizes of 250 or 300 per group provide somewhat greater advantage. At the control-group value of 0.100, the minimum detectable effects for these samples are 0.081 and 0.073, respectively.

For purposes of this evaluation design, we have adopted a per-group sample of 250 as the minimum acceptable size. Although such a judgment is ultimately arbitrary, the information in Exhibit 4-1 and reasonable expectations about effect sizes for meaningfully defined outcomes make it difficult to defend an experimental data collection effort that provides less statistical power than shown for per-group samples of 250.

Exhibit 4-1 Minimum detectable effects under alternative sample sizes

Control-group value	Sample size per group
	200	250	300	500
	Minimum detectable effect (treatment-control difference)
0.100	0.093	0.081	0.073	0.054
0.200	0.113	0.100	0.091	0.069
0.300	0.124	0.110	0.100	0.076
0.400	0.129	0.115	0.104	0.080

Explanatory note: Assumes 80 power and 10 percent significance for a two-sided test. See text.

It is important to note that the sample size of 250 per group (500 in total) applies to the number of individuals for which one obtains complete information over a multiyear follow-up period. This requires that the number of individuals initially recruited—for baseline data collection and then random assignment into the research sample—be even higher. We assume here that baseline interviews can be completed with 95 percent of the eligible program applicants recruited by the experimental site.[12] We also assume that multiyear follow-up data can be collected for 75 percent of those enrolled in the research sample.[13] These two assumptions imply that the number of initial program recruits must be 1.40 times as large as the final sample of 500—where 1.40 equals 1/(.95)(.75). The number of initial program recruits must thus be 700.

4.2.2 Random assignment methodology

The foundation of this and any other experimental design is the process by which subjects are assigned at random to a treatment and control group. The integrity of the research—and thus the validity of the corresponding empirical estimates—requires extreme care in implementing and monitoring the random assignment process to ensure that all subjects face the same random probabilities of assignment. As a result, we have identified a set of guiding principles for implementing the experimental design. Implementation of the research design in accordance with these principles will require careful advance planning and continuous coordination with HHS and the AFIA program staff. The principles guiding random assignment include the following:

• random assignment must be placed at a point in the program's intake process where it will reliably measure impacts for groups of interest;
• the random assignment process must be carefully controlled to provide no opportunity for "gaming," i.e., the steering of particular individuals to one group or another; and
• the random assignment algorithm must be able to maintain a reasonably even split between treatment and control assignments, both to ensure an even flow of participants into the program, and to avoid "strings" of consecutive control group assignments that may lead to complaints from staff at the evaluation site.

To meet these requirements, we suggest a process similar to that used for the ongoing evaluation of the American Dream Demonstration in Tulsa, Oklahoma. The key requirements of random assignment—and the associated prior step of recruitment of research sample members through intake interviews conducted by the AFIA program staff—are as follows:

• The evaluation site will conduct program outreach to recruit approximately 700 applicants. (See previous section on sample size determination.) An intake process conducted by the AFIA program staff will determine whether each applicant is eligible and willing to participate.
• Applicants will be referred for the baseline interview only if: (a) they meet the local AFIA program's income limits and other eligibility requirements; and (b) they indicate that they would indeed participate in the program if offered the opportunity, although the program staff must indicate to the applicant that only a randomly selected subset of eligible applicants can participate. Criterion (a) ensures that no cost is incurred in interviewing ineligible applicants. Criterion (b) eliminates any selectivity bias in the estimation of program impacts and maximizes the extent to which the local AFIA program is able to fill the funded slots available for the experimental participants.
• Applicants will be informed at their intake interview that survey cooperation (for both the baseline and follow-up surveys) is a requirement of program participation. To ensure a high response rate for the baseline and follow-up interviews, participants must also sign an agreement stating their consent to this process.
• Applicants who are determined to be program eligible and who agree to cooperate with the survey component of the program will be referred to the evaluation contractor for administration of the baseline interview.
• Applicants will be randomly assigned only after completing their baseline interview. This is essential in eliminating any potential respondent bias or interviewer bias in administering the survey instrument.
• The software that executes the random assignment will use a blocked random assignment protocol that ensures that a 1:1 ratio will be maintained. (A 1:1 ratio will yield equal numbers of treatment and control cases). Unlike simple random assignment, the software needs to be designed to ensure that the assignments will remain balanced throughout the random assignment period. This is important to establish the credibility and fairness of the random assignment process in the minds of the AFIA program staff and applicants. For example, it is essential to avoid a situation in which, for a batch of 10 cases, only 2 are assigned to the treatment group. In blocked random assignment, the two outcomes (treatment and control) are randomly ordered within small "blocks" of slots, each of which has exactly the desired ratio of treatment to control slots. Because the ratio in each block is equal to the desired random assignment ratio, the overall assignment ratio cannot depart substantially from the target. If the evaluation site has multiple locations, the blocked random assignment approach can be used to ensure that the ratio is maintained for small blocks of cases by site.

Eligible program applicants will be referred to the evaluation contractor, whose telephone interviewers will attempt to contact and interview them. After the individual has completed the baseline interview, the random assignment process would work as described below:

• On a weekly basis, the list of cases completing the baseline interview becomes subject to random assignment.
• This case list will then be entered into the random assignment software, which will be pc-based and operated by a trained staff member from the evaluation contractor.
• The random assignment software will prevent multiple assignment of the same individual, by checking the incoming list against a compiled list of all previously assigned cases by social security number.
• The software then executes the random assignment, using a blocked random assignment protocol that ensures that a 1:1 ratio will be maintained.
• A weekly report is then provided to the site on the outcome of the random assignment, listing the cases by their assigned demonstration status.

Our proposed approach to random assignment, coupled with our recommended strategy for sample recruitment by the site program staff, thus has the following features:

• It is sensitive to the need for "face validity" in the minds of the local AFIA program staff.
• It shows commitment to the ethical treatment of respondents by offering all incoming cases the same opportunity to participate in the AFIA program as a member of the treatment group and by informing them at the outset that not all applicants can participate.
• It protects the interviewer as well as the program staff from any appearance of having influenced the assignment process.

4.2.3 Baseline data collection procedures

Another critical component of the experimental evaluation will be the procedures for collecting baseline data. The effort must be well-planned to not only satisfy baseline data needs but also to collect information that facilitates future data collection activities. The procedures must also be carefully executed, following a standard set of steps to ensure that data will be collected consistently and in a timely manner for both treatment and control group members over the evaluation period. The following procedures will be followed in collecting the baseline information:

• baseline data will be collected prior to random assignment.
• contact information from eligible program applicants must be transmitted to the evaluation contractor in a regular and timely fashion.
• notification of completion of the baseline interview and status of the random assignment must be transmitted to the evaluation site so that they can notify applicants of their status in the program in a regular and timely fashion.
• data tracking the activities of all sample members (both treatment and control group members) must be comprehensive and accurate. This process is described in more detail in Section 4.2.5.

Baseline data must be collected prior to random assignment. It is crucial to obtain accurate and consistent baseline information on both treatment and controls before the point of random assignment. This is to ensure that the variables and their reporting are not influenced by either random assignment outcome or the intervention. It is also critical to obtain written informed consent of the individuals who are determined eligible for the program prior to random assignment to ensure that they understand and agree to the implications of random assignment and the requirements of the program. To accomplish this, the consent information should be collected during the intake process, after the individual is determined to be eligible for the program.

To satisfy the baseline data needs, we envision the need for three types of data collection forms:

• a participant enrollment agreement (informed consent);
• a contact information form; and,
• a baseline survey instrument.

We assume that all requisite information for random assignment and the baseline interview are provided by the sites on a one-page form. Those data items include:

• applicant name, SSN, date of birth
• address and telephone number
• contact information for the applicant
• contact information for friends and family members who will be likely to know how to reach the applicant over the next two years.

See Exhibit 4-2 for an example of a form that could be used for this purpose. The form will be reviewed by site staff prior to referring it to the evaluation contractor. Once the case is referred, telephone interviewers will attempt to contact and interview the referrals. Once a week the newly interviewed cases will be randomly assigned and the results faxed or sent electronically to the site. Weekly reports will be provided to the site, listing the cases still pending. Exhibit 4-3 illustrates how this process will work.

 

 

I have read and understood the description of the Assets for Independence Act study. I agree to allow the researchers conducting this study to obtain information from my records at government agencies, including unemployment insurance, social security earnings records, cash assistance, food stamps, and military records. I understand that this information will be used only for the purposes of the study, except if required by law, and will be kept strictly confidential. ________________________________________________________ Signature of Applicant ____________________ Date

This form has been reviewed by:_________________________ Time and date to be interviewed:_________________ Telephone # for interview: 01 Same as above or Area Code (_________)_______________________________ _________________________________________________________________________________________________ Signature of Applicant & Date

 

Exhibit 4-3

Model for Random Assignment

The proposed baseline survey and follow-up survey to be used in the experimental impact analysis are the surveys that Abt Associates is currently administering to the research sample from the American Dream Demonstration (ADD) in Tulsa, Oklahoma. Appendix D contains a copy of the ADD follow-up survey. A focused set of measures flows directly from the research questions posed for the study. These key measures will be collected using baseline and follow-up surveys:

• Effects on savings and asset accumulation
  • Savings level at baseline and followup
  • Self-investment between baseline and follow-up
  • Matching funds received (treatment group only)
  • Funds from any other sources
  • Net savings increase: savings at follow-up, minus savings at baseline, plus self-investment between baseline and follow-up
  • Home ownership and improvement/maintenance
  • Business startup
  • Other assets and their value (e.g., vehicles, property, other accounts)
  • Own educational activity, including employment training
  • Debts, by type
• Effects on employment and income
  • Employment status
  • Earned income
  • Hours worked per week and hourly wage
  • Other private (own) income
  • Public assistance use (cash assistance, food stamps, Medicaid)
  • Other income sources
• Effects on personal well-being
  • Outlook (feelings of self-efficacy, regard for the future, expectations for children)
  • Financial well-being / avoidance of hardship
  • Activities to improve status (e.g., looked at home purchase or job change opportunities)
  • Financial planning activities (e.g., budgeting, goal-setting, encouraging children to save)

4.2.4 Follow-up data collection

For this evaluation we propose conducting two annual follow-up surveys. The first follow-up survey will be conducted approximately one year from the completion of the baseline survey. The second follow-up survey will be conducted approximately two years from the completion of the baseline survey. The surveys for both follow-ups will be very similar to the baseline survey, with the addition of a treatment module.

4.2.5 Tracking the research sample

One of the most critical aspects of any longitudinal research program is sample retention, maintaining up-to-date locating information for all treatment and control group members. A strong tracking strategy must be developed to ensure that all the sample members can be reached in the future for follow-up surveys.

Passive tracking methods (which involve no direct contact with the respondent) include collection of contact information from sources such as postal address updates, directory assistance, reverse directories, credit bureau data, and public agency administrative data. Passive tracking resources are comparatively inexpensive and generally available, although some sources require special arrangements for access. Active tracking involves direct contact with respondents, either by contact in-person, by telephone or mail. Periodically, active contact with sample members confirms or renews their address and contact information.

Because we will be conducting two annual follow-up surveys, it is important to consider how, and how often the sample will be tracked. We recommend at a minimum, an annual verification mailing to respondents. This mailing should occur at the approximate midpoint between surveys (or approximately six months after each interview has taken place). In addition, after the first follow-up interview, we will verify the contact information obtained for the respondent, as well as collect any new contact information on the respondent, or on family members or friends who will know how to reach the respondent in the future.

 top of page

---

4.3 Data Analysis Plan

The impact analysis will examine the following effects of participation in an AFIA-funded IDA program:

• effects on savings and asset accumulation
  • savings account balances
  • home purchases
  • vehicle purchases
  • business startup or expansion
  • educational advancement
  • other assets held
  • debts held
• effects on employment and income
  • employment status
  • earned income
  • hours worked per week and hourly wage
  • other private income
  • publicly funded assistance (cash assistance, food stamps, Medicaid)
  • total income
• effects on personal well-being
  • personal outlook
  • financial well-being, hardship avoidance
  • financial planning activities
  • community and civic involvement

Random assignment of AFIA-eligible persons to a control group will provide an appropriate counterfactual. The data collected from control-group members can be interpreted as representing what would have happened to the treatment group in the absence of their participation in an AFIA-funded program. Any observed differences between treatment and control group members can therefore be attributed with confidence to the IDA program.[14]

The statistical power provided by "unadjusted" comparisons of treatment-control differences is potentially increased through multivariate regression techniques, which can reduce the amount of unexplained variation in outcomes. For example, by using a set of baseline explanatory variables that can explain 25 percent of the variance in an outcome—in statistical terms, would have an R-squared of 25 percent, absent the treatment—one achieves the same effect on the precision of the impact estimate as increasing the sample size by a third. Hence, even though multivariate analysis is not necessary to obtain unbiased impact estimate, it enables one to increase statistical power—i.e., the ability to detect a treatment effect. One should not expect a very large degree of explanatory power from the baseline descriptors, however, for several reasons. First, the target population in the experimental site may be fairly homogeneous. Second, the determinants of savings behavior among the poor are not well understood. Finally, the baseline questionnaire of necessity can collect only a limited amount of information on individuals' attitudes and past behavior.

Our general approach is to estimate models of the form:

y_i = b₀ + b₁ T_i + b₂ X_i + u_i ,

where

y_i is the outcome measure for individual i,
T_i is a treatment group indicator (1=treatment, 0=control),
X_i is a vector of baseline characteristics, such as the individual's age, race, education, household composition, and employment status, and
u_i is the regression residual.

This linear model, although unbiased, is not statistically efficient for outcomes that are dichotomous or highly skewed. For dichotomous variables, such as an indicator that a person has bought a home, we will use logistic regression:

log (p_i (1-p_i )) = c₀ + c₁ T_i + c₂ X_i + v_i ,

where

p_i is a probability between 0 and 1, and
v_i is the regression residual.

The coefficient on the treatment indicator, c, cannot be interpreted directly. To obtain the impact of the treatment on the probability of the event, we multiply the logistic coefficient by p x (1-p), where p is the control group mean of the outcome (e.g., home ownership). The resulting product tells us the impact of the treatment in percentage points on the likelihood of homeownership for a "typical" control group member.

For skewed outcomes such as earnings and total savings, our preference is to estimate a pair of logistic regressions. First, we determine the impact of the treatment on whether a respondent had any earnings or savings. (Depending on the experimental site chosen, we may expect a sizable proportion of participants not to have any earnings or savings at follow-up.) Then, we determine the impact of the treatment on whether the respondent had sizable earnings or savings—where "sizable" could be defined as exceeding either the control group median of non-zero values or some other functionally meaningful level. Both of these impact estimates are based on the full sample—including zeros, small increases, and sizable increases.

This paired logistic regression approach is very robust with regard to outliers, which could be a serious problem in a study of this sort—if, for instance, a handful of treatment or control group members did extraordinarily well for a reason unrelated to the intervention. Savings and earnings are likely to have so much variability over the sample population that it would be very difficult to distinguish changes in the mean from random noise. The proportion of individuals who save or earn more than a given amount, however, can be measured much more precisely.

Our proposed approach provides answers to the two most important questions about such outcomes:

• Did the intervention lead to more individuals achieving a nonzero outcome? and
• Did the intervention lead to more individuals achieving a meaningfully positive outcome—i.e., above some specified threshold?

It does not attempt to answer the sticky and confusing (and to our mind, subsidiary) question of the exact shape of the distribution of earnings or savings, beyond these summary statistics.

In a sample with equal numbers of treatment and control group members, the regression-adjusted treatment group mean is the equal to the overall sample mean plus one-half the estimated treatment effect, while the regression-adjusted control group mean is the sample mean minus one-half the estimated impact.

 top of page

---

4.4 Cost Estimate

This section provides the estimated costs associated with conducting the experimental impact analysis as a component of the AFIA evaluation.

The cost estimates, as shown in Exhibit 4-4, are based on the following assumptions:

• Over a year-long period (April 2001-March 2002), the experimental site will recruit 700 eligible applicants and will refer them to the evaluation contractor, who will administer a 40-minute baseline interview by telephone and randomly assign each respondent to either the treatment group or the control group. Assuming a 95 percent response rate at the baseline interview and a 1-to-1 random assignment ratio (treatment-to-control), the enrolled sample will consist of 333 treatment group members and 333 control group members.
• This sample will complete a first- and second-round follow-up interview (at 12 months and 24 months after random assignment, respectively), using computer-assisted telephone/personal interviewing (CATI/CAPI). The assumed interview length is 50 minutes for treatment cases and 40 minutes for control cases. Respondents will receive $35 for their participation.
• In the year preceding each follow-up interview, each sample member will receive two tracking letters, to update the contact information. Those who complete and return the second of these tracking letters at each round (mailed two months prior to the expected interview month) will receive $10.
• As shown in Exhibit 4-5, the expected response rate is 82 percent for the first-round follow-up interviews and 75 percent for the second-round follow-up interviews, both computed as a percentage of the enrolled baseline sample of 666.
• In each year of the data collection, we assume two trips to the experimental site. In the first year, this is primarily to monitor sample recruitment, baseline interviewing, and random assignment and to arrange for the transmission of data between the grantee and the evaluation contractor. In the subsequent years, the trips are to confirm that the site is properly maintaining the operational distinction between the treatment and control groups.
• The second Interim Report (September 2002) will present an analysis of the baseline survey data, including a comparison of the characteristics of the treatment and control groups. The third Interim Report (September 2003) will present the findings of an econometric estimation of experimental impacts, based on data from the first-round follow-up interviews. The fourth Interim Report (September 2004) will present the complete impact estimates, based on data from the first- and second-round follow-up interviews.

Exhibit 4-4 Experimental Impact Analysis -
Estimated Costs by Year

Item			Rate	Year 1		Year 2		Year 3		Year 4		Total
				Units	Cost	Units	Cost	Units	Cost	Units	Cost	Units	Cost
Staff Labor
		Class I - Senior		440	$20,178	260	$11,924	316	$14,492	196	$8,989	1212	$55,582
		Class II - Associate		1104	$32,971	1113	$34,741	1034	$31,814	829	$24,219	4080	$123,745
		Class III - Intermediate		0	$0	0	$0	0	$0	0	$0	0	$0
		Class IV - Junior		352	$5,586	240	$3,809	240	$3,809	240	$3,809	1072	$17,013
		Class V - Clerical		128	$2,339	120	$2,192	120	$2,192	0	$0	368	$6,723
	Labor Inflation Adjustment		4%		$2,443		$4,298		$6,531		$6,288		$19,559
	Subtotal Staff Labor				$63,517		$56,964		$58,839		$43,304		$222,623
	Fringe and Overhead				$69,970		$62,751		$64,817		$47,704		$245,242
Total Staff Labor				2024	$133,487	1733	$119,715	1710	$123,655	1265	$91,008	6732	$467,865
Other Direct Costs
	Survey Direct Costs				$43,161		$146,054		$188,663		$85,770		$463,647
	Travel				$5,964		$2,982		$2,982		$2,982		$14,910
	Telephone and Computer				$14,001		$11,823		$11,746		$10,250		$47,820
	Duplicating and Delivery				$1,870		$1,465		$1,465		$1,535		$6,335
	Payments to Respondents				$11,655		$21,210		$18,358		$8,803		$60,025
	ODC Inflation Adjustment		3%		$655		$991		$1,501		$1,853		$5,001
Total Other Direct Costs					$77,306		$184,525		$224,714		$111,193		$597,737
G&A and Fee					$55,101		$79,528		$91,064		$52,855		$278,548
Total Estimated Costs					$265,893		$383,768		$439,433		$255,056		$1,344,151

na = not applicable
[1] As a percentage of the 700 recruited applicants.
[2] As a percentage of the enrolled sample (666 in total, 333 per group)

 top of page

---

Notes

[11] Other statistical assumptions must also be made to assess different sample sizes. These assumptions pertain to the possibility that the "null hypothesis" is true—i.e., that the treatment has no effect. One such assumption is the specified "significance level" of one's test of the null hypothesis—that is, the likelihood that one's test will lead to mistakenly rejecting the null hypothesis when it is true. Here, we have assumed the significance level to be 10 percent for a two-sided test. This is a conventional assumption, implying a 90 percent chance of a correct judgment—i.e., not rejecting the null hypothesis when it is true. The two-sided ("nondirectional") nature of the test merely indicates that one allows for the possibility that the treatment effect could be either positive or negative. [Return to Text]

[12] This assumption is consistent with the 96 percent completion rate for baseline (Wave One) interviews achieved by Abt Associates at the Tulsa experimental site for the American Dream Demonstration (ADD). See Donna DeMarco and Gregory Mills, Evaluation of the American Dream Demonstration: Semi-Annual Progress Report, July-December 1999, Abt Associates, Cambridge, Mass., February 9, 2000, p. 7. [Return to Text]

[13] This assumption is drawn from the projections now used by Abt Associates for the data collection at the Tulsa ADD site. Ibid., p. 8. [Return to Text]

[14] Although the treatment and control groups are statistically equivalent at the outset, differential attrition could render the groups less comparable over time. To the extent that overall response rates are high, however, the potential for bias is minimized. [Return to Text]