Assets for Independence Act Evaluation:
Design Phase, Concept Paper
February 16, 2000
8. |
Overall Implications For Evaluation Design |
||
| Timing Of Evaluation Activities | |||
| Balancing The Statutory Requirements With Evaluation Resources | |||
This concluding
section highlights a series of issues relating to the scope and timing
of the evaluation activities that would commence following the design
phase, which ends in August 2000. These aspects of the evaluation
strategy will require further discussion with the Task Order Officer
and other HHS staff.
Timing Of Evaluation Activities
As noted in Section 1, a literal interpretation of Section 414 of the Act would call for the final evaluation report to be completed in 2009 (i.e., "not later than 12 months after the conclusion of all demonstration projects conducted under this title"). Presumably, however, the Department will want to present some intermediate findings available at the time that Congress is considering the reauthorization of the Act, scheduled for 2003.
Extensive findings will indeed be available by 2003 from the process analysis, in-depth participant interviews, and the program and participant tracking and monitoring. These findings will reflect project experience from the first, second, and third grantee cohorts. By 2003, the findings from the impact analysis would at best reflect only 12 to 24 months of followup data on an enrolled experimental sample. These are the findings that would then form the basis for the benefit-cost analysis.
This issue of timing is important with respect to upcoming decisions regarding the scope of the impact analysis, and thus the extent of findings ultimately available for the benefit-cost analysis. Simply stated, the question is whether 2003 should be regarded as the date for a final evaluation report, or whether the evaluation would extend further. As discussed below, the answer to this question has implications for selecting one or more experimental sites and for considering possible nonexperimental impact analysis.
Selection of one or more experimental sites. If experimental impact findings are to be available by 2003, the experimental site(s) must be chosen from the first or second cohort of grantees-i.e., those announced in September 1999 and those to be announced during the year 2000. The enrollment of a randomly assigned research sample would take place during 2000-2001, and data would then be collected for followup periods of 12 to 24 months. This schedule would require that the experimental site(s) be selected in mid-2000. If the timeframe for impact findings is extended beyond 2003, it allows the selection of the experimental site(s) to incorporate more information about the variation in program models that states and localities are implementing under the Act.
Consideration of nonexperimental impact analysis. If 2003
is regarded as the date for a final evaluation report, it raises questions
as to the feasibility of undertaking any nonexperimental impact analysis
of the type discussed in Section 6. The time lags in availability
of SIPP data from the year 2000 panel (the first panel to be observed
within the AFIA period) suggest that one could make use of information
on assets and liabilities from the Wave 3 topical module (covering
the first followup year for panel members), but not the Wave 6 topical
module (covering the second followup year).
Balancing The Statutory Requirements With Evaluation Resources
One of the products of this task order (under Subtask 3.4, due in late April 2000) will be "an overall cost estimate for implementing and conducting all aspects of the evaluation project." At this early stage, we can already foresee that the estimated costs to fulfill the statutory requirements will exceed the funds available through the 2 percent evaluation setaside, even if one assumes that the statutory mandate is minimally met and even if one assumes that annual appropriations are as high as $20 million (twice the level of annual appropriations thus far). This suggests that possible enhancements to the evaluation design, such as more than one experimental site or nonexperimental impact analysis, may not warrant serious consideration unless there is some expectation of additional evaluation funding.
