Table of Contents | Previous | Next |
Appendix B
METHODOLOGY FOR CHAPTER ELEVEN SYNTHESIS
This appendix provides additional detail regarding the synthesis of the random assignment and econometric studies presented in Chapter 11. In that chapter, we synthesize the results of the studies we reviewed in Chapters 4 to 10 according to the major policy or groups of policies they evaluate, as well as for reform as a bundle. The assignment of random assignment studies and econometric studies to the 11 policy rows shown in Table 11.1 is recorded in Table B.1.
The summary entries in each cell of Table 11.1 convey both the qualitative effect of the particular policy on the particular outcome and the depth of the knowledge base on which the entry is based. Both were determined by the signs and the significance of the underlying impact estimates reviewed in Chapters 4 to 10 and the number and quality of the studies that produced them. The procedure involved first assigning a direction (or sign) and significance indicator to each study, both experimental and econometric, that addresses each policy-outcome pair; tabulating the estimates by sign, significance, and quality of the underlying study; and then using that tabulation to assign an overall direction and knowledge-base indicator to each cell.
With a very small number of exceptions, which we discuss below, we first assigned a single direction and significance level to each study that addressed a particular policy-outcome pair. This was to avoid double-counting the studies for which the tables present multiple estimates, which would arbitrarily give them more weight than the others. If all the estimates for a study agreed in direction, then that direction was assigned to that study. Most of the studies fell into this category. If any of the estimates were significant, then the study was coded as having a significant effect. Among studies with mixed estimates, three approaches were taken. If the only significant estimates all had the same sign, then the study was assigned that direction and coded as significant. If none of the estimates were significant, then the direction for that study was assigned based on the majority of the estimates and coded as insignificant. If there was no majority, then no direction was assigned. If the study had significant estimates that were both positive and negative, then the study contributed two significant entries, one in each direction, to the second-stage tabulation.
Once each study had been scored, the results were tabulated for all studies within a policy-outcome cell using the assignments shown in Table B.1. Studies were categorized according to direction (positive or negative), significance (significant or not), and quality (high or moderate). The summary direction for the cell and the indicator of the depth of the knowledge base for the cell were based on this tabulation.
For the most part, the direction for the cell was easy to assess, since most studies within a cell tended to have the same direction. However, there were some cases where the underlying studies yielded results that varied enough that it was difficult to reasonably classify the direction of the impact. For example, in the case of Medicaid use (column (E)) for policy row (5) (mandatory work-related activities and strong financial work incentives), MFIP leads to a statistically significant positive impact, while FIP finds a statistically significant negative impact. In this case and others like it, the cells in Table 11.1 are labeled as "mixed." In two cases, we label a cell as "no change." These occur in cells F3 and G3 for the impact of mandatory work-related activities on marriage and fertility, where a dozen or so high-quality studies all find a statistically insignificant impact, thus providing a relatively high level of confidence that the null hypothesis of no impact is true.
Assessing the depth of the knowledge base within each cell is more subjective and necessarily involves some judgment. The extremes are reasonably clear. Some cells are populated by a substantial number of high-quality studies. When nearly all their results point in the same direction, and nearly all are significant, that clearly provides a deep knowledge base about the effect of that policy on those outcomes. In other cases, there is only a single high-quality study that addresses a particular policy-outcome combination and provides a statistically significant impact. That cell has a more shallow knowledge base. Other cells are empty or nearly empty because the few studies in that cell provide no statistically significant results.
Although some of the cases in between are hard to classify, we adopt a four-level scale to relate how much is known about the effect of a particular policy on a particular outcome. The criteria for the levels and their representation on Table 11.1 are as follows:
- No evidence (blank cells): No studies at all.
- Little evidence (unshaded cells): A single high-quality study that yielded a significant result or two moderate-quality studies that yielded significant results of the same sign. Similar combinations also fall into this category, such as two high-quality studies that yield results of the same sign, only one of which is significant. We also use this shading to denote cases when results are mixed because there are two high-quality studies with statistically significant impacts of opposite sign or there are three or more high-quality studies with mixed signs but only one is statistically significant.
- Some evidence (lightly shaded cells): All nonempty (or nearly empty) cells that fall neither in the "little" or "much" (defined below) categories reflect cells about which we know "some."
- Much evidence (darkly shaded cells): At least 4 high-quality studies that yield significant results of the same sign.113 This shading is also used in two cases signed as "no change" where 12 high-quality studies are nearly evenly split between positive and negative insignificant impacts with magnitudes close to zero, and in one other case signed as "mixed" where 13 high-quality studies are nearly evenly split between positive and negative impacts and only one impact of each sign is significant.
In addition to the blank cells in the first category above, we also leave some cells unsigned even though there may be one or more studies in the cell. These cases, denoted by an asterisk, occur when there are up to three moderate and/or high-quality studies with no significant impacts or a single moderate-quality study with a significant impact. We felt these cells provided too little evidence to assign a direction of impact with even a minimal level of confidence.
|
Policy or Policy Bundle |
Random Assignment Studies |
Econometric Studies |
|
|---|---|---|---|
|
1. Financial Work Incentives |
CWPDP |
CEA (1997) |
Horvath and Peters (1999) |
|
2. Financial Work Incentives Tied to Hours Worked |
New Hope |
||
|
3. Mandatory Work-Related Activities |
L.A. Jobs-First GAIN Atlanta LFA IMPACT-Basic Track |
CEA (1997) |
MaCurdy, Mancuso, and O’Brien-Strain |
|
4. Sanctions for Noncompliance |
CEA (1997) |
MaCurdy, Mancuso, and O’Brien-Strain |
|
|
5. Mandatory Work-Related Activities and Strong Financial Work Incentives |
MFIP |
||
|
6. Mandatory Work-Related Activities and Weak Financial Work Incentives |
WRP |
||
|
7. Time Limits (Before Recipients Reach Limit) |
CEA (1997) |
Kaushal and Kaestner (2001) |
|
|
8. Time Limits (After Recipients Reach Limit) |
FTP |
Meyer and Rosenbaum (2001) |
|
|
9. Family Cap |
AAWDP |
CEA (1997) |
Kearny (2001) |
|
10. Parental Responsibility |
PPI |
Horvath and Peters (1999) |
|
|
11. Reform as a Bundle (Before Recipients Reach Time Limits) |
EMPOWER |
Bartik and Eberts (1999) |
Kaushal and Kaestner (2001) |
|
NOTES: The eleven rows for individual reform policies or groups of policies correspond to the rows in Table 11.1. For full names of and citations for the random assignment programs, see Table 3.4. |
113 The number four is arbitrary, but we note there were only four studies of female-family heads from the Negative Income Tax experiments, the results from which have been widely regarded as conclusive (Burtless, 1986).(back)
| Table of Contents | Previous | Next |