The implications of measurement error differ depending on whether one is using natural or standardized units. One panelist noted that psychological outcomes are typically measured in arbitrary units, whereas economic indicators are typically measured in natural units. Measurement error for an outcome measured in natural units will affect standard errors and significance tests, but will not bias coefficient estimates. In contrast, measurement error for outcomes measured in standardized units is more problematic and introduces bias in coefficient estimates. In addition to reporting the measurement procedures, it is important to identify the intervention study design (e.g., random assignment versus pre-post) since these factors can influence the magnitude of the effect size expected. Standardized tests provide useful benchmarks from which to contextualize outcomes. How the contextualization markers are used depends on the goals of the comparison. When calculating effect sizes, researchers are comparing an impact to a standard deviation, but there are many standard deviations from which to select depending upon the research question. Some examples of reference points from which to select include: criterion reference points; social, normative, and moral standards; outcomes; and cost. Though usually not presented in detail, cost of an intervention is an important factor, though not the only factor, when interpreting the magnitude of the effect size. In terms of policy implications, intervention benefits for one population should not suggest that the intervention does not have potential negative effects for another population. Analyses must probe to find potential subgroup differences in effects.
- PDF Panel 2 Summary (10.36 KB)