Introduction
Federally funded systematic reviews of research evidence play a central role in efforts to base policy decisions on evidence. Historically, evidence reviews have reserved the highest ratings of quality for studies that employ experimental designs, namely randomized control trials (RCTs). However, RCTs are not appropriate for evaluating all intervention programs. To develop an evidence base for those programs, evaluators may need to use non-experimental study designs.
Purpose
This brief describes the need for non-experimental study designs when a randomized control trial is not appropriate. It also summarizes important considerations for using non-experimental studies.
Key Findings and Highlights
This brief identifies key considerations for assessing the causal validity of non-experimental studies systematically and reliably:
- Defines causal validity,
- Provides examples of threats to causal validity and methods that can be used to address those threats,
- Discusses causal validity ratings in HomVEE and WWC, and
- Summarizes key considerations for developing standards to assess the causal validity of non-experimental designs.
Citation
Deke, John (2018). Causal validity considerations for including high quality non-experimental evidence in systematic reviews (OPRE Report 2018-63), Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.