Introduction
For nearly 100 years, the null hypothesis significance testing (NHST) framework has been used to determine which findings are meaningful (Fisher 1925; Neyman and Pearson 1933). Under this framework, findings deemed meaningful are called “statistically significant.” But the meaning of statistical significance is often misinterpreted. Additionally, statistical significance cannot estimate the likelihood that an intervention has an impact. Statistical significance is the probability of estimating an impact of at least the observed magnitude when the true effect is zero; it cannot address the converse. To address the risk of arriving at misleading conclusions by using the NHST framework, the authors recommend using Bayesian methods and applying the BAyeSian Interpretation of Estimates (BASIE) Framework.
Purpose
The purpose of this brief is to describe the potential misinterpretations that may result from using null hypothesis significance testing through an illustrative example and propose the BASIE Framework in response. This brief also describes some of the challenges associated with applying the BASIE Framework, especially choosing appropriate priors.
Key Findings and Highlights
The BAyeSian Interpretation of Estimates (BASIE) Framework has five components, which are summarized below:
- Probability: In this framework, one’s personal belief should not influence probability.
- Priors: Evaluators should draw upon earlier evidence, or priors (rather than beliefs), to inform the probability that an intervention has an effect of a particular size. Flat priors, or priors that assume equal probabilities of any outcome, may lead to misinterpretation. Yet, specifying an evidence-based prior comes with challenges.
- Point estimates: Report both the impact estimated using only data from the intervention AND the intervention impact estimated using both data from the intervention and prior evidence.
- Interpretation: Use prior evidence to interpret the impact estimate.
- Sensitivity analysis: Evaluators should check whether using different prior evidence affects the conclusions they draw about the impact of their intervention. This analysis is an important way of addressing the challenges associated with choosing an appropriate prior.
BASIE may be an important way to answer a key question: “what is the probability that an intervention worked?” and be less susceptible to inappropriate interpretation.
Citation
Finucane, Mariel, John Deke (2019). Moving Beyond Statistical Significance: Moving Beyond Statistical Significance: THE BASIE (BAyeSian Interpretation of Estimates) Framework for Interpreting Findings from Impact Evaluations, OPRE Report # 2019-35, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.