Bayesian Methods for Social Policy Research and Evaluation

Publication Date: July 3, 2018
Current as of:
Bayesian Methods for Social Policy Research and Evaluation

Download Report

Download PDF (485.38 KB)
  • File Size: 485.38 KB
  • Pages: N/A
  • Published: 2018

Introduction

Probability (p) values are widely used in social science research and evaluation to guide decisions on program and policy changes. However, they have some inherent limitations, sometimes leading to misuse, misinterpretation, or misinformed decisions. Bayesian methods, which use probabilistic inference to determine the importance of a finding, are becoming the primary alternative approach to p-values. But, many researchers lack the knowledge and training to confidently implement a Bayesian analysis. Given the increasing attention to and use of Bayesian methods in social science research, it is essential to understand the underlying assumptions, tradeoffs, validity, and generalizability of results in a Bayesian framework, and the circumstances under which there may be advantages to using it rather than, or in addition to, a frequentist approach.

Purpose

In the fall of 2017, OPRE brought together a diverse group of participants from federal agencies, research firms, foundations, and academia to discuss Bayesian methods for use in social policy research and evaluation. This brief summarizes the meeting, which focused on four topics:

  • The advantages and disadvantages of Bayesian methods
  • Bayesian model building and evaluation
  • Examples of Bayesian applications in the field
  • How social policy researchers and stakeholders can use Bayesian methods to better support decision-making

Key Findings and Highlights

  • Advantages of Bayesian methods include the ability to:
    • Incorporate existing information and update conclusions as new data become available
    • Improve precision in studies with small samples or small subgroups
    • Communicate findings intuitively, using probabilistic statements (e.g., “there is a 90% chance that the program reduced costs”)
    • Quantify multiple types of uncertainty
    • Better evaluate possible models
  • Challenges to using Bayesian methods include:
    • Translating probabilistic inferences into yes/no responses that may be needed in a policy context
    • They are computationally intensive
  • Example applications include meta-regression, creating statistical benchmarks, meta-evaluation, and Bayesian model averaging.
  • Future directions include:
    • Building capacity among researchers and decision-makers to understand Bayesian methods
    • Developing transparent, objective criteria to identify high-quality Bayesian analyses

Citation

Holzwart, R., & Wright, D. (2018). Bayesian methods for social policy research and evaluation (OPRE Report 2018-38). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.