Introduction
This report details new and promising approaches to subgroup analysis for evaluators of employment programs. It discusses how two Bayesian methods—a Bayesian hierarchical linear model and a Bayesian causal forest—can potentially address limitations of standard subgroup analysis.
The report uses data from four experimental evaluations of employment programs in the Evaluation of Employment Coaching for TANF and Related Populations, a project sponsored by the Office of Planning, Research, and Evaluation in the Administration for Children and Families, U.S. Department of Health and Human Services. The results suggest that Bayesian methods can complement traditional methods of conducting subgroup analyses in impact evaluations.
Purpose
The purpose of this study is to inform evaluators of employment programs about new and promising approaches to subgroup analysis by illustrating how Bayesian methods for subgroup analysis can complement traditional methods. Drawing on real-world data, the study provides evidence on (1) how Bayesian methods can be used to reinterpret subgroup impact estimates for a single evaluation; (2) how Bayesian methods can be used to reinterpret subgroup impact estimates for multiple evaluations; and (3) the extent to which Bayesian methods can be used to identify subgroups that were not previously specified by the evaluator conducting the analysis. The study aims to identify methods that can draw more nuanced insights from subgroup analyses in the context of evaluations. Such insights can help practitioners and policymakers better serve and design programs for specific groups.
Key Findings and Highlights
The report found that:
- Compared to null hypothesis testing, the Bayesian hierarchical linear model approach can (1) suggest more nuanced conclusions about the differences between subgroups and (2) lead to estimates that are less sensitive to small deviations in the data.
- When applying the Bayesian hierarchical linear model approach to multiple evaluations of employment programs, the impact estimates from separate programs influence each other, highlighting how a Bayesian hierarchical linear model draws on information across programs.
- The Bayesian causal forest approach can potentially identify subgroups that had not been pre-specified by the evaluator conducting the analyses in evaluations with large sample sizes.
Methods
The report includes:
- A traditional subgroup impact analysis based on linear regressions and null hypothesis testing.
- A Bayesian hierarchical linear model approach to reinterpreting traditional impact estimates.
- A Bayesian causal forest approach to identifying subgroups that were not previously specified by the evaluator conducting the analyses.
Citation
Tim Kautz, Christina Kent, and Dan Thal. Using Bayesian Methods to Conduct Subgroup Analysis in Evaluations of Employment Programs. OPRE Report #2024-027. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.