Methods, Challenges, and Best Practices for Conducting Subgroup Analysis

Publication Date: February 5, 2021
Cover image for Methods, Challenges, and Best Practices for Conducting Subgroup Analysis

Download Report

Download Report PDF (434.75 KB)
  • File Size: 434.75 KB
  • Pages: 12
  • Published: 2021

Introduction

Policy and program evaluation research often seeks to estimate the effects of interventions across broad populations. In such studies, a focus on average treatment effects may obscure important variation across subgroups of the treated population. Subgroup analysis allows researchers to determine whether policy or program impacts vary across groups. This approach enables policymakers to efficiently allocate resources by developing a better understanding of which interventions are most effective for particular groups of participants under specific circumstances. 1, 2

 

1 Haegerich, T. M., & Massetti, G. M. (2013). Commentary on subgroup analysis in intervention research: Opportunities for the public health approach to violence prevention. Prevention Science, 14(2), 193–198.

2 Supplee, L. H., Kelly, B. C., MacKinnon, D. M., & Barofsky, M. Y. (2013). Introduction to the special issue: Subgroup analysis in prevention and intervention research. Prevention Science, 14(2), 107–110.

Purpose

This brief has two main goals:

  • Describe the features of a well-designed and implemented subgroup analysis that uses a multiple regression framework.
  • Provide an overview of recent methodological developments and alternative approaches to conducting subgroup analyses.

The brief builds on a 2009 meeting of experts Visit disclaimer page convened by the Administration for Children and Families’ Office of Planning, Research, and Evaluation and a corresponding 2012 publication in a special issue of Prevention Science Visit disclaimer page (MacKinnon, Supplee, Kelly, & Barofsky, 2012).

Key Findings and Highlights

  • Research plans for confirmatory subgroup analysis should be informed by a theory of change (e.g., a well-constructed logic model) and clear research questions. The plan should also address the following:
    • Experimental or quasi-experimental research design
    • Sample selection and statistical power
    • Adjustments for multiple comparisons
  • Analytic approaches to subgroup analysis should consider incorporating the following:
    • Multiple regression approaches with interaction terms
    • Estimation techniques for quasi-experimental data
  • Subgroup analysis results should be interpreted and presented with care to prevent inaccurate interpretations of the data and ill-informed policy recommendations.
  • Researchers may encounter several challenges and other considerations when planning and implementing subgroup analysis:
    • Multiple hypothesis testing may generate false positives.
    • Research context may cast doubt on findings (i.e., subgroup analyses that contradict previous findings should be interpreted with caution).
    • Researchers may discover post hoc subgroups that were not specified during the study planning phase but should not report those analyses as confirmatory.
    • Researchers may consider preregistering their study plans before beginning to conduct subgroup analyses to increase rigor and transparency.
  • Recent methodological developments in subgroup analysis have addressed previous limitations of this method and include the following:
    • Meta-analysis
    • Bayesian methods
    • Machine learning

Methods

This paper was based on a literature review.

Recommendations

Subgroup analyses to identify heterogeneous treatment effects are important efforts useful for informing policy creation and program design. Study designs for conducting effective and credible subgroup analysis adhere to several best practices:

  • Be explicit about whether subgroup analyses are confirmatory versus exploratory.
  • Motivate subgroup analysis with theoretical rationale and prior research.
  • Determine subgroups and subgroup comparisons early in the design process.
  • State hypotheses prior to collecting any data or conducting any analysis phases.
  • Calculate minimum necessary sample size or minimum detectable effect during the design stage.
  • Measure subgroup variable at baseline.
  • Stratify randomization to treatment by subgroup.
  • Test interaction effects of subgroups and treatment.
  • Limit the number of subgroup analyses and adjust for multiple comparisons.

Citation

Breck, A., & Wakar, B. (2021). Methods, challenges, and best practices for conducting subgroup analysis (OPRE Report #2021-17). Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services

OPRE Research Topics:
Types:
Topics:
Tags: