Introduction
Many awardees of the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program use outcome evaluation to understand their initiatives’ impact on children and families. Well-designed evaluations contribute knowledge to the field of home visiting and inform data-driven policy. Awardees proposing to conduct an outcome evaluation must select a rigorous evaluation design with a suitable comparison to the treatment group. Randomized controlled trials are often considered the gold standard for establishing treatment and comparison groups, though they are not always feasible. Evaluators can also use matching methods like propensity score matching to craft a sound comparison group using a quasi-experimental design.
This brief:
- Defines propensity score matching and other matching methods
- Covers steps in the matching process
- Offers suggestions for decreasing bias
- Presents a hypothetical example of matching in the home visiting context
- Recommends resources to support high-quality matching
Purpose
Measuring Program Effects in Home Visiting Evaluation: Improving Estimates with Propensity Score Matching serves distinct purposes depending on readers’ familiarity with the topic.
- For those not currently using propensity score matching: It explains propensity score matching for awardees to consider as a rigorous approach to MIECHV outcome evaluation.
- For those already using propensity score matching: It offers insights and suggestions for how to decrease bias with matching, leading to higher-quality evaluations.
Key Findings and Highlights
Key takeaways include the following:
- Matching can achieve unbiased estimates of program impact when there is no significant difference between the groups on variables associated with both treatment and outcomes.
- Trustworthy results rely on a good match between treatment and comparison groups, including careful selection of variables used for matching. Evaluators should match on a combination of demographics, treatment predictors, confounders, and a baseline measure of the outcome, if available. Evaluators can use programmatic data, staff observations, research literature, and directed acyclic graphs to identify matching variables.
- Evaluators should work to ensure that data are available for all key matching variables. If data are not available for an important matching variable, the evaluator should note this as a limitation in the evaluation report and discuss its potential impact on estimates.
- Four ways to use propensity scores for matching include pair matching, weights, stratification, and covariate adjustment.
- Once the treatment and comparison groups are identified through matching, evaluators must test for equivalence or balance between the groups. Common forms of testing include standardized bias testing, variance ratio testing, and plotting. Evaluators can modify the matching model, adjust the list of matching variables, or prune cases to achieve balance.
Methods
The authors first reviewed MIECHV state-led evaluation reports to identify areas for improvement. The brief reflects literature findings and standards of the field for using propensity score matching.
Citation
McCombs-Thornton, K. & Poes, M. (2021). Measuring program effects in home visiting evaluation: Improving estimates with propensity score matching. OPRE Report #2021-55. Office of Planning, Research, and Evaluation, U.S. Department of Health and Human Services. Produced by James Bell Associates.
Glossary
- Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program:
- Administered by the Health Resources and Services Administration in partnership with the Administration for Children and Families, the MIECHV Program was established in 2010 to support voluntary, evidence-based home visiting for at-risk pregnant women and parents with children up to kindergarten entry. The program provides grants to states, U.S. territories, and tribes, which conduct needs assessments to identify eligible at-risk communities and serve priority populations.
- Propensity score matching:
- Process of comparing treatment group members with comparison group members based on similar propensity scores, typically by pairing across groups, using weights, or stratifying the data.
- Propensity score:
- The probability that each case participates in the program, given the set of observed traits that are used to predict the score.