Using Within-Site Experimental Evidence to Reduce Cross-Site Attributional Bias in Connecting Program Components to Program Impacts

March 3, 2017
Topics:
Self-Sufficiency, Welfare & Employment
Projects:
Health Profession Opportunity Grants (HPOG) Evaluation Portfolio | Learn more about this project, Health Profession Opportunity Grants (HPOG) Impact Study, 2011-2018 | Learn more about this project
Types:
Reports
HPOG Cover Graphic
Download report (pdf)
  • File Size 1mb
  • Pages 45
  • Published 2017

Introduction

Randomized experiments—in which study participants are randomly assigned to treatment and control groups within sites—give researchers a powerful method for understanding a program’s effectiveness. Once they know the direction (favorable or unfavorable) and magnitude (small or large) of a program’s impact, the next question is why the program produced its effect. Multi-site evaluations offer a chance to “get inside the black box” and explore that question.

This paper considers a new method, called Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings (CAMIC), which seeks to reduce bias in analyses that researchers use to understand what about a program’s structure and implementation leads its impact to vary.

First, researchers estimate the overall impact of the program without selection bias or other sources of bias, and then use cross-site analyses to connect  program structure (what is offered) and implementation (how it is offered) to the magnitude of the impacts. However, these estimates are non-experimental and may be biased.

The CAMIC method takes advantage of randomization of a program component in only some sites to improve estimating the effects of other program components and implementation features that are not or cannot be randomized. The paper describes the method for potential use in the Health Profession Opportunity Grants (HPOG) program evaluation.

A simulation analysis of CAMIC shows that the method does not consistently reduce bias and, in some cases, increases bias. Nevertheless, we argue that presenting details of the method is useful. We urge other researchers to consider other settings where the method might be successfully applied in order to help evaluators learn more about what works.

Research Questions

  1. 1 Can the CAMIC method improve our ability to detect, without bias, which program components and implementation features are essential to a program’s success?

Purpose

In job training evaluations, the program components (such as a given curriculum or support service) are rarely randomized to sites; and most implementation features (such as the dynamism of a site administrator) cannot be randomized. Instead, each site chooses its own configuration of program components to adopt and each possesses its own set of implementation features. As a result, the reasons that a particular combination of program components and implementation features exists in a site are also correlated with the program’s impact. For example, the local program director’s enthusiasm and leadership might be associated with both the choice of a particular program component and how well the component is implemented. Better implementation, in turn, might lead to greater program impact. But if that implementation feature is not measured and therefore is excluded from the researcher’s cross-site attributional analysis, then estimates may overstate the influence of the program component on impact.

Multi-site experiments can facilitate an understanding of the effects of both program components and implementation features.  This is the motivation for this work:  to test whether a new method can help researchers better estimate the contributions of program components and implementation features to overall program impact.

Key Findings and Highlights

The paper describes how the CAMIC method uses the experimental estimate of the influence of a particular program component to specify the statistical model used for understanding the influence of other, non-randomized program components and implementation features. The goal is to identify the model that is least biased. However, the theoretical work demonstrates that there may be no single model that is the least biased for all estimates. Depending on the specific correlations among measured and unmeasured program components and implementation features, the model that produces the least biased measure of the influence of one program component may produce the most biased model of the influence of another.

Simulation work investigated how often the CAMIC method could select the least biased estimate of the influence of a particular program component. These simulations were unable to find generalizable conditions under which the CAMIC method is likely to reduce bias. Across all simulations, results were favorable to the CAMIC method for 47 percent of the parameters tested.

Methods

The Health Profession Opportunity Grants (HPOG) program’s impact evaluation is assessing whether providing access to health sector career pathways training improves participant outcomes overall. To do this, individuals are randomized to the HPOG treatment group or to a control group that does not have access to HPOG-funded services. Importantly, in some sites there are two treatment groups, and this allows researchers to focus on a given program component’s relative impacts. In particular, one treatment group has access to HPOG while the second treatment group has access to HPOG enhanced with one of three additional program components: facilitated peer support groups, emergency assistance for specific needs, or noncash incentives that encourage desirable program outputs and outcomes. These three studies can provide strong experimental evidence of the relative contribution of peer support, emergency assistance, or noncash incentives to the HPOG program’s impact.

The CAMIC method is designed to exploit three-armed randomization of certain program components to estimate the effect of other components or intervention features through cross-site non-experimental attributional analysis. That is, can having experimental evidence on one of HPOG’s experimentally evaluated enhancements improve our ability to gauge the effectiveness of other HPOG program components in situations where we observe these same program components naturally occurring in the program?

The method’s basic approach involves the following:

  1. Estimating the experimental impact of the program component (e.g., peer support) added to HPOG through a second treatment arm;
  2. Calculating several alternative, non-experimental estimates of the impact contribution of the program component (e.g., peer support) from cross-site analysis models that control for various sets of site-level influences in each analysis;
  3. Choosing the model that minimizes the difference between the experimental and non-experimental estimates of the impact contribution of the added program component (e.g., peer support); and
  4. Applying that model to estimate the contribution to program impact of other program components (not peer support but other components naturally occurring in the HPOG program such as intensive case management or the presence of career pathways principles) or implementation features (e.g., the program’s administrative structure, or case workers’ client orientation) when these other components or features do not vary randomly among individuals within sites or across sites.

Citation

Stephen H. Bell, Eleanor L. Harvill, Shawn R. Moulton, and Laura Peck. (2017). Using Within-Site Experimental Evidence to Reduce Cross-Site Attributional Bias in Connecting Program Components to Program Impacts, OPRE Report #2017-13. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Last Reviewed: November 13, 2018