Building Strong Families (BSF), a program of relationship skills education for unwed parents, has been found in a rigorous random assignment evaluation to have limited effects on couples who signed up for the program (Wood, McConnell, et al. 2010). Averaging results across the eight local programs that participated in the evaluation, BSF had no effect on the couples’ relationship quality or on the likelihood that they would remain romantically involved or get married 15 months after they enrolled in the program. When impacts were examined separately for the eight programs, only one was found to have a consistent pattern of positive effects on couples’ relationships, while another was found to have negative effects.
These results, however, leave us with an unanswered question of wide interest, because not all couples randomly assigned to receive BSF services actually participated. The core BSF service was group workshops on relationship skills, and across all evaluation sites about 45 percent of the couples assigned to the program group never attended even one workshop session. BSF was a voluntary program and voluntary programs, particularly those serving low-income families, often have low participation rates (McCurdy and Daro 2001; Myers et al. 1992; Garvey et al. 2006). Even so, it is natural to ask whether BSF had any effects on the couples who did attend group sessions.
This question is not answered in the BSF 15-month impact report, which is based on analyses of all couples who signed up for BSF, including those assigned to the program group who never attended a group session (Wood, McConnell, et al. 2010). The impacts reported thus represent the average effect on all program applicants of being offered BSF services and not the effect of actually attending group sessions. Such “intent-to-treat” (ITT) impact estimates are widely used in large-scale evaluations, for two reasons. First, they preserve the key strength of a random assignment research design—specifically that one can be confident that the program and control groups were similar at baseline, and that differences in outcomes that emerge (and that are large enough to be unlikely to be the result of chance) can be attributed to the program. If some sample members who were randomly assigned are excluded from the analysis, one can no longer be certain that the two research groups are similar and that differences in outcomes between them represent the effect of the program. Second, ITT estimates address a policy-relevant research question: What is the effect of offering a program in the “real world,” where one can anticipate that not everyone in the target population will participate in all program services? Nonparticipation may limit a program’s ability to affect outcomes in the target population the program seeks to help, despite extensive efforts to promote attendance, and it is appropriate to reflect this limitation in estimates of a program’s impact. Even so, it is natural to ask whether BSF had any effects on the couples who did attend group sessions.