Table of Contents | Previous | Next |
PART V: KEY ISSUES OF EVALUATION
High quality program evaluations will be critical to the next stages of program implementation. Policymakers and program developers will need to know which types of interventions, in which of the different settings, and for whom in the population yield the most positive results. Below we outline a number of issues that arose in our project that relate to the evaluation agenda for healthy marriage services. We utilize examples from our investigation of programs, where applicable, to illustrate some of the issues. This offers a first attempt at highlighting some of the major issues that may surface in developing strategies to evaluate programs that support healthy marriage.
Environment
The environments in which interventions are implemented vary greatly. The sheer variation of healthy marriage activities creates a challenge for evaluation. We found that interventions exist in geographic areas that vary by culture, religion, socioeconomic status, level of participation in marriage initiatives, and support for marriage programming. Understanding the contribution of the environment to the success or failure of an intervention will be important in future evaluations, as will documenting the environment to consider issues of generalizability. For example, some communities already have community marriage initiatives underway, while others seem hesitant or opposed to these initiatives. Organizing an intervention in a welcoming community could provide additional funding sources, an existing network of interested providers, and potential clients who have already been exposed to messages about marriage, arguably all factors that could lead to program success. The same intervention in a community without the political capital for marriage initiatives may attract fewer clients and less funding, perhaps making the intervention appear less successful than it could be.
The counterfactual set of services may be difficult to uncover. Evaluations of marriage interventions will have to go to great lengths to consider the counterfactual to the interventions of interest. Environments may be rich in related types of services, and the patterns of use of these services will not be documented prior to the evaluation.
Setting
Mixed exposure to research or evaluation. While healthy marriage service providers often cite the desire to provide “research-based” services, their assessment of the research tends to rely on the claims of the authors of curricula. And few providers themselves conduct evaluations of their own programs. Providers may, however, have experience with collection of data in their settings. They may collect information at first contact with clients, and they also may assess progress of those clients through post-tests at the end of the intervention. They have some experience with informal follow-up, as clients seek their advice on future relationship issues. Yet most providers have little experience with several other key elements of evaluation: longer-term follow-ups, data collection from control groups or participants who leave an intervention, and assessments of behavioral change. We observed that current program providers tend to evaluate based on knowledge of skills or program satisfaction rather than evidence of changes in attitudes or behaviors.
Of the evaluations that do exist, these studies commonly occur in academic settings where the marriage service is provided for the purpose of the research. Generally programs have not been rigorously evaluated in field settings. Given the policy interest in funding ground efforts toward marriage programming, it is important that rigorous evaluations examine programs like the ones we observed in the field.
The direct service providers may be highly trained and experienced or may be relatively new to the field. The motivation to deliver healthy marriage services comes from professional theories, as well as faith and practical concern about the decline of marriage and the effects on our society. As an evaluation issue, this raises key concerns about the inability to disentangle the provider from the intervention. Simply stated, there may be interventions of theoretical interest that are not implemented with skilled or engaging providers and consistent organizational leadership. These interventions may show very different outcomes when implemented by stronger programs and providers. Likewise, there may be strong and experienced providers who are not using a well-developed intervention. The effect of strong providers may overstate the usefulness of a weak intervention. Ideally, major evaluations would have many sites and would be able to explore these patterns and add to the knowledge base about the interaction of the intervention with its provider.
Intervention
Defining the piece of the intervention that is a “Healthy Marriage Service” is difficult. A central challenge in evaluation is defining the intervention of interest. If the key question is “does it work?”, it is crucial to define “it.” Evaluation projects have to define clearly the boundaries of the types of interventions they are evaluating. Healthy marriage services are connected to— and often embedded within— a range of other services. In some communities we found “ stand -alone” services. Sometimes we found stand-alone services that were part of a network of services. We often saw programs that simultaneously provide a range of services, one of which is a healthy marriage service. In any of these cases, it is important to distinguish between the components that make up the healthy marriage services and the components that are related to— but not part of— the same intervention. Evaluations will need to be clear about which component or package of components they are evaluating, as well as what they are evaluating them against. The availability of new funding for marriage programs and their evaluation may help guide this process. If to receive funding programs have to meet a certain set of criteria, these criteria could serve to narrow the definition of what a marriage intervention will be.
Interventions are small and may face challenges reaching larger scale. Many of the interventions we saw deal with fewer than 100 couples per year. Most interventions do not exclusively serve couples with children, either. Due to the variation in interventions (and their settings and environments), it may not be possible or advisable to pool findings across sites. Therefore, evaluations will require much larger samples to detect possible impacts on couple relationships and child well-being. Further, if evaluators seek to detect impacts on child well-being, with a large sample, evaluators could isolate a subgroup of couples with children to examine child outcomes.
However, we observed that individual sites may face considerable challenges “ ramping up ” to serve more clients, especially if the intervention targets or the evaluation requires a relatively homogeneous group. Organizations that offer services are often minimally staffed and already face challenges with the logistics of providing services. In these cases, the level of an organization’s commitment to marriage services as opposed to other services may determine how often staff offer marriage programs and the degree to which they prioritize them in their daily work.
Dosage may be too low to affect longer-term change. The interventions of healthy marriage services are short-term and may not meet the interests of ACF in developing programs to create longer-term behavioral change and, ultimately, improvements in child well-being. Providers noted that they ideally would provide more intensive services but felt that the market could not support intensive efforts. They believe few people have the time or desire for longer-term interventions.
Dropouts may be common. Not only are dosages low, but some participants attend inconsistently, and dropping out occurs occasionally within the programs we observed. Longer-term interventions may exacerbate the problem of attrition and its implications for program evaluation. Evaluations will have to be attentive to enrollment and subsequent service receipt for both members of the couple. Evaluations will also have to weigh the difficulty of tracking particular groups of couples, such as couples preparing for marriage, that tend to be transient. One church-based premarital program, while near a university, reported that only 10 percent of the couples receiving its service stayed in the area after marrying.
Training and the potential consistency of services across sites. Training in this field varies greatly. The interest in providing services has come, to date, from the ground up. Providers perceive that they have received the training they want and can afford, both in terms of time and money. However, knowledge of the existence of different curricula can vary vastly by provider. The great variation means that providers often do not implement a curriculum in a standard way across sites. They also frequently adapt curricula to meet the needs of their particular clients.
The stability of programs over time is unclear. The commitment of significant evaluation resources is best made when programs reach a point of stability, where early implementation kinks have been worked out, and when the program is operating as it would be expected to operate if expanded. Healthy marriage programs have not had sufficient funding over the past two decades to reach such stability. Programs that have remained in existence have evolved— driven either by opportunities for funding or by the evolution of the thinking of their creators and operators. While significant new Federal dollars for healthy marriage services may stabilize programs, such stability may not come for several years.
Clients
Client flow and point of randomization. Entry into healthy marriage programs is ill-defined when compared to many other human service areas. Because programs lack the standard application process of many human services systems, couples may drift toward the service and may attend without formally applying. In addition, because the service is intended to treat a couple, confirming the intent to participate of both members of the couple may be difficult. Randomization schemes will have to look at the market for services in a particular site and consider changing the application and attendance process to attempt to insert randomization at the best point between interest in the program and the beginning of services. These changes to the front end of services may concern service providers, who do not have a history of dealing with evaluation.
Evaluating voluntary services and the role of such services in non-voluntary settings. ACF hopes to explore the value of healthy marriage services made available on a voluntary basis to couples that desire such services. The voluntary nature of the service affects some evaluation design issues. Those who voluntarily seek the service may voluntarily walk away from it at any stage. In addition, the nature of the interaction within the couple in seeking and staying connected to the service is important to understanding outcomes. The decision to sign up or drop out may be driven by one partner more than the other.
Though healthy marriage programs are voluntary, they sometimes exist within non-voluntary systems. In particular, programs within the prisons and court-ordered child welfare or domestic violence services operate within a non-voluntary system. It is unclear whether evaluations should consider services within such compulsory systems as voluntary or comparable to other voluntary interventions. Moreover, though providers refer to such programs as voluntary, the incentive structure for participants must be fully understood. For example, participation within a prison setting may be linked to other benefits of prison life, such as time away from the prisoner’s cell or unstructured mealtime with a spouse. The additional benefits a participant may receive above and beyond the benefits of the intervention itself should be understood as additional incentives to participate that would not exist in truly voluntary settings. The relationship between providers in non-voluntary settings and clients may also be of interest. If a client sees her relationship with a provider, such as a worker in a TANF office, as adversarial, she may be less likely to respond to a marriage program within that setting.
The difficulty of operationalizing outcomes. Identifying and measuring outcomes of interest in programs for low-income clients may be difficult. For instance, providers often note that their premarital programs are successful if they discourage marriages that would be unhealthy. Evaluations must address whether and how they would evaluate individuals who leave a relationship during or after an intervention. Are the former partners better prepared for future relationships? Are children in a safer environment? In addition, the timing of follow-ups needs to be matched to the purpose of the intervention and the nature of the clientele, such as whether they are premarital, seeking enrichment, or at a point of crisis. Administrative data could be useful if reliable measures of a “ healthy marriage ” were available. Reliability and the source of outcome data are also important, whether it is the provider, the clients themselves or an outside assessor. Clients’ ability to accurately assess their own progress would need to be considered. If providers or outside assessors attempt to measure couple outcomes, evaluators must deal with inter-rater reliability and potential evaluator bias.
The developmental stages of relationships. The match of services and outcome measures to developmental stage is key. Evaluations should take into account the developmental stages of relationships in the same way that early childhood service evaluations must be sensitive to child development. Couples and their relationships change over time, just as individuals do.2 Healthy marriage interventions treat couples in many stages of relationships, from early relationship to pre-marriage to end-of-marriage services. Outcome measures, too, should be selected to be appropriate for assessing the particular stage of the couples’ relationship.
Demand for the services and lack of obvious oversubscription. Random assignment is most justifiable when a service is clearly oversubscribed and some sort of rationing process is already in place. It is often not clear with marriage interventions whether there are waiting lists or whether oversubscription exists. Interventions have little incentive to collect information on who is turned away, if anyone is. Outreach may only be conducted to the point of filling the service, not to document excess need for the service.
Evaluations may have to adapt the intake processes to clarify the excess demand. One drawback in this approach is that in changing the intake process to identify or create oversubscription, the resulting target population may change. As a result, evaluators may need to change the intervention they had intended to evaluate. Additionally, organizations that have a mission to serve anyone who is interested in their services, particularly couples in crisis, may resist turning people away for an evaluation. In cases where oversubscription does not exist or organizations wish to serve everyone, dual treatment evaluation designs are a possibility. With these designs, evaluators could examine the effects of one treatment compared to another.
Tracking low-income populations. Low-income populations tend to be more transient, which poses a very significant challenge to evaluation. Improper follow-up of a large percentage of clients can create a selection problem— who are we unable to follow, what do we know about them, and how does that taint the results of the evaluation? For instance, one provider noted that one in four applicants to his job training program for low-income individuals, which spans roughly one month from application to graduation, actually completed the program. How rates of attrition for marriage programs would compare to those of programs for low-income individuals is unknown. On the one hand, married couples might be more stable than single individuals and more likely to complete an intervention. On the other hand, the process of maintaining both individuals of a couple in an intervention may be even more difficult than for programs for singles. Attrition rates in studies that follow-up with those who complete the program may also be high, particularly among young, transient populations. A popular strategy to reduce attrition is to provide financial incentives for participation.
Targeting. Current studies of the effectiveness of marriage programs for middle-income populations may not be generalizable to low-income populations. Low-income couples deal with a variety of unique stressors in their lives, such as unemployment, incarceration, lack of transportation and housing, and economic instability, that may not affect middle-income populations as intensely. Furthermore, low-income populations are not homogeneous. Targeting low-income populations raises issues of how to match services to subpopulations and then generalize these results. For instance, an intervention that is successful for married couples with one spouse in prison may not necessarily be successful for unmarried, unemployed parents. Similarly, if you evaluate couples at a particular point in their relationships, it is unknown whether those findings would be generalizable to couples in other stages of relationships.
Providers may also resist targeted programs, particularly if they work within organizations charged with serving the whole community, such as community colleges or Cooperative Extension. Current programs can have very loose requirements for program participation. Premarital programs may include cohabiting couples that have only vague intentions to marry in the future, relationship skills programs may serve singles and couples of all sexual orientations, and marriage programs may mix young newlyweds with “empty nesters.” While group heterogeneity presents evaluation challenges, some providers see it as a desirable asset. In order to overcome this challenge, evaluations may choose to assemble large sample sizes in order to examine subgroups in a single site. Additionally, if evaluators can match or control for environmental factors, they could organize multiple sites and analyze an aggregate subgroup composed of samples from all sites.
2 For a more detailed description of research that has been done to trace marital quality over time and the factors that affect it, see Karney and Bradbury’s The Longitudinal Course of Marital Quality and Stability: A Review of Theory, Method, and Research (1995). (back)
| Table of Contents | Previous | Next |