Introduction
Research Questions
- What can the literature tell us about different methodological approaches evaluators have taken to assess the effectiveness of community-level change efforts?
- What can the literature tell us about the ways in which evaluations of community-level change efforts have identified viable indicators and sources of data for baseline measurement?
- What research and evaluation tools might entities undertaking community-level change efforts use to measure or assess the outcomes of their community-level change efforts?
Addressing many of the most difficult social problems requires an approach that combines the efforts of multiple organizations, often operating in different sectors, coordinating across an entire community. Numerous programs overseen by the Administration for Children and Families (ACF) engage in these community-level initiatives or community-level change efforts. These include efforts to improve local economic conditions, improve the quality of early childhood education in a community, or increase community-wide levels of adult literacy, among other objectives.
These initiatives, however, are challenging to evaluate. For a start, many—if not most—community-change efforts are multiyear endeavors. And although a grantee may be clear about the efforts’ ultimate objectives, the grantee must also find ways to capture annual progress toward achieving those objectives, identifying the appropriate, incremental outcomes—and indicators of those outcomes—to do so. In addition, grantees often conduct community-change efforts as part of a collaboration with other organizations and must find ways to account for the impact of their own contribution to the overall effort.
To inform ACF on how to better capture these initiatives’ impact, the Office of Planning, Research, and Evaluation (OPRE) at ACF commissioned this literature review to see what can be learned from recent efforts that have not only attempted to achieve community-level change, but have also attempted to produce credible evidence about the impact of these efforts. This literature review attempts to identify lessons that ACF and its grantees may be able to apply to evaluations of community-change work from both the conceptual literature on measuring community-level change as well as from the empirical approach specific community-level initiatives have taken to evaluating their efforts. To draw out these lessons, this review begins by providing an overview of the methodological challenges community-level initiatives pose for assessing causal impacts. It then highlights some key lessons gleaned from recent evaluations of community-level change efforts.
Purpose
The purpose of this literature review is to identify lessons that ACF and its grantees may be able to apply to evaluations of community-change work from both the literature on measuring community-level change and evaluating specific community-level initiatives. In reviewing this literature, this review captures lessons on various approaches to defining the boundaries of a community as well as on how to define and measure desired outcomes. The review also highlights the expertise that may be required to conduct evaluations of community-change efforts, what data collection methods and strategies are likely to yield useful information, and how these data might be aggregated or compiled to provide a useful view at the state and federal levels.
Key Findings and Highlights
- Credibly assessing the causal impact of initiatives on outcomes of interest is a long-standing challenge in the evaluation literature. The most credible way to determine causality is through random-assignment studies. However, such studies are often infeasible or impractical for evaluating community-level initiatives.
- Impact evaluations for community-level initiatives involve special challenges. Evaluation designs must account for these challenges, including the difficulty of defining the “treatment,” the importance of spillover effects within communities, the interactions between different initiative programs and services, and the difficulties of detecting effects in community-level outcomes.
- New methods are strengthening our ability to measure the effects of community-level initiatives. Recent advances in quasi-experimental methods suitable for community-level evaluation are promising. In addition, theory-based or mechanism evaluations—i.e., evaluations focused on uncovering the casual links between an initiative’s activities and its outcomes—can be especially helpful for learning about an initiative’s impacts. Moreover, mixing methods to include qualitative and quantitative approaches can strengthen causal claims.
- Recent evaluations of community-level initiatives such as the Community Development Block Grant, Promise Neighborhoods, and the Urban Health initiative, among others, offer particularly useful lessons for how to approach community-level evaluations.
Methods
This literature review drew on key practitioner and empirical literature from a number of different disciplinary fields: public and nonprofit management, social work, sociology, community development, program evaluation, public health management, governance networks and network management, and collective impact. The primary search strategy involved a keyword search of certain databases including Google (and Google Scholar), EBSCOHost, and JSTOR to identify relevant literature including books and book chapters, working papers, dissertations, government-commissioned reports, foundation-commissioned reports, academic and professional association magazines, and peer reviewed journals. The project also used a snowballing search technique to follow-up on promising citations from literature already reviewed and review of materials posted on key web sites. The set of case studies of recent efforts to evaluate community-change efforts was identified through these methods, as well.
Citation
Congdon, William, Margaret Simms, and Carol De Vita (2020). Assessing the Impact of Community-Level Initiatives: A Literature Review, OPRE Report #2021-4, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.