Rapid Learning: Methods to Examine and Improve Social Programs

Publication Date: October 22, 2019
Current as of:
Cover of the Rapid Learning: Methods to Examine and Improve Social Programs brief include the logos for the Office of Planning, Research, and Evaluation, Insight Policy Research, and the text from the first page of the publication.

Download Report

Download PDF (322.84 KB)
  • File Size: 322.84 KB
  • Pages: N/A
  • Published: 2019

Introduction

Research Questions

  1. What are rapid learning methods?
  2. What should be considered when designing a rapid learning approach?
  3. How can federal agencies promote the effective use of rapid learning methods?
 
Rapid learning methods aim to expedite program improvement and enhance program effectiveness. They use data to test implementation and improvement efforts in as close to real-time as possible. Many rapid learning methods leverage iterative cycles of learning, in which evaluators and implementers (and sometimes funders/policymakers) discuss findings, interpret them, and make adaptations to practice and measurement together. These methods can support data-driven decision-making in practice, in the spirit of ongoing improvement.

On October 25 and 26, 2018, OPRE brought together a diverse group of participants from Federal agencies, research firms, academia, and other organizations for a meeting titled, Rapid Learning Methods for Testing and Evaluating Change in Social Programs . This brief is based on a presentation at the meeting.

Purpose

This brief and the accompanying presentation provide an orientation to rapid learning methods, including: (1) a definition of rapid learning methods, (2) a guiding framework of questions to design an optimal rapid learning approach, and (3) suggested steps that federal agencies can take to promote the effective use of rapid learning methods.

Key Findings and Highlights

Core traits of rapid learning methods include:

  1. Start with program objectives
  2. Identify a strategic change to test
  3. Hypothesize potential improvements
  4. Determine the appropriate measurement
  5. Analyze the outcomes and make decisions

Core questions to consider when designing a rapid learning approach include:

  • What am I trying to understand?
  • What outcomes do I want to change?
  • How will I use the results? Who else will use them?
  • What are the organization’s priorities and where does this issue fit?
  • How confident do I need to be in the results?
  • How hard is it to implement the innovation?
  • How much will-building and engagement of program delivery staff do I need?
  • What data are available?
  • How soon do I need to know the results?
  • What time is needed to observe an impact? Is there a near-term proxy for that impact?

Recommendations

To promote the effective use of rapid learning methods, federal agencies can:

  • Establish expectations around evaluation and learning.
  • Combine rapid learning with other forms of program evaluation.
  • Write evaluation procurements that foster rapid learning.
  • Facilitate trainings on rapid learning approaches.
  • Create venues to share results from rapid learning.

Citation

Cody, S., & Arbour, M. (2019). Rapid learning: Methods to examine and improve social programs (OPRE Report 2019-86). Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.