OPRE Research and Evaluation Technical Assistance Resources

Introduction

While conducting rigorous research and evaluation projects studying ACF programs, OPRE develops a wide range of resources — including reports, toolkits, newsletters, and webinars — on providing technical assistance to ACF award recipients, evaluators, and technical assistance (TA) providers. However, many of these resources focus on specific programs or content areas and may not be generalizable to all ACF programs. Therefore, OPRE aimed to develop a series of new and updated resources that could be applied across all of OPRE’s content areas and help build the capacity of OPRE and its network of award recipients, evaluators, and TA providers to employ cutting-edge evaluation and research methods.

Purpose

The primary purpose of these TA resources is to build the capacity of ACF award recipients, evaluators, and TA providers to apply recent advances in research and evaluation methodologies to their work. The resources have three primary audiences:

• ACF award recipients, evaluators, and technical assistance providers, who can use the resources to support the implementation of evaluation and research activities
• Staff at the U.S. Department of Health and Human Services, who can use the resources to inform research and evaluation design and build awareness about new methodologies
• Individuals in the general public interested in learning more about program evaluation

The resources were also developed to complement and expand on topics addressed in The Program Manager’s Guide to Evaluation, including:

• Applying an equity framework to research and evaluation
• Engaging communities in the design and implementation of research activities
• Using innovative qualitative methods to complement more conventional quantitative approaches and incorporate the lived experiences of program participants and communities

Resources

• Assessing and Improving Program Performance (OPRE Report 2024-067). (PDF) This resource provides an overview of program improvement and its importance, practical guidance for implementing methods to measure and improve program performance, and resources to learn more.
• Centering Equity in Program Evaluation (OPRE Report 2022-211). (PDF) This resource provides an overview of how evaluators can develop an equity approach to their work and apply it across program evaluation phases. The brief complements the “Practice Culturally Responsive and Equitable Evaluation” sections of The Program Manager’s Guide to Evaluation.
• Engaging Community Representation in Program Evaluation (OPRE Report 2022-169). (PDF) This brief provides an introduction to involving representatives of the community in a program evaluation, including why, the benefits, guiding principles, and additional reading.
• Equity Bibliography for Program Evaluation (OPRE Report 2022-178). (PDF) This annotated bibliography supplements the Centering Equity in Program Evaluation resource with materials from the growing body of literature on applying equity principles to research and evaluation.
• Qualitative Approaches to Program Evaluation (OPRE Report 2023-302). (PDF) This brief introduces how to use qualitative research in program evaluations, including what it is, when to use it, and how to combine it with quantitative research for a more robust evaluation. It provides an update to a previous OPRE-funded brief: Qualitative Evaluation.
• Rapid Learning Approaches for Program Improvement and Evaluation (OPRE Report 2023-187) (PDF). This brief provides an overview of different rapid learning approaches, guidance on selecting the potential methods used within these frameworks, considerations for project staff responsible for the design and implementation of rapid learning efforts, and a repository of resources and materials.
• Using Administrative Data in Social Policy Research (OPRE Report 2022-163).  (PDF)This brief provides a definition and examples of administrative data, basics of administrative data (i.e., access and capacity), exploration of individual- and aggregate-level administrative data, using administrative data in visualizations, and principles of equity in administrative data. It provides an update to two previous OPRE-funded briefs: Issues in Accessing and Using Administrative Data and Using Aggregate Administrative Data in Social Policy Research.

Methods

Prior to developing the resources, the team completed an assessment of the strengths and gaps of OPRE materials published on the OPRE Resource Library since 2013. The team categorized the materials in a resource map, identifying the number, recency, and length of resources across different topics and subtopics.

The team then assembled five expert groups to gather feedback on updates to the resources and preliminary topic recommendations: (1) ACF management staff; (2) ACF program staff; (3) intermediary organizations (e.g., TA providers), including staff who work with ACF programs and practitioners; (4) practitioners and researchers, including those who work in an applied setting and have experience developing and using evaluation and TA resources with ACF and related programs; and (5) senior OPRE staff. The groups were consulted on resources that would complement The Program Manager’s Guide to Evaluation, address high-priority topics, and fill gaps of existing OPRE materials.

Final resources were selected to be developed based on several criteria, including that the TA resources should:

• Build on The Program Manager’s Guide to Evaluation by identifying key topics and addressing them in greater depth;
• Develop topics The Program Manager’s Guide to Evaluation does not directly address but that complement its use;
• Address other high-priority TA topics expert groups identified by filling gaps or updating resources;
• Highlight related themes across the TA resources to create an integrated package of products

Citations

Crocker, J., Powell, K., Wagner, H., & Wright, D. (2024). Assessing and improving program performance (OPRE Report 2024-067). Prepared by Westat Insight. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Gutuskey, L. (2022). Centering equity in program evaluation (OPRE Report 2022-211). Insight Policy Research; U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Hyra, A. (2022). Engaging community representation in program evaluation research (OPRE Report 2022-169). Insight Policy Research; U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Kline, N. (2022). Using administrative data in social policy research (OPRE Report 2022-163). Insight Policy Research; U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Mihovich, C., & Welch, R. (2023). Qualitative approaches to program evaluation (OPRE Report No. 2023-302). U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation. 

Steigelman, C., & Gutuskey, L. (2022). Equity bibliography for program evaluation (OPRE Report 2022-178). Insight Policy Research; U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Worden, M., & Crocker, J. (2023). Rapid learning methods for program improvement and evaluation (OPRE Report 2023-187). Insight Policy Research. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.