Methodological Advancement

2013-2027

Project Overview 

The mission of the Office of Planning, Research, and Evaluation (OPRE) is to produce rigorous evidence for improving child and family well-being. OPRE’s portfolio has long included work using innovative methodology in social science research. The passage of the Foundations for Evidence-Based Policymaking Act of 2018, also known as the Evidence Act, amplified the need to build awareness of methodological advancements for evaluating social programs and policies and to strengthen the capacity of human services programs to evaluate their effectiveness. In 2013, OPRE launched the Methodological Advancement project to address these needs. 

OPRE Methods Meeting 

The Methods Meeting brings together expertise from various disciplines, policy fields, academia, government, and the private sector to explore innovations in research methodology, study design, analytic techniques, and data measurement to advance the Government’s use of rigorous research methods. These meetings ensure that OPRE‐supported research continues to represent the most scientifically advanced approaches to determining effectiveness and efficiency of ACF programs. The meetings also build evaluation capacity more broadly. The OPRE Methods Meeting typically takes place each year in the fall. 

The contractor that facilitated 2014-2016 methods meetings was RTI International. The contractor that facilitated 2017 to 2023 methods meetings was Insight Policy Research. MEF Associates will facilitate methods meetings for 2024-2026. 

Visit the OPRE Innovative Methods Meetings website  to access resources and materials from previous meetings. 

Point(s) of contact: Kristyn Wong VanDahm & Paula Daneri 

Related Resources

Learn about the opportunities and challenges of probability and non-probability samples in survey research.

In this session, Dr. Nishimura described weighting, imputation, and other analytic strategies researchers can use to minimize the impact of unit missingness—the failure to obtain survey information from an intended respondent—on survey data.

Find resources to learn more about survey unit missigness topics, including the potential for increased nonresponse bias and the implications of missing data for human services research.

On October 28—29, 2020, the Administration for Children and Families’ Office of Planning, Research, and Evaluation convened a virtual meeting for participants from Federal agencies, research firms, academia, and other organizations to discuss core components approaches.

This summary document highlights key themes and presentations from the virtual meeting.

In this video, Dr. Katherine Corker of Grand Valley State University defines pre-registration and outlines how and why to pre-register a study. Dr. Corker gave this presentation at OPRE’s 2019 Methods Meeting, Methods for Promoting Open Science in Social Policy Research.

In this video roundtable, government experts and experienced researchers discuss the opportunities and challenges presented when using administrative data for social policy research. Topics include: tips for planning administrative data research; working with (federal and state) data custodians; negotiating data...

This brief has two main goals:

  • Describe the features of a well-designed and implemented subgroup analysis that uses a multiple regression framework.
  • Provide an overview of recent methodological developments and alternative approaches to conducting subgroup analyses.

The brief builds on a 2009 meeting of experts convened by the Administration for Children and Families’ Office of Planning, Research, and Evaluation and a corresponding 2012 publication in a special issue of Prevention Science (MacKinnon, Supplee, Kelly, & Barofsky, 2012).

“Open science” represents a broad movement to make all phases of research—from design to dissemination—more transparent and accessible. The scientific community and Federal agencies that support research have a growing interest in open science methods. In part this interest stems from highly publicized news stories and journal articles that cast doubt on research credibility...

 

“Open science” represents a broad movement to make all phases of research—from design to dissemination—more transparent and accessible. The scientific community and Federal agencies that support research have a growing interest in open science methods in response to highly publicized news stories and journal articles that cast doubt on research credibility...

Social service program stakeholders need timely evidence to inform ongoing program decisions. Rapid learning methods, defined here as a set of approaches designed to quickly and/or iteratively test program improvements and evaluate program implementation or impact, can help inform such decisions. However, stakeholders may be unsure which rapid learning methods are most appropriate for a program’s specific challenges and how to best apply the methods...

Rapid learning methods aim to expedite program improvement and enhance program effectiveness. They use data to test implementation and improvement efforts in as close to real-time as possible. Many rapid learning methods leverage iterative cycles of learning, in which evaluators and implementers (and sometimes funders/policymakers) discuss findings, interpret them, and make adaptations to practice and measurement together. These methods can support data-driven decision-making in practice, in the spirit of ongoing improvement.

On October 25 and 26, 2018, OPRE brought together a diverse group of participants from Federal agencies, research firms, academia, and other organizations for a meeting titled, Rapid Learning Methods for Testing and Evaluating Change in Social Programs . This brief is based on a presentation at the meeting.

Social service program stakeholders need timely evidence to inform ongoing program decisions. Rapid learning methods, defined here as a set of approaches designed to quickly and/or iteratively test program improvements and evaluate program implementation or impact, can help inform such decisions. However, stakeholders may be unsure which rapid learning methods are most appropriate for a program’s specific challenges and how to best apply the methods. Additionally, they may be unsure how to cultivate a culture of continuous, iterative learning.

Bayesian methods are emerging as the primary alternative to the conventional frequentist approach to statistical inference. This brief provides an overview of the Bayesian perspective and highlights potential advantages of Bayesian inference over frequentist inference. In light of the increasing value and viability of Bayesian methods to contemporary...

For nearly 100 years, the null hypothesis significance testing (NHST) framework has been used to determine which findings are meaningful (Fisher 1925; Neyman and Pearson 1933). Under this framework, findings deemed meaningful are called “statistically significant.”  But the meaning of statistical significance is often...

Federally funded systematic reviews of research evidence play a central role in efforts to base policy decisions on evidence. Historically, evidence reviews have reserved the highest ratings of quality for studies that employ experimental designs, namely randomized control trials (RCTs). However, RCTs are not appropriate for evaluating all intervention programs. To develop an evidence base for those programs, evaluators may need to use non-experimental study designs.

There is a growing understanding that there are some inherent limitations in using p-values to guide decisions about programs and policies. Bayesian methods are emerging as the primary alternative to p-values and offer a number of advantages...

Probability (p) values are widely used in social science research and evaluation to guide decisions on program and policy changes. However, they have some inherent limitations, sometimes leading to misuse, misinterpretation, or misinformed decisions. Bayesian methods...

In the fall of 2016, OPRE brought together a diverse group of participants from federal agencies, research firms, foundations, and academia to discuss alternatives to randomized controlled trials and their assumptions, trade-offs, benefits, and challenges.

Policymakers and practitioners have a growing interest in answering questions beyond simply “does a program work?” They are also interested in learning how programs work. Mediation analysis is one tool that researchers can use to identify elements of an intervention that do, or do not, lead to improved participant outcomes. Researchers can use the results of a mediation analysis to build knowledge to improve programs...

Policymakers are increasingly interested in using administrative data to address policy-relevant research questions. While researchers generally prefer individual-level administrative data in order to provide maximum flexibility to their analyses, it can be both difficult and costly to obtain. When individual-level data are not available or are too difficult or costly, aggregate administrative data can address many policy-relevant research questions...

Policymakers are increasingly interested in using administrative data to address policy-relevant research questions. In order to make use of administrative data for social policy research, there are multiple issues to consider...

Administrative data have the potential to help us answer pressing social policy questions. Government stakeholders and researchers are exploring the promises of using administrative data for research purposes.

This brief summarizes an Innovative Methods Meeting that was organized by OPRE in the fall of 2015 that considered the potential benefits and pitfalls of using administrative data for research purposes...

In the fall of 2014, OPRE organized an Innovative Methods meeting to explore cutting-edge applications of methods and analytic techniques that can inform social program practice and policy. This brief summarizes the meeting and includes..

In this video, Dr. Katherine Corker of Grand Valley State University defines pre-registration and outlines how and why to pre-register a study. Dr. Corker gave this presentation at OPRE’s 2019 Methods Meeting, Methods for Promoting Open Science in Social Policy Research.