Introduction
Many federal agency research staff rely on surveys as part of their research and evaluation efforts to understand how federal programs are serving participants. Although federal staff often recognize that information from surveys should be collected in ways that are accessible, respectful, and responsive to community context, federal staff and their contractors may not always have a fully informed perspective on how to achieve these goals.
In September 2021, OPRE contracted with the Urban Institute to conduct the Advancing Contextual Analysis and Methods of Participant Engagement (CAMPE) project. The goal of the CAMPE project was to aid OPRE in better understanding, incorporating, and advancing equitable research practices in OPRE projects. As part of this project, the team convened a 13-member community advisory board (CAB) to review and comment on multiple documents and processes used in the conduct of OPRE research and evaluation projects. One OPRE process that the CAB explored was survey development and administration.
Purpose
This report focuses on the CAB’s review of an excerpt of a sample survey from an OPRE-funded research study. The OPRE-funded study used the sample survey, along with other data, to evaluate the effectiveness of programs that seek to help adults navigate employment challenges and achieve economic stability. This report details the CAB’s recommendations on developing and administering surveys in ways that reflect an understanding of and respect for survey participants. The report offers specific recommendations for person-centered survey development and administration, developing surveys with community engagement, providing incentives to survey participants, and maximizing survey recruitment and retention. Operationalizing the CAB’s recommendations may require additional work on the government’s part.
Key Findings and Highlights
- Survey development and administration processes should reflect an understanding of and respect for survey participants.
- Person-centered approaches to survey development and administration can help build trust between participants and researchers, and in turn, improve the quality of data collected.
- Person-centered approaches to surveys include demonstrating that researchers care about the person taking the survey, preparing for questions and concerns participants might have, using trauma-informed and culturally sensitive terms, prioritizing survey access, being thoughtful about survey delivery, and engaging with the community to develop and administer the survey.
- Community engagement is a powerful tool to increase the likelihood of a survey being person-centered.
- Re-imagine incentives as compensation, rather than a thank you.
Methods
The goals of the CAB’s engagement on survey development and administration were to provide general feedback on survey processes and specific feedback on select survey measures that focused on work, compensation and benefits, barriers to work, and education and training program outcomes. OPRE provided the CAB with an example baseline and follow-up survey from a previously completed OPRE project. The CAB reviewed three sections: employment status and challenges, employment and earnings, and education and training programs.
To ensure that CAB members had the necessary foundation to provide relevant and actionable feedback, the project team structured the engagement to include multiple trainings related to survey development and administration. The trainings included two-hour sessions with the CAB, post-session homework, and small group meetings with the project team supporting the CAB. The sessions were scheduled on a bi-weekly basis at various times to best accommodate CAB members’ schedules across four different time zones. Over the course of five sessions, the CAB members received training in the following topics: introduction to survey methodologies; community-engaged survey creation; and survey development and administration in multisite studies.
After the trainings, the CAB members and the project team discussed what they were learning. In addition to this discussion-based learning, CAB members piloted the example OPRE survey. Before piloting the survey, the project team updated the baseline and follow-up survey sections to incorporate recommendations from the CAB. CAB members then piloted the survey either by engaging someone from their personal or professional networks or by partnering with a project team member.
Citation
Farrell, Lauren, Mikaela Tajo, Elsa Falkenburger, and Soumita Bose (2024). Strengthening Federal Surveys through Person-Centered Development and Administration. OPRE Report #2024-280. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- CAB:
- Community advisory board. CAMPE is directly engaging people with lived experience by convening a Community Advisory Board (CAB). The term “CAB” is commonly used in research and evaluation practice and does not refer to a Federal Advisory Committee; OPRE will not seek consensus advice from the CAB, but rather will seek a variety of perspectives to inform the project.
- ACF:
- Administration for Children and Families
- OPRE:
- Office of Planning, Research and Evaluation
- CAMPE:
- Advancing Contextual Analysis and Methods of Participant Engagement