Introduction
Implementation studies of employment programs typically involve interviews with managers, frontline staff, and partners, and sometimes interviews or focus groups with participants. Although evaluation teams might view some activities, observations of interactions between staff and participants are often limited. Yet for many programs, these interactions are central to the intervention.
This brief discusses the use of video recordings of interactions between program staff and participants as a data source for an implementation study of four employment coaching interventions for people with low incomes. The study team piloted this data collection as part of the Evaluation of Employment Coaching for TANF and Related Populations. The brief describes the process for video recording coaching sessions and analyzing the videos, provides examples of what was learned from the data, and offers lessons for researchers considering using video recordings as a data source in future studies.
Purpose
This brief is intended to inform researchers about how video recordings can be used as a data collection strategy in implementation studies of employment programs and offers lessons for using video data collection in future studies.
Key Findings and Highlights
- Overall, the study team found that collecting and analyzing video data for employment programs is feasible and can be a useful addition to implementation studies.
- Collecting video data with consent from participants can help evaluation teams understand the implementation of interventions that rely on staff—participant interactions.
- Researchers should be sure to video enough staff—participant interactions to support conclusions.
- In addition to using a sampling plan, monitoring how the videos are collected and having multilingual coders can help ensure the recording sample is representative.
- Tailoring the coding form to best capture the unique elements of each intervention being studied and using highly trained coders is important when observing multiple interventions with nuanced program elements.
Methods
In spring 2019, the study team asked each coaching intervention to record 30 coaching sessions with consent of the participants, with the aim of obtaining 15 quality videos for each intervention for analysis. Before site visits, coaches were sent tablets and tripods to record the videos. Then, while study team members were on-site to interview staff, they trained the coaches on how to use the equipment. In order to observe a range of coaches, have multiple recordings from each coach, and limit the burden on any one coach, it was requested that about five coaches participate from each intervention. The coaches were asked to record the appointments they had after the training, if the participant consented. The study team also instructed the coaches to send the videos to the team using a secure file-sharing software and then delete the video from the tablet once the study team confirmed the video had been received.
While watching the videos, coders completed a coding form that was standardized across interventions. After finishing the coding, a researcher input the data into an Excel workbook for analysis. The video recordings were also used as a source of coaching examples for this project’s implementation study.
Citation
Joyce, Kristen (2021). “Using Video Recordings as a Data Collection Strategy: Lessons from the Evaluation of Employment Coaching.” OPRE Report #2021-226. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.