Introduction
Research Questions
- What is the effect of offering relationship skills education as part of the regular school curriculum?
- How does abbreviating the curriculum influence program effects?
Since the mid-2000s, the Office of Family Assistance (OFA) in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (DHHS) has supported grants to provide healthy marriage and relationship education (HMRE) programming for youth and adults. More than half of grantees receiving HMRE funding since 2011 have offered relationship education to youth. The current grantee cohort, awarded in 2015, includes 47 HMRE programs, 32 of which serve youth in high school and/or community-based settings. To date, ACF has funded three rigorous multisite evaluations involving HMRE grantees, but none have focused on programs for youth. In 2015, the Office of Planning, Research, and Evaluation (OPRE) within ACF contracted with Mathematica Policy Research and its partner, Public Strategies, to conduct the Strengthening Relationship Education and Marriage Services (STREAMS) evaluation. The STREAMS evaluation includes random assignment impact studies and process studies of five HMRE programs funded by OFA in 2015. The evaluations focus on understudied populations and program approaches not covered in OPRE’s prior federal evaluations, including relationship education programs for youth.
This process study report presents findings on the implementation of the Relationship Smarts Plus 3.0 (RS+) curriculum in two high schools in suburban Atlanta, Georgia. More than Conquerors, Inc. (MTCI), a nonprofit social service provider with a long history of delivering HMRE programming with funding from OFA, led the implementation. RS+ is a widely-used relationship education curriculum for youth from 13 to 18 years old. The full curriculum includes 12 lessons designed to help youth better understand themselves, plan for the future, learn the characteristics of healthy relationships, and develop skills to form and maintain healthy relationships. MTCI is implementing two versions of the curriculum: the full 12-lesson curriculum and an 8-lesson summary version.
Purpose
This process study examined MTCI’s implementation of RS+, including program design; processes for hiring, training, and supervising facilitators; service delivery; and youth engagement.
Key Findings and Highlights
Findings from this process study will provide context and help interpret impact evaluation findings. Key highlights include:
- MTCI developed strong systems for hiring, training, and supervising facilitators that emphasized fidelity. Systems included a facilitator hiring process that included a rigorous, multi-day preservice training program on RS+ and facilitation skills, organization-wide emphasis on curriculum adherence, frequent classroom observations, and regular facilitation practice and curriculum review.
- MTCI worked closely with the curriculum distributor to develop the 8-lesson summary version of RS+ and develop an implementation plan. The 8-lesson version excluded four lessons on communication and conflict and sexual decision-making. The curriculum developer created special materials and workbooks for the shortened version and led two staff trainings.
- MTCI facilitators implemented RS+ with fidelity, maintaining intended differences between the full and summary versions and the control group. Facilitators delivered the intended number of class sessions and total hours of instruction for both versions of RS+. In slightly more than 90 percent of class sessions, facilitators reported using all of the intended curriculum materials and following all of instructional guidance in the manual.
Methods
This report is based on analysis of data from the following four sources, collected during MTCI’s first two semesters of delivering RS+ (fall 2016 and spring 2017).
- Semi-structured interviews, focus groups, and observations: In February 2017, the STREAMS evaluation team conducted a three-day site visit to collect qualitative data. Researchers interviewed 14 MTCI and partner staff, including MTCI facilitators, managers, directors, and staff from the two high schools and county health department; observed two RS+ curriculum sessions (one at each school); and held three focus groups with 19 students in total who received either the full or summary version of RS+.
- Staff survey: All eight of the program facilitators delivering RS+ completed a web-based survey in April 2017. The survey asked about their work roles and experiences, feelings towards the program, impressions of the quality of their supervision, training opportunities, and organizational climate.
- nFORM data: nFORM is the client management information system that OFA provided to MTCI and other HMRE grantees. MTCI staff entered data on youth attendance and activities. After class, MTCI facilitators completed a short self-assessment about their ability to deliver that day’s planned content and engage students. The STREAMS evaluation team analyzed nFORM data on 1,098 students who received RS+ during the fall 2016 and spring 2017 semesters.
- STREAMS baseline survey: At the beginning of each semester, youth completed a baseline survey administered during health class. The STREAMS evaluation team analyzed survey responses from 1,085 youth who enrolled and completed the survey in the fall 2016 and spring 2017 semesters.
Citation
Baumgartner, Scott and Heather Zaveri (2018). Implementation of Two Versions of Relationship Smarts Plus in Georgia, OPRE Report # 2018-121, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.