Download BriefDownload Report PDF (3,024.00 KB)
- File Size: 3,024.00 KB
- Pages: 87
- Published: 2021
- What was the reliability (consistency of ratings across different observers) of the draft observation tool?
- What was the concurrent validity of the draft observation tool (agreement between observers’ ratings and educators’ self-assessments)?
- Is a single training sufficient to build observers’ knowledge of self- and co-regulation behaviors and become familiar with the draft observation tool and its procedures?
- How might the items in the draft observation tool and observation procedures be improved?
Youth need support to process emotions, cope with stress, and for self-regulation—managing thoughts and feelings to achieve goals and make healthy decisions in the moment and for the future. Caring adults such as parents, guardians, teachers, and coaches support the development of self-regulation skills from infancy through young adulthood through a process called co-regulation. This brief describes a draft observation tool that was developed and piloted as part of a formative study to translate theory about co-regulation into practice in youth-serving Healthy Marriage and Relationship Education (HMRE) programs.
The Administration for Children and Families (ACF) sponsored the Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Programs for Youth (SARHM) project to bring more focus to self-regulation development in programs for youth. Specifically, SARHM’s aim was to build on developmental psychology and prevention research on the adult role in youth self-regulation development to create resources for educators in Healthy Marriage and Relationship Education (HMRE) programs for youth (defined as ages 14 to 24).
SARHM is a formative research project to develop and test the integration of co-regulation approaches in two youth-serving HMRE programs. It is funded by the Office of Family Assistance (OFA) and overseen by the Office of Planning, Research, and Evaluation (OPRE). SARHM focused on youth and young adults aged 14 to 24 because this is a time of rapid brain change and development. It also is a time of many transitions, risks, and opportunities.
Adult support is critical for helping young people make healthy decisions, engage in prosocial behaviors, and prepare for the future. Although many educators are likely practicing elements of co-regulation in their interactions with youth, educators’ influence may be more powerful when they are trained to apply specific co-regulation strategies in a group session. To ensure proper implementation, it is important for programs to have a standardized way to assess educators’ application of co-regulation strategies. Observations provide insight into how educators engage with youth and the extent to which they recognize and act on opportunities to support youth self-regulation. Observations also provide a picture of the group environment and signal to educators that program supervisors and managers are invested in how the program is delivered and that co-regulation is important for strong facilitation.
The tool was designed as a part of SARHM to measure educators’ co-regulation during group sessions, including their use of specific, theory-based co-regulation strategies. By sharing the draft observation tool and our recommendations for next steps, this brief seeks to advance emerging lessons about the importance of co-regulation for youth-serving programs.
Key Findings and Highlights
The SARHM team found that:
- Observers demonstrated moderate to substantial levels of agreement in their ratings. For some items where there was less agreement, a revised tool should include clearer guidance on the time it took the educator to get a class back on track after disruptions.
- Observers’ reports and educators’ self-reports were weakly correlated. Additional training may be beneficial, though differences between educators’ and observers’ reports may also reflect the fact that educators did not always report on the same workshops that were observed.
- The observation tool made observers more aware of the use of co-regulation in their programs. They found the initial two-day training helpful, but wanted more opportunities to practice using the tool.
- Observers reported that some of the tool’s procedures were challenging, particularly related to the timing of different sections and the number of behaviors they had to track. A revised tool could be simplified and improved by reducing the number of behaviors to rate or the amount of time spent observing educators.
- Observers had concerns about the cultural relevance of some items, such as recording how frequently educators provided direct and individualized praise to youth. A revised tool could allow communities or programs to make culturally-appropriate adaptations to the measure.
The SARHM observation measure pilot occurred in three phases:
Phase 1 — We developed a draft tool, informed by three main activities. First, we conducted a review of key ACF publications to develop a co-regulation conceptual framework that depicts the relationship between co-regulation and youth self-regulation development. Second, informed by this framework, we conducted literature reviews to identify: (1) supportive strategies that youth-serving HMRE practitioners could use to build co-regulation into their programs, and (2) existing measures of self- and co-regulation. Third, we consulted with an expert panel on the structure of the tool and the items it contained.
Phase 2 — We partnered with two youth-serving HMRE programs: a program serving a population in suburban high schools and a community-based organization serving young adults who were formerly in foster care. Staff at each site were trained to pilot a set of co-regulation strategies. They then used a draft of the observation tool to measure use of the strategies and provided feedback on the strategies and the feasibility of using the tool. We used the feedback to adapt the tool for programs whose educators had not been trained on co-regulation strategies.
Phase 3 — We conducted pilot tests of the observation tool with three youth-serving HMRE programs. The pilot tests included a two-day training for observers about self- and co-regulation behaviors and observation tool procedures. We conducted a total of three learning cycles per site in which we analyzed feedback, refined the strategies, and tested them again. Observers and educators in these programs completed questionnaires to assess their knowledge and beliefs about youth self-regulation and their own self- and co-regulation behaviors at the beginning of the pilot period. Educators also completed a weekly self-assessment form about their use of co-regulation in workshops.
Next steps for developing and testing co-regulation strategies include:
- Revise challenging items so they can be interpreted and answered reliably, while still gathering detailed information about behaviors. When making revisions, researchers should also consider the cultural relevance of the items for the context in which they plan to use the tool.
- Simplify observation procedures and instructions with input from program staff.
- Continue and expand pilot testing to determine whether the observation tool measures what it is supposed to measure in varied settings. After additional pilot testing, a next step would be to conduct a field test with more youth-serving programs to further assess the tool’s reliability and validity.
- Enhance training for observers with more opportunities for practice and a certification process. This can improve the reliability of the tool.
Through the process of developing and pilot testing the observation tool, we also identified several lessons about how to develop observational measures and conduct observations of educators’ behaviors:
- Identify a small set of behaviors to measure so that observers will be able to accurately rate them.
- Define observable behaviors with examples, so that observers can rate them consistently over time.
- Ensure observers agree on what counts as evidence of a behavior so they use the observation tool in the same way.
- Practice observing together to refine standard observation procedures, and keep a record of decisions about how best to rate behaviors.
Alamillo, J., Baumgartner, S., Frei, A., & Herman-Stahl, M. (2021). Measuring Co-Regulation: A Draft Tool for Observing Educators in Youth Serving Programs, OPRE Report #2021-09. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
- The act of managing thoughts and feelings to enable goal-directed behavior.
- The supportive process between an adult and a young person that promotes self-regulation. Co-regulation integrates three key types of support: (1) providing warm, responsive relationships; (2) helping youth find and create supportive environments; and (3) coaching and modeling self-regulation skills.
- Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Programs for Youth
- Healthy Marriage and Relationship Education