Introduction
Research Questions
- What are the challenges of measuring self-regulation skills in the context of evaluations of employment programs that serve low-income populations?
- What general criteria should evaluators use when selecting measures of self-regulation skills in this context?
- What are the tradeoffs between different approaches for measuring self-regulation skills in this context?
People’s ability to find, keep, and advance in a job depends on self-regulation skills in addition to education, work experience, and technical skills (Almlund et al. 2011). Self-regulation skills include the ability to finish tasks, stay organized, and intentionally control emotions and behaviors. Research has shown that these skills are important in attaining goals and in determining life outcomes, including those related to employment (Almlund et al. 2011). Research has also shown that interventions can both strengthen self-regulation skills and encourage their use (Kautz et al. 2014).
In response to this research, some employment programs, including those offered as part of the Temporary Assistance for Needy Families (TANF) program, use strategies designed to strengthen and boost participants’ use of self-regulation skills (Cavadel et al. 2016; Kautz et al. 2014). To assess the effectiveness of these strategies, evaluators need a way to measure self-regulation skills. However, measuring self-regulation skills for such programs presents unique challenges.
This report discusses the challenges in measuring self-regulation skills and provides guidance on selecting measures to use in evaluations of employment programs for low-income populations. It complements a brief entitled “New Perspectives on Practice: A Guide to Measuring Self-Regulation and Goal-Related Outcomes in Employment Programs,” by Cavadel et al. (2018) that focuses on providing guidance to practitioners by encouraging them to consider self-regulation outcomes and by introducing measures of goal-related and self-regulation skills. This report focuses on providing guidance to researchers on how to measure self-regulation skills in evaluation settings, highlighting ways to address unique challenges that arise in this application.
Purpose
This report discusses issues related to measuring self-regulation skills in evaluations of employment programs for low-income populations.1 First, it presents an overview of self-regulation skills and their importance for employment programs. Second, it introduces approaches to measuring self-regulation skills. Third, it discusses challenges when measuring self-regulation skills in evaluations of employment programs for low-income populations. Fourth, it provides criteria and recommendations for selecting measures.
Key Findings and Highlights
Four challenges arise when measuring self-regulation skills in evaluations of employment programs for low-income populations. First, measures of self-regulation skills can reflect aspects of a person’s situation (for example, his or her background or financial resources) in addition to his or her skills. Second, most existing measures were developed for purposes other than program evaluation, such as describing characteristics of populations generally or for diagnosing people with severe problems. Third, most existing measures were not designed for use with low-income populations. Fourth, some measures take a long time to administer or require special technology.
For use in evaluations of employment programs, we suggest that measures of self-regulation should: (1) relate to employment outcomes of interest; (2) capture skills that could be influenced by the program; (3) account for confounding factors that affect measurement but not skills, and (4) be feasible to administer in an evaluation.
To meet these criteria, we suggest using a set of both general measures of self-regulation as well as ones that are specific to the employment context, collecting information on other aspects of the participants’ situations that can be affected by the program, modifying measures to fit the target population, and conducting analyses to assess the reliability and validity of selected measures.
Methods
The report includes:
• A brief review of self-regulation skills and general approaches to measuring them
• A discussion of key challenges of measuring self-regulation skills in the context of evaluations of employment programs for low-income populations
• A critical analysis of various measurement approaches for this context
• Recommendations for measuring self-regulation skills in this context
Citation
Measuring Self-Regulation Skills in Evaluations of Employment Programs for Low-Income Populations: Challenges and Recommendations. OPRE Report # 2018-83. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services