Introduction
Research Questions
- What challenges did HMRF grantees face in collecting, analyzing, understanding, and using data to monitor, improve, and evaluate their programs?
- Based on the challenges that grantees face, which technical assistance (TA) strategies could build their data capacity? What are the likely audiences, outputs, outcomes, and timing of the recommended TA activities?
Since 2005, Congress has funded $150 million each year in healthy marriage (HM) and responsible fatherhood (RF) grants designed to improve the well-being of children and families. The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services has awarded and overseen three cohorts of these grants (2006–2011, 2011–2015, and 2015–2020). ACF’s Office of Family Assistance (OFA) works with ACF’s Office of Planning, Research, and Evaluation (OPRE) to implement ACF’s learning agenda, which includes conducting research on the best ways to serve families through these grants.
High quality data are essential to supporting ACF’s learning agenda for the HMRF grant program. Data are critical for systematically monitoring and improving programs, evaluating them to learn what works, and building the field’s knowledge base. However, collecting, analyzing, understanding, and using data can be challenging for many grantees and their staff. Frontline staff—who typically focus on serving and interacting with clients—could particularly benefit from training and support related to collecting and using data.
This report is part of the Fatherhood and Marriage Local Evaluation (FaMLE) Cross-Site Project, led by OPRE in collaboration with OFA. ACF has partnered with Mathematica to conduct the FaMLE Cross-Site project.
Purpose
ACF sought to deepen its understanding of the data capacity challenges faced by the 2015 cohort of HMRF grantees. This enhanced understanding could inform a TA approach for supporting future HMRF grantee cohorts. Data capacity is a grantee’s ability to collect, analyze, understand, and use data on its grant programs and data
Key Findings and Highlights
The study team identified the following as the grantees’ main challenges:
- Developing procedures and protocols for complete and accurate data collection, entry, and reporting; and accessing and using HMRF grantees’ required management information system (the Information, Family Outcomes, Reporting, and Management system [nFORM])
- Analyzing data from nFORM or other sources; working with multiple data systems, including nFORM and the grantees’ own data systems; using nFORM’s reporting features; understanding nFORM’s calculations; and prioritizing analyses given the breadth of available data
- Using data for reasons beyond performance monitoring, such as program improvement, continuous quality improvement, sustainability, and evaluation; communicating findings; and telling the story of their programs
- Developing staff buy-in for collecting data, building consensus between grantee staff and evaluators, hiring a skilled team to both deliver services and use data, training a team on the purpose of research and data collection and the associated procedures and protocols, and using research and data from outside the programs
To support future cohorts of HMRF grantees in addressing these challenges, the study team developed a recommended TA approach that (1) uses different modes of TA delivery that escalate in intensity; (2) sequences TA content to progress from foundational to more advanced material; and (3) features frequent communication between TA providers and the federal staff who work with grantees. The recommended TA approach features up to 14 specific TA activities per grant year. This includes one written material and corresponding training webinar per quarter; quarterly peer collaborative meetings; and sessions at grantee conferences and meetings. The team also recommends that one-on-one TA be available for grantees who need more support. A full list of activities is in the Appendix along with associated outputs and outcomes. Although the study team specifies topics for the activities, there is flexibility to cover new topics and issues that arise. A companion report summarizes recommendations for improving data about HMRF grantees. The report focuses on performance measures, which are a key data source for ACF and grantees to monitor performance and progress; functionality of the management information system developed for HMRF grantees, called nFORM (Information, Family Outcomes, Reporting, and Management); and the TA activities to support future grantees in collecting the measures and using nFORM.
Methods
Data collection methods included:
- Five grantee focus groups with a total of 21 grantees
- Three focus groups with other stakeholders: (1) ACF staff who oversee the grantees, (2) programmatic TA providers, and (3) evaluation TA providers
- A session at the 2019 HMRF grantee conference in which grantees identified their data and analysis needs and potential ways to fill them
- A review of grantees’ requests for help with performance measures or nFORM
The study team systematically identified themes across the data sources. The team first assessed the challenges grantees faced in developing their data capacity, then developed a recommended TA approach to match the challenges. The approach was mainly based on solutions that emerged in the process of data collection for this report.
Citation
Friend, Daniel J, Rebecca Kleinman, Megan Hague Angus, Hannah McInerny, Leah Pranschke, and Sarah Avellar (2020). Building data capacity in Healthy Marriage and Responsible Fatherhood grantees: Challenges and recommended support. OPRE Report 2020-95, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.