Download ReportDownload Report PDF (1,847.85 KB)
- File Size: 1,847.85 KB
- Pages: N/A
- Published: 2020
- What changes to the HMRF performance measures would improve the quality, utility, and clarity of the information collected?
- What nFORM changes and enhancements would improve the user experience, access to data and reports, and data quality?
- What training and technical assistance (TA) activities would help grantees use nFORM and collect high quality performance measures data?
Since 2006, the Office of Family Assistance (OFA), within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services, has awarded and overseen federal funding for three cohorts of healthy marriage (HM) and responsible fatherhood (RF) grant programs (2006–2011, 2011–2015, and 2015–2020). HM grantees promote HM and relationships through eight legislatively authorized activities, such as marriage and relationship education. RF grantees’ legislatively authorized activities promote responsible parenting, HM, and economic stability.
OFA works with ACF’s Office of Planning, Research, and Evaluation (OPRE) to conduct research on how to best serve families through these grants. As part of ACF’s learning agenda to increase understanding of HMRF programs and to support program monitoring and improvement, ACF requires grantees to submit performance measures data collected from people served by the program (clients) and program staff. For the 2015 grantee cohort, ACF approved a common set of performance measures on characteristics and outcomes of clients, services provided, and program operations. Grantees submit data via a web-based data collection system called Information, Family Outcomes, Reporting, and Management (nFORM).
To better understand what worked well and identify improvements for the HMRF grantees’ performance measures and data collection, ACF conducted a review to identify potential changes to the performance measures, nFORM functionality, and data collection support activities. This report describes the process to review the existing system and identify improvements and describes the planned changes for the next cohort of HMRF grantees, which ACF expects to start in 2020.
This work is part of the Fatherhood and Marriage Local Evaluation (FaMLE) Cross-Site Project, led by OPRE in collaboration with OFA. ACF has partnered with Mathematica to conduct the FaMLE Cross-Site project.
To prepare for the 2020 cohort of HMRF grantees, ACF conducted a review to identify potential changes to the performance measures, nFORM functionality, and activities that support data collection. ACF aims to expand and improve on what it learned about the 2015 cohort of grantees by refining data collection. High quality data about grantees’ operations and clients is essential for ACF’s monitoring of grantees’ performance; helps grantees track and improve their program implementation; and can inform future program services, development, and investments.
Key Findings and Highlights
Performance measures and reports
Based on its review, ACF plans to change the existing data collection instruments in five ways:
- Clarify and align question or item wording with programs and clients’ experiences.
- Modify response categories to address cases in which survey items did not fully capture clients’ experiences.
- Replace some measures with alternatives suggested by stakeholders.
- Drop unnecessary items deemed less relevant or useful to ACF, grantees, and other stakeholders.
- Add new questions or items to capture information useful to grantees and/or ACF.
The report includes the modified instruments. These are draft instruments that ACF will submit to OMB for review and approval prior to collecting data. As part of that process ACF will publish a Federal Register Notice inviting public comment on the instruments and measures. Thus, the instruments in these appendices might change based on the feedback received during the OMB process.
ACF plans for multiple enhancements to nFORM based on feedback from stakeholders and best practices in survey methodology, data collection, and designing and implementing management information systems. These enhancements aim to support accurate, complete, and timely data and improve the user experience while adhering to security requirements for the system.
nFORM TA and support
As with the 2015 cohort of HMRF grantees, ACF plans to provide individualized and group-based TA, as well as written and digital resources, to the 2020 cohort on collecting data and using nFORM. ACF has specific plans to provide the TA and resources:
- Individualized TA. ACF plans to retain the same overall strategy from the 2015 cohort for the 2020 cohort, featuring a help desk that responds to grantees’ questions by email or phone.
- Written and digital resources. ACF plans to provide the 2020 cohort with a range of written and digital resources that describe nFORM functionality and best practices for using the system, including a data dictionary, email tips, and trainings videos among other resources.
- Group-based TA. ACF plans to continue webinars and other trainings. It also plans to continue office hours as a core part of nFORM TA. Office hours include tutorials and time to answer participating grantees’ questions.
A separate report about building HMRF grantees’ data capacity to collect and use data will provide additional information on planned TA and support.
Several information sources informed the potential changes to the performance measures, nFORM, and training and TA activities:
- Targeted analysis of existing measures. To examine the quality of the current measures, the FaMLE Cross-Site team analyzed performance measures survey data from the 2015 cohort of HMRF grantees. The targeted analysis examined (1) item-missing rates, (2) ceiling or floor issues (when a high proportion of respondents selected the maximum or minimum response or score for an item), (3) variation in responses, (4) whether respondents used open-text fields and provided usable data, and (5) outlier or illogical responses.
- Public comments. In late 2019, ACF solicited public comments on the performance measures, survey instruments, and nFORM. ACF invited feedback related to the quality, utility, and clarity of the information collected and ways to reduce the volume or burden of the data collection for the next cohort of grantees. ACF received 57 comments.
- Listening sessions with stakeholders. The FaMLE Cross-Site team held six listening sessions by phone to hear feedback directly from grantee staff in various roles (five sessions) and OFA federal staff (one session). Each session focused on a specific topic and included staff whose roles gave them experience in the topic.
- Discussions with grantee-affiliated experts. To explore measurement issues in more depth, the FaMLE Cross-Site team obtained expert input through two group calls—each with three to five experts in HM or RF—on improvements to the performance measures, continuous quality improvement, and/or evaluation.
- Pre-tests of measures. The FaMLE Cross-Site team conducted pre-test interviews to assess whether clients understood and interpreted the new and revised measures for the client surveys as intended. Pre-test staff completed 19 in-person interviews with clients from two current HMRF programs.
The FaMLE Cross-Site team systematically reviewed the feedback and developed recommendations for ACF. ACF and the project team discussed various criteria for changes to the performance measures, such as alignment with HMRF policies and program priorities, data completeness and quality, respondent burden, and survey best practices. The report reflects ACF’s decisions on these items at the time of the report’s release.
Strong, Debra, Mathew Stange, Grace Roemer, Sarah Avellar, and Brianna Noonan (2020). Supporting Program Progress: Performance Measures, Data System, and Technical Assistance for the 2020 Healthy Marriage and Responsible Fatherhood Grantees, OPRE Report #2020-64, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.