Introduction
In 2014, the Administration for Children and Families (ACF) contracted with Abt Associates and its partner Urban Institute to provide recommendations for evaluating the second round of its Health Profession Opportunity Grants (HPOG 2.0) Program awards and to support the grants and national evaluation. One of the main project activities was to design, implement, and maintain a federally funded management information system (MIS) to support HPOG 2.0. The system, named the Participant Accomplishment and Grant Evaluation System (PAGES), operated on Microsoft Dynamics 365 US Government, a cloud-based customer relationship management (CRM) platform.
PAGES was designed to serve multiple purposes for HPOG 2.0, especially data collection for evaluations, performance reporting, and case management. Using PAGES, HPOG 2.0 grantees recorded participant activities, conducted case management, and tracked participant outcomes. Grantees also used PAGES to monitor their own organizational progress and to analyze data for ways to improve their programs. PAGES enabled ACF to oversee grantees by generating detailed performance progress reports and to produce annual reports. In addition, PAGES collected high-quality data for program evaluation and randomly assigned participants for the experimental Impact Evaluation.
Purpose
This brief describes insights and lessons learned by the HPOG team while creating and operating PAGES. The team had members at ACF, Abt Associates, and Urban Institute—all of whom contributed to this brief, along with the HPOG 2.0 grantees themselves. This brief provides federal agencies and other organizations with recommendations for implementing data systems that support federally funded time-limited grants, demonstration projects, and evaluations similar to HPOG 2.0.
Key Findings and Highlights
The findings described in this brief are organized around four steps for establishing a federally funded MIS. The findings include:
- As the first step, gather and document program needs and system requirements. Dedicate time and resources to define the desired system, request input from system and content-area experts, and review user feedback from prior systems.
- Second, when selecting a platform and vendor, use a competitive selection process if time and resources allow. Assess whether commercial-off-the-shelf (COTS) software will meet the needs of the data system. COTS software provides immediate access to pre-built system features that can be easily modified; however, it introduces risk if combined with customized code. Confirm whether the system meets federal security requirements and seamlessly integrates with other systems or tools users rely on.
- At the third step, develop, launch, stabilize, and maintain the system. Plan for a stabilization period after the system launches as well as unexpected maintenance needs over time. Establish efficient processes and sufficient staffing for both the initial development and ongoing maintenance of the system.
- Finally, deliver user support through multiple channels: training, documentation, and individualized troubleshooting. Ensure that users have access to resources in varied formats and tailor trainings to users’ roles. Calibrate individualized support to the urgency with which users need to access the system.
Methods
This brief describes themes from interviews and written feedback gathered from HPOG 2.0 grantee staff, ACF, Abt Associates, and Urban Institute in the summer of 2021.
Citation
Annie Leiter and Nathan Sick. (2022). Creating a Data System to Manage and Evaluate Grant Programs: Insights from the Participant Accomplishment and Grant Evaluation System. OPRE Report 2022-213. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.