Descriptive Evaluation Design Report for the National Evaluation

March 23, 2018
Topics:
Self-Sufficiency, Welfare & Employment
Projects:
Health Profession Opportunity Grants (HPOG) Evaluation Portfolio | Learn more about this project, National Evaluation of the 2nd Generation of Health Profession Opportunity Grants (HPOG 2.0 National Evaluation), 2015-2025 | Learn more about this project
Types:
Reports
Descriptive Evaluation Design Report for the National  Evaluation Cover
Download report (pdf)
  • File Size 887kb
  • Pages 46
  • Published 2018

Introduction

The Health Profession Opportunity Grants (HPOG) Program is designed to deliver high-quality training in the health professions to eligible individuals. A National Evaluation of 27 grants awarded in 2015 as part of the second round of HPOG grants (HPOG 2.0) is currently underway. The National Evaluation will include a Descriptive Evaluation of the implementation, outcomes, and local service delivery systems of the grants as well as an Impact Evaluation of the grants’ impacts on participants and the HPOG Program’s costs and benefits.

This report presents a research design plan for the Descriptive Evaluation, which includes three related studies: (i) the Implementation Study, (ii) the Outcome Study, and (iii) the Systems Study. Each of these studies makes important independent contributions to the National Evaluation:

  • The Implementation Study will describe HPOG 2.0 Program design and implementation, including Program context, administration, costs, education and support services, and employment assistance services.
  • The Outcome Study will describe HPOG 2.0 participant characteristics, program experiences, and educational and employment outcomes.
  • The Systems Study will describe how local service delivery systems (i.e., community resources, funding availability, economic conditions, and policies) may have influenced HPOG program design and implementation and how HPOG implementation may have influenced these local systems.

Research Questions

  1. 1 How is HPOG 2.0 designed and implemented and what innovative and/or promising strategies have programs introduced in the following focus areas: employer engagement, providing training in the career pathways framework, providing work-readiness training, and providing for program sustainability after grants end?
  2. 2 What are the characteristics of HPOG 2.0 participants; at what rate do they engage in program activities, training courses, and support services; and what are their education, employment, and earnings outcomes?
  3. 3 What are the local service delivery systems in which HPOG programs operate and what was the mutual influence of the HPOG programs and the local service delivery systems and vice versa?

Purpose

This design report presents detailed plans for the Descriptive Evaluation of HPOG 2.0. It specifies a logic model comprising the domains and constructs and the hypothesized relationships among them that characterize the theory of change underlying the HPOG 2.0 Program. The document then specifies the research questions and the data sources and collection strategies needed to describe how the logic model is realized. The report closes with an account of the future products of the Descriptive Evaluation, including an analysis plan, a report, and a series of case studies focusing on topics of particular policy and practice interest.

Key Findings and Highlights

The Descriptive Evaluation includes three related studies (i) the Implementation Study, (ii) the Outcome Study, and (iii) the Systems Study. Each of these studies makes important, independent contributions to the National Evaluation:

  • The Implementation Study will describe HPOG 2.0 Program design and implementation, including Program contexts, administration, costs, education and support services, and employment assistance services.
  • The Outcome Study will describe HPOG 2.0 participant characteristics, program experiences, and educational and employment outcomes.
  • The Systems Study will describe how local service delivery systems (i.e., community resources, funding availability, economic conditions, and policies) may have influenced HPOG program design and implementation and how HPOG implementation may have influenced these local systems.

Methods

The Descriptive Evaluation will use multiple sources of primary and secondary data. The primary data sources are:

  • Program administrative data collected in the Participant Accomplishment and Grant Evaluation System (PAGES), a management information system developed for the HPOG 2.0 Program and Evaluation.
  • Two rounds of telephone interviews with HPOG 2.0 grantee and program representatives and key partners that provide training courses or support services.
  • Site visits to up to ten programs with promising or innovative strategies for the five focus areas of interest noted in the first research question above.
  • In-depth interviews with program participants.

The secondary data sources for the Descriptive Evaluation are:

  • Evaluation Design and Implementation Plans (EDIPs), which describe in detail how the Impact Evaluation will be integrated into each local HPOG 2.0 program’s operations. EDIPs also include summary information about program operations, training courses, and services, as well as control conditions.
  • Site monitoring reports prepared by the research team based on their ongoing contact with grantee staff, which describes program changes and issues.
  • Grantee applications, which provide some institutional background as well as the grantee’s objectives and rationale for a grant award.
  • Performance Progress Reports (PPRs), which compare outcomes against quantitative performance goals, and provide grantee narrative descriptions of their programs.
  • U.S. Bureau of Labor Statistics data on healthcare employment and wages, which will be used for information on local labor markets in the healthcare industry.
  • National Directory of New Hires (NDNH) federal administrative data on employment and earnings.

Results from the Descriptive Evaluation will be shared through a variety of reports, briefs, and presentations.

Citation

Werner, Alan, Robin Koralek, Pamela Loprest, Lauren Eyster, and Gretchen Locke. (2018). National and Tribal Evaluation of the 2nd Generation of Health Profession Opportunity Grants (HPOG 2.0): Descriptive Evaluation Design Report for the National Evaluation, OPRE Report # 2018-07. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Glossary

Career Pathways
a framework for occupational training that combines education, training, and support services that align with the skill demands of local economies and help individuals to enter or advance within a specific occupation or occupational cluster
HPOG, HPOG Program
the national Health Profession Opportunity Grants initiative, including all grantees and programs
HPOG Grantee
the entity receiving the HPOG grant and responsible for funding and overseeing one or more local programs
HPOG Program
a unique set of services, training courses, and personnel; a single grantee may fund one or more programs
HPOG Partners
other organizations directly involved in the operations of an HPOG program
HPOG program participants
enrollees in education and training programs and related services supported by HPOG grants
Contextual factors, or “system”
the economic and service delivery environment in which an HPOG program operates
Outcomes
end goals for HPOG, including employment and earnings in general and in healthcare specifically
Last Reviewed: October 10, 2019