Research Questions
- COVID-Era Applicants: What are short-term outcomes (for the treatment group) and impacts, overall and by subgroups, for study members enrolled during COVID (May 2020 through September 2021)?
- Cross-Cohort Comparisons: At five quarters after study entry, how do outcomes and impacts for COVID-era applicants compare with applicants from the pre-COVID period, overall and by subgroups?
- COVID Era: Across a range of follow-up intervals, how do outcomes and impacts vary in the COVID era relative to outcomes and impacts in the pre-COVID era, for a given follow-up interval?
- COVID Severity: How do outcomes and impacts, overall and by subgroup, vary with proxies for the severity of COVID, across place and time?
Purpose
The purpose of this report is to describe a plan for answering the HPOG COVID-Cohort Study’s key research questions. The document operationalizes key outcome measures, briefly describes key sources of data for the study, and provides details of the methods that will be used. By specifying these details in advance, this document serves as a public commitment to the planned analysis.
Key Findings and Highlights
- The study’s estimation of impact will build on random assignment conducted by all 27 non-Tribal HPOG 2.0 grantees.
- The document describes the outcomes that will be considered; how these outcomes will be measured; and how impacts on multiple outcomes will be prioritized in the summary of program effectiveness.
- The document describes the econometric models that will be used to explore how outcomes and impacts vary with the local severity of COVID.
- The document also provides a draft outline for the corresponding report, to be titled COVID and the Impact of HPOG 2.0.
Methods
Eligible applicants to local HPOG 2.0 programs were assigned randomly either to a treatment group that had access to HPOG 2.0 training and services or to a control group that did not have access to HPOG during the study period but did have access to other training and services in the community. Building on this random assignment, the report will estimate impacts with software that controls for baseline differences between the treatment and control groups, gives more influence to programs that served larger numbers of students, and captures the extra uncertainty due to variation in design and implementation across programs.
Most of the analyses build on a pair of surveys conducted 15 months after study entry: the Short-Term Follow-Up Survey (STS) surveying some of those who entered the study in 2017 and 2018, well before COVID, about their experiences in a period before COVID; and the COVID-Cohort Follow-Up Survey (CCS) surveying those who entered the study during COVID about their experiences after the outbreak of COVID. Through 15 months after study entry, this pair of surveys support analyses of outcomes and impacts among COVID-era applicants (RQ1) and changes in outcomes and in impacts between the pre-COVID- and COVID-era applicants (RQ2).
Other analyses use administrative data to compare pre-COVID and COVID outcomes and impacts at longer follow-up (RQ3). Finally, a set of regression models explore how outcomes and impacts vary with the severity of COVID, while holding constant end-of-grant effects—which otherwise limit the evaluation’s ability to attribute pre-COVID/COVID differences to COVID (RQ4).
Citation
Judkins, David Ross, Jacob Alex Klerman, and Radha Roy. (2023). Analysis Plan for the Health Profession Opportunity Grants (HPOG 2.0) COVID-Cohort Impact Report. OPRE Report 2023-048. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- ACF:
- Administration for Children and Families
- OPRE:
- Office of Planning, Research, and Evaluation
- HPOG:
- Health Profession Opportunity Grants
- IEDP:
- Impact Evaluation Design Plan
- STIR:
- Short-Term Impact Report
- ITIR:
- Intermediate-Term Impact Report
- CCIR:
- COVID-Cohort Impact Report