Testimony from Naomi Goldstein on OPRE Studies

Publication Date: January 10, 2014

Naomi Goldstein, Ph.D.
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services

Before the

Committee on Energy and Commerce
Subcommittee on Health
U.S. House of Representatives

January 9, 2014

Chairman Pitts, Ranking Member Pallone, and Members of the Subcommittee, thank you for the opportunity to testify today.  My name is Naomi Goldstein, and I have served since 2004 as Director of the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF), Department of Health and Human Services (HHS). 

ACF promotes the economic and social well-being of families, children, individuals and communities through a broad array of programs carried out in partnership with states, territories and tribes, with other federal agencies, and with community-based organizations and local governments.  I am pleased to share with you today information about the activities of a few of these programs, and what we are learning from them.

ACF’s Office of Planning, Research and Evaluation studies ACF programs and the populations they serve.  OPRE conducts its work primarily through competitively awarded grants and contracts for research and evaluation projects.  We aim to make our work both rigorous and relevant, and to disseminate it in ways that are useful for policy-makers and practitioners.

ACF appreciates your interest in our work, and welcomes the opportunity to discuss with you the Health Profession Opportunity Grants, Abstinence Education, and Personal Responsibility Education Programs.  I will also speak about ACF’s collaboration with the Health Resources and Services Administration (HRSA) on the evaluation of the Maternal, Infant and Early Childhood Home Visiting Program.

Health Profession Opportunity Grant Program

The Health Profession Opportunity Grants (HPOG) Program funds training in high-demand healthcare professions targeted to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals.  The program is designed to meet the demand for healthcare workers in communities and improve the job prospects for adults from hard-working families, matching careers in a growing field with people who are eager to fill them.

The program was established by the Affordable Care Act (ACA).  In 2010, ACF awarded five-year funding to 32 grantees in 23 states.  Five of the grantees are tribal organizations.  ACF provides approximately $67 million annually to these grantees.  HPOG grantees are post-secondary educational institutions; workforce investment boards (WIBs), state and local government agencies, and community-based organizations.  Grantees coordinate services with state and local WIBs, state and local TANF agencies and federal and state offices of Apprenticeship, among other partners.
Career pathways (CP) programs have developed over the past decade as a comprehensive framework of adult developmental and vocational education and supportive services designed to address the challenge of providing post-secondary skills training to low-income and educationally disadvantaged populations.  This framework builds on past research showing that similar programs can improve employment and earnings. , , , ,

As of December 2013, approximately 25,800 participants have enrolled in HPOG programs.  Of the more than 12,000 participants who have completed an occupational or vocational training program, more than 10,000 participants have become employed since the program began.  Among those who became employed, their average wage is $12.37 per hour.    

The majority of HPOG participants were single females at program entry, with one or more dependent children.  While most were not TANF recipients at enrollment, most had a household income of less than $20,000 when starting the program, and almost two-thirds received some form of public assistance at program intake.  The most common training among participants is preparation to become a nursing assistant, aide, orderly, or patient care attendant, generally short training courses that can be the first step in a longer career pathway.  Other common trainings included instruction to be a licensed or vocational nurse, registered nurse, and medical assistant.  HPOG participants also engaged in pre-training college study skills and basic skills education classes.  Grantees provide a variety of support services including case management and counseling services; financial assistance for tuition, books, and fees; and social service supports, including assistance with transportation, child care and emergency assistance.  Grantees also provide employment assistance in the form of job search workshops, career coaches, and placement and retention assistance. 

ACF is using a multi-pronged research and evaluation strategy to examine outcomes and impacts for HPOG participants as well as program implementation and systems change resulting from HPOG programs.  The HPOG impact evaluation uses a rigorous random assignment design that will show how variations in program services affect program impacts.  The HPOG impact evaluation report, to be released in 2016, will report on education impacts such as credential attainment and impacts on employment and earnings as well as job quality for participants   15 months after program entry. 

All HPOG grantees are participating in a companion study on program implementation, systems change, and outcomes.  ACF will complete and release interim reports from this study in 2014 and a final report in 2017, as well as interim and final tribal program evaluation reports in 2014 and 2015. 

While the formal evaluations are still in progress, we have already heard first-hand about grantees that are addressing barriers to employment through innovative strategies and partnerships.  For example, Bergen Community College is the lead organization for a consortium that includes ten community colleges in Northern New Jersey and has designed a “boot camp” curriculum that provides participants with an orientation to healthcare occupations.  As another example, Central Susquehanna Intermediate Unit in Pennsylvania is using “Google hangouts” to facilitate real-time tutoring and homework assistance in a highly rural ten-county service area to support students who are completing healthcare training programs.  In California, the San Diego Workforce Partnership initiated the formation of a “common customer workgroup” that brings together workforce and human services agencies to streamline their efforts while also helping participants navigate the system more easily.   

Personal Responsibility Education Program

Teen birth rates have fallen significantly in recent years.  Nevertheless, births to teens remain relatively common in the U.S.  Preliminary data for 2012 indicate that more than 300,000 children were born to mothers between the ages of 15 and 19.  Teen births are associated with a range of negative outcomes for teen parents and their children.  For example, teen parents use public assistance more often and finish high school less often.  Furthermore, it is estimated that teen childbearing in the U.S. costs taxpayers billions a year in lost revenue and increased expenditures for foster care, public assistance, and criminal justice services.

Congress authorized a new evidence-based teen pregnancy prevention program called the Personal Responsibility Education Program (PREP) through the ACA.  The program is designed to educate adolescents on “both abstinence and contraception for the prevention of pregnancy and sexually transmitted infections” and to prepare youth for adulthood by addressing topics such as healthy relationships, adolescent development, and healthy life skills.  It is funded at $75 million a year through fiscal year 14 and is administered by the Family and Youth Services Bureau (FYSB), within ACF. 

The program design was guided by research and evaluation that has demonstrated what works to reduce teen pregnancy.  In 2010, HHS sponsored a transparent, systematic review of the teen pregnancy prevention evidence base, in order to independently identify teen pregnancy prevention programs with evidence of impacts on teen pregnancies or births, sexually transmitted infections, or associated sexual risk behaviors.  The review identified, assessed, and rated the rigor of program impact studies and described the strength of evidence supporting different program models.  The review is ongoing and partially supported by PREP funds.  Based on the review, HHS identified evidence-based programs, defined as those with:  (1) studies with designs that have the best chance of finding unbiased impact estimates; and (2) a positive, statistically significant impact on sexual activity, contraceptive use, sexually transmitted infections, pregnancies, or births.  There are now 31 different program models that have met the review criteria for evidence of program effectiveness.  Most youth served through PREP formula funding (93 percent) will participate in one of these evidence-based programs.  We released a report last fall on how states are scaling up these evidence-based programs; the report also highlights how some states are reaching their target populations.

Let me highlight three key accomplishments of the PREP program to date.  First, the reach of the program is quite broad.  States plan to serve a total of 300,000 youth through formula grant funding over the course of the five-year grant period.  These youth are being reached through over 300 different program providers operating in over 1,300 different sites across the country.   Second, most state grantees are focusing on high-risk youth.  Three-fourths of state program providers operate in high-need geographic areas.  And third, state PREP grantees are creating an infrastructure to support successful replications of evidence-based programs through training, technical assistance, and monitoring.

The PREP program includes two key components – formula grants for evidence-based programs and competitive grants for promising programs.  The majority of the funding ($55 million a year) is available via formula grants for states and territories.  Programs funded through these grants are required either to be evidence-based or to substantially incorporate elements of evidence-based programs.  Forty-nine states and territories draw down formula grant funding.  In the states and territories that have not taken up formula grant funding, unallocated funds are awarded to organizations within the state or territory via competitive grants.  Within these states and territories, a total of 37 grantees receive competitive funding.  In addition, $10 million was made available through PREP for competitive grants to implement and evaluate promising new teen pregnancy prevention strategies.  Twelve grantees receive funding through these competitive grants for innovative strategies.  Finally, about $3 million a year is available for competitive grants to tribes and tribal organizations.

We now have the opportunity to add to our knowledge about what works to reduce teen pregnancy – and to learn more about what it means to scale up evidence-based programs– through an independent evaluation that ACF is sponsoring of the PREP program.  Mathematica Policy Research, our evaluation contractor, is:  (1) conducting a descriptive study to document how PREP programs are designed and are implemented by states; (2) collecting and analyzing performance measure data for all formula-grant funded PREP programs; and (3) assessing the effectiveness of four specific PREP-funded programs, with an eye to filling gaps in the teen pregnancy prevention evidence base.  In 2013, we released a report from the descriptive study (which I discussed earlier).  The report documents key decisions states made about the design of their PREP programs.  Further findings from the evaluation will be released on a rolling basis, culminating in short-term and long-term impact findings from the four selected sites in 2016 and 2018, respectively.

Abstinence Education

Through the Title V State Abstinence Education Program, $50 million per year is available via formula grants to states “to enable the State to provide abstinence education, and at the option of the State, where appropriate, mentoring, counseling, and adult supervision to promote abstinence from sexual activity.”  The program was first authorized in 1996 by the Personal Responsibility and Work Opportunity Reconciliation Act, and was most recently authorized by ACA.  In FY13, 39 states and territories drew down Title V funding.  The program also is administered by FYSB’s Adolescent Pregnancy Prevention Program.

The program provides funds to states to teach young people the social, psychological and health gains to be realized by abstaining from sexual activity.  States are encouraged to develop flexible, effective abstinence-based plans that are responsive to their specific needs.  As part of those plans, states are encouraged to use abstinence education models that are evidence-based and all models must provide medically accurate information.

Many states focus directly on youth in foster care, and one state, Kansas, has dedicated the entire program to abstinence education for youth in foster care and the parents, adoptive parents, agency staff, and community professionals impacting the lives of children in foster care.  Most grantees also include mentoring, counseling or adult supervision in some capacity.

In the 1990s and 2000s, HHS funded an evaluation of four programs, which showed that youth in the program group were no more likely than control group youth to have abstained from sex; at the same time, program group youth were no more likely to have engaged in unprotected sex than control group youth.

More recently, HHS has reviewed the current evidence base for teen pregnancy prevention programming and found three abstinence program models to meet the criteria for evidence of effectiveness.  These models are:

• Promoting Health Among Teens – Abstinence Only;
• Making a Difference; and
• Heritage Keepers Abstinence Education.

The Abstinence Education statute provides no authority for dedicated research and evaluation funding.  However, HHS is supporting evaluation of abstinence education through some of its broad teen pregnancy prevention research and evaluation activities.  For example, a PREP grantee, Lighthouse Outreach in Virginia, is conducting an evaluation including an abstinence curriculum and a character-development curriculum.

Research and evaluation in the Maternal, Infant and Early Childhood Home Visiting Program (MIECHV)

As Dr. Lu mentioned, HRSA and ACF collaborate on implementing the MIECHV program.  Dr. Lu mentioned our collaboration on research and evaluation for this program, and I will provide some additional detail.  The MIECHV program is based on a large body of research on the effectiveness of home visiting for pregnant women and families with young children.  Impacts have been seen across a broad range of outcomes, including maternal health, school readiness, parenting, prevention of child maltreatment, and family economic self-sufficiency.  The MIECHV statute calls for a rich set of research and evaluation activities to continue to generate new knowledge.  First, it requires the Secretary of HHS, to establish criteria for evidence of effectiveness and to reserve the majority of program funding for home visiting models that meet those criteria.  Second, the statue requires a national evaluation of MIECHV.  Third, it calls for rigorous evaluation of promising approaches implemented by grantees, that is, home visiting models that don’t meet the evidence criteria.  Fourth, it calls for the collection of performance management data by grantees.  Finally, it calls for an ongoing portfolio of research and evaluation activities.

Following an opportunity for public comment, in 2010 the Secretary established criteria for evidence of effectiveness of home visiting models.  ACF awarded a contract to conduct a thorough, transparent, systematic review of the evidence on models of home visiting, applying these criteria.  This project is known as the Home Visiting Evidence of Effectiveness, or HomVEE.  It conducts an exhaustive literature search for impact studies, determines the quality of the studies based on their ability to produce unbiased impact estimates, and assesses whether the available evidence for particular home visiting models meets the HHS criteria.  The project conducts annual reviews to update the evidence on models that have already been reviewed and to review emerging evidence on models not yet reviewed.  To date, the project has reviewed 35 models and found 14 to have evidence of effectiveness.

The statute directs HHS to conduct a national evaluation of MIECHV and includes specific requirements related to the evaluation.  First, it requires the establishment of an Advisory Committee which has reviewed the design of the study and outline of the Report to Congress, which is due in March, 2015.  Second, the law requires that the evaluation examine the states’ needs assessments, address all the outcome domains noted in the legislation, examine impacts across different models and populations, and include a cost study.  The evaluation, known as the Mother and Infant Home Visiting Program Evaluation (MIHOPE), uses a rigorous random assignment design to answer questions of overall impacts as well as impacts for individual home visiting models.  It will examine features of models and their implementation that lead to stronger impacts, and will include information on the costs of implementing home visiting models and the cost effectiveness of MIECHV. 

In order to support grantees in evaluating promising approaches and collecting benchmark performance management data, we have provided technical assistance to grantees on establishing benchmarks, creating data systems, reporting performance management data, building continuous quality improvement processes and conducting rigorous evaluations. 

Finally, the legislation calls for an ongoing portfolio of research and evaluation.  ACF and HRSA have undertaken activities including a tribal research center, investigator-initiated grants, and a home visiting research network to build on the prior work and expand the knowledge base.

Thank you again for inviting me to testify.  I would be happy to address any questions.