Table of Contents | Previous | Next |
Introduction
This report presents interim results from a rigorous evaluation of New York City’s Personal Roads to Individual Development and Employment (PRIDE) program, a large-scale initiative that provided specialized employment services to welfare recipients with medical problems that limited their employability. PRIDE operated from 1999 to 2004, serving more than 30,000 people.
The PRIDE evaluation is part of the national Employment Retention and Advancement (ERA) project. Conceived and funded by the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS), the ERA project is testing innovative programs across the county that aim to promote steady work and career advancement for current and former welfare recipients and other low-wage workers. MDRC, a nonprofit, nonpartisan research organization, is conducting the ERA project under contract to ACF and is producing a similar interim report for each site in the project.
This introduction provides background on the national ERA project and describes key components of the PRIDE program. It also describes the research design for the evaluation and the characteristics of the study participants.
Overview of the National ERA Project
In the wake of the 1990s welfare reforms –– which made long-term welfare receipt much less feasible for families –– policymakers and program operators have sought to learn what kinds of services and supports are best able to help long-term recipients find and keep jobs and to help former recipients stay employed and increase their earnings. The ERA project was developed to increase knowledge on effective strategies to help both these groups move toward self-sufficiency.
The project began in 1998, when ACF issued planning grants to 13 states to develop new programs. The following year, ACF selected MDRC to conduct an evaluation of the ERA programs.1 From 2000 to 2003, MDRC and its subcontractor, The Lewin Group, worked closely with the states that had received planning grants, and with several other states, to mount tests of ERA programs.
Ultimately, a total of 15 ERA experiments were implemented in eight states, including New York. Almost all the programs target current or former recipients of Temporary Assistance for Needy Families (TANF) –– the cash welfare program that mainly serves single mothers and their children –– but the program models are very diverse. One group of programs targets low-wage workers and focuses on advancement. Another group (which includes PRIDE) targets individuals who are considered “hard-to-employ” and primarily aims to move them onto a path toward steady employment. Finally, a third group of programs has mixed goals and targets a range of populations, including former TANF recipients, TANF applicants, and low-wage workers in particular sectors. Some of these programs initiate services before individuals go to work, while others begin services after employment. Appendix Table D.1 describes each of the ERA programs and identifies its goals and target populations.
The evaluation design is similar in most of the sites. Individuals who meet ERA eligibility criteria (which vary from site to site) are assigned, at random, to a program group –– also called the “ERA group” or, in this case, the “PRIDE group” –– or to a control group. Members of the ERA group are recruited for the ERA program (and, in some sites, are required to participate in it), whereas members of the control group are not eligible for ERA services. The extent and nature of the services and supports available to the control group vary from site to site. The random assignment process ensures that the two groups were comparable at study entry. Thus, any differences in outcomes that emerge between the two groups during the follow-up period can be validly tested for the likelihood that they arose because of the program and not by chance variation.
The PRIDE Program
Rules requiring welfare recipients to work or prepare for work have existed for nearly 40 years, but most states did not begin enforcing these requirements until the 1980s. Even then, a large proportion of welfare recipients were exempt from work-related requirements, either because they had young children or because they had health problems that limited their ability to work.
In the 1990s, many states expanded work requirements to a much broader share of the welfare caseload. The federal Personal Responsibility and Work Opportunity Reconciliation Act of 1996 accelerated this process by requiring states to ensure that a specific proportion of all recipients were working or preparing for work and by limiting most families to 60 months of federally funded assistance. By 2004, about a third of the states had adopted a “universal engagement” philosophy and taken steps to require all TANF recipients to engage in work activities.2 Changes to the TANF program that were passed by Congress in January 2006 are likely to put additional pressure on states to deliver employment services to all or most TANF recipients.3
As states began to work with a larger share of the TANF caseload, and as caseloads declined dramatically, many states began to focus more attention on the substantial barriers to employment facing those recipients who remained on the welfare rolls. Some states began to develop new employment-oriented programs for recipients with mental health problems, drug and alcohol abuse, physical disabilities, and other serious behavioral and health problems. Evaluations of broadly targeted welfare-to-work programs in the 1990s found that such programs were able to increase employment for long-term recipients with low levels of education and work experience but that outcomes for these recipients were much worse than for recipients with fewer employment barriers.4 Little is known about the effectiveness of the newer, more targeted approaches.
New York City has been particularly aggressive in attempting to ensure that all welfare recipients are engaged in work activities.5 The city’s policies assume that virtually everyone on welfare should either participate in work-related activities, take specific steps to stabilize a medical problem, or apply for Supplemental Security Income (SSI) benefits.6 As part of this effort, beginning in the late 1990s, the city’s welfare agency, the Human Resources Administration (HRA), developed a set of tailored programs for populations facing particularly serious barriers to employment. One of these initiatives, PRIDE, was an ambitious program designed to serve welfare recipients who had medical problems and had previously been exempt from work-related requirements. Another goal in establishing PRIDE was to give recipients who had medical limitations an equal opportunity to obtain employment and move toward self-sufficiency.
The program, started as a pilot in 1998, was a partnership of the TANF and vocational rehabilitation systems — along with several other agencies — and reflected the belief of both systems that even people with serious disabilities can work. The program operated citywide from 1999 to 2004, when it was replaced by a new program, WeCARE, that builds on the PRIDE model. (“WeCARE” stands for the Wellness, Comprehensive Assessment, Rehabilitation, and Employment program.) More than 35,000 recipients were referred to PRIDE while it operated. PRIDE is one of two New York City programs that are being evaluated as part of the ERA project.7
Under PRIDE, recipients who reported that they were unable to work due to a medical problem were required to undergo an HRA medical evaluation. If the evaluation determined that the recipient was “employable with limitations,” he or she was referred to one of several contracted PRIDE service providers (referred to as “vendors”). After additional assessment by vendor staff, recipients were assigned to one of two service tracks: a vocational rehabilitation (VR) track for those with more serious medical problems or a Work-Based Education (WBE) track for those with less severe medical barriers but low literacy or education levels and/or a lack of English language skills. Recipients in both service tracks were required to work in exchange for their welfare benefits –– an activity known locally as “work experience” –– and many were also required to attend education classes. In addition, in both service tracks, participants were provided with job search and placement assistance as well as employment retention services.
PRIDE’s emphasis on work experience reflects New York City’s long-running commitment to this activity. In fact, New York is one the few jurisdictions in the United States that has mounted large-scale work experience programs for welfare recipients. There have been few rigorous studies of work experience programs in recent years. Evaluations of less intensive work experience programs in the 1980s found that such programs were generally seen as fair by recipients, but they also cast doubt on whether work experience generates increases in employment and earnings in regular jobs.8
Participation in PRIDE was mandatory for recipients who were deemed appropriate for the program. Those who failed to participate were referred back to HRA and could be penalized (sanctioned) by having their welfare grant reduced or closed. Thus, the PRIDE program represented a fairly radical departure from “business as usual” for these clients, mandating participation in work activities and placing them in work experience positions. The hope was that providing these recipients with an extensive assessment of their condition as well as work designed to accommodate that condition would lead to a successful experience in the work experience positions and, ultimately, a transition to unsubsidized employment.
The New York City Context
New York City is the largest city in the nation, with about 8.1 million residents. The city’s population makes up over 40 percent of the population of New York State. The city also ranks high in terms of diversity: 45 percent of its inhabitants are white; 27 percent are black; another 27 percent are Hispanic; and 10 percent are Asian.9 In contrast, the city’s public assistance caseload is more heavily weighted toward black and Hispanic families and individuals.
The two key cash assistance programs in New York are the Family Assistance program (New York’s TANF program) and the state- and locally funded Safety Net program. (PRIDE served recipients from both programs.) Previously called Home Relief, the Safety Net program serves childless adults and, more recently, Family Assistance recipients who have reached the 60-month time limit on federally funded benefit receipt. Unlike many other states, New York State does not impose time limits on cash assistance receipt for families but, rather, moves cases to the Safety Net program after the 60-month point. In addition, New York does not use full-family sanctions (which cancel a family’s entire welfare grant) to enforce work requirements in its TANF program; rather, recipients’ grants are reduced in response to noncompliance with work requirements.10 (Safety Net recipients who do not have children can have their entire case closed in response to noncompliance with work requirements.)
The city’s TANF (Family Assistance) caseload has fallen by over 50 percent since 1997, from about 270,000 cases in January 1997 to fewer than 110,000 in fall 2006, the latter number including former TANF cases that had transitioned to the Safety Net program.11 The traditional Safety Net caseload (consisting of childless adults) has also fallen by nearly half since 1997, although it has increased somewhat in recent years, from 150,000 in 1997 to 77,000 in 2002 to 90,000 in 2006. TANF cases began to reach the 60-month time limit in December 2001, and recent data indicate that about one-fourth of the Safety Net caseload consists of cases that were converted from TANF after reaching the time limit. As of fall 2006, for example, there were 119,000 Safety Net cases, about 29,000 of whom were converted TANF cases.
The PRIDE evaluation began just after September 11, 2001, when New York City’s economy was feeling the aftereffects of the attack on the World Trade Center. Although the unemployment rate in New York City has fallen since 2002, it remains slightly above the national average, at 5.6 percent in December 2005, compared with 4.6 percent nationally. Unemployment rates in the entire metropolitan area are somewhat lower than those in the city.12 Finally, the poverty rate for families in New York City was 18.5 percent in 1999, compared with 12.4 percent nationally.13
The Target Population
PRIDE targeted an “in-between” group of TANF and Safety Net recipients: individuals whose medical problems were too severe to allow them to participate in the city’s regular welfare-to-work program but were not severe enough to make them eligible for federal disability benefits. Locally, these individuals were referred to as “employable with limitations.” Before PRIDE, these recipients were exempt from work requirements.
In the evaluation, results for single parents (whether they were receiving TANF benefits or had transitioned to the Safety Net program after 60 months of assistance) are presented separately from results for childless adults. Although data on the sex of the recipients are not available, it is assumed that most single parents are female and that the majority of childless Safety Net recipients are men.
Table 1 presents selected characteristics of both groups of sample members at random assignment, or baseline –– the time when individuals entered the study. Most of these data are drawn from welfare agency administrative data. With an average age of 39, the single-parent sample members were considerably older than most samples in the ERA evaluation and in other welfare-to-work studies, which may reflect the higher incidence of health problems with age. Over 40 percent of this group were older than age 41. (The Safety Net childless sample members were even older, with an average age of 47.) Consistent with their age profile, this sample also had older children; most had no children under age 6. They also had more children than the typical recipient family; 30 percent had three or more children. Although not shown, a key difference between the single parents on TANF and those who had moved onto Safety Net is the number of children: Nearly 40 percent of the Safety Net parents had three or more children, compared with 24 percent of the TANF parents. Single parents with several children may have more difficulty working steadily, or they may find the child care costs associated with work prohibitive, resulting in longer stays on welfare.
Most people in both research groups are either Hispanic or black, with the single-parent sample having a higher fraction of Hispanic parents and the childless sample having a higher fraction of black adults. Most also reported living in unsubsidized rental housing, although a substantial share reported living in temporary or emergency arrangements. Finally, few had recent work experience, at least in formal jobs covered by the unemployment insurance (UI) system. Only one-fifth of single parents, for example, worked in a UI-covered job during the year before study entry.
By definition, it was expected that the PRIDE target group would suffer from poor health. Table 2 presents a more detailed look at the health status of the single-parent sample, using data from the ERA 12-Month Survey, which was administered a year after sample members entered the study. (Survey data are not available for the childless sample.) Although the data were measured well after individuals entered the study — and are based on self-reports rather than direct medical evidence — they give some indication of the health barriers faced by the PRIDE population.
Not surprisingly, the majority of survey respondents rated their health as “fair” or “poor.” In addition, 45 percent of respondents were obese, while another 31 percent were classified as overweight. Obesity has been found to be an important deterrent to work and is correlated with being disabled.14 Nearly half of the respondents also reported problems with pain: 46 percent said that pain interfered “a lot” with their daily work (housework or work outside the home). Finally, over one-third of the sample reported having experienced severe psychological distress in the month prior to the survey, as determined using a well-known diagnostic scale that measures depression, anxiety, and other mental health conditions.15
The lower panel of Table 2 presents data on the specific health conditions that survey respondents had at the time of study entry. This information is based on respondents’ recollection of the medical evaluation that occurred prior to random assignment or on any conditions that they remembered having at the time of random assignment, if they did not recall the details of the evaluation. The most common physical health ailments were orthopedic conditions, such as back pain or herniated disk. The prevalence of these conditions may explain the large proportion of respondents (46 percent) who said that pain interfered with daily life. The next most commonly cited ailments related to mental health, including depression, anxiety, and posttraumatic stress disorder. Significant fractions of the sample also reported having asthma or high blood pressure.
| Variable | Single Parents | Safety Net Recipients Without Children | |
|---|---|---|---|
| Age (%) | 20 years or younger | 0.9 | 0.4 |
| 21 to 30 years | 17.3 | 5.5 | |
| 31 to 40 years | 37.6 | 14.3 | |
| 41 years or older | 44.2 | 79.8 | |
| Average age (years) | 39 | 47 | |
| Race/ethnicity (%) | Hispanic | 49.5 | 37.1 |
| Black | 36.6 | 47.4 | |
| White | 10.5 | 12.3 | |
| Other | 3.4 | 3.2 | |
Number of children in household (%) |
0 | 2.7 | NA |
| 1 | 36.7 | NA | |
| 2 | 30.7 | NA | |
| 3 or more | 29.8 | NA | |
| Average number of children in household | 2 | NA | |
| Age of youngest child in household (%) | Less than 3 years | 21.3 | NA |
| Between 3 and 5 years | 17.1 | NA | |
| More than 6 years | 61.6 | NA | |
| Housing status (%) | Rent, public housing | 17.3 | 16.7 |
| Rent, subsidized housing | 16.1 | 6.5 | |
| Rent, other | 48.4 | 69.8 | |
| Owns home or apartment | 0.1 | 0.0 | |
| Emergency/temporary housing | 17.2 | 2.7 | |
| Other housing arrangements | 0.8 | 4.4 | |
| Worked in UI-covered job during year prior to random assignment (%) | 20.4 | 24.3 | |
| Sample size | 2,648 | 540 | |
| SOURCES: MDRC calculations from the New York Welfare Management System (WMS) and unemployment insurance (UI) wage records from the State of New York. |
| Outcome (%) | Control Group | ||
|---|---|---|---|
| Overall health | Self-rated health | Excellent | 2.5 |
| Very good | 5.5 | ||
| Good | 18.6 | ||
| Fair | 47.8 | ||
| Poor | 25.1 | ||
| Body weighta | Normal weight | 21.6 | |
| Overweight | 30.7 | ||
| Obese | 44.6 | ||
| Over past month, how much did pain interfere with work? | None | 14.7 | |
| A little | 38.4 | ||
| A lot | 46.1 | ||
| Experienced serious psychological distress in the past monthb | 36.5 | ||
| Specific health conditions at study entryc | Orthopedic problems | 63.1 | |
| Mental health issues | 39.3 | ||
| High blood pressure | 31.7 | ||
| Asthma | 31.4 | ||
| Diabetes | 14.8 | ||
| Neural conditions | 10.0 | ||
| Other problems | 35.6 | ||
| Sample size | 379 | ||
| SOURCE: MDRC calculations from responses to the ERA 12-Month Survey. NOTES: See Appendix F. a National Institutes of Health weight categories. b Based on the K6 scale that includes six questions about how often a respondent experienced symptoms of psychological distress during the past 30 days. The response codes (0-4) of the six items for each person are summed to yield a scale with a 0-24 range. A value of 13 or more for this scale is used here to define serious psychological distress. (Web site: http:\\hcp.med.harvard.edu/ncs/k6_scales.php.) c Categories sum to more than 100 percent because many respondents reported more than one health condition. |
About the ERA Evaluation in New York City
Research Questions
The ERA evaluation focuses on the implementation of the sites’ programs and their effects, or impacts. Key questions addressed in this report include the following:
-
Implementation. How did HRA and its partners and contractors execute the PRIDE program? What services and messages did the program provide and emphasize? How did staff and case managers spend their time?
-
Participation. Did the PRIDE program succeed in engaging a substantial proportion of individuals in services? What types of services did people receive? To what extent did the program increase service levels above the levels that would “normally” be received, as represented by the control group’s behavior?
-
Impacts. Within the follow-up period, did the PRIDE program –– compared with earlier rules and services for this population –– increase employment and employment stability and reduce reliance on TANF?
The Random Assignment Process
As shown in Figure 1, random assignment of recipients began in December 2001 — about two years after PRIDE was implemented citywide — and lasted for one year.16 As noted earlier, the first step for HRA was to identify, through a medical evaluation, TANF and Safety Net recipients who were “employable with limitations.” These recipients were then required to attend an intake meeting with HRA and PRIDE staff. At the meeting, staff conducted a brief interview to determine whether the recipient was, indeed, eligible to participate in PRIDE — for example, he or she had not experienced a change in medical condition, was not caring for a disabled child, and had child care in place. (Sometimes, recipients were rescheduled and were given help finding child care.) Once a recipient was deemed eligible and ready for PRIDE, staff placed a brief phone call to MDRC, which conducted the random assignment. Recipients were assigned either to the PRIDE group (and were referred to the PRIDE vendor that was most convenient to their residence) or to the control group.17
The Counterfactual: What Is ERA Being Compared With?
Recipients who were randomly assigned to the control group –– who represent the counterfactual for the study –– were not referred to a PRIDE vendor and were told that they were not required to participate in work-related activities; that is, they were treated as they would have been treated before PRIDE began. They were, however, given a list of employment services in the community and were referred back to their HRA caseworker for assistance with other, nonemployment issues. Control group members could have been required to participate in work activities during the study period if their status changed to “nonexempt” (that is, if their medical condition improved) –– as would have been the case under prior rules.
Thus, the key differences between the conditions faced by the PRIDE group and the control group were that the PRIDE group members (1) received a more in-depth assessment of their conditions, including literacy and education levels, and (2) were required to work in tailored work experience positions and to participate in other work-related activities or faced the possibility of sanctioning if they did not.
[D] |
Data Sources
The data sources for the analyses presented in the report are described below.
Baseline Data: Administrative Records
At the point of study entry, selected demographic data for the sample members were obtained from the New York Welfare Management System (WMS), the automated welfare database. In addition, data on employment and welfare receipt prior to study entry are available from New York State unemployment insurance (UI) records and from TANF and food stamp records. These baseline data are used to describe the sample (as in Table 1) and to identify the key subgroups for whom program effects might differ.
Follow-Up Data: Administrative Records
Effects on employment were estimated using automated UI wage records data. Data on average earnings are shown for descriptive purposes but are not used to estimate program effects.18 Effects on public assistance receipt were estimated using automated TANF and food stamp administrative records. When the analyses for this report were conducted, two years of follow-up data were available for all sample members.
The ERA 12-Month Survey
Information on sample members’ participation in program services and their employment, income, and other outcomes was gathered by the ERA 12-Month Survey, which was administered to a subset of single parents approximately 12 months after random assignment.
Program Participation and Implementation Data
Information on recipients’ participation in work-related activities and other services was obtained from the ERA 12-Month Survey. MDRC also conducted a special “time study” of PRIDE staff, tracking their activities over a two-week period. Information on program operations was gathered from interviews with PRIDE and HRA staff and caseworkers and from reviews of participants’ case files, conducted during several site visits. Finally, participation data were also obtained from NYCWAY, a database that is maintained by HRA and that tracks all events for a given case.
Sample Sizes
As shown in Table 3, a total of 3,188 people entered the evaluation between December 2001 and December 2002. This includes 2,648 single parents and 540 traditional Safety Net recipients without children. The study focuses mainly on the sample of single parents, which includes 1,615 TANF recipients and 1,033 former TANF recipients (who had reached their time limits and were transferred to the Safety Net program).19
A total of 1,043 single parents — a subset of the 1,704 parents who were randomly assigned between July and December 2002 — were targeted for the ERA 12-Month Survey and are called the “fielded sample.” Of this group, 759 people responded to the survey (the “respondent sample”), resulting in a response rate of 73 percent.20
| PRIDE Group | Control Group | Total | ||
|---|---|---|---|---|
| Single parents | 1,553 | 1,095 | 2,648 | |
| TANF recipients | 945 | 670 | 1,615 | |
| Safety Net recipients | 608 | 425 | 1,033 | |
| Fielded sample | 524 | 519 | 1,043 | |
| Respondent sample | 380 | 379 | 759 | |
| Safety Net recipients without children | 356 | 184 | 540 | |
| SOURCE: MDRC calculations from the New York Welfare Management System (WMS). |
1 The U.S. Department of Labor has also provided funding to support the ERA project. (back)
3 Under the 1996 law, states were required to ensure that specific percentages of TANF recipients were participating in work activities. However, the required “work participation rates” facing states were reduced by 1 percentage point for each percentage point reduction in a state’s TANF caseload. Because caseloads fell dramatically in the late 1990s, most states faced very low required rates. When Congress reauthorized TANF in 2006, it restructured the “caseload reduction credit” so that most states need to significantly increase the number of recipients in work activities or else risk fiscal penalties. (back)
4 Michalopoulos, Schwartz, and Adams-Ciardullo (2000). (back)
5 For general information on New York City’s welfare reform efforts, see Nightingale et al. (2002) and Besharov and Germanis (2005). (back)
6 SSI is a federally funded means-tested program that provides cash assistance to needy elderly, blind, or disabled people. (back)
7 The second program provides case management and services to individuals who have substance abuse problems. (back)
8 Brock, Butler, and Long (1993). (back)
9 U.S. Department of Commerce, Bureau of the Census (2000). (back)
10 Partial sanctions are calculated by removing the adult from the grant calculation. A client’s first sanction lasts for up to three months but can be lifted at any time if the client comes into compliance. The second sanction lasts for a minimum of three months, and the third and subsequent sanctions last for a minimum of six months. (back)
11 New York City Department of Social Services, Human Resources Administration (2005). (back)
12 U.S. Department of Labor, Bureau of Labor Statistics (2006). (back)
13 U.S. Department of Commerce, Bureau of the Census (2000). (back)
14 Cawley and Danziger (2005); Kaye (2003). (back)
15 Harvard School of Medicine (2005). (back)
16 Random assignment for Safety Net recipients who did not have children was completed in June 2002, while random assignment for single parents continued through December 2002. (back)
17 For logistical reasons, only a relatively small subset of the recipients who went through the process described above were targeted for the study. Initially, only one unit of PRIDE intake staff was designated to conduct random assignment. Cases were distributed among intake units in a relatively random manner, so there is no reason to believe that the research sample is not representative of the broader PRIDE population. However, because ACF was particularly interested in results for families with children, the study oversampled parents. (back)
18 Earnings data were supplied by the state as averages for groups of sample members, rather than for individuals. The number of group averages is too small to test for program effects. (back)
19 Until August 2002, two-thirds of those who went through the random assignment process were assigned to the PRIDE group, and one-third were assigned to the control group. From August to December 2002, one-half of those who went through the process were assigned to each group. (back)
20 Appendix H presents an analysis of the response bias for the ERA 12-Month Survey. (back)
| Table of Contents | Previous | Next |