Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

Chapter 1

Introduction

Overview of the National ERA Project

For over a decade, policymakers and program operators have sought to learn what kinds of services, supports, and incentives are best able to help low-income working parents retain steady employment and move up to better jobs. This issue has assumed even greater urgency in the wake of the 1990s welfare reforms, which made long-term welfare receipt much less feasible for families. Yet, while a great deal is known about alternative approaches to job preparation and placement, there is still relatively little hard evidence about effective strategies to promote employment retention and advancement. Previous studies on retention and advancement efforts — notably, the Post-Employment Services Demonstration (PESD), a four-site project that tested programs providing follow-up case management to welfare recipients who found jobs — generally failed to improve employment retention.1

The Employment Retention and Advancement (ERA) project was designed to improve on past efforts to learn what works in this area, by identifying and testing innovative models designed to promote employment stability and wage progression among welfare recipients or other low-income groups. The project began in 1998, when the U.S. Department of Health and Human Services (HHS) issued planning grants to 13 states to develop new programs. The following year, MDRC was selected by HHS to conduct an evaluation of the ERA programs.2 From 2000 to 2003, MDRC and its subcontractor, The Lewin Group, worked closely with the states that had received planning grants — and with several other states — to mount tests of ERA programs. MDRC, The Lewin Group, and Cygnet Associates also provided extensive technical assistance to some of the states and program operators, since most were starting programs from scratch, with no proven models on which to build.

Ultimately, a total of 15 ERA experiments (also called “tests”) were implemented in eight states. Almost all the programs target current or former recipients of Temporary Assistance for Needy Families (TANF) — the cash welfare program that mainly serves single mothers and their children — but the ERA program models are extremely diverse. One group of programs targets low-wage workers and focuses strongly on advancement. Another group of programs targets individuals who are considered “hard to employ” and aims primarily to place them in stable jobs. Finally, a third group of programs has mixed goals and targets a diverse set of populations, including former TANF recipients, TANF applicants, and low-wage workers in particular firms. Some of these programs initiate services before individuals go to work, while others begin services after employment. Appendix Table A.1 describes each of the ERA programs and identifies its goals and target populations.

The evaluation design is similar in most of the sites. Individuals who meet ERA eligibility criteria (which vary from site to site) are assigned, at random, to a program group — also called “the ERA group” (in this report, “the PASS group”) — or to a control group. Members of the program group are recruited for (and, in some sites, are required to participate in) the ERA program, while those in the control group are not eligible for ERA services but are eligible for other services and supports available in the community. The extent and nature of the services and supports available to the control group vary from site to site, but it is important to note that, in most sites, the ERA program is not being compared with a “no services” control group. The random assignment process ensures that any differences in outcomes that emerge between the two research groups during the follow-up period can be attributed to the ERA program rather than to differences in the characteristics of people in the groups. Differences in outcomes are known as “impacts.” To track both groups over time, MDRC is using surveys and administrative records (data on quarterly earnings in jobs covered by unemployment insurance and records of TANF and food stamp payments).

The Riverside PASS Project

Origins and Goals of the Riverside PASS Program

The Riverside County, California, Department of Public Social Services (DPSS) created the Post-Assistance Self-Sufficiency (PASS) program model in 2001 to complement Phase 2, its existing postemployment intervention for working TANF recipients.3 (DPSS crafted the Phase 2 model as a response to the limitations of Phase 1, the county’s welfare-to-work program, which excelled at connecting TANF recipients to work but not at helping them stay employed or enhance their career development.)

PASS was designed to provide voluntary postemployment services to TANF recipients who left aid and were working, in order to assist them in keeping their jobs, remaining off TANF, and advancing their earning potential. PASS included, but was not limited to, the following services: case management, which involved the assessment of clients’ needs and referrals to appropriate program services; counseling and mentoring; reemployment activities, such as supervised job search, assistance in preparing a résumé, and the provision of job leads; life skills workshops; referrals to education and training slots; arranging supportive service payments for such items as child care, transportation, books, tools, and uniforms; and — as requested by clients — referrals to social service programs for help with such problems as domestic violence, substance abuse, and mental health.

Five service providers delivered PASS program services in their communities. Three of the PASS providers were community-based organizations (CBOs):

1. Center for Employment Training (CET), serving Indio, Coachella, and Temecula

2. Volunteer Center, serving Corona, Norco, and Lake Elsinore

3. Valley Restart, serving Hemet, San Jacinto, and Perris

The other two PASS providers included a community college and a DPSS office:

1. Riverside Community College (RCC), serving Riverside and Moreno Valley

2. DPSS Rancho Mirage, serving Palm Springs and Rancho Mirage

The Counterfactual: What Is PASS Being Compared With?

DPSS designated a number of Phase 1 (welfare-to-work program) case managers in each of its offices to provide a core set of postemployment services — such as providing job leads and arranging supportive services — to the members of the control group.4 Individuals who were assigned to the DPSS control group had to contact these workers in order to receive these services. In other words, unlike the staff at the PASS service providers, these Phase 1 workers did not actively recruit control group members into the program. In addition, control group members were not eligible for the enhanced services that were offered through the PASS service providers.5

Sample members in both research groups were eligible to receive services for up to 12 months after their date of random assignment. In addition, sample members in both research groups retained full eligibility for food stamps, transitional child care, and Medi-Cal (California’s Medicaid program), as well as for TANF (if they returned to the welfare rolls), in accordance with the rules of those programs. If sample members in either group returned to TANF, they would first be referred to Phase 1 (if unemployed) or to Phase 2 (if they were still working but receiving TANF as well). Sample members in either group would not be referred again to PASS unless they subsequently left TANF with employment.

The External Environment

Riverside County is one of the most diversified areas in California. The metropolitan Riverside area (which includes Riverside, Corona, Norco, and Moreno Valley) dominates the western portion of the county and possesses its most broadly based economy. Hemet, San Jacinto, and Perris — located in the central region of the county — are part of a rural area quickly transforming into a bedroom community for metropolitan Los Angeles. Palm Springs and Rancho Mirage and, farther to the east, Indio and Coachella are in the sparsely populated but rapidly growing eastern sector of the county. The leisure industry dominates the Palm Springs and Rancho Mirage economies, while the agriculture sector predominates in the Indio and Coachella areas. The county’s population increased steadily throughout the study, growing from 1,645,300 in 2002 to 1,776,700 in 2004.6 The unemployment rate decreased slightly during this period, declining from 6.3 percent in 2002 to 5.8 percent in 2004.7 Regional unemployment rates ranged from 6 percent in the metropolitan Riverside area to around 9 percent in the central and eastern regions.8

The single-parent TANF caseload in Riverside County averaged about 9,500 during the period of the evaluation.9 TANF grant levels increased from $647 for a family of three in 2002 to $671 in 2004.10 Because of California’s relatively high TANF grant levels and generous earned income disregards, TANF recipients can earn a significant amount of money before becoming ineligible for this assistance. For example, in 2002 and 2003, a family of three — which is the typical size of a PASS sample member’s family — could earn up to $1,514 per month before losing TANF eligibility. At this level of earnings, the net earned income of $1,418 (after taxes) was more than the federal poverty guideline of $1,252.11

As many California county welfare departments experienced, Riverside DPSS encountered significant budgetary problems in 2002 and 2003. While DPSS opted to renew the contracts of the PASS service providers in November 2002 for another year, continuing fiscal challenges ultimately contributed to the agency’s decision to end the contracts as of December 2003. Although PASS group members could continue to receive program services from the control group’s case managers at their local DPSS office through June 2004, participation rates during those last six months appear to have been quite low.

About the Evaluation

Research Questions

The Riverside PASS evaluation includes three major components: (1) an implementation analysis, which studies the way the program operates; (2) a participation analysis, which examines the extent to which the PASS and the control group members received postemployment services and financial supports; and (3) an impact analysis, which assesses what economic difference the program made relative to what would have happened in the absence of the program. A benefit-cost analysis — which will compare the financial benefits and costs of the PASS program for participants and for the government budget — is also planned.

This report focuses on the following questions.

• Implementation: How did the PASS service providers launch and operate their programs? What services and messages did PASS offer? How did PASS case managers spend their time?

• Participation: Did the PASS providers succeed in engaging a substantial proportion of individuals in program services? What types of services and financial supports did people receive? How did the PASS group’s participation levels compare with the levels of the control group?

• Impacts: Within the follow-up period, did the PASS group, relative to the control group, experience increases in employment retention and earnings and reductions in public assistance receipt? Did individuals’ measured income increase as a result of the program?

The PASS Research Design and Random Assignment Procedures

As discussed above, to produce reliable estimates of the effectiveness of the Riverside PASS program, the evaluation uses a random assignment research design. The random assignment process ensures that there were no systematic differences in the characteristics, both measured and unmeasured, of sample members in the two research groups. Thus, any differences between the two groups that emerged over time — for example, in employment rates or average earnings — can be attributed to the ERA program. These differences in outcomes are known as impacts.

Figure 1.1 illustrates the research design of the Riverside PASS program, including the random assignment process. DPSS Information Technology (IT) staff developed a software program to identify individuals who left TANF with employment within the GAIN Employment and Activity Reporting System (GEARS; “GAIN” stands for “Greater Avenues for Independence,” California’s welfare-to-work program). Once these people were identified, DPSS staff in the Research and Evaluation Unit (REU) used the PASS research module (designed and created in-house, with MDRC guidance) to randomly assign them into the study.12 Individuals were eligible for random assignment if they were (1) ineligible for cash aid for the current month and had been eligible for cash aid in the prior month13 and (2) were determined to be employed.14

Two groups of individuals who had left TANF were excluded from random assignment: (1) Phase 2 ERA study sample members and (2) clients who lived in a ZIP code area that was not served by one of the five service providers. If these clients left TANF with employment, the module would identify and automatically refer them to the program operated by the local DPSS office. (The number of individuals who fit these categories is very small.)

Random assignment ratios were based on case manager staffing levels at each PASS service provider. To prevent providers’ caseloads from building up too quickly, DPSS and MDRC agreed to limit the number of clients randomly assigned to a specific provider to no more than 20 clients per full-time case manager per month. Remaining clients were assigned to a nonresearch group that was eligible for the same range of services as control group members.

Once REU staff performed random assignment for a group of TANF leavers, they uploaded the research statuses for these people to GEARS, which then generated different service notification letters for the PASS and the control group sample members, telling them that they were eligible to receive services through a specific agency: either the local PASS provider or the local DPSS office. In addition, GEARS electronically referred the designated sample members either to their local PASS service provider or to their local DPSS office. These referrals contained the sample members’ names, contact data, and other information. The PASS provider would then attempt to contact and enroll them into the program. Control group members were not subject to such outreach and recruitment efforts; they needed to request services from their case managers. The sample buildup period ran from July 1, 2002, to June 30, 2003.

`

Characteristics of the PASS Research Sample

Table 1.1 shows selected demographic characteristics of sample members at the point they entered the study. (For a breakout of these characteristics by research group, see Appendix Table A.2.) As anticipated, given the program’s design, nearly 90 percent of the sample members were employed and had earnings at random assignment.15 Among these employed individuals, about two-thirds were working full time (32 hours or more per week), and about one-half were earning between $7 and $10 per hour. (As noted in above in the discussion of the program’s external environment, California’s relatively high TANF grant levels and generous earned income disregards meant that many sample members could have been employed but still have remained eligible for a reduced welfare grant.)

The sample is overwhelmingly female (90 percent), and almost half of them are Hispanic. At their time of random assignment, more than half had never been married — a somewhat lower proportion than in most of the other ERA study sites. Sample members in Riverside had an average of two children in their households at random assignment, and over half had at least one child age 5 or younger, suggesting a need for child care while employed.

The Employment Retention and Advancement Project
Table 1.1
Selected Characteristics of Single-Parent Families at Random Assignment
Riverside PASS

Characteristic			Total
Gender (%)	Female		90.0
	Male		10.0
Age (%)	20 years or younger		8.4
	21 to 30 years		42.2
	31 to 40 years		33.1
	41 years or older		16.3
Average age (years)			31.4
Race/ethnicity (%)	Hispanic		49.4
	Black, non-Hispanic		16.4
	White, non-Hispanic		31.6
	Native American		0.7
	Asian		1.9
Primary language (%)	Spanish		12.4
	English		87.0
	Other		0.6
Assistance group (%)	Single-parent family		97.0
	Child-only casea		3.0
Marital status (%)	Never married		56.8
	Married, living with spouse		8.6
	Married, separated from spouse		21.3
	Widowed		2.7
	Divorced		10.5
Number of children in household (%)	None		0.9
	1		38.7
	2		30.5
	3 or more		29.8
Average number of children			2.1
Age of youngest child in household	2 years or younger		36.1
	3 to 5 years		23.9
	6 years or older		40.0
Currently employedb(%)			89.9
Hours worked per week (%)	Less than 20		4.1
	20-31		32.2
	32 or more		63.6
Average hours worked per week			32.1
Hourly wages	Less than $6.25		2.5
	$6.25 - $6.99		36.5
	$7.00 - $9.99		50.2
	$10.00 or more		10.8
Average hourly wages ($)			7.81
Catchment areas (%)	Center for Employment Training		24.2
	Volunteer Center		23.1
	Valley Restart		27.3
	Riverside Community College		21.3
	Rancho Mirage		3.9
Length of prior AFDC/TANF receipt, as of most recent CalWORKs/GAIN appraisalc (%)	NA (applicant)		4.8
	Less than 1year		39.6
	1 year or more		12.0
	2-5 years		24.6
	6-10 years		12.0
	Over 10 years		7.1
Education (%)	California High School Proficiency Exam (CHSPE)		0.3
	General Educational Development (GED) certificate		11.1
	High school diploma		41.4
	Technical/associate's degree/2-year college		2.9
	4-year college (or more)		1.0
	None of the above		42.7
	Other		0.5
High school diploma/GED or higher (%)			56.8
School grades completed (%)	Grade 11 or below		45.3
	Grade 12		44.5
	College/postsecondary, 1-3 years		8.5
	College/postsecondary, 4 years		1.0
	None		0.5
Sample size (total = 2,770)
Additional characteristics recorded at entry into Riverside Phase 2d
Speaks English adequately for employment (%)			93.6
Months employed in past 3 years (%)	Did not work		4.4
	Less than 6		23.9
	7-12		20.1
	13-24		22.3
	More than 24		29.3
Type of employment in past 3 years (%)	Mostly part time		35.4
	Mostly full time		50.9
	About the same		13.7
U.S. citizen (%)			90.7
Housing status (%)	Rent, public housing		5.9
	Rent, subsidized housing		6.1
	Rent, other		74.1
	Emergency/temporary housing		2.8
	Owns home or apartment		2.7
	Other		8.5
Sample size (total = 1,584)

SOURCE: Riverside PASS baseline data. NOTES: See Appendix D. aThis category consists of adults who were receiving TANF on behalf of dependent children. bSample members are identified as employed if they had an hourly wage and/or hours worked greater than zero. cCalWORKs/GAIN is the welfare-to-work program operated by Riverside DPSS. Mandatory for most TANF recipients, CalWORKs/GAIN requires an appraisal of participants upon their entry into the program. Appraisal data are entered on GEARS, the DPSS automated program tracking system used by CalWORKs/GAIN and Phase 2. If PASS sample members had an appraisal recorded on GEARS within one year prior to their random assignment into the PASS study, then DPSS sent MDRC these records (all PASS sample members had such records). dRiverside Phase 2, the postemployment program operated by DPSS and the local workforce agency, is also being studied as part of the ERA evaluation. DPSS staff asked clients the questions denoted by this note at the point when they could have been randomly assigned for the Phase 2 sample, and their responses were entered into GEARS. Because of a surplus of individuals eligible for random assignment, a nonresearch group was created for the Phase 2 study. Persons placed in this Phase 2 nonresearch group could be eligible for random assignment for PASS. (Phase 2 randomly assigned sample members were not eligible for PASS random assignment.) If PASS sample members had Phase 2 records entered into GEARS within two years of their random assignment for the PASS study, then DPSS sent MDRC these records. About 57 percent of the PASS sample members possessed such records (1,584 sample members).

 

 

Sample Sizes and Data Sources

This report covers all single-parent sample members who were randomly assigned for the study from July 2002 through June 2003 — a total of 2,770 individuals (1,627 in the ERA group and 1,143 in the control group).16 Most of the report’s findings cover a two-year follow-up period. The data sources examined for each type of analysis in the report are described below.

• Baseline Data. For each sample member, demographic characteristics — such as gender, race/ethnicity, educational background, and welfare history — were collected from the Machine Budgeting System (MBS) and GEARS at the time that the sample members were randomly assigned into the study.

• Unemployment Insurance (UI), TANF, and Food Stamp Records Data.
  Employment, earnings, and public assistance impacts were estimated using automated state UI wage files and county TANF and food stamp eligibility and payment records. One year of follow-up for TANF and food stamp records and two years of follow-up for UI wage records were available for all sample members.17

• Participation Data. Data of two types were collected by the DPSS-created “P3” program tracking computer system, which both PASS and DPSS case managers used to record and track program activities and to authorize supportive service payments:

  • Participation and supportive service payment records — including referrals to and participation in component activities as well as supportive service reason codes and payment amounts — were collected from the program tracking system for both PASS and control group members for the period from July 2002 though August 2004.

  • PASS monthly management reports — which include information on the type and number of staff-client contact attempts and the types of program services and referrals used by clients — were collected for the same period, that is, from July 2002 through August 2004.

• ERA 12-Month Survey Data. MDRC also conducted a client survey for a subset of PASS and control group members 12 months after their date of random assignment. From those who were randomly assigned between October and December 2002, a total of 300 sample members were selected for the survey, and 224 (75 percent) completed it. The survey explored clients’ participation in employment activities, the characteristics of their employment and jobs, their household composition and income, their child care use, and other experiences. MDRC subsequently determined that the outcomes recorded for survey respondents (from the 12-month survey and other sources) are not representative of the full research sample from which the respondents were drawn. As a result, the report presents only a small number of outcomes based on survey responses. (Appendix H presents the rationale for not fully using the survey data in the analysis of program participation.) Nonetheless, survey data are used in a limited fashion to supplement the participation analysis, especially with respect to receipt of program services. (See Appendix Table E.8.)

• Child Care Payment Data. Monthly payment data were collected for PASS and control group members who qualified for Stage 2 child care assistance from the DPSS child care payment system. (Chapter 3 describes these payments in detail.) The Stage 2 program offered child care assistance payments for former TANF recipients. These payment records encompass the period from July 2002 through August 2004.

• Time-Study Data. MDRC designed and administered a two-week time study of case managers at the five PASS service providers. The time study collected detailed information on the nature of staff-client interactions and on the topics covered in these interactions. In addition, the study collected information on how case managers typically spend their time each day. The time study was administered confidentially, using an MDRC-assigned ID number to protect the identity of case managers. All 10 case managers at the five PASS providers participated in the time study from October 27, 2003, to November 7, 2003.

• Field Research Data. Starting in 2001 and running through late 2003, MDRC staff periodically interviewed personnel at the PASS service providers and DPSS program administrators to learn about the goals, structure, and operations of the PASS program. MDRC researchers collected information on a range of topics, including marketing and outreach approaches used to recruit prospective clients, the types of program services and supportive service payments offered to participants, management philosophies and structures, and the relationships and collaborations between PASS service providers and DPSS. As part of this work, MDRC also reviewed a number of sample members’ case files at each service provider.

Roadmap of the Report

This report focuses on program implementation and early impact findings. Chapter 2 provides more detail on the design, implementation, and operation of the Riverside PASS program. Chapter 3 describes the impacts of PASS on receipt of postemployment services and supportive service payments. Chapter 4 presents early information regarding the program’s impacts on employment, earnings, and other outcomes.

---

1Rangarajan and Novak (1999). (back)

2The U.S. Department of Labor has also provided funding to support the ERA project. (back)

3MDRC is studying the Riverside Phase 2 program as part of the ERA evaluation.(back)

4Control group members were eligible to receive the same type of support service payments as the PASS group members. (back)

5Riverside DPSS selected one specialized case manager in its Rancho Mirage office to work with individuals randomly assigned to the PASS group. People randomly assigned to the control group in the Rancho Mirage area were referred to a DPSS Phase 1 case manager in the Indio office, as described. (back)

6California Employment Development Department, Labor Market Information Division (2002, 2004). (back)

7California Employment Development Department, Labor Market Information Division Web site.(back)

8California Employment Development Department, Labor Market Information Web site. Regional averages were estimated across annual unemployment rates from 2002 to 2004. (back)

9California Department of Social Services (2005).(back)

10 California Department of Social Services (2001, 2003). (back)

11California Department of Social Services (1998), which contains the CalWORKs earned income disregard regulations; and California Department of Social Services, Research and Development Division Web site. (back)

12Since the research design called for the random assignment of individuals, rather than entire TANF cases, a portion of sample members’ cases — approximately 15 percent — continued to receive TANF after their random assignment date. These sample members would have had their needs removed from the case, but they still received a reduced grant on behalf of their dependent children, who remained eligible for TANF. (back)

13The PASS study used an atypical definition of a TANF leaver, compared with similar evaluations. Most such studies define a TANF leaver as an individual whose entire case has been closed for at least two months. (back)

14The module determined employment status by checking the following items, in the order given: (1) had earnings that were used in calculating the current or prior month’s grant, (2) had a TANF termination code that signified employment or increased earnings, or (3) had an open employment record in GEARS (the Phase 1 and Phase 2 program tracking system) with a start date either in the month of TANF termination or in the month prior. While two of the three employment verification criteria allowed for clients to be randomly assigned into the PASS study without verified earnings, the vast majority of the sample (90 percent) left TANF with confirmed earnings and work hours. (back)

15MDRC defined sample members as employed if their baseline records specified an hourly wage rate and weekly work hours. According to Riverside DPSS staff, county TANF eligibility technicians (ET) can use their discretion for closing TANF cases if a client requests that the case be discontinued because the person started working. ETs can close the case either because of the earnings or because of the client’s closure request. If a client fails to submit a Quarterly Income Report, the case will close automatically after three days, regardless of any underlying reasons. Thus, a case could be closed without the person’s wages and hours being confirmed by the ET and could be entered into the Machine Budgeting System (MBS), the county’s TANF and food stamp tracking system. Further, TANF leavers could be randomly assigned into the PASS study on the basis of their employment status, rather than their actual earnings. (The preceding section gives a detailed description of the eligibility criteria for random assignment into the PASS study.) As a result of these two procedures, approximately 10 percent of the sample members did not have verified wages and hours at the time they were randomly assigned into the study.

In addition, MDRC matched the PASS research sample to California’s unemployment insurance records and discovered that approximately 15 percent of the sample members did not have UI wage records and, conceivably, were still eligible for TANF. While this development probably increased the percentage of unemployed sample members, it is feasible that some of these individuals were employed in non-UI-covered positions (such as “off-the-books” jobs, some agricultural jobs, and federal government jobs) and had their wages and hours verified — and entered into MBS — by their ETs. (back)

16In all, 3,226 individuals were randomly assigned as part of the Riverside PASS study. A small number of adults who were in two-parent families — 456 individuals — are not analyzed in this report. (back)

17Year 2 TANF and food stamp records were not available for this report because DPSS was transitioning to a new automated data system at the time the report was written. Year 2 TANF and food stamp records will be available and will be analyzed for future ERA reports that include PASS. (back)

 

Table of Contents | Previous | Next