Evaluation to Determine the Effectiveness of PARIS – Executive Summary of the Final Report

Evaluation to Determine the Effectiveness of the Public Assistance Reporting and Information System – Executive Summary of the Final Report

Submitted to:
Administration for Children and Families
Office of Financial Services/Division of Financial Integrity
370 L’Enfant Promenade SW
Aerospace Building, 6th Floor-East
Washington, DC 20447

Submitted by:
Health Systems Research, Inc, an Altarum Company
1200 18th St. NW, Suite 700
Washington, DC 20036

June 30, 2007

I. Introduction

II. Evaluation Design

III. Key Findings Related to PARIS Participation

A. State Decisions to Participate in PARIS
B. Implementing Technical Requirements of the PARIS Match
C. Scope of State Participation in PARIS
D. Organizational Location of PARIS
E. Adequacy of Current Staff Level

IV. Key Findings Related to Follow-up on Matched Data

A. Ability of States To Track PARIS Matches
B. Use of Filters to Clean Data Files
C. Usefulness of Matched Data

V. Developing Cost and Benefit Models

A. Developing Cost Models
B. Creating Models for Calculating Benefits

VI. Future Expansion of PARIS

A. Expansion of Programs Included in the PARIS Match
B. Expansion of the Nmber of States Icluded in PARIS

VII. Conclusion

Final Report

I. Introduction

The Public Assistance Reporting Information System (PARIS) is a voluntary computer data matching and information exchange system administered by the Administration for Children and Families (ACF) to provide States with a tool to improve program integrity in the administration of public and medical assistance programs. The PARIS project is designed to match State enrollment data from the Temporary Assistance to Needy Families (TANF) Program, the Food Stamp Program, and Medicaid, with data from other participating States and from a selected group of Federal databases. Using the client’s Social Security number (SSN) as the unique identifier, the files submitted by the States are matched against:

The Interstate match, where social security numbers of public assistance clients are submitted by participating PARIS States and matched with data from all other participating States to determine if participants are enrolled in two or more States.
The VA match, which provides States with information on the eligibility of their clients for veterans’ benefits and also allows States to confirm if their clients are receiving income and medical assistance payments from the Department of Veterans Affairs (VA).
The Federal match which matches State data with information from the Department of Defense and the Office of Personnel Management to determine if clients are receiving income from any of these sources or are eligible for Federal health care coverage.

The purpose of this evaluation is to assess the PARIS Project by examining a number of factors related to the administration, cost, and overall usefulness of the information provided. The three main goals of the PARIS evaluation are to:

Assess the effectiveness of State implementation efforts to determine what factors facilitate or hinder the execution of PARIS.
Examine how States have implemented PARIS, and how data from the various PARIS matches are used to support program integrity efforts.
Develop cost and benefit models in order to identify, where possible, consistent methods by which costs and benefits of the PARIS system can be calculated.

II. Evaluation Design

As of June 2007, there were 43 jurisdictions that had signed PARIS agreements. The evaluation design for this study was cross-sectional, using interviews with a sample of PARIS officials in 14 States, as well as current and historic programmatic and cost data to assess the implementation, process, and effectiveness of PARIS. Effectiveness is being measured in several ways, including assessing the ease or difficulty of State decisions to participate in PARIS, how well States have been able to implement the various components of PARIS, the extent to which effective follow-up is conducted on PARIS matches, and an examination of potential models for conducting a consistent cost-benefit analysis.

In particular, four areas of potential benefits to States from PARIS participation were examined in detail. These potential benefits include:

Closing cases reported as active when a participant has moved from one State to another, but has not reported the move (interstate match). Interstate matches result often identify when a participant moves from one State to another without reporting the move to officials in the original State of residence. This allows State officials to close these cases. Case closures bring cost savings from the halting of future TANF payments and Food Stamp Program benefits, and eliminates any payments for that client to managed care organizations participating in the Medicaid program.
Examination of potential dual participation for the purpose of detecting fraud and recovering overpayments (interstate match). One of the important uses of the PARIS interstate match is to identify clients who may intentionally be receiving benefits in more than one State and to take appropriate action to recover potential duplicate benefits.
Verifying income from the VA and other Federal sources (VA and Federal matches). The VA and Federal matches provide States with information on client earnings classified by type and source of income from Federal employment databases and through payments to veterans made by the VA. By using data from the VA and Federal matches, local offices can verify whether or not income was reported, if it was reported accurately, and whether or not the income should have been counted in determining eligibility.
Coordination of benefits between Medicaid and other Federal insurance coverage (VA and Federal matches). One of the more innovative uses of PARIS data is to ensure that the proper agency is covering the cost of a client’s health insurance benefits. A few States use the VA and Federal matches to determine if clients are eligible for health insurance coverage provided through the VA or through the Federal military retirement system (TRICARE). If a client is eligible for both Medicaid and one or more of these Federal insurance programs, State Medicaid officials can work with these entities to coordinate the payment of benefits. This coordination results in savings to State Medicaid programs, particularly in cases where high-cost benefits such as long-term care are being provided.

III. Key Findings Related to PARIS Participation

A. State Decisions to Participate in PARIS

Some agencies surveyed reported that deciding to participate in PARIS, and gaining approval from high-level policy makers, was relatively simple and informal. States reporting relative ease of approval cited the strong potential for increased program integrity and the potential for cost savings as the key element that facilitated the approval process. However, these experiences with ease of approval may have been the exception rather than the rule for many other PARIS States. During the course of the study, information was provided by both ACF and PARIS Board members that the approval process was often lengthy and somewhat difficult in many States. Most of the difficulty in obtaining approval seemed to center around staff resource issues to implement and conduct follow-up on PARIS matches. This difficulty was reflected in the amount of time ACF and PARIS board officials spent working with States to gain official approval for the States to participate.

B. Implementing Technical Requirements of the PARIS Match

None of the 14 State agencies reported substantial challenges to participating in PARIS from the standpoint of implementing the technical requirements and submitting data. The States had the support of the relevant policy and IT decision-makers who saw the benefits of the initiative. There were no issues with signing the PARIS agreement, and the time frame from deciding to participate to submitting the application varied from 2 weeks to 6 months. No State reported that the process was unusually difficult or cumbersome.

C. Scope of State Participation in PARIS

Full participation in PARIS was defined as participation in all three matches with appropriate follow-up activities being conducted. One other challenge that is important to note is that some States have limited the scope of their PARIS activities so that they do not participate in all three of the matches. Almost all 43 of the participating States participate or intend to participate in the interstate match, but fewer participate in the VA and Federal match. The major reasons for limiting the scope of PARIS activities included a lack of resources to manage follow-up activities, the difficulty of understanding and working certain files, and the need to coordinate with an organization (mostly Medicaid) that are located in a different department than the Food Stamp or TANF program.

D. Organizational Location of PARIS

In almost every case, States felt that they had good reason for choosing to locate PARIS where they did. States cited the following considerations when making the decision on an organizational location:

Ability to mobilize resources. One of the key issues in being able to conduct PARIS activities is the ability of the lead PARIS agency to mobilize the resources needed to conduct all of the PARIS activities. This often means coordinating with multiple programs or local offices that have significant autonomy.
Contribution to the ongoing work of the agency. Many agencies have an ongoing responsibility for conducting program matches other than PARIS to determine benefits eligibility. By incorporating PARIS into the office that administers these other match programs, PARIS gives these agencies another data matching tool. It also allows for sharing of staff across multiple activities related to data matching and program integrity.
Easily integrated. Several agencies attributed their successful PARIS implementation to the fact that the tasks associated with the program were easily integrated into their normal everyday workflow.
Access to Data. Some States find it useful to house PARIS within the agency that also controls the data systems for eligibility determination. This allows for better coordination of follow-up activities and limits the number of data entry activities needed to close or amend a record.
Motivated staff. Though not a substitute for workflow integration or available resources, a motivated PARIS coordinator and staff in some cases determined the organizational location of the PARIS program.

One of the disadvantages to any organizational location involves the limitations States place on the use of PARIS data. As discussed in Chapter I, there are many potential uses of PARIS data, including detecting and preventing fraud, coordinating benefits between Medicaid and Federal insurance systems, and removing inactive cases from a State’s benefit system. However, the ways in which States actually use PARIS data often reflect the specific priorities of the implementing agency rather than the full range of possible uses for the data. For instance, States that place PARIS in an audits and investigations division tend to emphasize using PARIS data for fraud detection and prevention and not for coordination of benefits or case file “cleaning,” while the opposite occurs when PARIS is located within a program office that focuses on certification policies and procedures.

E. Adequacy of Current Staff Level

PARIS participation requires two sets of staff: one that focuses on creation and distribution of the PARIS match files, and one that oversees the use of these data for follow-up activities. Most States have divisions or agencies that are responsible for the preparation and distribution of the PARIS files, while assignment of follow-up staff varies across States, often reflecting the structure of regional and local field offices within a State.

States consistently reported adequate staff levels for project coordination and the creation and distribution of PARIS files. No States reported shortages among technical staff. Other than time spent on initial programming for the data submission, technical staff members across all States reported spending only a small percentage of their time working on PARIS, typically ranging from a few hours to a few days per quarter.

However, most States reported not having enough staff resources for conducting the types of follow-up activity that they would like to be doing. Follow-up activities on PARIS matches often compete with other priorities, resulting in States limiting the scope of PARIS participation or limiting the amount of follow-up activity assigned to staff.

IV. Key Findings Related to Follow-up on Matched Data

A. Ability of States to Track PARIS Matches

Follow-up activities can be conducted at the State level, through local offices, or a combination of both. Of all the PARIS States reporting location of follow-up activities, 44 percent conduct their PARIS follow-up activities at the State level. Thirty-six percent reported that PARIS follow-up activities occurred locally, while 19 percent use a combination of State and local staff. PARIS matches are often combined with other program integrity data for follow-up. This means that most States view PARIS as simply another tool to protect program integrity, and do not track outcomes of PARIS matches. As a result, only half of the States included in the sample were able to track through and provide specific outcomes to PARIS matches, which meant that the other half of the States could not document results and costs/benefits attributed to PARIS match follow-up.

B. Use of Filters to Clean Data Files

PARIS data “filters” are specific criteria that a State uses to reduce the size of its matched data files prior to follow-up activities being conducted. Filters can be a very effective tool in reducing the number of files for which follow-up must be conducted, prioritize follow-up activity, and eliminate files with incomplete or erroneous information. Filters also may help States focus their follow-up efforts on cases that stand a better chance of being resolved.

C. Usefulness of Matched Data

When asked about the usability of the matched data, most States felt that, overall, the files were fairly easy to use and that the data quality was fairly high. All States reported that PARIS is a major contributor to improving program integrity. However, there were some specific issues cited by States that caused them to limit the scope of follow-up activities:

States have the most difficulty understanding and working with the Federal file. Many of the States feel that the Federal file is confusing and can be difficult to understand in terms of identifying sources of income and the applicable time period for which the amounts apply.
States that conduct follow-up on the VA file found it less difficult to use than the Federal file, but also reported that it requires more time to work than an interstate match if there is a discrepancy between what is reported by VA and what the client reported.
The interstate match data are only as good as what is submitted by the other States. Poorly prepared match data in one State can result in another State wasting time and effort on follow-up.
Some States report that contact information for follow-up staff is very frequently missing, both for other States and for the Federal and VA agencies.

Cross-State communication issues also appear to be a point of contention for several States. Workers involved in follow-up in some States have an established network of out-of-State contacts to call; these workers generally reported no major communication issues when working a match. However, workers in other States were not as positive. They reported that some States do not provide the number of an individual contact person on their PARIS match file, but instead use either an 800 number or a generic State eligibility hotline number, which makes it difficult to find someone who is familiar with the case. In addition, calls placed to generic phone numbers can result in a conversation in which the individual answering the phone has a strictly limited amount of time to answer questions.

V. Developing Cost and Benefit Models

One of the key objectives of this study is to create a standardized model that States can use for calculating the cost and benefits of PARIS participation. However, because of the diverse nature of PARIS participation, a single cost/benefit model will not work for PARIS. This is because States differ in both the scope and frequency of their PARIS participation (limiting both number and frequency of matches), and cost and benefit calculations will differ among States. States are not required to match their data every quarter, so some States will only match once or twice a year. States with more frequent matching will accrue benefits more frequently. Additionally, some States limit their matching to one or two of the three possible PARIS matches. Those States that match against all three files will likely have a greater potential savings than those that do not.

The study team approached the development of cost/benefit models from the standpoint of creating reasonable models based on the ability of a State to collect the type of data necessary for the calculations. In the next sections, we describe the proposed models for calculating costs and benefits, explain the limitations of the models, and provide examples of how existing PARIS States can populate the models.

A. Developing Cost Models

To reflect a true representation of the total cost of PARIS, cost models were grouped into three areas, the cost of PARIS start-up; the ongoing cost of creating the master file, submitting the file to DMDC, and preparing and filtering the file once it is returned; and the cost of staff time devoted to follow-up activities, such as conducting coordination of benefits or trying to capture overpayment recoveries.

1. Start-up Costs

It is important for States that are considering joining PARIS to understand the costs involved to program and implement the PARIS data matching protocols. Start-up costs encompass those costs related to the programming of the initial PARIS files to match the required PARIS format. Additional start-up expenses also might include the costs of incorporating follow-up activities into policy and procedure manuals, training staff on how to conduct follow-up, and setting up tracking systems for reconciling PARIS results.

2. Ongoing State-level PARIS Costs

All of the States interviewed reported that the ongoing cost of creating PARIS files, submitting the data to DMDC, and preparing and filtering the results are relatively small. One State, which conducts an extensive coordination of benefits effort with the VA and TRICARE reported more significant costs for these activities. However, for most States, ongoing costs of PARIS operations are not commonly tracked, as most States incorporate PARIS activities into a more general, higher-level compliance activity. None of the States interviewed had a tracking system that would allow one to track costs specifically to PARIS activities. However, some States were able to provide estimates of the amount of time needed to create the files, submit the match, and filter and distribute the results.

3. Cost of Local Follow-up

As was noted earlier, two of the States interviewed tracked local costs. However, only one of these States conducts follow-up on all three matches. When examining costs for local follow-up in the State where all three matches are worked, we noticed that costs do vary by type of match. In general, it took local staff longer to work an average Federal match than it did for them to work an interstate or VA match, thus creating a variance in the average cost per case worked between the three matches. Matches that were tracked in the second State were only interstate matches, and thus there was more consistency in the average cost per case work. The methods use by both States, however, can provide a framework for calculating local costs, even if they were calculated using a sampling method. This could be done by devoting resources to conducting a time and results study based on a representative sample of PARIS matches. Once an average cost per match is identified, this average can be applied to all PARIS matches.

4. Coordination of Benefits and Cost Recoveries

Two of the States have special projects by which they track PARIS activities related to coordination of benefits and establishment of financial recoveries. Because these costs have their own coding as part of a special project, the PARIS coordinators have access to accounting records to rack the costs from month to month. This allows the PARIS coordinators to produce reports to justify the ongoing funding of these projects.

B. Creating Models for Calculating Benefits

Ideally, a standardized benefit calculation model would not only allow an individual State to calculate benefits for their own purposes, but would allow ACF to conduct comparisons across States. Additionally, they would allow ACF to make national estimates of savings to document the success of PARIS. While the findings of this study support the idea that a consistent methodology can be implemented, there are some challenges that will need to be addressed as the implementation proceeds.

For example, the diversity in approaches used by States to implement PARIS makes it extremely difficult to develop a single model for calculating benefits that can compare one State with another. Some States only work all three matches, while other States only work matches for one or two of the programs included in PARIS. Some only work interstate matches and close cases based upon unreported moves, while others use PARIS as an income verification system and use the data to verify eligibility.

Creating a standardized method for calculating cost and benefits of PARIS requires some flexibility in approach. By developing models for specific activities rather than an overall single model, the applicable model(s) can then be applied by each State individually to allow for an approach to suit their own customized implementation of PARIS. The models can then be adjusted to meet frequency and scope issues within an individual State, and thus create a cost-benefit level for each State. As a result, four separate potential benefit models have been created;

A model that calculates savings related to closing cases from unreported moves on the part of clients
A model for calculating savings related to the recovery of overpayments
A model for calculating savings as a result of benefit reduction as a result of clients not reporting all income
A model to calculate savings as a result of coordination of health benefits with Medicaid.

VI. Future Expansion of PARIS

There are a number of opportunities for expansion of PARIS in the future. Because of changes in the file format, additional programs can be added as the PARIS community decides there is a need for expansion. In addition to new programs, there is also room for expansion of how existing PARIS data are being used by some States. Described below are some of these opportunities, and some issues to consider when making expansion decisions.

A. Expansion of Programs Included in the PARIS Match

PARIS is currently planning to expand the PARIS match to include State Workers Compensation payments for the interstate match. By adding workers compensation to PARIS, States would be able to use the data to verify income that might not be reported because the source is from another State, or detect duplication of payments (fraud) that might occur. However, there are some issues that must be resolved before it can be fully embraced by the PARIS States. First and foremost, Workers Compensation is often handled by a State Government Department separate from where all other PARIS activities take place. This means that State officials must negotiate with the separate department to incorpor;)ate their data into the PARIS match and make arrangements for follow-up activity to take place. Second, training will need to take place in order to ensure that State and local officials can read and understand the file that is created, and clearly identify how income is displayed, time limits, and any other limitations on the data.

ACF is also working to expand PARIS to include child care payments in the database. This process will likely need to be refined by ACF and the PARIS Board of Directors to both demonstrate the value of the match as well as how best to conduct follow-up before many States decide to participate.

One method by which expansion can be promoted is through the publishing of PARIS success stories on the PARIS website. By posting successes of States participating in the Workers Compensation and Child Care program matches, other States will see the value of participating and may well join.

B. Expansion of the Number of States Included in PARIS

It is a stated goal of the PARIS board of Directors to include all States in the PARIS match. As of the date of this report, there were still nine States that had not signed PARIS membership agreements. Some non-participating States have indicated an interest in participating. As has been noted, there is the potential for significant benefits to all States joining PARIS. For example, if one were to examine a single benefit that could potentially result from PARIS participation, it is in the area of closing Medicaid cases where a client enrolled in managed care has moved, but not reported the move. If a client moves, and does not report the move, the State continues to make payments to the Managed Care Organization (MCO). By identifying these unreported moves, the State can realize a savings from closing Medicaid managed care cases where capitation payments are being made.

VII. Conclusion

The PARIS project is viewed by participating States as a beneficial effort that helps improve program integrity. The findings of this study support this belief. PARIS provides States with a tool that has the potential for not only saving money, but also helping to improve the overall integrity of public assistance programs. With the PARIS project almost doubling in size since this study started, there are even more opportunities for States to match with new partners. While some States are currently limiting their applications of PARIS, the potential for both expanded use of existing data as well as growth in the types of programs participating in PARIS is excellent. The sharing of ideas among States, along with States’ development of creative uses for PARIS data will help promote expansion in the future and will help PARIS to reach its full potential as a tool to preserve program integrity