Child Care Bureau: CCDF: Measuring Improper Payments in the Child Care Program

Second Error Rate Pilot Report

Download this document in Word (1,076 KB) or PDF (1.28 MB)

VI. RECOMMENDATIONS

A goal of the error rate pilots was to test a methodology that could provide States with useful information on areas for improvement in administrative policies and procedures. All nine States acknowledged learning additional useful information about the quality of their eligibility processes.

The two most frequent errors in the Second Pilot occurred in the calculation of hours of care and treatment of income. The three most costly errors occurred in the treatment of income, calculation of hours of care and application forms. Although the error coding of missing documentation varied between the States, it was the primary error cause in the second pilot.

As a result of this pilot, each of the five States has planned action steps or has implemented several new systematic changes to improve monitoring and reduce improper payments. Highlights of some of the State strategies include:

Strengthen supervision of new eligibility workers;
Clarify selected standards with eligibility workers;
Improve IT system elements to:

prevent or decrease calculation errors;
generate exception reports to highlight areas of potential problems or concern;
implement automatic income calculation; and
enhance the capability of extracting data from other data systems.

Conduct extensive technical assistance in counties to address error-prone areas;
Institute changes in the monitoring process;
Introduce statutory changes to simplify access to other State databases; and
Examine State policies to determine what changes may be necessary to provide more consistent application of policies and procedures.

The other goal of the error rate pilot was to test the error rate methodology in different States to produce useful lessons learned regarding management of the methodology process to result in reduction of State burden and improvement in the accuracy of the review findings. Based on the experience of nine pilot States, this chapter provides the implications for replication or expansion of the error rate process including recommendations regarding:

Management of the error rate methodology process;
Provision of instructions and managing the review process;
Use of the instruments;
Modifications to improve reliability across States; and
Implications for ongoing technical assistance.

A. Management of the Error Rate Methodology Process

1. State Project Team Formation and Responsibilities

To coordinate the error rate methodology process, a State needs to create a State Project Team (Project Team). A critical factor in the quality of the review process in the second pilot was the leadership role played by the Project Team. The more successful teams had similar characteristics. Leadership of the Project Team was assigned to a high level management staff member. The Project Team leader’s responsibilities included establishing and adhering to timelines; customizing the Record Review Worksheet; training the reviewers; and arranging Project Team meetings.

In addition to the Project Team leader, the Project Team membership should include the State Lead Agency administrator, staff representatives from program, policy, training, information technology, and the Review Team leader.

The inclusion of the State Lead Agency administrator provides the appropriate level of importance to the Project Team’s assignment. Program and policy team members provide critical input into the customization of the Record Review Worksheet, logistics of the record review process, record review process training, coordination of the re-review process, consolidation of the data analysis and error findings, and completion of the State Response Form. IT staff provide critical logistical support to the sample selection process, as evidenced in the sampling issues New Jersey confronted, and automation of the Record Review Worksheet.

2. Provision of Time to Complete All Steps of the Process

In managing the review process a State must allow enough time to adequately complete the error rate methodology. The pilots provided illustrative data on the amount of time needed to complete the methodology, but it should be remembered that these statistics are related to (1) a 150 case sample for a single sampling frame and (2) a project completed with considerable technical support, particularly in the sample selection and data analysis.

While sample size and technical support clearly play a role in time estimates the length of time for States to complete the error rate methodology is largely dependent on selecting and providing sufficient staff resources to the actual record review component. Those pilot States with Review Team members who were experienced in any local operations or compliance monitoring function (Quality Control (QC), audit, technical support) accomplished the record review efficiently and accurately.

Setting timelines for completion of the error rate methodology process requires consideration of several other factors. If the State decides to expand the pilot methodology and conduct reviews more frequently than once a year, arrangements must be made in advance with IT for additional sampling and with county agencies or local contractors to obtain additional case records for the review process. Likewise time will be a factor if a State decides to include in the review additional client information sources (social security and wage data bases, motor vehicles, etc.) and attendance and payment files. Arrangements would need to be made in advance to coordinate with other parts of the agency to obtain records or payment files.

Three States captured review completion information (date assigned/date completed) in their Record Review Worksheet. Two of these had large State Review Teams and conducted the review centrally and the other was one of the smallest State Review Teams and read records on site. Based on that information, the review of the 150 case sample with a single review month took one month for all three States. All States agreed with the estimate given at the site visit. It is recommended that the methodology timeframe allow for one hour for each case record review.

The decision of whether to conduct the case record review process locally or centrally will have a time and workload effect at the county agency or local contractor level. The State timeline needs to provide the time necessary for each reviewer’s travel or the shipping of case records to a central location for reviews. It is important for States to estimate adequate time for the re-review process, an important reliability check used by the pilot States. Because the re-review component of a centralized record review has fewer practical problems than if the records are read at the local site, it is especially important for the State to include a step in the timeline for a re-review method if its decision is to review records locally. West Virginia addressed the re-review issue through the use of conference calls on error cases.

The time needed for sample selection is dependent on a State’s IT resources and whether or not the sample selection can occur centrally. For some pilot States, sample selection was dependent on cooperation with county agencies or local contractors who may have separate data systems or none at all.

B. Provision of Instructions and Managing the Review Process

Under the leadership of the Project Team, a State needs to establish a State Review Team (Review Team). The State will need to consider existing review processes and staff assignments when making decisions about the actual composition and size of the Review Team.

1. State Review Team Selection and Training

The States made different choices in the type of staff assigned to the record review. Those States that assigned the task to central office administrators, auditors and policy specialists had significantly higher review team costs than those States where existing review staff completed the review. The two States in the second pilot that used existing Quality Control staff for the review (Kansas and Oregon) had two of the lowest record review team costs of all nine States.

Kansas assigned the task to three of its existing quality assurance reviewers. These staff members had extensive experience conducting reviews in the TANF and Food Stamps programs and had conducted an earlier review of child care cases. Oregon also chose existing Food Stamp review staff members. The Florida review staff members included Performance and Program Analysts, who normally conduct audits and reviews (grants, financials, contracts, etc.) of the Early Learning Coalitions (ELCs). West Virginia assigned the task to two Field Consultants and a central office Policy Specialist. Staff members from the State’s Central office audit and policy divisions completed the Colorado review. Arkansas chose local office staff members as reviewers.

The staff resources assigned to the actual record review varied among the States in the pilots. The West Virginia and Colorado reviews were completed by three staff members and the Ohio review by four staff members. New Jersey and Arkansas were at the high end of the range with 13 and 10 staff members respectively. A determinant of the team size is the composition of the team. To the assigned staff members the record review is best described as a “special project.” Some consideration had to be given to their normal responsibilities as program manager, policy specialist, auditor or Quality Control reviewer. Only West Virginia, Ohio and Illinois managed to combine record reviews with the staff members’ normal field visits.

Regardless of the existing review processes and the composition of the Review Team, formal training is necessary to achieve inter-rater reliability. The training needs to include actual case reviews to increase the consistency of error definition, attribution, and interpretation.

States need to establish a process to conduct re-reviews of a subset of all cases to enhance inter-rater reliability of error interpretation. The findings from the second pilot underscored the need to incorporate some form of re-review or joint review to enhance inter-rater accuracy. It is recommended that the re-reviews be drawn from a sub-sample of all cases and not limited to just cases with an improperly authorized payment.

2. Instructions for Error Interpretation and Coding

Feedback from the pilot States and review findings underscore the critical importance of training the reviewers in State rules, policies, and procedures.

Even with staff experienced in conducting case record reviews, it should not be assumed that all members of the Review Team agree on the interpretation of policy and, as a result, “what is” or “is not” an error. Many State eligibility policies are replete with “mays” and “shoulds.” In contrast, the quality of the information produced by the case record review process depends on the consistency of error definition and coding.

It is also very important for training to emphasize the need for sufficient detail in Column 3 of the Record Review Worksheet so case information and error findings can be understood. Without training, reviewers will rely on their own experiences and perspectives in deciding errors and causes. For example, several site visits included case reviews as an agenda item in the review process overview. During the discussion, it was common for the reviewers to express very different decisions about the case situations. This was particularly true if the Review Team consisted of Quality Control (QC) staff members who may be accustomed to very strict or narrow interpretations regarding failing the case based on missing documentation.

A major emphasis for any future pilot is to provide formal training on standardizing error interpretation, descriptions and coding in the Record Review Worksheet to achieve inter-State reliability of data. The quality of the information produced by the case record review process depends on the consistency of error definition and coding. Inconsistencies across pilot States in error interpretation dramatically influenced the outcomes. States in the second pilot were not consistent in their error coding. There were instances within a State where cases with similar findings had very different error results, in terms of improper payment vs. a correct case. Consistency of error definitions and coding has a direct impact on the accuracy of the data and as a result the development of strategies to address improper payments.

3. Record Review Site

States determined in their review processes whether to review records centrally or in the local office. The choice of whether to conduct the record review at the local office or centrally does not appear to have a major role in deciding the number of staff members assigned to the project. Both West Virginia (three State Review Team members) and New Jersey (thirteen State Review Team members) reviewed the records in the local office. Arkansas, with the second largest State Review Team, read the records centrally.

A concern specific to centralized reviews is the shipment of case records. It is not uncommon for record material to be missed when a record is shipped, for example, failure to obtain retired material or recently received but un-filed documentation. The findings from the two pilots do not indicate that the decision to review the records centrally, rather than at the local office, was related to the frequency of missing documentation as a cause for errors. A recommendation is to include a response time within the review cycle to allow local agencies the opportunity to provide the missing documentation. The Quality Control procedures in TANF, Food Stamps, and Medicaid all include local response requirements. Because the re-review component of a centralized record review has fewer practical problems than if the records are read at the local site, the State needs to include a re-review method if its decision is to review records locally.

C. Use of the Instruments

1. Record Review Worksheet Development and Use

The pilot States customized several versions of the Record Review Worksheet that can be a reference for other States (Exhibits A-1 – E-1). The different versions were practical in terms of use by the reviewers, but still included the level of detail needed for data analysis. Some observations of the various versions are:

The Record Review Worksheet template contains a section requiring Client ID, State, County, and Date. Four of the nine States (Arkansas, Colorado, Kansas, and New Jersey) developed a face sheet for their Record Review Worksheet. The face sheets included summary information of the case being reviewed. Examples of the information captured are case name, address, case number, the date, and type of the most recent case action (application or recertification), and particulars of the other household members (name, date of birth, relationship). These face sheets also contained information related to the review processing such as reviewer name, date assigned, and completion date. Face sheets are a useful addition to the Record Review Worksheet, since they provide a valuable overview of the case and its review. Each of the nine States modified the Record Review Worksheet template. It is recommended that States add a face sheet to the Record Review Worksheet in order to summarize key case review information. Within the review elements, most States substituted predefined codes, check-off blocks, “Yes” and “No” fields, and specific questions for written narrative summary. States may choose to make changes such as these to simplify and standardize staff worksheet completion.
A quality summary of the error finding is critical for data analysis. Pilot States differed in the quality of the review narratives in error cases. In some States, the narratives lacked the detail necessary to understand the case information or the error finding. In contrast, the narrative in other reviews provided a thorough synopsis of the case and the review results. Reviewers must provide detail in their error findings in Column 3 of the worksheet. Boilerplate language, “Yes” or “No” fields, and check-offs alone in Column 2 of the worksheet provide sufficient detail for non-errors, but too little detail for error findings. Review Team staff training must stress the importance of the error narrative.
Three of the States (Florida, New Jersey, and West Virginia) created electronic versions of the Record Review Worksheet. Compared to hard copies, the obvious advantages of an electronic version are standardization, ease of analysis, ease of report submission, and ease of archiving. State reviewers' access to laptops, portable memory, and shared network files is necessary to eliminate duplicate data entry or manually filling out worksheets and later entering the data electronically. All States need to make every effort to commit their Record Review Worksheet to an electronic format. (See Appendices A–1 to E-1 for each State’s Record Review Worksheet.)

2. Data Entry Form

The Data Entry Form summarizes the findings contained in the Record Review Worksheet. Because the information from the Data Entry Form must be standardized if used for inter-State pilot purposes, there are no recommendations pertaining to this form.

Most States in the second pilot combined the Data Entry Form with the Record Review Worksheet to assist reviewers in reliably recording the required documentation resulting from the case review findings. West Virginia, New Jersey, and Florida combined the Data Entry Form at the end of the Record Review Worksheet. Kansas kept the Data Entry Form as a separate attachment. (See Appendices A–1 to E-1 for each State’s Record Review Worksheet.)
The Data Entry Form is designed to capture the minimum amount of review information necessary to calculate error measures. Since the information in the form may not be modified, a State may wish to consider methods for collecting a greater level of detail when developing its Record Review Worksheet. Each State can decide its information needs in the development of targeted corrective action strategies after consolidation of the data and calculation of the error measures.

3. Cost Analysis Form

To determine the feasibility of expanding the review methodology, it was necessary to establish the costs associated with both the site visits and the record review process. The States had the flexibility to modify the costs worksheet, if modifications helped them to collect the necessary cost information. In the first pilot, the contractor instructed the States to gather all costs incurred by the State, counties, and CCR&R agencies and contractors that participated in the pilot. The contractor provided a slightly modified worksheet for the cost analysis in the second pilot. The second pilot did not ask States to gather separate county, CCR&R agency, and contractor costs, since local staff members did not participate in the second pilot. Several States in the second pilot stated that the costs were a “best guess” estimate.

The Federal Project Team provided a substantial amount of flexibility to States during both pilots in gathering their costs. The results of the two pilots demonstrated that most States, given existing workloads and the additional work from the pilot, were unable to collect the time and costs data while, at the same time, conducting the methodology. Instructions for States in the first pilot were to estimate time and cost data retrospectively. This retrospective estimation resulted in less precision regarding accounting of time spent and actual costs. State review procedures must include a clearly defined method for collecting cost data that places emphasis on reviewer’s time recording starting at project outset. It is recommended that a State use a standardized time/cost worksheet for all time and costs. It is also important that the leadership reinforce the importance of collecting all time and cost. With sufficient data, it would be possible to develop and use a time/cost standard and thereby reduce the State burden of tracking and compiling actual task data.

Instructions for States in the second pilot were to keep track of the length of time to review each case during the actual case record review process. Only three States in the second pilot captured review completion information (date assigned/date completed) in their Record Review Worksheet. Two had large review teams and conducted the review centrally and the other was one of the smallest teams and read records on site. Based on that information, the review of the 150 sample cases took one month for all three. All States agreed with the estimate given at the site visit that the case review required 45 minutes to one hour to complete, but only three States systematically collected this information.

As the States move forward to address improper payments, an accurate cost accounting of both fixed and variable costs is important to the error rate process. State consistency in data collection of time and costs from the beginning of the process is critical to accurately establish the true cost of this effort. Recommendations for future time and cost accounting include use of a standardized time/cost worksheet for all State/county participants and administrative leadership to reinforce the importance of collecting all time and costs from the outset of the error rate process.

4. State Response Form

New for the second pilot, the contractor shared the findings with each of the five pilot States following the calculation of error measures and data analysis. After reviewing its error rate findings and site visit summary description, each pilot State submitted a written response to the following three topic areas included in the State Response Form:

What are the causes of the improper payments that have been identified? What are the actions that will be taken to correct those causes in order to reduce errors in the future?
Describe the information systems and other infrastructure that assist the State in identifying and reducing improper payments. If the Lead Agency does not have these tools, describe actions to be taken to acquire the necessary information systems and other infrastructure, as required in the Improper Payments Information Act of 2002; and
What steps will the State take to ensure that the Lead Agency and eligibility workers will be accountable for reducing improper payments?

Recommendations for States to consider for planning responses to the three topic areas include:

For the first topic area, analyze the error data to determine the causes and how to reduce the errors going forward. West Virginia completed an analysis that included error elements, error cause and CCR&R specific results.
For the second topic area, analyze ways that automation could have prevented errors detected in the review process and steps that the State is taking to address needed changes to the automated system. In response to the second topic area, only two States in the second pilot listed that changes were needed to their automated systems to reduce improper payments. There are several reasons for this limited response. Most of the States in the pilots have statewide systems with multiple payment accuracy features. The exceptions were New Jersey and Florida. New Jersey plans to improve its system’s links with other data sources to verify information and plans to request additional programming to increase up-front edits to reduce data entry errors. Florida is designing and implementing a new automated system that will improve the accuracy of the eligibility process.
A recommendation related to the third topic area is to describe how the State intends to monitor staff performance over time to reduce payment errors. In response to he third topic area, pilot States offered several new initiatives including:

New Jersey plans to link error rate findings to contractor performance.
West Virginia CCR&R agency Quality Assurance Plans will now require inclusion of payment accuracy initiatives if a CCR&R agency has an error rate that is above the State mean.
West Virginia child care providers with incorrect or improper billing forms must attend a retraining session on proper billing procedures.
Florida provided each Early Learning Coalition (ELC) with its individual results of the pilot review process along with recommendations for corrective action.

The State Response Form questions provide States a starting point in the summation of the review data and the resulting corrective action strategies. The contractor provided the causes for the errors as an example of the types of analyses States may want to complete with the data they collect from the record review process. See Chapter IV (pg. 36-37) for a summary description of the error causes and reasons provided to each State.

D. Modifications to Improve Reliability across Future Pilot States

An initial error rate measure provides both the States and the CCB with a baseline against which to compare future findings. More importantly, the findings from the two pilots suggest the following recommendations for revising the pilot methodology in order to achieve a more consistent review process across States.

Provide for all pilot States formal training on standardizing error interpretation and coding. A detailed review of the error findings in the area of missing documentation shows that States had similar findings with very dissimilar error results. Great latitude was given to the pilot States in defining errors. Some States interpreted missing or out-of-date verification as causing authorized payment errors, while other States ruled the missing or out-of-date verification as a procedural error or no error. This inconsistency between reviewer’s interpretations also occurred within a State’s findings.
Establish within each State a process to conduct re-reviews of a subset of all cases to enhance inter-rater reliability of error interpretation. State training sessions must emphasize reviews of sample cases as part of the training process.
Develop consistent standardized policy and procedures as part of ongoing or routine auditing processes. The State responses to the error findings in both pilots include numerous initiatives to improve the quality of case work and the frequency of reviews. Training to increase staff awareness of error-prone areas, knowledge of policy, interviewing skills, and the quality of routine case reviews are the most effective strategies States can use to prevent or reduce procedural or policy errors.
Focus State review strategies on both the most common error types and those error causes which produce errors of greater amounts, such as ineligibility. Both pilots’ findings suggest that there is little, if any, correlation between States’ percentage of cases in error, the percentage of cases in error that had an improperly authorized payment, and the percentage of improperly authorized payments. In comparison to New Jersey, Kansas had relatively high numbers of cases with errors and cases with an improperly authorized payment. Yet the Kansas percentage of improperly authorized payments was the lower of the two States. In the first pilot, Arkansas had a percentage of improperly authorized payments that exceeded both its percentage of cases with errors and percentage of cases with errors that had improperly authorized payments. For these reasons, State improper payment strategies need to focus on both the most common error types and those error causes which produce errors of greater amounts, such as ineligibility.

Although the treatment of missing documentation varied between the States, missing documentation was the primary error cause in the second pilot and accounted for the largest percentage (57%) of the total improperly authorized payments in the first pilot. As an error type, missing documentation is not affected by some corrective action strategies such as exception reports, system edits, and data matches. Training to increase staff knowledge of appropriate documentation and policy, coupled with routine case reviews targeted on problem areas can assist States in reducing missing documentation errors.

E. Implications for ongoing Technical Assistance

If future pilots do not include consultant assistance, States will need to consider accommodations to reflect additional State functions. States may choose to incorporate the following:

Work with IT departments or a local university for assistance on generating the universe and a random sample of cases for review. Most IT departments have access to programming for random number generators for sample selection. When planning the error rate methodology, States need to decide on the frequency of sample selection and the definition of the universe of cases from which to select the sample. For example, how many times per year will the sample be drawn and what will be the sample size?
Regarding sample size, consult with a statistician to achieve the desired sample size. For example, California worked with a local university to determine a sample size for calculating an error rate that met the IPIA guidelines²¹.
For data consolidation, designate a research analyst or program specialist who can create a database or an excel spreadsheet that includes at a minimum, all of the variables listed in the Data Entry Form and several error categories determined to be most important, such as the review element, missing or insufficient information, data entry errors, or calculation errors. An Excel spreadsheet provides all the formulas needed to compute the error measures described in the pilot methodology that if combined with an electronic Record Review Worksheet, can aggregate individual review data automatically.

This second pilot demonstrated that improvements to administrative practice to reduce administrative errors can result in substantially preventing or deterring improperly authorized payments. Combined with those of the first pilot, these error findings are promising as a baseline test of the methodology. The pilot methodology also was useful for States to gain evaluative information on potential areas for improvement in administrative policy in the CCDF block grant program.

_______

²¹ The Improper Payment Information Act recommends using a formula to determine the sample size for calculating error rates in Federally funded programs, leading to an error rate with a 90 percent confidence interval of +/- 2.5 percent. Return to text.

Appendix A. Florida >>

June, 2007