Child Care Bureau: CCDF: Measuring Improper Payments in the Child Care Program

Second Error Rate Pilot Report

Download this document in Word (1,076 KB) or PDF (1.28 MB)

II. METHODOLOGY

This chapter describes the various components of the second pilot methodology and the process that the States used to measure improperly authorized payments. The chapter begins by providing an overview of the key components of the error rate methodology, followed by a detailed description of how each was implemented with the five pilot States. The final section of this chapter describes an additional area of inquiry in the second pilot. Following the data analysis, the contractor shared the findings with the five pilot States and asked the States to respond to three questions about the causes of the errors, the role of automation in reducing errors, and any corrective actions to be taken as a result of the error rate pilot.

A. Overview of the Process to Measure Improper Payments

The second pilot methodology for measuring improper authorizations for payment focused on client eligibility and employed a case record review process to determine whether child care subsidies were properly authorized. This methodology deviated from the first pilot methodology only in the addition of one element to the Record Review Worksheet and formalizing attention to findings and intended actions of States by requesting response to a State Response Form.

The methodology focused on improper authorizations for payment during the eligibility and eligibility redetermination processes. Eligibility and authorization are the first steps in the child care subsidy process and errors made at this stage in the process are likely to affect the administration of the entire program.

The process to measure improper payments in five pilot States consisted of several phases, discussed in more detail within the remainder of this chapter. They include:

Fieldwork preparation by States, including creation of State Project Teams, formal planning conference calls and site visits with each pilot State by the Federal Project Team;
Generation of a sample of 150 cases;
Customizing Record Review Worksheets by States to reflect State-specific policies and procedures;
State training of case record reviewers, case record reviews using customized Record Review Worksheets, and selected re-reviews of cases;
Completion of Data Entry Forms;
Computation of error rates and findings; and
Completion of State Response Forms to include anticipated changes in response to the error rates and findings.

As with the first pilot, a contractor performed several key functions throughout these phases:

Facilitating planning conference calls with States;
Planning and facilitating site visits;
Receiving State sampling frames and selecting the statewide samples;
Providing ongoing technical assistance regarding customization of the Record Review Worksheet, conducting record reviews, completing Data Entry Forms and State Response Forms; and
Receiving States' Record Review Worksheets, Data Entry Forms, and using them to compute error rates and analyze the findings.

B. Fieldwork Preparation

To prepare for the second pilot, each of the five pilot States formed a State Project Team and identified a project coordinator. Each State Project Team and project coordinator identified who would be responsible for planning for a one-day site visit with the Federal Project Team, generating a universe of cases to send to the contractor, customizing a Record Review Worksheet, forming and training members of the Record Review Team, and assuring that completed Data Entry Forms were submitted for analysis. Members of State Project Teams also agreed to consider findings and determine next steps.

State Project Team members participated in Error Rate Pilot Study overview conference calls. Each planning conference call began with a brief overview of the error rate pilot followed by specific discussions to select the most appropriate sampling plan, consider approaches to customize that State's Record Review Worksheet, and schedule the one-day site visit. To accomplish the fieldwork preparation and site visits within a six month timeframe, States worked quickly with the Federal Project Team to choose their sampling method, generate a universe of cases, and schedule site visits.

C. Generating the Sample

1. Pilot Sample Size

Similar to the first pilot, the primary unit of measurement for this pilot was a child authorized to receive a child care payment¹⁰ during a single month (October 2005 for the second pilot). This did not include children who were denied or terminated, or were billed but not served during October 2005.¹¹ States would review a sample of 150 cases, a sample size designed to produce a statistically valid estimate of erroneous payments. The Improper Payment Information Act of 2002 recommends using a formula to determine the sample size for calculating error rates in federally funded programs, leading to an error rate with a 90 percent confidence interval of +/-2.5 percent.¹² A sample of 150 cases per State yields an estimate with approximately +/-6 percent confidence intervals with 90 percent confidence for each State, adequate for a pilot intended to establish feasibility and estimate the cost of the methodology.

2. Sampling Options

States chose from two basic sampling techniques—a single-stage random statewide sample and a two-stage random statewide sample.¹³ States selected the sampling method most appropriate for their circumstances. Florida, Kansas, and West Virginia elected to use the statewide random sample, while Oregon and New Jersey used the two-stage random sampling approach. Oregon and New Jersey chose the two-stage approach to minimize the number of counties selected for the pilot, thereby decreasing the number of counties from which cases would be drawn and reducing the amount of travel necessary to conduct record reviews in the counties, while still ensuring an equal chance of selection for any child in the population.

a. Creating a Sampling Frame and Child Identification Number

Each State created a sampling frame consisting of the universe of children that met the criteria for a child authorized to receive a child care payment during the month of October 2005. Each case was identified by:

Sequential number;
Child identification (ID); and
County of service.

The sequential number refers to the numerical order that the case appears on the list of cases comprising the sampling frame. Each State determined its own parameters for child IDs. Specifications for the child ID were:

Each child in the sampling frame received a unique ID. If there were several children within one family, case record, or household, each child had a unique ID.
The ID could contain both letters and numbers.
The unique ID did not contain identifying information; however, it was linked to a county or State data system so that the county or State could pull the record if the case was selected for the pilot.

The identification of cases by county made it easier for States to pull the randomly selected sample cases for review.

The contractor randomly selected a sample of 150 cases for each State. In addition, a 20-case supplemental sample was randomly selected as record replacements in Florida, Kansas, and West Virginia. Because New Jersey and Oregon chose to use a random two-stage sample, 30 records were needed for its supplemental random sample. More replacement cases were selected for the two-stage samples to assure enough replacements were available within the county sample, while 20 replacement cases across the State was deemed more than sufficient under the single-stage approach. States used the replacement records only if the case selected did not meet the pilot criteria for valid reasons.

b. Single-Stage Sample

The steps of the single-stage sampling method, used to select a random statewide sample for Florida, Kansas, and West Virginia, were:

Calculation of a sampling interval based on the size of the sampling frame or universe of cases authorized to receive a payment during October 2005. To do this, the total number of cases within the sampling frame (i.e., all cases authorized to receive a child care payment during October 2005) was divided by the number of cases selected for the sample (i.e., 150). (For example, if the sampling frame included 15,000 cases, the sampling interval was 15,000/150 or 100—every 100^th case was selected.)
Consultation of a random number table to determine a random starting point within the sampling frame. From that starting point, cases within each sampling interval were selected, as described above. (For example, if the random start was seven and the sampling interval was 100, then the contractor selected the case listed seventh, the case listed 107^th, the case listed 207^th, and so on.) After randomly selecting one case for each sampling interval, the sample included 150 cases.
Forwarding of a list of the 150 sample cases—including sequential number, child ID, and county of service—to the designated State representative.
Removal of the sample of 150 cases from the overall sampling frame and use of the steps described above to select this second sample of 20 replacement cases.

c. Two-Stage Sample

The sampling plan for New Jersey and Oregon followed a two-stage approach, as follows:

Provision by New Jersey and Oregon of a sampling frame that listed all cases authorized to receive a child care payment during October 2005, grouped by county. The sampling frame was sorted by county size (defined as the number of cases the county served during October 2005)—from largest to smallest counties.
Selection of 15 counties from which to draw samples of 10 cases (15 counties X 10 cases = sample of 150 cases). A first-stage sampling interval was determined by dividing the total number of cases represented in the sampling frame by 15. A random starting point within the sampling frame was determined by consulting a random number table. From that starting point, counties were selected within each sampling interval. (For example, if the total number of cases listed in the sampling frame was 30,000, the sampling interval would be 30,000/15 or 2,000. If the random start was seven, the contractor selected the county in which the case listed seventh lived, the county in which the case listed 2,007^th lived, the county in which the case listed 4,007^th lived, and so on.) Through this process, counties were selected from which subsamples of 10 cases would be drawn during the second stage of the sampling approach. By this process, some large counties would be selected for at least one subsample of 10 cases with certainty and, in some instances, a county with a relatively large number of cases could be selected for two or three subsamples of 10 cases.¹⁴
Selection of a series of county-level random samples, as the second stage of the sampling approach. For each of the counties selected in the first stage, the contractor determined a unique sampling interval based on the number of cases served by the county during October 2005 and the number of subsamples that would be selected from the county. (For example, a large county may have been selected two times in the first stage, meaning that two subsamples of 10 cases would be selected from this county. If the population of the county was 3,000 and two subsamples of 10 cases (20) were selected, the sampling interval for that county would be 3,000/20 or 150. If the random start was seven, the contractor selected the case listed seventh, the case listed 157^th, and so on.) This process was repeated for every county selected in the first stage.

D. Customizing Record Review Worksheets

The CCDF block grant allows maximum flexibility for States to set critical policies and procedures. Eligibility criteria, rates, regulation of child care providers, and payment mechanisms vary widely among jurisdictions. Because each State has developed its own child care and data collection policies, it is important that all reviewers within a State and all States participating in this pilot define and collect data elements in a similar fashion to increase the validity of the pilot findings.

To determine errors based on standards contained in current statutes and applicable regulations, the contractor developed a Record Review Worksheet template, which was a composite of existing review sheets used by several States to conduct Child Care, Food Stamps, and Temporary Assistance for Needy Families (TANF) reviews. States participating in the first pilot agreed that it was a workable template to develop State-specific worksheets. The Record Review Worksheet template captures the detail for each element of eligibility, the calculation of and amount of authorized subsidy for the review month, and any resulting errors (See Appendix F for a template of the Record Review Worksheet.) States were encouraged to customize the template to reflect State-specific policies and procedures.

The template worksheet consists of four columns:

Column 1 lists the basic elements of eligibility and resulting computations;
Column 2 contains the findings of the record review and notes any pertinent facts, questions, or conflicts in information;
Column 3 provides a space for the reviewer to note the ultimate findings of the analysis and a summary of any error; and
Column 4 contains error coding—in the event that a review element contains more than one error, the largest dollar error is the one coded.

Each pilot State customized a Record Review Worksheet to capture client information (name, address, case number, client ID, and household members), provider information, payment data, and error findings. Each State customized the data elements within the Record Review Worksheet to capture the analysis, investigation, and results of the individual eligibility and payment elements. The most common modifications were made in Column 1 and were designed to capture the State’s specific eligibility policy and procedures.

In the first pilot, Colorado reordered elements in Column 1 of the worksheet to follow its workflow more efficiently. Arkansas used the changes made by Colorado and added some content modifications to the elements in Column 1. Ohio reformatted the template by adding an additional column indicating the citation authority for each element. Illinois created a “check-off” rather than a written summary of the record material in Column 2. Another common alteration was to include a section addressing the review findings of the subsidy payment process. (See Appendices A–D in the first Error Rate Study Final Report for each State’s Record Review Worksheet: Available at http://www.acf.hhs.gov/programs/ccb/ta/ipi/phase2/phase2.pdf.)

In the second pilot, States included additional descriptive information in the first two columns of the worksheet to assist reviewers in reliably identifying the required documentation to support the eligibility element. West Virginia specified all applicable forms or data screens required for each element including form numbers and requirements. Florida modeled its worksheet after West Virginia, specifying each applicable form required under elements of eligibility. Kansas included additional prompts or check-offs for reviewers to indicate the presence of applicable evidence. New Jersey and Oregon made changes to the narrative elements in Column 1 of the Record Review Worksheet to reference relevant State policies. See Appendices A–1 to E-1 for each State’s Record Review Worksheet.)

E. Site Visits

Following the fieldwork preparation stage of the pilot, and after the States had selected their pilot samples and developed their Record Review Worksheets, the Federal Project Team visited each of the five States. The Federal Project Team consisted of Federal Central Office staff members, Regional Office staff representatives, and contractor staff members. The purpose of the one-day site visit was to explain and review the pilot methodology for the case record reviews with the State Project Team. The site visits followed the basic agenda presented in Appendix J. (See Appendices A–E for information obtained during the site visits.)

During their site visits, Florida and New Jersey presented and discussed each element of the Record Review Worksheet explaining how they would interpret potential errors according to State policy and procedure. In both New Jersey and Oregon, the site visit occurred before their Record Review Worksheet had been finalized. Although local staff had met several times previously to customize the worksheet, the site visit provided an additional opportunity to reinforce the review methodology. During the site visit, Kansas decided to scale back the scope of its review and to modify its Record Review Worksheet accordingly. West Virginia completed its case record review process prior to the Federal Project Team site visit. West Virginia’s site visit presentation included a review of several examples of eligibility errors discovered and lessons learned during the case record review process. In all States, review team members attended a portion of the site visit and described their planned process for reviewing the cases. Most teams conducted a training following the site visit that would involve group discussion and/or re-review of cases to assure inter-rater reliability. This process allowed for greater consistency within the State regarding error determination.

F. Conducting Record Reviews

The purpose of the record review process is to ensure that all elements are addressed during the case review and to document the findings as the review is conducted. Reviewers need to be experienced with child care program regulations and the quality control techniques and methodology common to the review of other Federal programs, such as Medicaid and Food Stamps. Training and reliability checks need to be conducted to avoid inconsistencies in verification techniques and/or the identification and calculation of errors.

Using the Record Review Worksheet as a guide, States conducted record reviews of the random sample of 150 cases authorized to receive a child care payment during October 2005. Florida, Kansas and Oregon conducted the training of the reviewers and the case record review process centrally. All three States also conducted a re-review process, involving a second team member reviewing each case to increase inter-rater reliability and consistency. New Jersey and West Virginia conducted training centrally; however, the reviewers conducted the record review process regionally at the local CCR&R agencies.

G. Completing the Data Entry Form

State reviewers recorded the findings from each Record Review Worksheet onto a Data Entry Form. (See Appendix G)

The Data Entry Form summarizes the composite results of the record review documented on the Record Review Worksheet. Below is a list of the information included on the Data Entry Form:

State;
County of service;
Child ID;
Pilot period month (October 2005);
Date of data collection;
One or more administrative errors during the pilot period;
Cause of improperly authorized payment;
Total amount of improperly authorized payment during review month; and
Total amount of authorized payment during review month.

For both pilots, States calculated and recorded the authorized payment amount based on the eligibility determination process as indicated within the case record. In both pilots the term “payment” refers to the amount authorized for payment.

In the second pilot, two States combined the Data Entry Form with the Record Review Worksheet. West Virginia and Florida combined the Data Entry Form at the end of the Record Review Worksheet. Kansas and New Jersey kept the Data Entry Form as a separate attachment. (See Appendices A–1 to E-1 for each State’s Record Review Worksheet.) Combining the Data Entry Form with the Record Review Worksheet reduces the number of forms used in the review of a case and assists in the reliable transfer of information from the Record Review Worksheet to the Data Entry Form. However, once the Data Entry Forms have been completed and the task becomes the calculation of the five error measures the combined Record Review Worksheet and Data Entry Form can be a cumbersome document.

H. Computing Error Rates

As a final review, the contractor matched the information contained in the Data Entry Forms with the companion Record Review Worksheet. Project Teams sent copies of all Data Entry Forms and each Record Review Worksheet to the contractor. Using these data, the contractor computed four error rate measures—the percentage of cases with an error, the percentage of cases that have an improperly authorized payment¹⁵, payment error rate, and average amount of improperly authorized payment. (See Appendix G for the Data Entry Form.)

Percentage of cases with an error—This percentage is based on the number of cases with an error, regardless of whether it resulted in an improperly authorized payment or not, compared to the total number of cases in the sample. This percentage is determined by dividing the number of sampled cases with an error by the total number of cases reviewed in the sample and then multiplying by 100.
Percentage of cases that have an improperly authorized payment—This percentage is based on the number of cases with an error that resulted in an improperly authorized payment compared to the total number of cases in the sample. The percentage is determined by dividing the number of cases with an error that resulted in an improperly authorized payment by the total number of cases in the sample and then multiplying by 100.
Payment Error Rate (Percentage of authorized payments in error for the review period)—The payment error rate is the percentage of the gross amount of improperly authorized payments (overpayments plus underpayments) for the review period compared to the total amount of authorized payments in the sample. This rate is determined by dividing the gross amount of improperly authorized payments by the total dollar amount of authorized payments for the sample cases and then multiplying by 100.
Average amount of improperly authorized payment—The average amount of improperly authorized payments is the average amount of money the State authorized for payment improperly on a per child basis during the designated review month. This rate is determined by dividing the gross amount (overpayments plus underpayments) of improperly authorized payments in the sample by the number of cases with errors that had an improperly authorized payment.

I. State Response Form

An additional area of inquiry in the second pilot included sharing the findings with the five pilot States following the data analysis. After reviewing its error rate findings and site visit summary description, each pilot State submitted a written response to the following three topic areas included in the State Response Form:

Causes of the improperly authorized payments identified in the record review process;
Use of information systems that assist the State in identifying and reducing improperly authorized payments; and
Actions to be taken in order to reduce errors in the future.

Appendix I contains the State Response Form. Chapter IV contains a summary of each State’s responses to the State Response Form within the section entitled Next Steps. Appendices A–E also contain summaries of the next steps States are taking as indicated in their responses.

________

¹⁰ For both pilots, States calculated and recorded the authorized payment amount based on the eligibility determination process as indicated within the case record. In both pilots the term “payment” refers to the amount authorized for payment. Return to text.

¹¹ For the purposes of this pilot a child randomly selected for review is referred to as a case. Return to text.

¹² The Improper Payment Information Act, Public Law No: 107-300, § 2, 116 Stat. 2350. Return to text.

¹³The single-stage random sample selects cases directly from the statewide universe of cases authorized to receive payments during the sample month. The two-stage random statewide sample selects a set of counties first (first stage), from which subsamples (second stage) of cases are drawn such that the total cases from all of the subsamples equals the total sample size of a single-stage sample for the sample month. Return to text.
¹⁴ Some counties served 10 or fewer cases during October 2005. The pilot team grouped into clusters counties that served fewer than 25 cases, such that at least 25 cases were in the cluster. (For example, if county A served 6 cases in October 2005 and county B served 19 cases, together they served 25 cases in that month. If this cluster were selected via the random process, a subsample of 10 cases would be selected randomly from among the 25 cases in the cluster. This process ensured that an entire county would not be included in a subsample due to its small size.) Return to text.
¹⁵ For both pilots, States calculated and recorded the authorized payment amount based on the eligibility determination process as indicated within the case record. In both pilots the term “payment” refers to the amount authorized for payment. Return to text.

Measuring Improper Payments in Five States >>

Posted July 6th, 2007