Skip Navigation
acfbanner  
ACF
Department of Health and Human Services Administration for Children and Families
          
ACF Home   |   Services   |   Working with ACF   |   Policy/Planning   |   About ACF   |   ACF News   |   HHS Home

  Questions?  |  Privacy  |  Site Index  |  Contact Us  |  Download Reader™Download Reader  |  Print Print      


The Child Care Bureau   Advanced
Search

Second Error Rate Pilot Report

Download this document in Word (1,076 KB) or PDF (1.28 MB)

Executive Summary

Congress responded to the issue of improper payments in Federal programs by enacting the Improper Payments Information Act (IPIA) of 2002. This law requires Federal agencies to identify programs that are vulnerable to improper payments and to estimate annually the amount of underpayments and overpayments made by these programs.

The Executive Branch also has worked to address the improper payments issue. A goal of the President’s Management Agenda (PMA) is “eliminating improper payments.” To this end, agencies are to establish a baseline of the extent of improper payments and to set goals to reduce improper payments.

Because the PMA and IPIA provide for establishing a child care error rate as one on-going measure of program efficiency, the Child Care Bureau (CCB) launched the project: "Measuring Improper Payments in the Child Care Program" to identify and describe methods that could help States identify, measure, and prevent improper payments in the administration of the Child Care and Development Fund (CCDF). A major focus of the CCB project was to work in partnership with States to determine the feasibility of computing a child care improper payments error rate.

The CCDF is a block grant that allotted over $ 5 billion to States, Territories, and Tribes to support child care subsidies for low-income working families during Fiscal Year (FY) 2006.1 The CCDF block grant allows maximum flexibility for States, Territories, and Tribes to set policies regarding eligibility and fiscal management approaches, as well as define administrative structures that allow maximum choice for parents. As a result, States' eligibility criteria, rates, regulation of child care providers, and payment mechanisms vary widely among jurisdictions.

During FY 2005, the CCB contracted with Walter R. McDonald & Associates, Inc. (WRMA) to develop and pilot test in four States, a common methodology that States could use to compute an error rate. The CCB, in consultation with an initial workgroup of “partner States,” chose to focus the analysis on administrative error related to eligibility in order to measure an element that is common to every State and to mitigate some of the variation among State policies and procedures.

The methodology of the first pilot had four main components:

  • The contractor assisted each pilot State to select a random sample of up to 150 cases (children), using a sampling frame of all children in the State authorized to receive a child care payment2 during October 2004. The sample size was designed to produce a statistically valid estimate of erroneous payments.
  • Pilot States customized a Record Review Worksheet template to reflect child care policies in the State. The States used this instrument to guide a record review of the sampled cases to identify administrative errors in eligibility determination. States collected data regarding the number of cases with errors and whether the errors led to an improperly authorized payment. Although all of the worksheets contained common elements, the definitions pertaining to those elements varied from State to State.
  • The contractor conducted site visits and provided technical assistance to the pilot States who then conducted record reviews and collected data. The contractor computed the error rates using the data submitted by the pilot States.
  • The contractor also conducted telephone interviews to gather additional information about improper payment activities in five States. These States were Arizona, California, Kansas, Nebraska, and New Hampshire.

The purpose of the FY 2006 pilot was to further test an error rate methodology in five States. Similar to the first error rate pilot, the methodology of the current pilot focuses on client eligibility and employs a case record review process to identify cases with errors, cases with errors that result in improperly authorized payments, and percentages of improperly authorized payments. However, in the second pilot the methodology provides pilot States with their results in order to analyze the types and sources of error. Following an analysis of the findings from the case record review process, the contractor forwarded the results to the pilot States for their internal review. Pilot States responded to a short survey providing an explanation for the causes of the errors and a description of next steps or corrective actions to be taken as a result of participation in the error rate pilot.

Definitions of the error rates calculated for this pilot and the relevant findings are as follows:

  • Percentage of cases with an error—This percentage is based on the number of sampled cases with an error, regardless of whether it resulted in an improperly authorized payment or not, compared to the total number of cases in the sample. In the second pilot, the percentage of sampled cases with errors ranged from 19 percent to 35 percent. The four States in the first pilot had a slightly wider range, with the percentages of cases with errors ranging from 12 percent to 44 percent.
  • Percentage of cases that have an improperly authorized payment—This percentage is based on the number of sampled cases with errors that have an improperly authorized payment, compared to the total number of cases in the sample. In the second pilot, the percentage of sampled cases with errors that had an improperly authorized payment ranged from 11 percent to 35 percent. The four States in the first pilot had a slightly lower range, with the percentages of cases with errors that resulted in an improperly authorized payment ranging from 6 percent to 32 percent.
  • Payment Error Rate (Percentage of improperly authorized payments for the review period)—The payment error rate is the percentage of the gross amount of improperly authorized payments (overpayments plus underpayments) for the review period compared to the total amount of authorized payments in the sample. The estimated percentage of improperly authorized payments in the five pilot States ranged from a low of 2 percent to a high of 18 percent. The results are very similar to the percentages of improperly authorized payments in the first pilot which ranged from 4 percent to 20 percent.
  • Average amount of improperly authorized payment—The average amount of improperly authorized payment is the average amount of money the State improperly authorized on a per child basis during the designated review month. The average amount of the improperly authorized payment made in the five pilot States ranged from a low of $89.42 to a high of $215.82. The four States in the first pilot had a somewhat wider range, with the average amount of the improperly authorized payment ranging from $88.99 to $289.53.

The findings from both pilots include several sources of administrative error, such as incomplete or missing documentation and inaccurate calculation of income. The three predominant error types in the second pilot were incomplete or missing documentation, the miscalculation of income (both earned and unearned), and incorrect hours of care. All five States in the second pilot documented these three error types in significant numbers, which accounted for 79 percent of the total dollars in error. Although the treatment of missing documentation varied between the States, missing documentation was the primary error cause and accounted for 28 percent of the total improperly authorized payments in the second pilot and 57 percent of the total improperly authorized payments in the first pilot.

One goal of the error rate pilots was to test a methodology that could provide States with useful information on areas for improvement in administrative policies and procedures. All nine States acknowledged learning additional useful information about the quality of their eligibility processes. As a result of this pilot, each of the five States has planned action steps or has implemented several new systematic changes to improve monitoring and reduce improper payments. The State strategies include:

  • Strengthen supervision of new eligibility workers;
  • Clarify selected standards with eligibility workers;
  • Improve IT system elements to:
    • Prevent or decrease calculation errors,
    • Generate exception reports to highlight areas of potential problems or concern,
    • Implement automatic income calculation, and
    • Enhance the capability of extracting data from other data systems;
  • Conduct extensive technical assistance in counties to address error-prone areas;
  • Institute changes in the monitoring process;
  • Introduce statutory changes to simplify access to other State databases; and
  • Examine State policies to determine what changes may be necessary to provide more consistent application of policies and procedures.

Some of the State specific actions are:

  • West Virginia Technical Assistance staff will continue to perform random quarterly case audits. Prior to the pilot, the FACTS electronic record was the object of the review. An expanded review will now include the physical case record.
  • Oregon continues to conduct a series of “Accuracy Summits” in all areas of the State that focus on techniques to reduce errors in the Food Stamp program. Beginning with the next series of summits in July 2007, workshops to improve payment accuracy in child care will be included in the summits.
  • Oregon plans to modify the Food Stamp targeted review process and database that contains review information to ensure that income errors identified in the Food Stamp case are corrected in the companion Employment Related Day Care (ERDC) case.
  • Kansas continues to complete monthly Child Care QC reviews and uses the results of these reviews to inform training needs. Kansas is updating the Child Care Personnel Trainer and Training Academy to emphasize case documentation and computation of hours of care needed and income. Kansas supervisors now complete monthly case reviews for the Child Care Assistance Program.
  • Kansas contracted with a firm to develop software to track all case review findings and provide aggregate review data. The web based system tracks aggregate case review data for the mandatory Food Stamp reviews and the child care reviews. The child care reviews began in July 2006. The software cost $75,000 and there is an 18 percent annual maintenance costs agreement.
  • Florida developed a desk reference tool for eligibility procedures, initiated focused monitoring and training on eligibility procedures on an annual basis and is providing each Early Learning Coalition (ELC) with its individual results of the pilot review process along with recommendations for corrective action.
  • New Jersey is implementing a process to conduct electronic matching of the automated child care client database with records on other wage, SSI, and child support information systems to identify and reduce the number of improper payments.
  • New Jersey plans to continue conducting child care record reviews utilizing the pilot monitoring tool, using a methodology that guarantees a statistically valid sample size of randomly selected cases, and will hire or allocate quality assurance staff needed.
  • Recognizing the limitations of its legacy automated system, KIDS (Key Information Delivery System), Arkansas designed and developed a new automated eligibility system called Keying in Day Care Accurately, Reliably, and Efficiently (KIDCare), to be fully operational as of July 2005. Arkansas designed KIDCare to determine eligibility based on program specific guidelines and has incorporated numerous edits to prevent inaccuracies from occurring on the front end of eligibility determination.
  • Illinois developed a resource guide for workers to outline acceptable forms of documentation or verification needed to determine eligibility accurately. The guide provides clarification for workers to use with clients during the eligibility process when the client cannot produce the required documentation. A worker can place a case in a pending status while awaiting necessary documentation.
  • As a direct result of the first pilot, Ohio began to retool its monitoring and technical assistance processes with the county agencies, implemented a quality control process for the child care program, and examined policy for possible revisions to strengthen and clarify procedures.
  • In response to the first pilot, Colorado planned to implement an automatic income calculation into Colorado’s automated Child Care Tracking System. State staff also conducted training and feedback with all participating counties to address deficiencies identified in the pilot. Staff members then shared the results of the error rate analysis at the State child care conference and changed statutory language to simplify access to other State databases such as those for new hires and unemployment compensation. Colorado also examined State policies to determine where changes may be necessary to provide more consistent application, particularly in the area of self-employment.

The other goal of the error rate pilot was to test the error rate methodology in different States to produce useful lessons learned regarding management of the process to reduce State burden and improve the accuracy of the review findings. Based on the experience of nine pilot States, recommendations for replication or expansion of the error rate process based on the pilot findings3, are:

  • Create a State Project Team (Project Team) to coordinate the error rate methodology process. Leadership of the Project Team should be assigned to a high level management staff member. The Project Team leader’s responsibilities involve managing the entire process, including: recruiting additional Project Team members; establishing and adhering to timelines; customizing the Record Review Worksheet; training the reviewers; arranging Project Team meetings; answering questions and consulting with Project Team members as needed.
  • Allow enough time to adequately complete the error rate methodology. The pilots provided illustrative data on the amount of time needed to complete the methodology, but it should be remembered that these statistics are related to (1) a 150 case sample from a single sampling frame and (2) a project completed with considerable technical support, particularly in the sample selection and data analysis.
  • Establish a State Review Team (Review Team) under the leadership of the Project Team. While sample size and technical support clearly play a role in time estimates, the length of time for States to complete the error rate methodology is largely dependent on selecting and providing adequate staff resources to the actual record review component. Those pilot States with Review Team members who were experienced in any local operations or compliance monitoring function (Quality Control (QC), audit, technical assistance) accomplished the record review efficiently.
  • Provide thorough training to the reviewers in State rules, policies, and procedures. Even with staff experienced in conducting case record reviews, it should not be assumed that all members of the Review Team agree on the interpretation of policy and, as a result, “what is” or “is not” an error. The training needs to include actual case reviews. It is also very important for training to emphasize the need for sufficient detail in Column 3 of the Record Review Worksheet so case information and error findings can be understood.
  • Provide formal training on standardizing error interpretation, descriptions and coding in the Record Review Worksheet to achieve inter-State reliability of data among future pilot States. The quality of the information produced by the case record review process depends on the consistency of error definition and coding. Inconsistencies across pilot States in error interpretation dramatically influenced the outcomes. States in the second pilot were not consistent in their error coding. There were even instances where States had cases with similar findings, but with very dissimilar error results. Consistency of error definitions and coding has a direct impact on the results.
  • Involve some form of re-review or joint review to enhance inter-rater accuracy, regardless of any existing review processes and the composition of the Review Team. It is recommended that the re-reviews be drawn from a sub-sample of all cases and not limited to just cases with an improperly authorized payment. Oregon’s re-review changed the review findings in three cases from an error to a correct case.
  • Make careful up-front decisions about the error information desired at the conclusion of the review, to reduce the time needed for data analysis. The pilot findings indicated that States’ choices in customizing the Record Review Worksheet impact the timeline and costs of that portion of the error rate process. Additional data fields can be incorporated into the customized Record Review Worksheet, reducing the analysis time.
  • Develop consistent standardized policy and procedures as part of ongoing or routine auditing processes. The State responses to the error findings in both pilots include numerous initiatives to improve the quality of case work and the frequency of reviews. Training to increase staff awareness of error-prone areas, knowledge of policy, interviewing skills, and the quality of routine case reviews are the most effective strategies States can use to prevent or reduce procedural or policy errors.
  • Focus State review strategies on both the most common error types and those error causes which produce errors of greater amounts, such as ineligibility. Both pilots’ findings suggest that there is little, if any, correlation between States’ percentage of cases in error, the percentage of cases in error that had an improperly authorized payment, and the percentage of improperly authorized payments. In comparison to New Jersey, Kansas had relatively high numbers of cases with errors and cases with an improperly authorized payment. Yet the Kansas percentage of improperly authorized payments was the lower of the two States. For these reasons, State improper payment strategies need to focus on both the most common error types and those error causes which produce errors of greater amounts, such as ineligibility.
  • Make accommodations to implement the methodology independent of a pilot4, such as:
    • Work with IT departments or a local university for assistance on generating the universe and a random sample of cases for review. Most IT departments have access to programming for random number generators for sample selection. When planning the error rate methodology, States need to decide on the frequency of sample selection and the definition of the universe of cases from which to select the sample. For example, how many times per year will the sample be drawn and what will be the sample size?
    • Regarding sample size, consult with a statistician to achieve the desired sample size. For example, California worked with a local university to determine a sample size for calculating an error rate that met the IPIA guidelines5.
    • For data consolidation, designate a research analyst or program specialist who can create a database or an excel spreadsheet that includes at a minimum, all of the variables listed in the Data Entry Form and several error categories determined to be most important, such as the review element, missing or insufficient information, data entry errors, or calculation errors. An Excel spreadsheet provides all the formulas needed to compute the error measures described in the pilot methodology that if combined with an electronic Record Review Worksheet, can aggregate individual review data automatically.

Combined with those of the first pilot, the second pilot error findings are promising as a baseline test of the methodology. The pilot methodology was also useful for States to gain evaluative information on potential areas for improvement in administrative policy for the CCDF block grant program.

________

1 Child Care Development Fund Fact Sheet (October 2006) available on the Child Care Bureau website: http://www.acf.hhs.gov/programs/ccb/ccdf/factsheet.htm. Return to text.

2 For both pilots, States calculated and recorded the authorized payment amount based on the eligibility determination process as indicated within the case record. In both pilots the term “payment” refers to the amount authorized for payment. Return to text.

3 See Chapter IV for the specific findings about administrative policies and procedures upon which these findings are based. Return to text.

4 This would apply if a State wishes to apply the methodology in a review program independent of a CCB pilot. Return to text.

5 The Improper Payment Information Act recommends using a formula to determine the sample size for calculating error rates in Federally funded programs, leading to an error rate with a 90 percent confidence interval of +/- 2.5 percent. Return to text.

Introduction >>

June, 2007