Phase II Findings: Error Rate Methodology Pilot
Executive Summary, I. Introduction, II. Methodology, III. Measuring Improper Payments in Nine States, IV. Findings and Next Steps, V. Recommendations, Appendices: Appendix A: Arkansas, Appendix B: Colorado, Appendix C: Illinois, Appendix D: Ohio, Appendix E: Record Review Worksheet, Appendix F: Data Entry Form, Appendix G: Telephone Discussion Guide, Appendix H: Arizona, Appendix I: California, Appendix J: Kansas, Appendix K: Nebraska, Appendix L: New Hampshire
Section V. Recommendations
This pilot study was an exploratory effort to develop and test a methodology that States can use to compute a statistically valid error rate estimate. The methodology was successfully tested in four States, representing both State- and locally-administered management structures. This chapter outlines recommendations about the methodology for the Child Care Bureau (CCB) to take under advisement.
A desired outcome of this study is to recommend a strategy that will assist States in measuring the reduction of improper payments. Used as a performance measure, an error rate can be part of an effective strategy to monitor progress on error reduction. To reduce improper payments, many States have invested administrative resources into the infrastructure needed to design, evaluate, and monitor program compliance, quality, and accountability. An error rate is a useful tool for these States to assess progress toward the goal of reducing improper payments.
For those States that have implemented error rate measurement, the results have been instructive, prompting States to reexamine policy or regulations, clarify procedure, and develop new business practices.[13] The four study States that implemented error rate measurement as part of this pilot are planning to reexamine their monitoring processes, provide training, or clarify policy and procedure in those counties where problems or issues were identified.
The findings also reveal useful feedback regarding challenges encountered in the pilot study of the methodology. For example, States need a backup plan when cases from a participating county are unobtainable. Thirty-eight cases from one county were removed from the Ohio sample. That county was in the process of implementing imaging technology and those cases could not be removed from the warehouse. The study team could not select replacement cases within the study timeframe to fix this problem with the study sample. Rather than penalize the State by failing every missing case, the study team analyzed the results without these missing cases, thus reducing the statistical validity of the Ohio sample. Anticipating these types of problems prior to future compliance reviews, States can avoid such large numbers of missing records by giving counties sufficient notice prior to compliance reviews so the cases can be retrieved in a timely manner.
Another challenge was the variation in the ways that States interpreted the error findings. States need to standardize the interpretation of error findings to have more reliability across raters and results that are more consistent. For example, Illinois has defined the documentation required for eligibility specifically enough that raters know exactly how to interpret the missing data and whether it would constitute an eligibility error. Each State could define hierarchies of errors, such that certain types of errors constitute less concern or do not necessarily result in an improper payment. The study team detected some variation in the thresholds for what constituted an error from State to State, but this was an expected outcome of this pilot effort. Each State review team can provide very useful feedback for improvements to the record review process to increase the reliability and consistency of error interpretation.
As a result of increasing standardization of what constitutes an error, States can begin to construct different hierarchies for different types of errors. Evaluation of the data indicates that the most common error found by reviewers involved a lack of sufficient documentation. This result is consistent with the findings from the California error rate analysis.[14] However, the interpretation of what missing documentation constituted ineligibility must be clearly defined by each State prior to initiating the review process. The rigor in this interpretation may have varied from State to State. For many cases the errors due to missing documentation did not result in an improper payment. In some cases, however, missing documentation resulted in the case being deemed ineligible and the entire subsidy ruled an improper payment. States need to standardize what missing documentation results in ineligibility and how State reviewers should compute the resultant improper payment.
States that have historical experience with compliance reviews suggest that the more error prone areas of eligibility are caused by clients or providers. To explore this area of error estimation, Colorado extended its record review process. This additional study yielded findings that indicate that there is value in an extended study of both client and provider caused error. The study methodology can be refined in future reviews to include State-specific techniques of external verification of both provider and client error. States may want to pilot these additional areas in their error reviews, not limiting the review to administrative error. However, extended reviews can be costly due to travel, staff time, and related costs. States will need to customize auditing techniques to pursue additional investigation of cases identified as being problematic or at greatest risk for an improper payment.
States can also implement improvements to administrative practice to reduce administrative error. These improvements include independent confirmation of employment or training and intermittent reconfirmation of participation, as well as attendance or sign-in sheet comparison with claim forms.
An initial error rate measure provides both the States and the CCB with a baseline against which to compare future audit findings. The pilot findings underscore the importance of regularly conducted compliance and evaluative reviews by States to improve administrative practice. Federal monitoring of improper payments compliance reviews would not have to occur every year. Audits in some programs, such as title IV-E, are not conducted on an annual basis. States could self-certify on the results of regularly held compliance reviews, and reporting on the results of improper payment monitoring could occur every 2 years as part of the State Plan submission process.
While there is an emphasis on regularly conducted reviews to comply with the Improper Payment Act, the study underscores the value of findings derived from a smaller yet still statistically valid sample. The pilot study used a sample to 150 cases per State, adequate for a pilot study intended to establish feasibility and estimate the cost of the methodology. However, to achieve the level recommended by the Office of Management and Budget would mean selecting much larger sample sizes per site. The burden on the States to implement a methodology that requires a larger sample size would be much more costly than for the pilot study.
This study cannot address in a scientific manner the issue of costs to conduct a statewide error rate analysis, but describes in a general way the level of effort required by the States. However, the study did make some estimates. The study team asked the participating States to track the time spent reviewing each record. Time spent ranged from less than 1 hour to several hours.[15] State review teams varied in size from 3 to more than 10 reviewers. Regularly reviewing cases to monitor compliance requires a significant investment in infrastructure at both the State and local levels. One audit approach States may consider to avoid unreasonable costs is to select a first sample and set a threshold that if a certain number of cases “fail” then another sample is pulled. This would prevent using a more intensive approach unless it is needed.
To support such a standardized methodology for error rate measurement, the CCB has a strong group of committed States and stakeholders who can provide additional feedback on improvements to the methodology to assure that it can achieve the desired outcomes in each State. The importance of involving the partner States in developing and shaping an error rate methodology for all States cannot be underestimated. The contribution of the four study States to demonstrating the feasibility of this methodology for child care in the current study was exemplary.
13 California and Connecticut are the only States known to the study team to date that regularly use error rate as a performance indicator in the child care subsidy program. (
Return to Text)
14 CalWORKS and Alternative Payment Child Care Programs Error Rate Study Report Required by Chapter 229, Statutes of 2004, (Senate Bill 1104, Committee on Budget and Fiscal Review) April, 2005, p.2. ( Return to Text)
15
Colorado incurred the longest review times, but they did reduce the amount of time spent on case reviews over the duration of the study and they did conduct a more detailed analysis with a sub sample of cases. ( Return to Text)
<< Previous Page | Table of Contents | Appendix A: Kansas >>