Child Care Bureau: CCDF: Measuring Improper Payments in the Child Care Program

Second Error Rate Pilot Report

Download this document in Word (1,076 KB) or PDF (1.28 MB)

V. COST ANALYSIS

In order to estimate the burden to States to implement the pilot error rate methodology, each of the nine pilot States (including the four States from the first pilot and five States from the second pilot) agreed to work with the Child Care Bureau (CCB) to determine the costs associated with the methodology implementation. This chapter reviews the costs incurred by all nine pilot States to implement the methodology with a sample size of 150 selected from the universe of cases for one month.

A. Objectives of the Cost Analysis

The CCB asked the contractor to expand upon the cost analysis findings from the first pilot of the error rate methodology to include the additional five States from the second pilot. In addition to the data from the two pilots, this cost analysis also draws upon data from California, where staff computed cost information and provided it to the CCB. The inclusion of the California data is useful because the California legislature mandated implementation of an on-going child care error rate methodology. The California methodology closely mirrors the methodology used in the two CCB pilots and, therefore, California’s results lend additional substance to the analysis.

During any pilot, two important questions are how much effort did the State expend and was that effort worth the results that the State achieved. The first pilot provided an initial baseline analysis of costs incurred by the four States. The first pilot findings also underscored the importance of regularly conducted compliance reviews by States to improve administrative practice. In order to further that analysis, this chapter examines the estimated costs for the initial four pilot States, the costs for the five States that participated in the second pilot, and the additional cost information from California.

B. Conceptual Design

The conceptual design for the cost analysis involved two general types of cost variables—fixed and variable (usually per capita). For example, costs associated with training the Review Team are essentially fixed, one-time costs for a specific year’s cost analysis, provided all of the Review Team members can be trained during a single session. If multiple sessions are required for review of a one-month sample of cases, which was not the case for the nine pilot States, additional sessions would involve only the costs associated with presenter and participant time, since the preparation time will have already been incurred (e.g., development of a training agenda and training materials).

Site visit costs to the pilot States are relatively fixed, with the major exception that there were two-day site visits to the four States in the first pilot and only one-day site visits to the five States in the second pilot. In the first pilot, the Federal Project Team visited one or more county locations in some of the States. Therefore, the first pilot includes county costs for the additional day(s), travel, and county staff burden. The second pilot did not include a second day to visit a county or local child care office/Child Care Resources and Referral (CCR&R) agency, so no county costs are reflected in the preparation, site visit, and training costs below.

In both pilots, costs also were broken out by general levels of pay (hourly) rates, plus benefits, to achieve better precision. The principal cost variable is the average per capita cost to review a record. The review costs would also include the staff costs for the re-review of records in some states. Additional per capita costs would be the cost of follow-up verification, as one example, but this only occurred for a subset of records in the State of Colorado in the first pilot.

Finally, the contractor calculated the costs for the nine pilot sites based on the sample size of 150 cases reviewed; however, when viewing the California costs it is important to note that California used a sample of 1,744 cases due to State legislative requirements.

C. Measuring the Costs to Conduct the Error Rate Pilot

To begin the cost analysis in the first pilot, the contractor contacted each of the pilot States and California to explain the expanded¹⁷ cost analysis. Each of the States was eager to continue the work begun during the pilot and pledged to provide the data on costs incurred during the process. The next step was to develop a tool to collect data regarding the costs associated with measuring improper payments in the child care program. For the initial four pilot States, the contractor also captured the costs associated with conducting a site visit in association with measuring improper payments, as well as the estimated costs associated with conducting the record review.

D. Cost Analysis Worksheet Development

The contractor developed a draft worksheet template to capture the relevant cost information and shared it with the initial four pilot States. After the States commented on the draft worksheet, the contractor modified the worksheet and sent it to the participating States. After discussions with the pilot States, a consensus set of cost categories emerged. The cost categories were staff time, materials, copying, postage and site visit costs. For the first pilot, the worksheet included :

Costs incurred by different State, county, CCR&R agency and contractor staff for conducting the record review process;
Costs related to participating in the site visits; and
Federal participants estimated costs for the site visit only.

The contractor used a slightly modified cost analysis worksheet in the second pilot in order to capture greater detail on the costs incurred by different staff to complete each of the error rate pilot tasks:

Preparation and training of the Review Team (including the Federal Project Team site visit);
Drawing the sample;
Record review process;
Consolidating the data on the Data Entry Forms;
Reviewing the findings and responding to the three questions; and
Discussion of causes/strategies and report preparation.

The second pilot did not ask States to gather county, local agency, and contractor costs, since local staff did not participate in the second pilot.

Staff time was divided into categories according to job classification—Review Team, Computer Support, and Clerical Support—to allow for differing salary ranges. Within these job categories, costs were estimated for the above activities.

E. Preparation and Site Visit Costs

For estimating the costs involved in preparation and training for the error rate project, the second pilot States estimated the costs associated with participating in the technical assistance planning calls with the contractor, customizing the Record Review Worksheet template, planning for and participating in a one-day site visit, selecting and training the record review team, conducting the case record review, and completing the Data Entry Form.

The first pilot focused on the costs associated specifically with the site visits. In the first pilot, the Federal Project Team consisted of two contractor staff members, the Federal Regional Office representatives, and several Federal Central Office staff members. During the second pilot, the Federal Project Team consisted of two contractor staff members, Federal Regional Office representatives, and Federal Central Office staff. Only one Federal Central Office staff member was present in Kansas and New Jersey, two Federal Central Office staff members were present in Florida, and three were present in West Virginia.

In both pilots, the focus of the site visit was to review the tools that States would use to conduct the record review process, discuss the universe of cases (sampling frame) and the selected sample, and provide technical assistance that might be necessary for a successful record review. In some of the States, the site visit team also conducted a review of several of the records. This process allowed for questions and clarifications concerning the tools and definitions prior to beginning the actual review. This training process increased inter-rater reliability and ensured consistency across States regarding error interpretation.

Exhibit 9 highlights the aggregate State and County costs for the two-day site visit activities in the first pilot and the State only costs for preparation, training, and the one-day site visit in the second pilot.

Exhibit 9. Preparation, Training and Site Visit Costs for Nine Pilot States


Initial Pilot States
Arkansas	$7,352
Colorado	$7,140
Illinois	$5,000
Ohio	$10,295
Sub Total	$29,787
Second Phase Pilot States
Florida	$10,061
Kansas	$3,814
New Jersey¹⁸
Oregon	$3,173
West Virginia	$2,154
Sub Total	$19,202
TOTAL COSTS	$48,989

In the first pilot, the consensus among all participants was that the time spent discussing the Record Review Worksheets and reviewing the definition of each element was essential to establish consistency, both within and across States. In the second pilot, the site visit had a dual focus of reviewing State eligibility practices regarding improper payments in addition to providing technical assistance around sampling and the review methodology.

Costs were higher in the first pilot because the site visit occurred over two days and involved visiting one or two local child care offices. County staff time accounted for a substantial number of hours and costs in the first pilot. The second pilot focused on gauging State costs only involved with preparation, training, and the one-day site visit. Costs were lowest for the States, such as Kansas, Oregon, and West Virginia, that used existing quality assurance and monitoring staff to conduct the child care reviews. While Florida recorded approximately the same number of hours as the other States for preparation and training, Florida used a higher average hourly (salary + benefits) rate to estimate the cost of staff time.

F. Record Review Costs

Although each of the participating pilot States engaged in the same basic process to conduct the record reviews, there was some variation in implementation.

1. Four States in the First Pilot

Ten staff members from county offices in Arkansas comprised the Review Team. The reviews occurred in Little Rock. The Review Team members received an overview and training on the process used to review the records. The Review Team conducted the record reviews in a large room and questions about individual records were resolved by asking questions within the group.
In Colorado, staff from the counties first reviewed the records before they submitted them to the State. The Review Team then conducted an in-depth review of all records. For the record review process, two staff members from two different State offices—one from the Audit Division of the Office of Performance Improvement and one from the Child Care Division—participated in the pilot.
In Illinois, staff from the Program Integrity and Quality Assurance Unit comprised the Review Team. This review was an extension of the regular duties and functions of this unit. Little modification or variation from the normal processes was necessary to meet the record review requirements. Because of the volume of cases in the Cook County CCR&R agency, the Cook County CCR&R—Action for Children—reviewed the cases on site. The other CCR&R and site-administered programs sent their records to the Department of Human Services for review.
Ohio did not have an existing process in place to conduct the case record reviews. Prior to the pilot, the Research, Assessment, and Accountability Division, Quality Control Unit did not audit child care, believing that child care program requirements were too vague to lend themselves to a quality assurance review process. This unit, along with staff from the Office for Children and Families, Bureau of Child Care and Development, comprised the State Review Team that conducted the record reviews.

2. Five States in the Second Pilot

In Florida, the Review Team consisted of two Managers, two Performance Analysts, and three Program Analysts. Florida made substantial modifications to the Record Review Worksheet template. The review process occurred centrally, with all records sent to Tallahassee. A second member of the team re-reviewed all records for inter-rater reliability.
The Kansas Review Team consisted of three Quality Assurance Supervisors (Topeka, Wichita, Kansas City) and three Quality Assurance Specialists. The team also conducted a re-review of all cases by having a second team member conduct the review. When both team members agreed on an error, they sent a notice to the local Social Rehabilitation Services (SRS) office for corrective action. The Review Team reviewed all records centrally in Topeka. Kansas modified the Record Review Worksheet to include “Yes” or “No” questions to reduce the amount of written narration for the reviewers.
New Jersey substantially amended the Record Review Worksheet template to substitute coding and specific boilerplate language to guide the reviewer’s written case narrative. New Jersey trained the Review Team members centrally and then reviewers conducted the reviews in their assigned regions at the local CCR&R agencies. The Review Team consisted of three Supervisors and 13 Child Care Specialists.
Oregon made very few modifications to the Record Review Worksheet template. The Review Team consisted of five Quality Control (QC) Staff Persons who have responsibility for TANF and FS case reviews. Following training on child care policy and procedures, the Review Team reviewed the cases centrally. The State Team Leader reviewed all cases with errors a second time and discovered that the Review Team went beyond the level of a desk audit, expecting greater detail in documentation than what had been anticipated or required of the child care review process. As a result, the Review Team reviewed a subset of cases a third time and changed some review results accordingly.
West Virginia made substantial modifications to the Record Review Worksheet template, using check-off boxes, “Yes” and “No” fields and coding to eliminate narrative recording. West Virginia completed the record review process and compiled the results prior to the site visit. The Review Team consisted of two Field Consultants and a Policy Specialist. The reviews occurred on site at the regional CCR&R agencies. The reviewers had already read 10 cases during one of their regular CCR&R audits. To improve the review process, the three reviewers completed several reviews jointly and consulted by telephone frequently. Because the team reviewed the cases regionally, travel time was a considerable factor with several hours of driving time from one CCR&R agency to another.

Each of the four States in the first pilot submitted information concerning the costs for conducting the record reviews. Exhibit 10 presents the State and local/county costs for the record review process and site visits for the four States in the first pilot.

Exhibit 10. Cost of Record Review Process for Four States in the First Pilot

Task	Arkansas		Colorado		Illinois		Ohio
Task	State	County	State	County	State	County	State	County
Staff Time (Review Team)	$12,030	$5,742	$22,920	$10,465	$16,760	$6,475	$26,571	$9,173
Staff Time (Computer Support)	$150	$110	$560	$175	$1,338		$2,568	$4,275
Staff Time (Clerical and Other Support)			$170	$1,780	$2,094	$1,906	$1,545	$3,947
Site Visit	$1,920	$5,432	$5,970	$1,170	$3,500	$1,500	$9,341	$954
Materials	$50		$50	$50	$10	$115	$1,378	$726
Copying	$25		$1,378		$20	$138	$42	$865
Postage			$260	$260	$6	$212	$183	$412
Travel			$100		$4		$855
Other Costs			$500			$385	$4,298	$2,003
Subtotal	$14,175	$11,284	$31,908	$13,290	$23,732	$10,731	$46,781	$22,355
COMBINED TOTAL COSTS	$25,459		$45,198		$34,463		$69,136

In the first pilot, several factors contributed to differing totals among States. Arkansas reduced many of its costs by requesting that field staff members transport the records to Little Rock—a central location—for the record review. Illinois had an existing process in place for conducting record reviews; therefore, the costs in Illinois were less than in Colorado and Ohio, where a new process was developed and implemented. The two county-administered States incurred higher costs than the State-administered States, suggesting that the cost burden for county-administered States (Colorado and Ohio) may be substantially greater than for State-administered States (Arkansas and Illinois). Exhibit 11 shows that California, a county-administered State, corroborated this finding, because it had 11 times as large a sample size, compared with the pilot States, but it had a total cost of more than 20-30 times the costs for Colorado and Ohio and 50-60 times the costs for Illinois and Arkansas.

Exhibit 11 presents California’s total costs for the record review process. California’s total costs are substantially higher due to the larger sample size of 1,744 cases.

Exhibit 11. Cost of Record Review Process for California


Task	Time	Cost
Staff Time (Review Team)
Preparation		$142,333
Training		$71,167
Record Review		$640,500
Subtotal		$854,000
Staff Time (Computer Support)
Preparation		$500,000
Training		$0
Record Review		$50,000
Subtotal		$550,000
Staff Time (Clerical and Other Support)
Preparation		$10,833
Training		$5,417
Record Review		$48,750
Subtotal		$65,000
Materials		$0
Copying		$500
Postage		$0
Travel		$0
Other Costs		$100,000
TOTAL COSTS		$1,569,500

Because of the modified cost analysis worksheet in the second pilot, States tracked the time utilized by different staff to complete each of the error rate pilot tasks:

Preparation and training of Review Team (including the Federal Project Team site visit);
Drawing the sample;
Record review process;
Consolidating the data on the Data Entry Forms;
Reviewing the findings and responding to the three questions; and
Discussion of causes/strategies and report preparation.

The second pilot did not ask States to gather county, local agency, and contractor costs, since local staff members did not participate in the second pilot.

Exhibit 12 displays the aggregate costs summarized by tasks for four of the five States in the second pilot. New Jersey’s cost data were unavailable at the time of report submission.

Exhibit 12. Cost of Project Tasks for Five States in the Second Pilot


*Tasks*	*Florida*	*Kansas*	*New Jersey¹⁹*	*Oregon*	*West Virginia*
Preparation, Training, and Site Visits	$10,061	$3,814		$3,173	$2,154
Record Review Process	$14,550	$8198		$6,981	$2,260
Drawing the Sample	$2,225	$388			$655
Consolidation of The Data	$1299	$561			$310
Reviewing the Findings	$3533	$157		$1454	$717
Discussion of Causes/Strategies and Report Preparation	$1073	$179			$388
Materials	$279				$107
Copying	$225	$771			$19
Postage	$620	$128
Travel	$2,477				$690
Other Costs
Total Costs²⁰	$36,671	$13,603		$11,609	$7,686

Final analysis of the cost data does not include the data from New Jersey. For the remaining States, the costs incurred during the second pilot are less than those incurred by States which participated in the first pilot. Reasons for this include:

States in the second pilot had access to information and instrumentation published in the final report from the first pilot.
Four of the five States in the second pilot made significant modifications to the Record Review Worksheet to reduce the time needed for its completion.
Costs were lowest for the States, such as Kansas, Oregon, and West Virginia, that used existing quality assurance and monitoring staff members to conduct the child care reviews.
While Florida recorded approximately the same number of hours as the other States for preparation and training, Florida used a higher average hourly (salary + benefits) rate to estimate the cost of staff time.
The site visits in the first pilot lasted two days to allow time for local office visits (Ohio had two local office visits). The States estimated a cost of $9,000 for county staff for these visits. In the second pilot site visits were reduced to one day and the local office visits were eliminated.
The States made different choices in the staff assigned to the Review Team. Those States that assigned the task to central office administrators, auditors and policy specialists had significantly higher review team costs than those States where existing review staff completed the review. The three States in the second pilot that used existing Quality Control and monitoring staff members for the review had the lowest record review team costs among the nine States.
Total costs in the second pilot included a process to re-review cases to increase inter-rater reliability and to verify the accuracy and consistency of error determination. Some States included these costs as part of the record review process, while Oregon recorded time and costs for the re-review process as a separate activity. The re-review process did not add substantially to the total per case review time.
County costs were not estimated in the second pilot.

______

¹⁷ After the first pilot, the CCB asked the contractor to expand upon the findings and analyze the costs incurred by the State and Federal participants to conduct the site visits and record reviews. Return to text.

¹⁸ New Jersey data were not available at the time of final report submission. Return to text.

¹⁹ As of the submission of this report, New Jersey cost data were not available. Return to text.

²⁰ These are State costs. County costs were not estimated in this pilot. Return to text.

______

Recommendations >>

June, 2007