State Assessment of Internal Controls Final Report, May 2007

Download Guide in Word (1,622 KB) or PDF (676 3KB) format.

III. METHODOLOGY

This chapter describes the various components of the pilot methodology and the process that the Federal Project Team used to develop and pilot test the State Internal Control Self-Assessment Instrument. The chapter describes the make-up of the Federal Project Team, examines the conceptual design and Instrument development process, and explains the initial pilot in Kansas and the training for ACF staff in Regional Offices.

A. Project Team

Because of the intended use of the State Internal Control Self-Assessment Instrument for both States and ACF staff in Regional Offices, it was important to involve both State and ACF Regional staff in the development and pilot process. The Federal Project Team consisted of Region VII financial/grants management and program staff, CCB policy and program staff, and WRMA staff. Kansas agreed to participate in the design as well as conduct a pilot of the Instrument. After the project team finalized the Instrument, Kansas staff proceeded to select State team members.

B. Conceptual Design

The Federal Project Team conducted an extensive review and analysis of existing applicable documentation as the first step in developing a baseline tool. ^7 8 9 The Government Accountability Office (GAO) Internal Control Management and Evaluation Tool provided the conceptual model used to design the State Internal Control-Assessment Instrument. ¹⁰ In addition, the Project Team reviewed several other risk assessment Instruments, including tools used by Virginia, Illinois, and Federal agencies such as the Department of Transportation. Elements from these tools were included in the draft Instrument.

The purpose of the Instrument is to provide a supplemental guide for States to assess the adequacy of internal controls and to identify issues that may contribute to errors in the administration of the Child Care and Development Fund (CCDF). States can modify this tool to fit the circumstances, conditions, and risks relevant to each agency. States can review their internal controls and procedures, using the relevant factors and elements in the Instrument, and document responses as appropriate to their particular agency circumstances.

The Instrument has five sections corresponding to the five standards for internal control outlined by the GAO. ¹¹ The following section provides a brief description of each of the five standards. The third standard, Control Activities, is further broken down into three additional sections: one dealing with Common Activities and two dealing with Information Systems.

• Control environment: This standard addresses how the State establishes and maintains an environment throughout the organization that sets a positive and supportive attitude towards internal control and conscientious management. Major factors affecting the control environment include:
  • Integrity and Ethical Values;
  • Commitment to Competence;
  • Management Philosophy and Operating Style;
  • Organizational Structure;
  • Assignment of Authority and Responsibility;
  • Human Resource Policies and Practices; and
  • Oversight Groups.
• Risk assessment: This standard addresses the importance of establishing clear and consistent goals and objectives at both the organizational and program level. When an agency has established, well-articulated objectives, it can then assess any risks that could impede the efficient and effective achievement of those goals and objectives. Once the agency identifies and analyzes the potential risks, it can develop a plan to minimize, eliminate, or mitigate those risks. The design of this section or standard assists States in this effort.
• Control activities : This standard applies to the policies, procedures, techniques, and mechanisms used by States to mitigate the risks identified during the risk assessment process. Effective control activities are integral to an agency’s planning, implementation, and review processes. In addition, internal control activities are essential in achieving accountability, and efficient program results.
  • Common categories of control activities: Control activities include a wide range of diverse activities, such as approvals, authorizations, verifications, reconciliations, performance reviews, security activities, and the production of records and documentation. The agency can determine whether the proper control activities have been established, whether they are in sufficient number, and the degree to which those controls are operating effectively.
  • Control activities specific to information systems general: Some internal control activities apply specifically to information systems, including the structure, policies, and procedures that apply to the agency’s overall computer operations. These activities apply to all aspects of the information systems operations, including mainframe, network, Internet, and end-user environments. General control governs the environment in which the agency’s application systems operate. The six major factors or categories of information systems control activities within general information systems controls are: agency wide security management, access control, application software development and change, system software control, segregation of duties, and service continuity.
  • Control activities specific to information systems application control:Application control involves the structure, policies, and procedures designed or adopted by the agency to ensure completeness, accuracy, authorization, and validity of all transactions during application processing. Controls include both the routines contained within the computer programming code, as well as the policies and procedures associated with end-user activities. The four major categories of control activities are authorization control, completeness control, accuracy control, and control over integrity of processing and data files.
• Information and communications:The fourth standard assesses the degree to which the State has relevant and reliable information, both financial and non-financial, relating to external and internal events. Information and communication must be broad based, accountable, reliable, continuous, appropriate, and secure, whether manual or automated.
• Monitoring: The final standard allows the State to examine and evaluate the performance of contract and non-contract providers administering child care and other related services. This standard provides elements to gauge the effectiveness of the program, audits, and other monitoring activities.

Each section of the Instrument contains a list of major elements or criteria for States to consider when reviewing internal controls as they relate to each particular standard. By reviewing each element, States can first determine the applicability of the item to local circumstances. If applicable, the Instrument provides standards against which States can assess performance, identify any resultant gaps or weaknesses, and determine the extent to which the element influences the agency’s ability to achieve its mission and goals. The results of this assessment can guide States in developing processes, procedures, or other measures to improve internal controls in order to meet the intent of the IPIA and ACF's need to document improvements. State assessment results can help provide a framework for ongoing technical assistance, when shared with ACF staff in Regional Offices.

C. Instrument Development

The Federal Project Team, consisting of Federal CCB staff from Central Office and Region VII and WRMA, used an interactive approach to develop the Instrument. Beginning with the basic Internal Control Management and Evaluation Tool (GAO–01–1008G), the Project Team reviewed other assessment tools, including those developed by Illinois and Virginia, and verified that relevant information contained in those Instruments were also contained in the draft Instrument. This process took place over several weeks through conference calls and e-mails to produce and refine the initial draft Instrument.

Kansas participated as a partner State in the review, modification, and initial pilot test of the Instrument. Alice Womack, State Child Care Administrator, agreed to lead the State Self Assessment of Internal Controls Project Team, which included staff representing Program, Fiscal, Policy, Human Resources, Information Technology, and Auditing Divisions, at both the central office and field office levels.

Federal Project Team members consisting of Federal staff from Central Office, Region VII, and WRMA visited Kansas in November 2005. The Federal Project Team conducted a two-day working session to orient the Kansas Project Team to the project and work together to tailor the Instrument for use in reviewing the child care program . The Kansas and Federal Project Teams reviewed every section, element, and criterion in the initial draft Instrument and, through consensus, modified and tailored the Instrument. The Project Teams deleted several of the elements and their associated criteria from the original document. The Project Teams came to consensus on all modifications to the Instrument before making any changes. In all cases, the deletion of elements occurred because the team members considered the element redundant and so it did not add value to the Instrument.

Several elements had no associated criteria. The Project Teams reviewed each of these and only added criteria when the teams determined that the element needed clarification. The Instrument repeats some elements across issues and several sections have the same criteria, but used in a different context. The Project Team members chose to delay further refinement and modification of the Instrument until after State pilot implementation.

D. Initial Pilot State: Kansas

Kansas piloted the Instrument from November 21, 2005 to January 13, 2006. Following completion, the Federal Project Team conducted a debriefing call to discuss the Kansas experience. After considerable discussion, the Kansas Project Team made several recommendations for the future pilot States.

1. Initial Pilot Observations

The Kansas Project Team expressed that the pilot was “rewarding” and required only a nominal investment of State staff time and resources. Kansas recruited team members from a broad range of program areas to participate on the Kansas Project Team. The Kansas Project Team recognized that the Instrument had relevance across program offices and some of the team members indicated that they planned to share this information more widely in their respective divisions to make process improvements. Kansas anticipates that these improvements will occur as the other program areas absorb the ramifications of the Instrument.

Kansas offered the following recommendations for future pilots:

a. Composition and Responsibilities of the State Project Team

• Select one overall project coordinator. Make sure the coordinator is available during the entire process to answer internal questions, arrange conference calls with the team, consult with team members, and coordinate status and technical assistance call with the contractor when necessary.
• Have a high-level management staff member recruit team members and act as organizational sponsor.
• Form a broad-based team. Include members from: High and Middle Level Management, Audits, Personnel/Human Resources, Information Technology, Program Staff (including those working on the systems for programs if different than IT staff), Quality Assurance or Quality Control, Staff who do Eligibility Determination (workers and/or contractors), Licensing Staff, Finance/Accounting, Operations, Public Information Officer, Legislative Liaison, and County Management/Staff (if county administered). See Appendix B for a listing of potential team member’s functional assignments.
  • Designate adequate support/clerical staff to be involved from the beginning of the project, especially for tracking and compiling data and information.
  • See “Suggested Section Assignments,” Appendix C for more guidance on forming the State Project Team.

b. Orientation Meetings and Ongoing Communication

• When forming the State Project Team, hold an initial team meeting to explain the purpose, process, and sell the project (explain benefits of participation and uses for future work within the agency). Include the organizational sponsor to kick-off the meeting.
• Hold additional orientation meetings, if necessary, to ensure that all persons involved in completing sections of the assessment understand the purpose and process. This is particularly important if staff members who will complete sections of the Instrument are not at the site visit meeting.

c. Gathering and Recording Information

• Discuss the procedures for gathering documentation, including: the use of hyperlinks to both the State Internet and Intranet Web sites; and how hard copy attachment options are to be coordinated with all team members.
  • Provide all team members with an electronic version of the document to use in completing their assigned sections. Have them enter all information onto the electronic document and send to the coordinator who can then assign support staff to compile into one document. More than one team member may complete some sections, which will require integration by the coordinator.
  • When recording supporting documentation, use hyperlinks to the State’s Internet or Intranet sites (URL’s) to reduce the amount of hard copy attachments and to facilitate future updates.

d. Finalizing and Submitting the Completed Assessment

• Have support staff edit the completed Instrument for errors, grammar, and writing style consistency.
• When completed, hold a wrap-up meeting with all team members to share completed documents and receive comments and feedback; make changes to document as agreed upon in discussion.
• Have high-level management staff/team review the completed Instrument to ensure that it is consistent with agency mission and can be integrated with agency strategic or business plan . Kansas had high-level support for this project and shared all findings with top management.

2. Training for Federal Staff in ACF Regional Offices

Upon completion of the Kansas pilot, the CCB convened a one-and-a-half day training session in Kansas City, MO for Regional grants officers and program specialists from CCB Central Office and all ten ACF Regional Offices. Members of the Kansas Project Team also attended the training.

The session covered the background and genesis of the Instrument followed by an explanation of the initial pilot experience by members of the Kansas Project Team. Participants reviewed and discussed each element of the Instrument. The session concluded with a discussion of how the staff in the ACF Regional Offices would provide technical assistance to States during the Instrument pilot and how they could use the results for monitoring purposes. At the conclusion of the session, the CCB selected an additional nine States to invite to participate in the pilot.  

Exhibit 2. State, Commonwealth, and Regional Participation

                                                  * Kansas was the original pilot State

3. Site Visit Protocols

To plan the site visit, WRMA convened a planning conference call with each pilot State and corresponding ACF staff in the Regional Office. Prior to the call, WRMA shared with each pilot State an overview of pilot process and a copy of the Instrument completed by Kansas. These documents provided concrete examples of potential State responses and suggested the level and types of staff who needed to attend the initial technical assistance and training site visit. This information also provided useful “lessons learned” from the Kansas experience regarding team composition and the need for top-level administrative support to assure pilot success. Each State then identified and selected team members and scheduled a site visit.

The Federal Project Team conducted site visits in eight of the nine jurisdictions; Maine chose not to have a site visit due to timing issues. Instead, the Federal Project Team conducted a conference call with the Maine Project Team to answer questions and prepare team members for completing the Instrument.

While the Federal Project Team originally planned a one-and-a-half day site visit, it became apparent that one day was sufficient and remaining site visits were adjusted accordingly. This minimized the disruption to the State. The Federal Project Team consisted of representation from Central and ACF Regional Offices and staff from WRMA.

During the site visit, Federal staff and WRMA provided an item-by-item review of the Instrument. The members from the State Project Team identified as most knowledgeable in a given area volunteered to respond to each section of the Instrument. Team members also addressed the logistics and timeframes for Instrument completion. States also identified who would provide administrative assistance to consolidate the document.

7 Improper Payment Information Act of 2002, Pub. L. No. 107–300, § 2, 116 Stat. 2350. return to text

⁸ Office of Management and Budget. (2003). The Improper Payment Information Act, Public Law No: 107–300 [On-line]. Available: http://www.whitehouse.gov/omb/memoranda/m03-13-attach.pdf. return to text

⁹ Government Accountability Office. (June 2004) TANF and Child Care Programs: HHS Lacks Adequate Information to Assess Risk and Assist States in Managing Improper Payments. (GAO Publication No. GAO–04–723). Washington, DC: U.S. Government Printing Office. return to text

¹⁰ Government Accountability Office. (August 2001) Internal Control Management and Evaluation Tool (GAO Publication No. GAO–01–1008G). Washington, DC: U.S. Government Printing Office. return to text

¹¹ Government Accountability Office. (October 2001) Strategies to manage improper payments: Learning from public and private sector organizations (GAO Publication No. GAO–02–69G). Washington, DC: U.S. Government Printing Office. return to text

¹² Originally, the CCB intended for one State, Territory, Commonwealth, or District from each of the ten Federal Regions to participate in the next phase of the pilot process; however, due to a number of compelling factors, there was difficulty in securing a pilot state in Region III. After discussions, the CCB, decided to proceed with the pilot with eight States and the Commonwealth of Puerto Rico. return to text

Posted January 31, 2008