Child and Family Services Reviews Procedures Manual

	Questions?
	Privacy
	Site Index
	Contact Us

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Administration on Children, Youth and Families
Children's Bureau

Child and Family Services Reviews
Procedures Manual

Washington, D.C.
August 2000

Chapter 1. Framework for the Child and Family Services Reviews

A.		Purpose of the Reviews
B.		Conceptual Framework
C.		Structure of the Reviews
D.		Steps in the Review Process

Chapter 2. The Review Team

A.		Composition of the Review Team
B.		Functions of the Review Team
C.		Leadership of the Review Team

Chapter 3. Statewide Assessment

A.	Purpose of the Statewide Assessment
B.	Completing the Statewide Assessment
C.	Technical Assistance With the Statewide Assessment
D.	Reviewing the Statewide Assessment and Preparing the Preliminary Assessment
	D.1.	Making Decisions Regarding the Onsite Review
	D.2.	Preparing the Preliminary Assessment
	D.3.	Example of Preliminary Assessment
E.	Interim Statewide Assessments Between Full Reviews
	E.1.	State Responsibilities
	E.2.	Regional Office Responsibilities

Chapter 4. Onsite Reviews

A.	Purpose
B.	Onsite Review Activities
C.	Advance Preparation for the Onsite Review
	C.1.	Regional Office ACF Responsibilities
	C.2.	Central Office ACF Responsibilities
	C.3.	Peer Review Contractor Responsibilities
	C.4.	State Agency Responsibilities (Central Office)
	C.5.	Local Agency Coordinators' Responsibilities
D.	Case Selection and Review
	D.1.	Sample of Cases Reviewed
	D.2.	Location of Case Records
	D.3.	Preparation of Records for Review
	D.4.	Case Interviews
E.	State and Local Stakeholder Interviews
	E.1.	Local Stakeholder Interviews
	E.2.	State Stakeholder Interviews
F.	Team Briefings
G.	Instruments

Chapter 5. Final Report

A.	Purpose of the Final Report
B.	Format of the Final Report
C.	Preparation of the Final Report
D.	Content of the Final Report
	D.1.	Example of Final Report
E.	Distribution of the Final Report

Chapter 6. Substantial Conformity

A.	Substantial Conformity on the Outcomes
	A.1.	Rating Outcomes on an Individual Case
	A.2.	Establishing the National Standard on Statewide Aggregate Data
	A.3.	Determining Substantial Conformity for the State
	A.4.	Example of Determining Substantial Conformity on the Outcomes
B.	Substantial Conformity on the Systemic Factors
	B.1.	Example of Determining Substantial Conformity on the Systemic Factors
C.	Resolving Discrepancies Between the Statewide Assessment and the Onsite Review
	C.1.	Example of Resolving a Discrepancy

Chapter 7. Program Improvement Plans

A.		Criteria for Developing a Program Improvement Plan (PIP)
B.		Content of the PIP
C.		Preparation of the PIP
D.		Technical Assistance
E.		Role of Regional Office in Coordinating Technical Assistance
F.		Timeframes for Implementing the PIP
G.		Evaluating the Implementation of the PIP

Chapter 1
Framework for the Child and Family Services Reviews

A. Purpose of the Reviews

The 1994 Amendments to the Social Security Act authorized the U.S. Department of Health and Human Services to review State child and family service programs in order to ensure substantial conformity with the State plan requirements in titles IV-B and IV-E of the Social Security Act. Title IV-B of the Social Security Act and 45 Code of Federal Regulation (CFR) 1357.15 require States to submit child and family services plans (CFSP), i.e., State title IV-B plans, to the Administration for Children and Families (ACF) that include many of the requirements included in the child and family services reviews. The reviews cover child protective services, foster care, adoption, family preservation and family support, and independent living.

In addition to reviewing for the State's substantial conformity with applicable requirements, the reviews are designed to help States improve child welfare services and the outcomes for families and children who receive services by identifying strengths and needs within State programs, as well as areas where technical assistance can lead to program improvements. Other purposes of the reviews include the following:

Ensure that Federal funds are spent in accordance with Federal statute, regulation, and policy
Link the reviews to the joint planning, technical assistance, and program improvement processes that exist between States and Regional Offices
Assist States to become self-evaluating over time
Assemble data that will inform national policy
Provide timely and specific feedback to States that is directly related to program performance and outcomes

B. Conceptual Framework

The child and family services reviews are based on a number of central principles and concepts, including the following:

The child and family services reviews are a collaborative effort between the State and Federal governments. A review team composed of both State and Federal staff conducts the reviews and evaluates State performance.
The reviews rely on information from a variety of sources in making decisions about a State's performance. The sources of information include a statewide assessment, completed by State members of the review team; onsite reviews of a sample of children and families served by the State; statewide aggregate data; and interviews with State and community representatives.
The reviews examine State programs from two perspectives: First, the reviews cover outcomes of services provided to children and families served by the State agency. Second, they cover systemic factors that have an effect on the agency's ability to help children and families achieve positive outcomes.
The reviews are designed to capture both the strengths and needs of State programs. With a strong emphasis on using the reviews to drive program improvements, the reviews identify the strengths of State programs that can be used to make improvements in other program areas, where needed.
The reviews promote practice principles believed to support improved outcomes for children and families, such as family-centered practice, community-based services, strengthening parental capacity to protect and provide for their children, and individualizing services that respond to the unique needs of children and families.
The reviews emphasize accountability. While the review process includes opportunities for States to make program improvements before having Federal funds withheld due to nonconformity, there are significant penalties associated with the failure to make the improvements needed to attain substantial conformity.

C. Structure of the Reviews

There are two phases included in the child and family services reviews: the statewide assessment, which the State completes during the 6-month period prior to the onsite review, and the onsite review.

In examining outcomes, State child and family services programs are reviewed in both phases of the review process, with the information gathered from both phases contributing to the overall evaluation plan and plan for program improvement. The outcomes included in the reviews fall into three domains: safety, permanency, and child and family well-being. Within each domain, specific outcomes are assessed through the onsite review and statewide assessment. The outcomes are as follows:

Safety
Children are, first and foremost, protected from abuse and neglect.
Children are safely maintained in their homes whenever possible and appropriate.
Permanency
Children have permanency and stability in their living situations.
The continuity of family relationships and connections is preserved for children.
Child and Family Well-Being
Families have enhanced capacity to provide for their children's needs.
Children receive appropriate services to meet their educational needs.
Children receive adequate services to meet their physical and mental health needs.

In the first phase of the review, the State members of the review team complete a statewide assessment, using statewide aggregate data to evaluate the programs under review, and address the other requirements subject to review. In the second phase, the onsite review examines outcomes for a sample of children and families served by the State. The onsite review addresses specific outcomes through a focus on the quality of services and practice.

A set of performance indicators is used to evaluate each of the outcomes listed above (see appendix A), based on information from each case reviewed onsite and data in the statewide assessment. The performance indicators are in two forms: one includes qualitative items that are evaluated through case reviews conducted onsite in the State, and the other includes statewide aggregate data obtained from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and Neglect Data System (NCANDS).

A combination of performance indicators is used to assess the achievement of each outcome. For example, in evaluating the outcome "children are, first and foremost, protected from abuse and neglect," the performance indicators for this outcome include the following statewide aggregate data:

The recurrence of maltreatment statewide
The incidence of child abuse and/or neglect in foster care statewide

In addition, the reviewer considers the following performance indicators gathered on each individual case during the onsite review to evaluate this same outcome:

The timeliness of initiating investigations following the receipt of a report of suspected abuse or neglect
The recurrence of maltreatment of the child

In a similar manner, the review of systemic factors occurs at two levels. First, the statewide assessment includes an evaluation by the State members of the review team of the requirements related to each systemic factor covered in the review. Second, during the onsite review, members of the review team interview selected community stakeholders to determine how well each of the systemic factors functions in the State.

The systemic factors examined in the reviews include the following:

Statewide information system
Case review system
Quality assurance system
Staff training
Service array
Agency responsiveness to community
Foster and adoptive parent licensing, recruitment, and retention

The reviews evaluate a set of State plan requirements and other program requirements for each of the systemic factors. Decisions about the State's substantial conformity with the systemic factors are based on the satisfactory implementation of those requirements in the State.

D. Steps in the Review Process

A summary of the timeframes for the major activities in the child and family services review is provided in appendix J. The following steps in the review process include both State and Federal roles and some joint responsibilities:

Determine the dates for the review. The Central Office and Regional Office determine the dates for the review in collaboration with State officials. Reviews must be conducted within the timeframes specified in the final regulation governing the frequency of State reviews.
Form the review team. This involves identifying State and Federal members of the review team, along with individuals from the State who are not staff of the State agency. This step also includes identifying peer reviewers, selected from a nationwide pool of reviewers, maintained under contract, who supplement the Federal review team.
Transmit data profiles, pulled from AFCARS and NCANDS, to the State. In order to reduce the burden on States, the Federal government compiles data submitted by the State to NCANDS and AFCARS into reports that the States use in completing their statewide assessment. If the State does not have complete AFCARS or NCANDS data, it must provide the required data from an approved alternate source.
Complete the statewide assessment. State members of the review team, including those members who are not State staff, analyze the data provided through AFCARS and NCANDS and address the outcomes and systemic factors included in the statewide assessment.
Designate locations for the onsite review activities. The Regional Office and the State jointly identify three locations in the State where the onsite review activities will occur. The State's largest metropolitan subdivision is a required location, and the other two locations are determined from information in the statewide assessment (see chapter 3, D.1).
Select the sample and types of cases to be reviewed on site. The exact number of cases to be reviewed, by location and type of case, i.e., foster care or inhome, will be determined jointly by the Regional Office and the State, based on information in the statewide assessment.
Prepare and disseminate the preliminary assessment. The Regional Office prepares an analysis of the statewide assessment on the Summary of Findings Form (see appendix F). The Regional Office provides the preliminary assessment to the peer reviewer contractor for distribution to all members of the review team.
Prepare for the onsite review. The State, in collaboration with the Regional Office, schedules local stakeholder interviews, prepares reviewer schedules, and plans logistical arrangements, e.g., hotels and transportation for State review team members and space for meetings and review activities. The State also pulls the 30-50 cases for the onsite review from the total sample of 150 foster care cases and 150 inhome cases randomly identified by the Central Office. The Regional Office, in collaboration with the Central Office, the State, and the peer review contractor as needed, assigns team members to local sites, provides orientation or training needed by review team members, and disseminates review materials to the team members before the onsite review.
Onsite review in the State. The review is completed in 1 week by a joint State and Federal review team.
Complete and issue the final report. The Regional Office prepares and distributes the final report of the review, which includes the written determination of substantial conformity, within 30 calendar days of the onsite review or within 30 calendar days of resolving any discrepancies encountered between information in the onsite review and the statewide assessment.
Develop the program improvement plan. The State, in collaboration with the Regional Office, develops a program improvement plan that covers all areas of nonconformity, as determined in the review, and submits the plan to the Regional Office for approval within 90 calendar days of receiving written notice of nonconformity.
Implement the program improvement plan. The State implements the approved program improvement plan, technical assistance is provided through the sources identified in the plan, and the Regional Office monitors implementation of the plan through quarterly reports and other methods addressed in the plan.
Withhold Federal funds for nonconformity. If the State fails to make improvements needed to bring areas of nonconformity into substantial conformity, Federal funds are withheld from the State commensurate with the level of nonconformity.
Conduct subsequent reviews. For States determined to be in substantial conformity, rereviews are conducted at 5-year intervals, with an interim statewide assessment prepared by the State 3 years following the onsite review. States determined not to be in substantial conformity are rereviewed at 2-year intervals.

Chapter 2
The Review Team

A. Composition of the Review Team

The State should begin to identify the State members of the review team approximately 8 months prior to the onsite review (see timeframes in appendix J). The purposes of starting at that point are to have the key review team members from the State agency and those State members who are not staff of the State agency in place to begin the statewide assessment 6 months prior to the onsite review and to provide initial orientation of those members of the team. Some team members will not be identified until closer to the onsite review, i.e., staff of the local sites selected for the reviews based on the completed statewide assessment.

A team of individuals, including the following members, conducts the reviews:

Administration for Children and Families (ACF) Central and Regional Office staff
State central office and local office staff
Representatives external to the State agency with whom the State was required to consult in the development of its title IV-B State plan
Peer reviewers selected from a national pool of reviewers maintained under contract to supplement the Federal review team

The team may also include cross-Regional or cross-State representatives, at the expense of the Region or State sending staff on the review. States and Regions preparing to conduct reviews may find it helpful to send staff to reviews in other States and Regions to help prepare and train them. States or Regions interested in participating in other reviews will make their requests through the Regional Office that is leading the review.

Staff of the federally funded National Resource Centers and other technical assistance providers who may be working with the State on program improvement planning are not included on the review team, in order to avoid possible conflicts of interest if the States are later in a position to purchase technical assistance from the providers. The review teams comprise approximately 25 people, more if needed, at least half of whom are State representatives (both State staff and external representatives from the State). There is a designated Federal team leader for the review team, in addition to local team leaders for each of the three local sites in the State where review activities occur. Also, there are team members designated to perform quality assurance functions during the onsite review in order to ensure consistency and objectivity in reviewing cases.

The reviews encourage cross-system coordination and consultation in States, as promoted by the title IV-B, sub-part 2 planning process, by including review team members from outside the State agency who represent the planning team that develops the State's Child and Family Services Plan (CFSP). Examples of the external representatives the State agency may consider having on the team include foster parents, university social work educators, advocates, coordinators of State child welfare initiatives or projects, tribal representatives, and staff of private and public agencies.

Through arrangements with the Federal government, a contractor will maintain a pool of prospective peer reviewers nationwide. Having access to such a pool of reviewers will help ensure that the review team is composed of individuals who have the knowledge, background, and preparation to conduct qualitative reviews of State child and family service programs. The peer reviewers are not to be confused with State members of the review team who are not State agency staff members. The peer reviewers used through a Federal contract will be used to supplement Federal members of the review team.

B. Functions of the Review Team

Membership on the review team involves a significant commitment of time and effort. State members of the team must be in a position that enables them to work closely with the State in preparing the statewide assessment and fulfill all the responsibilities noted below through the onsite review.

The State members of the review team are responsible for completing the statewide assessment within the timeframes specified in the regulation by submitting the completed statewide assessment to the Regional Office for approval no later than 60 days before the onsite review. For that reason, the State members of the team should be identified before work begins on the statewide assessment. State members of the team who are not directly involved in writing the statewide assessment may be involved in other ways, such as reviewing the data included in the statewide assessment, attending meetings to provide input, submitting information on selected items, and reviewing and commenting on drafts as requested by the State agency.

Following the completion of the statewide assessment, the functions and responsibilities of both State and Federal review team members include the following:

Review the completed statewide assessment in preparation for the onsite review
Participate in any scheduled orientation or training sessions prior to or during the review
Remain present in the review site(s) for the entire week of the onsite review, including participation in all scheduled review activities from the entrance conference through the exit conference
Conduct all assigned activities associated with review of case records and interviews, including completion of the review instruments
Participate in focus groups, open forums, or other group meetings scheduled during the onsite review
Attend daily briefings of the local review teams, and present information on cases reviewed or stakeholders interviewed
Assist in compiling a summary of the team's findings during the onsite review
Turn in completed review instruments on assigned cases and interviews prior to departing the review site at the end of the onsite review
Remain available for telephone consultation following the review, if needed, in order to clarify or supplement information recorded on the instruments

C. Leadership of the Review Team

The team leader for the review will be a Federal staff member of the Regional Office. The major responsibilities of the team leader and staff assisting the team leader include the following:

Serve as liaison with State leadership in planning review activities
Ensure that the statewide assessment and accompanying data are sent to the State on a timely basis and received back in time to plan the onsite review
Prepare an analysis (preliminary assessment) of the statewide assessment 30 days before the onsite review and record it on the Summary of Findings Form
Plan the details of the onsite review with State and Central Office representatives, including arranging conference calls as needed and transmitting review instructions and procedures to State liaisons
Work in collaboration with the peer review contractor for the reviews to ensure that all documents needed for the review, including the preliminary assessment, instruments, handouts, instructions and orientation materials are available and delivered to the State administration and members of the review team in time for the onsite review
Provide leadership for entrance and exit conferences at the beginning and conclusion of the onsite review
Ensure that local site team leaders (see below) are aware of their responsibilities during the onsite review
Coordinate the development, revision, circulation, and release of the final report of the review (which includes the written notification to the State regarding substantial conformity) with the Central Office
Work with designated State staff toward development of the program improvement plan

During the onsite review, the review team will be divided into smaller "local" teams that are assigned to different geographical areas in the State. There should be a local site coordinator, who is a State staff member, in each of the three review locations in the State who handles logistical arrangements, sets up interviews, and so forth. This person may or may not actually be on the review team. Each local team will also have a designated local team leader, who is a Federal representative, with the following responsibilities:

Provide strong positive leadership to the team through setting the pace of work for the week, assisting members to resolve problems with their cases or schedules, supporting the team in all ways to complete the work, and promoting a positive and objective approach to the review
Coordinate the review schedule with local agency staff, e.g., communicate with the local site coordinator to: (1) confirm interview arrangements, and (2) develop schedules, including for local entrance and exit conferences
Coordinate team members' responsibilities for case record reviews; interviews; and other review activities in the local sites, including reassigning members with gaps in their schedules to other tasks
Provide leadership for entrance and exit conferences with staff of the local review sites
Convene and lead daily briefings with local team members
Provide leadership for any scheduled focus groups, open forums, or discussion groups arranged as part of the local review work
Perform a quality assurance review of all onsite review instruments (see appendix C) and stakeholder interview guides (see appendix D), as they are completed, in order to identify missing information and inconsistencies in completing the instruments, and ensure that the performance indicators are used correctly to rate outcomes (if the team leader must have assistance with this function, (s)he may select another team member who is experienced in the reviews to assist)
Ensure that team members complete all assigned review functions, including completing and submitting all instruments, prior to departing the local review site
Coordinate the completion of the Summary of Findings Form for the local team at the end of the review week, including providing all local team members with opportunities for input into the document
Submit local team members' completed instruments and the team's Summary of Findings Form to the designated team leader prior to departing the State at the end of the review week
Participate in review activities by reading case records and conducting interviews

Chapter 3
Statewide Assessment

A. Purpose of the Statewide Assessment

The statewide assessment is the first phase of the review process. It provides States an opportunity to examine data relating to their programs and to consider the data in light of programmatic goals and outcomes for children and families served by the State. The statewide assessment requires State review team members to consider the State's success in helping children and families achieve positive outcomes in the areas of safety, permanency, and well-being. The statewide assessment process leads to identification of the areas where the State is performing well and those areas that need further examination through the onsite review.

The statewide assessment includes data that the Administration for Children and Families (ACF) extracts from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and Neglect Data System (NCANDS) (Detailed Case Data Component data), or another approved source of data in the absence of these sources, and transmits to the State in report format. The data included in the statewide assessment assist States to examine service populations from two different perspectives:

First, the NCANDS and AFCARS data provide point-in-time profiles of the foster care and child protective service populations that include all children in the State who are included in the AFCARS and NCANDS databases. The profiles provide States with information on the status of the service populations as of a given date.
Second, AFCARS data are also used to provide States with information on the cohort group of children entering foster care for the first time in their lives during the period under review. Through these data, States have the opportunity to begin tracking progress in certain areas longitudinally.

In part, the value of examining data on the cohort group of children who enter foster care for the first time in the State is that the effects of the agency's practices and policies, particularly with respect to family preservation, time-limited reunification efforts, and permanency planning, are more easily discernible for a group of children recently entering foster care for the first time than for the entire foster care population. The State's total foster care population will include children who entered foster care several years ago and children who have had multiple entries into foster care. These differences among children in foster care make it difficult to identify the most typical experience of children recently entering foster care in the State. If the data are available, for each statewide assessment, we will provide data on the cohort group of children entering foster care for the first time in each of 3 years, so that the State can compare changes for first-time entries over time.

Since the cohort profiles only include data on children entering foster care for the first time during a given year, we must also use point-in-time data profiles that provide information on all children in foster care in the State. Using both profiles, States can evaluate the status of all children served by the agency, as well as gain insight into where program improvements are most needed based on recent data and practice.

The data profiles also include statewide aggregate data indicators that are used to determine substantial conformity. The Children's Bureau will establish national standards for each of the statewide aggregate data indicators used to determine substantial conformity. When a State is undergoing a child and family services review, the State and the Regional Office will compare the State's data for the period under review with the national standards and determine the State's compliance with the standards.

B. Completing the Statewide Assessment

The Regional Office transmits the statewide assessment instrument and accompanying data to the State for review and completion approximately 6 months prior to the onsite review (see appendix B). States also have ongoing access to the final version of the instrument through the Children's Bureau Web site. (State-specific data used in the statewide assessment are not available on the Internet.) The State must return the completed statewide assessment to the Regional Office no later than 60 days prior to the scheduled onsite review.

AFCARS and NCANDS submissions from the States do not correspond to the same time periods, i.e., AFCARS is submitted on a fiscal year basis and NCANDS is submitted by calendar year. Also, NCANDS submissions, which are voluntary, are a year behind AFCARS submissions. The data from these two sources will, therefore, be based on fiscal year AFCARS data and calendar year NCANDS data, leaving approximately a 3-month time difference in the data included in the permanency and safety profiles.

If a State does not submit data to NCANDS, the State and the Regional Office must agree on an alternate source of statewide data to be used in the safety profile for the statewide assessment. Also, for its first preview, if the State has incomplete AFCARS data, an alternate source of data, approved by the ACF, may be used to generate the foster care data profiles. In the absence of AFCARS or NCANDS data, the Regional Office must request that the State submit its alternate source of data to the Regional Office 8 months prior to the onsite review in order to allow time to approve the data and transmit it to the Children's Bureau to prepare the profiles. The Regional Office must either approve or disapprove the alternate data. The criteria for approving or disapproving the alternate data include:

The source of the data, i.e., the Regional Office must ensure that the data accurately represent the State's service population.
The alternate sources must provide the same data as would have been provided by NCANDS and AFCARS if the State had made complete submissions.

Since some of the data elements on the data profiles are used to determine the State's substantial conformity, failure to provide the needed data, in the absence of AFCARS and NCANDS data, could result in a determination that the State is not in substantial conformity on the outcomes to which those data elements apply.

When the Regional Office has approved the alternate source of data for the profiles, it will transmit the data to the statisticians in the Children's Bureau who will use it to prepare the profiles. The Children's Bureau will prepare the profiles and send them to the Regional Office. The Regional Office will then transmit the profiles and the statewide assessment form to the State 6 months prior to the onsite review.

In addition to statewide aggregate data, the statewide assessment includes a series of narrative-style questions. The State's review team should be in place to conduct the statewide assessment and should be the primary group that responds to the narrative questions.

The responsibilities of the State members of the review team at this point include the following:

Analyzing the data
Meeting to discuss the data and the issues behind the data
Developing responses to questions about the data, in consultation with sources outside the review team as needed

The review team must include representatives outside the State title IV-B/IV-E agency in order to provide a broad perspective on the items in the statewide assessment (see chapter 2). In completing the statewide assessment, States have the option of gathering information through focus groups with stakeholders or consumer groups, surveys, joint planning forums within the State, or otherwise connecting the statewide assessment with ongoing consultation through their title IV-B planning process.

In analyzing the data profiles, the State members engage in the following activities:

Review the statewide aggregate data related to each outcome. The team will identify areas of strength or need for further review onsite. The team members will attempt to identify the reasons for the status of certain data indicators. For example, under Permanency Outcome #1, "children have permanency and stability in their living situations," the State will examine the data on the number of placement settings experienced by children during their current episode of foster care. If the data are available through AFCARS and NCANDS, the State will have 3 years of data on this indicator and will be able to identify whether the indicator is moving in the desired direction. In order to understand the reasons behind the numbers, however, the State will have to look further. For example, the State may have other data that help explain the number of placement settings that children experience, or it may have to review a sample of cases to identify the reasons, or interview caseworkers or foster families.
Compare the State's performance on statewide aggregate data with the national standards, where applicable. For those statewide aggregate data indicators used to determine substantial conformity, the State will compare its data with the national standard and begin to determine the reasons behind the numbers as described above. The importance of understanding the factors that affect these numbers, in particular, lies in the requirement that the State implement a program improvement plan that addresses the numbers that fall below the national standard.
Use the data to identify areas of strength and areas needing further review. The State will need to examine the data in a manner that identifies the program areas that are most in need of further review onsite. For example, if the State members of the review team identify safety as a major concern, they can work with the Regional Office to structure the sample of cases for the onsite review in a way that is likely to capture the safety issues. In this same example, the State will also need to suggest those locations in the State, other than the State's largest metropolitan subdivision which is a required review location, where either the most typical or the most needed observations of the safety issues can be made, depending upon the strengths and needs identified in the statewide assessment.
Use supplemental data, other than the NCANDS and AFCARS profiles, to review other outcomes and systemic factors. Since the NCANDS and AFCARS profiles only cover the permanency and safety outcomes, any other data that the State can produce that address the remaining outcomes and the systemic factors will increase the State's ability to understand the factors that affect its performance.

C. Technical Assistance With the Statewide Assessment

The State has an opportunity to build its capacity for continuous program evaluation and improvement by using the statewide assessment to examine progress on the basis of data. Although some States have the analytical capacity in place to examine and interpret data, others will need assistance in interpreting and manipulating the data, comparing indicators, and relating indicators to outcome measures. To the extent possible, the Federal government will provide technical assistance to States in developing the ability to analyze the data. This assistance may be in the form of Federal staff helping to analyze the statewide assessment; conference calls to discuss the analysis of the information; or access to other sources of technical assistance, such as the National Resource Center for Information Technology in Child Welfare. States will be encouraged to integrate the statewide assessment process into their ongoing quality assurance and program review functions.

D. Reviewing the Statewide Assessment and Preparing the Preliminary Assessment

The Regional Office will review the entire statewide assessment to ensure that it is complete and it addresses all areas appropriately. If critical information is missing or not adequately covered in the statewide assessment, the Regional Office may ask the State to address those areas more completely.

Upon receiving the completed statewide assessment, the Regional Office will review the document and use the information in two ways.

First, in collaboration with the State, the Regional Office will make certain decisions about the onsite review, including:

Locations of review activities
Sample composition and size
Specific issues to target through interviews

Second, the Regional Office will use the information to prepare a preliminary assessment of the State's performance, as reported in the statewide assessment.

D.1. Making Decisions Regarding the Onsite Review

The onsite review activities are conducted in at least three locations in the State. The State's largest metropolitan subdivision is designated in regulation as a required location for the onsite review. This means the metropolitan subdivision in the State, whether that is a county or a city, that has the largest population in the State.

The other two locations are to be determined by the Regional Office in collaboration with the State, and the selection will depend upon issues raised by the statewide assessment. The Regional Office will consider the following criteria in making this decision:

It is not necessary to target the most troublesome geographic areas for the onsite review, although the locations selected must fairly represent practice in the State.
If the statewide assessment identifies particular geographic areas, program areas, or populations of children and families served that need the more intense review provided onsite, locations may be selected that will permit that type of review.
If there are no outstanding programmatic or systemic issues that will be the focus of onsite review activities, the criteria for selecting locations may be to identify the most typical practice in the State, including a mix of rural and urban sites, small and large sites, and so forth.

Because the circumstances in each State will vary, imposing additional requirements on site selection would impede the flexibility of the Regional Office and State to ensure that the onsite review is responsive to individual State issues and needs.

In using the statewide assessment to determine the composition of the sample of cases to be reviewed onsite, the State and the Regional Office determine whether there are program needs that warrant a weighting of the sample toward one program over another. In all State reviews, the sample must include both inhome cases and foster care cases, but the proportion of each case type need not always be equal. For example, if the statewide assessment raises significant safety concerns around children served in their own homes, but does not indicate problems in the foster care program, the sample might be structured to include a larger percentage of inhome cases than foster care cases. If there are not overriding concerns raised in the statewide assessment about one program over another, the sample should be divided evenly between inhome and foster care cases.

Finally, in using the statewide assessment to structure the onsite review, there may be issues raised around outcomes or systemic factors that warrant specific questions of stakeholders. Some examples encountered in the pilot reviews include State policies or practices around screening investigations of reports of child maltreatment that affect child safety, bifurcated systems of service delivery that affect the agency's responsiveness to the community, and contractual issues that affect service delivery. Where such issues exist and are within the scope of the Child and Family Services (CFS) reviews, the Regional Office may address this either by asking the State to schedule specific types of stakeholders for interviews or by advising the review team of specific issues that should be addressed during the interviews.

D.2. Preparing the Preliminary Assessment

The Regional Office will prepare a preliminary assessment of the State's performance on each of the outcomes and systemic factors, based on information from the statewide assessment. The preliminary assessment should be completed by the Regional Office 30 days prior to the onsite review in order to provide adequate time to disseminate it to members of the review team.

The preliminary assessment is based entirely on information from the statewide assessment and is simply a matter of recording information pertaining to the outcomes and systemic factors from the statewide assessment onto the Summary of Findings Form. Decisions about substantial conformity and the final report to the State will be based on information from both the statewide assessment and the onsite review. Recording information from the statewide assessment onto the Summary of Findings Form prior to the onsite review accomplishes the following objectives:

It provides all members of the review team, who will receive a copy of the preliminary assessment, with basic information about the State and the statewide assessment as they begin the onsite view.
It assists the team in combining information obtained onsite with the information in the statewide assessment for use in the exit conference.
It permits a quick identification of areas where there may be discrepancies between information obtained onsite and information from the statewide assessment, so that the discrepancy resolution process can begin immediately following the onsite review.
It identifies, before the onsite review, the State's performance level with regard to the statewide aggregate data and the national standards.
It provides an opportunity to begin preparing the final report, since the Summary of Findings Form is a part of the final report. Given the importance of providing timely feedback to the State following the review, this is an especially important consideration.

Completing the preliminary assessment involves the following steps:

The Regional Office records pertinent data and narrative information from the statewide assessment, describing each of the outcomes and systemic factors, on the Summary of Findings Form.
The Regional Office compares the State's aggregate data used to make determinations about substantial conformity with the national standards and records that information on the form.
The Regional Office works in collaboration with the State and the peer review contractor to see that all review team members receive a copy of the preliminary assessment prior to the onsite review (as part of the information package sent to review team members).

D.3. Example of Preliminary Assessment

The example that follows is one page from the Summary of Findings Form (see appendix F) that illustrates how information from the statewide assessment is used to prepare the preliminary assessment. This information is updated and supplemented during and after the onsite review and serves as the final report to the State for the CFS review.

Example of Preliminary Assessment

II. PERMANENCY

Outcome P1: Children have permanency and stability in their living situations.
Number of cases reviewed by the team according to degree of outcome achievement:
	Team 1	Team 2	Team 3	Total Number	Total Percentage
Substantially Achieved:	__________	__________	_________	_________	__________
Partially Achieved:	__________	__________	_________	_________	__________
Not Achieved or Addressed:	__________	__________	_________	_________	__________
Not Applicable:	__________	__________	_________	_________	__________
Conformity of Statewide data indicators with national standards:
	National Standard	State's Percentage	Meets Standard	Does Not Meet Standard
Foster care re-entries	13%	20%	_________	X
Length of time to achieve reunification	80%	87%	X	___________________
Length of time to achieve adoption	26%	34%	X	___________________
Stability of foster care placements	77%	73%	_________	X
Length of stay in foster care	12 months	9 months	N/A	N/A

Item 5: Foster Care Re-entries

_______ Strength ______ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates that more children (20%) in foster care have had multiple entries within the previous 12 months than the national standard for this indicator (13%). Percentage of children re-entering foster care has increased over the past 3 years from 16% to 20%.
According to the statewide assessment, the State has observed an increase in foster care re-entries as the length of stay in foster care has decreased. State indicates that most re-entries are by children discharged to reunification, as opposed to other discharge reasons. State is concerned about the availability of services to families following reunification.

Item 6. Stability of foster care placement

_______ Strength ______ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates fewer children in the State (73%) have no more than 2 placements in a 12-month period than the national standard (77%) for this indicator. The number of placement settings for children in foster care has not changed significantly over the past 3 years.
Statewide assessment indicates the average number of placements per child in the first episode of foster care is 3. Most children go to shelter care first and then to another placement setting. Foster care disruptions are increasing.

For those outcomes and systemic factors reviewed for which there are no data, the Regional Office will record other relevant information from the statewide assessment in the preliminary assessment. For example, under "Child and Family Involvement in Case Planning" (item 18 on the Summary of Findings Form), the Regional Office will summarize information from section IV-C of the statewide assessment related to this item. Under each of the systemic factors, the Regional Office will record relevant information addressing each factor from sections II and IV of the statewide assessment.

Some of the performance indicators used in the review are only collected on site, and are not addressed in the statewide assessment, e.g., current risk of harm to the child, current relationship of child in care with parents. Those indicators are not addressed in the preliminary assessment.

E. Interim Statewide Assessments Between Full Reviews

E.1. State Responsibilities

States determined to be in substantial conformity are required to complete interim statewide assessments between full reviews. There is no similar requirement for States determined not to be in substantial conformity, since those States are reviewed at 2-year intervals.

The State must submit to the Regional Office a completed interim statewide assessment 3 years from the date of the previous onsite review, meaning that the work on the interim assessment must begin approximately 6 months prior to that time. The process for completing the interim statewide assessment is the same as completing it at the time of a full review, including the participation of representatives external to the State agency.

E.2. Regional Office Responsibilities

The Regional Office initiates the statewide assessment process by transmitting the statewide assessment form and the data profiles to the State 6 months prior to the date the completed statewide assessment is due in the Regional Office. The Regional Office works with the Central Office of the ACF as needed to obtain the data profiles from NCANDS and AFCARS.

The Regional Office reviews the completed interim statewide assessment for indications of the State's status in relation to the outcomes and systemic factors subject to review. It is not necessary for the Regional Office to approve the interim statewide assessment, but if it is incomplete, the Regional Office will ask the State to provide additional information. In particular, the Regional Office will review the interim statewide assessment to determine whether the State is maintaining the level of achievement on the statewide aggregate data required to comply with the national standard. If the State drops below the national standard for the statewide aggregate data, or the agreed upon percentage of achievement for the State, the Regional Office will follow the procedures at 45 Code of Federal Regulation (CFR) 1355.32 (c) for reinstating reviews based on information that the State is not in substantial conformity. The Regional Office will first request that the State submit additional information. If the additional information submitted by the State continues to indicate nonconformity, the Regional Office, in consultation with the Children's Bureau, may initiate either a partial or a full review, as appropriate, to make the determination of substantial conformity.

Chapter 4
Onsite Reviews

A. Purpose

The onsite review is the second stage of the review process. The onsite review includes the examination of a sample of cases for outcome achievement and interviews with community stakeholders to evaluate the systemic factors under review.

The onsite review of cases is designed primarily to gather qualitative information. The cases reviewed on site include child-specific performance indicators that correspond to certain statewide aggregate data, such as foster care re-entries and the recurrence of maltreatment. Other performance indicators reviewed on site cannot be reported in aggregate form through databases, such as the risk of harm to children and the nature of the relationship between children in care and their parents; therefore, the onsite review is the only source of information for those indicators. Through the combination of aggregate data reported on the statewide assessment and case-specific information gathered on site, the review team is able to evaluate outcome achievement within programs and to identify areas where technical assistance is needed to make improvements.

B. Onsite Review Activities

The onsite review occurs over a period of 1 week. While the exact review schedule must be developed for each individual State, a sample agenda is provided in appendix G.

The State agency must schedule the following activities for the onsite review:

An entrance conference for Federal officials to meet with State staff and review team members to review the structure and agenda of the week's activities and provide opportunities to raise and clarify issues pertinent to the review (Note: If an entrance conference is impractical logistically, it may be forgone altogether or conducted by video or audio conference call.)
An orientation of the review team members at the start of the review, if not previously done, to the instruments used in the review and functions of the team (this should be done prior to the onsite review if at all possible)
Appointments with State-level stakeholders for interviews with the reviewers
A meeting of the entire review team at the end of the review week to conduct the final debriefing, compile the Summary of Findings Form for the State, and prepare for the exit conference
An exit conference for Federal officials to meet with State staff and review team members to provide an overview of tentative findings of the review, discuss next steps, and raise and clarify issues related to the review or the preliminary findings

Each local review site coordinator must arrange the following activities for the onsite review:

Informal entrance and exit conferences, where practical, with local officials and local team members at the local review sites within the State
Schedules for each reviewer that include time for case record reviews; stakeholder interviews; and travel time and means of transportation, if needed, to interviews
Appointments for reviewers to interview individuals in the cases being reviewed
Appointments for the reviewers to interview stakeholders
Daily team briefings at the local review sites

C. Advance Preparation for the Onsite Review

Preparation for the onsite review will be carried out by the Central Office of the Administration for Children and Families (ACF), the Regional Offices, the State central and local agencies, and the contractor handling peer reviewers and certain logistical arrangements for the reviews. The responsibilities of each of those parties are listed below:

C.1. Regional Office ACF Responsibilities

Assign an overall review team leader and other Regional Office team members
Provide Regional Office staff to serve on the review team, including leading or co-leading local teams
Consult with State and Central Office staff about the review team composition, including representatives from other States or Regional Offices if they are to be included on the review team
Notify the peer review contractor, 3 months prior to the onsite review, of the number of peer reviewers needed for the review and approve the list of peer reviewers selected for the review
Provide the peer review contractor with information needed to send out to review team members before the onsite review week
In collaboration with the State and Central Office, identify State-Specific systemic issues from the statewide assessment that need further review onsite, determine locations of the local review sites, and determine the sample size and composition
In collaboration with the State, ensure that all required State and local stakeholders are scheduled for interviews during the onsite review
Request a listing of cases that were open in the State for inhome services during the period under review, from which the sample of inhome cases will be drawn, and transmit the listing to the Children's Bureau for sample selection
Return to the State the random sample listings from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the list of inhome cases after the Children's Bureau has drawn the samples
Plan for the orientation and training of review team members
Develop the agenda for the entrance conference in collaboration with the State
Determine which individuals on each local review team will provide quality assurance functions to ensure consistency in ratings, correct application of criteria to performance indicators and outcomes, and accuracy in making review-related decisions

C.2. Central Office ACF Responsibilities

Identify Central Office review team members and make arrangements for all team members to arrive and be present for the entire review period
Arrange for the transportation and lodging of Central Office team members
Provide Central Office reviewers with training and preparation to participate as reviewers or local team leaders, including Regional Office staff by teleconference or otherwise where possible
Assign Central Office staff members to serve as local team leaders on the review, as needed
Review the statewide assessment and the preliminary assessment in order to become familiar with the State's performance before the onsite review
Consult with the State and Regional Office on sample size and composition, locations of review sites, and issues needing particular attention in the review
Identify a random sample of cases to be reviewed onsite from AFCARS data (foster care cases) and the listing of inhome service cases provided by the State, and transmit the sample listing through the Regional Office to the State
Assist, as needed, with conducting prereview orientation or training for team members

C.3. Peer Review Contractor Responsibilities

Identify peer reviewers for the review and ensure that they are trained and prepared to participate in the review
Provide the Regional Office with the names of peer reviewers 2 months prior to the onsite review
Obtain Regional Office approval of the peer reviewers selected for the review
Make logistical arrangements for the peer reviewers, e.g., transportation and lodging (Note: The contractor should coordinate these arrangements with Federal and State staff to ensure that peer reviewers are housed in the same locations as Federal and State members of the review team.)
Transmit advance packages of information to all review team members, including logistical information and other materials to prepare them for the review (A checklist of information that should be sent to review team members is included in appendix H.)
Make arrangements for copies of the review instruments and any other information needed by team members during the review week to be available in the State at the beginning of the review

C.4. State Agency Responsibilities (Central Office)

Assign a State coordinator for the review to act as liaison with the Regional Office and the contractor in making arrangements for the review
Identify State members of the review team and provide that information to the Regional Office
Assign local agency review coordinators in each of the local sites selected for the review: they may or may not be members of the review team; they are responsible for setting up interviews, making local arrangements, and ensuring that case records to be reviewed are available
Identify local review sites, including the State's largest metropolitan subdivision, in consultation with the Regional Office and on the basis of information from the statewide assessment
Ensure that all local agency review coordinators have a copy of the procedures manual and are well oriented to the review process
Prepare, and submit to the Regional Office 45-60 days prior to the onsite review, or as soon as the composition of the onsite sample has been determined, a listing of cases that were open for inhome services for any portion of the year under review, from which the sample of inhome service cases will be drawn
Transmit the total sample listings of 150 foster care cases and 150 inhome cases to local agency coordinators 30 days prior to the beginning of the onsite review; local agencies must have sufficient time to review the total sample listings, identify the 30-50 cases to be included using the criteria provided in this manual, contact the persons involved in the cases, and schedule interviews
Schedule State stakeholder interviews
Consult with Regional Office staff and local agency coordinators on logistical arrangements for the review, including:
Identification of lodging arrangements for onsite review team members
Identification of locations for entrance and exit conferences
Identification of space needed for other scheduled meetings and review activities during the week

C.5. Local Agency Coordinators' Responsibilities

Select the cases to be reviewed from the random sample pulled for the review, on the basis of criteria discussed below under "Case Selection and Review"
Orient local staff to the purposes and activities of the review
Ensure that workers assigned to the families selected for review are scheduled for interviews on their case(s)
Schedule and confirm interviews with each person involved in a case who is to be interviewed and orient those persons to the purposed of the review (see appendix E)
Schedule and confirm local stakeholder interviews and orient those persons to the purposes of the review
Schedule and confirm any planned focus groups or other meetings that will be a part of the local review activities
Prepare an agenda for each review team member that includes time to review cases assigned to that individual; the name, time and date of each scheduled interview and/or meeting; and time for the local entrance and exit conferences
Prepare maps or other written directions for each reviewer to get to the scheduled appointments
Reserve space for record review, interviews, and other planned meetings
Assemble all case records to be reviewed so that they will be accessible and ready for review at the onset of the review
Secure any releases of information or confidentiality forms needed to permit reviewers who are neither State nor Federal staff to access case records and interview the individuals associated with a case

D. Case Selection and Review

D.1. Sample of Cases Reviewed

The Regional Office collaborates with the State to determine the number and composition of the sample of case records to be pulled for the onsite review, on the basis of information in the statewide assessment. The sample of cases reviewed will include children in foster care and children receiving services in their own homes.

The sample of foster care cases reviewed on site is selected randomly from the AFCARS data submitted by the State, after the locations and composition of the sample have been determined. The sample of inhome cases is selected randomly from a listing of inhome service cases that were open to services for at least 60 days during the period under review, including cases that were subsequently closed. The State must provide that listing of cases, since the information is not currently available through the National Child Abuse and Neglect Data System (NCANDS) or other national data sources, and should submit it to the Regional Office 45-60 days prior to the onsite review. The listing of cases must include information that will permit selection of the sample by location in the State, since cases will only be drawn for those locations selected for the onsite review. The Regional Office will provide the list of inhome cases to statisticians in the Children's Bureau, who will select a random sample of 150 cases from the listing of inhome cases and 150 cases from the AFCARS data for those locations in the State where the onsite review activities will be conducted. The Children's Bureau will send the sample listings back to the Regional Office, which will then transmit them to the State.

The sample is selected from the universe of cases in the program area under review. The universe includes all children reported on the AFCARS and all families whose cases were open for services during the period under review in the locations selected for the onsite review. Cases of children in foster care will be pulled by child rather than by family, and cases involving inhome services will be pulled by family. Cases selected for the sample must have been open for services for at least 60 days during the period under review, although they may be closed at the time of the review. The Regional Office will advise the State of the specific timeframe covered by the review, which will most likely correspond to the Federal fiscal year of the AFCARS data used for the permanency profiles.

The total sample pulled will consist of 150 foster care cases and 150 inhome cases from the population of cases in the three locations to be reviewed on site. From these listings, a smaller subsample totaling 30 to 50 cases will be selected for review on site across the three locations. For example, approximately 10-20 cases will be reviewed in each of the three locations. The total of 30-50 cases reviewed across the three locations will include both foster care and inhome cases. The proportion of cases pulled from the foster care and inhome case listings will reflect the composition of the sample as decided jointly by the Regional Office and the State, on the basis of information in the statewide assessment. However, both inhome and foster care cases must be included in the sample of cases reviewed onsite. If necessary, to ensure that both types of cases are adequately represented in the sample, the size of the sample of cases reviewed may be increased.

Interviews may not be possible in all the cases in the sample. Therefore, local agency coordinators in the sites being reviewed will select the cases for the review from the random listings of 150 cases, using the criteria below:

The local review coordinators must begin with the random list of case records and select, in order, the first cases on the list for their locations that meet the criteria listed in the bullets below, bearing in mind that all key individuals may not be available for interviews in each case selected.
The local review coordinators will determine whether the individuals in the case who must be interviewed are willing to be interviewed and available and can be reached during the review week. The case may be rejected and the next case on the list considered only under the following circumstances:
If enough of the individuals to be interviewed in the case are either unavailable or completely unwilling to be interviewed that sufficient information cannot be obtained to complete the review instrument
If the case was pulled in error and had not been open for service at any point during the review period, or, for inhome cases, open at least 60 days during the review period
All cases selected for the sample must have some interviews with the relevant parties in the case. Otherwise, the instrument cannot be completed accurately.
The case must have been open for a long enough period of time, i.e., 60 days, during the year under review to have sufficient information to review the case. The listing of cases from which the sample is pulled should only include those cases open for at least 60 days during the period under review. If cases open less than 60 days appear on the list of 150 cases, they should not be considered for the sample of 30-50 cases that will be reviewed.
Local review coordinators must record the reasons for eliminating any cases from the sample of cases to be reviewed on site and make that information available to the Regional Office review team leader.

The cases in the sample of 150 cases that are not selected for review will be used:

To substitute for cases in the sample of 30-50 that cannot be reviewed for the reasons listed above

To provide a pool of additional cases to be reviewed, if needed, to resolve discrepancies between information in the statewide assessment and the findings of the onsite review

D.2. Location of Case Records

All case records being reviewed will be assembled in the local review sites so the reviewers will have access to the caseworkers assigned to the cases.

D.3. Preparation of the Records for Review

Case records must be as orderly and up-to-date as possible, including any files maintained separately, e.g., separate child protective service files or separate family and child records. If the agency uses electronic files instead of paper files, it will be necessary for the local review coordinator to either make computers and technical support available to the reviewers for viewing the electronic records or obtain hard copies of the files or the portions of the files containing information relevant to the review. The caseworkers assigned to each case must be available for interviews with the reviewers. If electronic files are used, staff of the local agency must also be available to assist reviewers in obtaining additional information from the files that might be needed in addition to the hard copy files. If necessary, the State agency will obtain confidentiality statements or releases of information required by the State agency prior to the onsite review, in order to permit reviewers to read case records and interview the relevant individuals in a case. The peer reviewer contractor will obtain signed confidentiality statements from peer reviewers participating in the review.

D.4. Case Interviews

The review team member(s) assigned a particular case is responsible for interviewing the individuals involved in the case, as well as reviewing the case record.

The following persons must be interviewed in a case unless they are unavailable or completely unwilling to be interviewed:

The child (if old enough)
The child's parent(s)
The child's foster parent(s) if the child is in foster care
The family's caseworker with the agency
Any major service providers involved with the child or family; where there are numerous service providers involved with a family, it may only be necessary to schedule interviews with those most recently involved, those most knowledgeable of the family, or those representing the primary services the family is receiving

As needed on a case-by-case basis, other individuals who have relevant information on the case may also be interviewed, such as the child's guardian ad litem, advocate, or other family members.

Only school-age children will be interviewed unless other arrangements are made with the State. Cases involving preschool-age children may be selected for the sample with no child interview scheduled. For example, a preschool-age child included in the review may be observed by the reviewer in the foster home while interviewing the foster parent(s).

If possible, interviews should be conducted where the persons to be interviewed are located, i.e., in the foster home or in the family's home. Where travel arrangements and the availability of reviewers do not permit the time needed to travel to those locations, or when persons to be interviewed prefer not to have the reviewer in their homes or offices, the local coordinators may arrange the interviews in a central location. Also, telephone interviews may be arranged for individuals who are located outside of the local review site.

The interviews with the individuals in a case should follow the time designated on the reviewer's schedule for reviewing the case record. Without first becoming familiar with the circumstances of the case through a review of the record, the reviewer will be unable to explore pertinent issues with the person being interviewed.

Local site coordinators will schedule the interviews not to exceed 1 hour per interview and allow for time between interviews for any necessary travel to the appointments. Maps or other written directions to the interview sites should be prepared in advance and provided to the reviewer. Unless there are specific concerns about a reviewer interviewing someone alone, the caseworker will not need to accompany the reviewer on the interviews. If there are concerns about safety or other issues related to the interview, the local coordinator will advise the reviewer and assist in taking whatever precautions are needed, e.g., scheduling two reviewers for a particular interview or arranging the interview in the office.

Persons to be interviewed must be prepared for the interview by the local coordinator or designee by helping them to understand the purpose of the review. They should be assured that their participation is voluntary, but that their participation is critical to the success of the review. Also, when the interviews have been scheduled, the appointments should be confirmed in writing. (See Preparation for Interviews and Sample Confirmation Letters in appendix E.)

E. State and Local Stakeholder Interviews

The onsite review includes interviews with community or State representatives, i.e., stakeholders, who are knowledgeable about the functioning of the agency in the State and community. The purpose of these interviews is to obtain information about the systemic factors under review and about how the systemic factors affect the outcomes for children and families in general, not on a case-specific basis. Information from the stakeholder interviews is used in combination with information from the statewide assessment to determine the State's conformity with State plan and program requirements for each of the systemic factors.

The review team will interview a complete set of stakeholders in each local review site for the local perspective. In addition, the team will interview stakeholders at the State level who can provide a broader, statewide perspective.

Stakeholder interviews are not to be confused with the interviews conducted on individual cases. Stakeholder interviews are not designed to elicit information on specific cases.

The perspectives and knowledge of individual stakeholders will vary and will determine which systemic issues they can and cannot address. It is unlikely that any single stakeholder will be able to cover each of the systemic factors with equal knowledge. Therefore, in each local review site, and at the State level, the review team must ensure that the combined information obtained from all the stakeholder interviews adequately addresses the three outcome areas and the seven systemic factors.

A Stakeholder Interview Guide is provided (see appendix D) to guide reviewers in interviewing stakeholders. The Regional Office team leader will complete the State-Specific Issues section of the Stakeholder Interview Guide prior to the onsite review so that reviewers will cover the same issues in the three review locations.

State or local agency coordinators may prefer to set up group meetings or focus groups with some stakeholders in place of individual interviews. If so, the meetings should generally be limited to 8-10 individuals whose interests and involvement in child and family services are similar, for example, groups of foster parents, or law enforcement or education representatives.

If possible, State and local stakeholder interviews should be scheduled during regular work hours since the review teams often meet in the evenings for team briefings. However, it may be impossible to arrange for all interviews to occur during regular work hours, and some may have to be scheduled in the evenings.

E.1. Local Stakeholder Interviews

In each local review site, prior to the onsite review, the local review coordinator will schedule a maximum of seven to ten stakeholder interviews. The interviews may be scheduled during the other review activities, depending upon the availability of individuals for interview. Stakeholder interviews should be scheduled for 1 hour each, and the schedule should allow for any necessary travel between appointments. Stakeholder interviews may be conducted either at the local agency or where the stakeholders are located. As in the case-specific interviews, the stakeholders to be interviewed should be prepared for the interviews, and the appointments confirmed in writing.

The following stakeholders must be scheduled for interviews in each local site:

Local child welfare agency administrator
Foster parents (preferably a small group meeting)
Juvenile court judge (or the judge's designated court representative)
Caseworker(s) from the local agency (preferably a small group meeting)
Guardian ad litem/legal representatives (individually or in a group)
Agency attorney(s) (individually or in a group)
Local representatives of administrative review bodies, e.g., foster care review boards, if they exist

Additional representative stakeholders may be selected from the representatives with whom the State consulted in the development of its State Plan, such as:

Tribal representatives
Law enforcement representatives
Youth services representatives
Major initiative/project representatives
Major service providers
Mental health representatives
Education representatives, including special education or early intervention coordinators
Local child and family advocates

E.2. State Stakeholder Interviews

In each State, interviews will also be scheduled with stakeholders who can address issues of concern to the State as a whole, as opposed to local issues. One or two reviewers of the review team will take responsibility for conducting the State interviews. If the location of the stakeholders relative to the review team presents a logistical problem, reviewers may conduct interviews by phone. Prior to the onsite review, the State coordinator for the review will schedule these interviews in collaboration with Federal staff. No more than five to 10 State stakeholder interviews should be scheduled.

The following State stakeholders must be scheduled for interviews:

State child welfare director
State child welfare program specialists (foster care, protective service, adoption, etc.)
State court system representative(s)
Major tribal representatives
State representative(s) of administrative review bodies, e.g., foster care review boards

Additional State stakeholder may be selected from the representatives with whom the State consulted in developing its State Plan, such as:

State education system
State youth services agency
State health department
State Medicaid program
State mental health agency
State child welfare advocates
University social work education program
Major initiative/project representatives
State foster parent association

F. Team Briefings

The local teams will meet daily during the onsite review to review the day's activities. One of the primary purposes of the briefings is for each local review team to complete, over the course of the review week, a Summary of Findings Form based on their case reviews and local stakeholder interviews.

The briefings are the designated forum for individual reviewers to discuss their cases and their rationale for assigning particular ratings. While individual reviewers assign ratings to the cases they review, the briefings provide an opportunity for the team leader and other reviewers to help ensure that all reviewers use consistent rating criteria and are able to substantiate their ratings with adequate information. The briefings must provide opportunities for case discussion within a structured agenda in order to ensure that all cases are adequately briefed and considered by the team.

The briefings should occur following the onsite review activities for the day, and should include the following activities:

Team members who have completed case reviews that day give a brief summary of the case to the local team and provide their ratings for each of the outcomes.
The local team leader records the ratings on the Summary of Findings Form.
Team members who have interviewed stakeholders give a brief summary of the interviews, addressing the systemic issues covered in the interviews.
Team members raise problems or concerns about the schedules, logistical arrangements, instruments, or other areas before the team.
The team leader determines whether all review activities are proceeding according to schedule and whether adjustments are needed.

During the daily briefings, the team leader uses the Summary of Findings Form to record the number of cases reviewed by the team, according to the degree of outcome achievement (substantially achieved, partially achieved, not achieved, or not applicable) for each of the safety, permanency, and well-being outcomes. Team members also discuss, and the team leader records on the form, the performance indicators that substantiate the ratings on the outcomes.

The local teams also summarize the information obtained from the local stakeholder interviews as it relates to the systemic factors and the outcomes. The team leader records this information in the section of the Summary of Findings Form dealing with systemic factors. By the end of the review week, all of the cases reviewed and information from all of the stakeholder interviews should be recorded on the form.

At the end of the onsite review, when the three local teams come together for the final briefing, the Regional Office review team leader will use the Summary of Findings Form as a guide for the final briefing, addressing each item with the entire review team. The information gathered by the three teams during the onsite review is consolidated with the preliminary assessment, which was prepared before the onsite review, for purposes of the exit conference. The Regional Office review team leader should be able to give the State a preliminary report on the outcomes and systemic factors. This verbal report provided at the exit conference should be presented as the team's tentative findings, since a complete analysis and compilation of the information will not be possible until after the onsite review. A determination of substantial conformity cannot be provided at the exit conference, and will be included in the written final report to the State following the onsite review.

G. Instruments

Standardized instruments and instructions are provided by the ACF for all phases of the review (see appendices B, C, D, and F). Orientation to the instruments used during the onsite review will be provided to review team members prior to or at the onset of the review. The following instruments are needed to complete the review:

Statewide assessment
Onsite Review Instrument
Stakeholder Interview Guide
Summary of Findings Form

Copies of the instruments and instructions will be provided for the review by the Regional Office through the peer review contractor. Also, copies of the instruments will be available through the Children's Bureau Web.

Chapter 5
Final Report

A. Purpose of Final Report

The final report is a compilation of the agency's strengths and needs for each of the outcomes and systemic factors reviewed. The primary purpose of the final report is to document for the State the determination of substantial conformity or nonconformity in each area reviewed. The State will use the information in the final report to develop a program improvement plan (PIP) for any area determined not to be in substantial conformity.

B. Format of the Final Report

The format for the final report is the Summary of Findings Form, which is used throughout the review process, supplemented by additional information described below. The preparation of the form begins with a preliminary assessment prior to the onsite review (described in chapter 3), and is then completed during the onsite review and immediately thereafter. The final version of the Summary of Findings Form serves as the final report of the review to the State.

C. Preparation of the Final Report

Information is gathered for the final report at three points in the process:

The preliminary assessment of the outcomes and systemic factors under review is completed by the Regional Office when the State submits the statewide assessment.
The preliminary assessment is updated and supplemented by the team leaders during the onsite review with information gathered on site from case reviews and stakeholder interviews.
The final compilation of the information from the onsite review and the statewide assessment is prepared by the Regional Office immediately following the onsite review.

Following the onsite review, the Regional Office completes the Summary of Findings Form and makes a determination about substantial conformity. Since the preliminary assessment includes information on the performance indicators from the statewide assessment, completing the final report consists of adding information from the onsite review, i.e., case reviews and stakeholder interviews.

D. Content of the Final Report

The information recorded under each item in the Summary of Findings Form must be specific to the item, clearly stated, and relevant to the applicable State plan requirements for the item. If possible, each item should be addressed by including information obtained from all three sources of information used in the reviews: the statewide assessment, the case reviews, and the stakeholder interviews. This will ensure that all of the necessary information will be used in making determinations about substantial conformity. Each item must be addressed using only that information obtained during the review process. The confidentiality of individual children, families, and representative stakeholders is protected in the report by not citing names of persons or organizations from which information was obtained.

The completed final report to the State will include:

Cover Letter. The cover letter includes a statement about substantial conformity; the amount of the penalty, if applicable; and the date by which a PIP must be submitted to the Regional Office, if applicable.
Executive Summary. This section is a summary of the major strengths and needs noted in each outcome and systemic area and the status of substantial conformity.
Introduction. This section provides an overview of the background and purposes of the review; the outcome areas reviewed: and dates and descriptions of the review activities, such as time periods involved, methods of completing the statewide assessment, review locations, review team representatives, and number and type of cases reviewed. (See appendix K for sample introductory language.)
Summary of Findings Form. This form is completed and noted above, along with the final sections of the form that address the areas of substantial conformity.

D.1. Example of Final Report

The example on the following page is one page from the Summary of Findings Form, illustrating how the preliminary assessment example in chapter 3 is updated with information from the onsite review for the final report.

Example of Final Report

II. PERMANENCY

Outcome P1: Children have permanency and stability in their living situations.
Number of cases reviewed by the team according to degree of outcome achievement:
	Team 1	Team 2	Team 3	Total Number	Total Percentage
Substantially Achieved:	7	7	12	26	81%
Partially Achieved:	3	1	1	5	16%
Not Achieved or Addressed:	0	1	0	1	3%
Not Applicable:	6	6	6	18	__________
Conformity of Statewide data indicators with national standards:
	National Standard	State's Percentage	Meets Standard	Does Not Meet Standard
Foster care re-entries	13%	20%	_________	X
Length of time to achieve reunification	80%	87%	X	___________________
Length of time to achieve adoption	26%	34%	X	___________________
Stability of foster care placements	77%	73%	_________	X
Length of stay in foster care	12 months	9 months	N/A	N/A

Item 5: Foster Care Re-entries

_______Strength ___X___ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates that more children (20%) in foster care have had multiple entries within the previous 12 months than the national standard for this indicator (13%). Percentage of children re-entering foster care has increased over the past 3 years from 16% to 20%.
According to the statewide assessment, the State has observed an increase in foster care re-entries as the length of stay in foster care has decreased. State indicates that most re-entries are by children discharged to reunification, as opposed to other discharge reasons. State is concerned about the availability of services to families following reunification.
Of 32 foster care cases reviewed onsite, ten had been discharged from foster care and re-entered within the past year. All ten had re-entered due to the same general reasons as the original entry. There was little evidence of post-reunification services to families, and no safety plans were developed for children being reunified.
Service providers interviewed in all 3 review sites indicated they receive few requests for post-reunification services. When requested, they typically provide individual or family counseling and parenting classes. There was no evidence of more specialized services to support reunification.

Item 6. Stability of foster care placement

_______ Strength ___X___ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates fewer children in the State (73%) have no more than 2 placements in a 12-month period than the national standard (77%) for this indicator. The number of placement settings for children in foster care has not changed significantly over the past 3 years.
Statewide assessment indicates the average number of placements per child in the first episode of foster care is 3. Most children go to shelter care first and then to another placement setting. Foster care disruptions are increasing.
Of the 32 children in foster care reviewed, 18 of the children had been in two placement settings, 9 had been in three settings, and five had been in four or more settings. Most of the children were initially placed in shelter care and later moved. Except for moving children from shelter, all but five of the other moves were unplanned disruptions.
Foster families interviewed indicated a need for more support services, such as respite care and behavior management. They ndicated that while caseworkers provide assistance when requested, they generally have infrequent contacts with their workers unless there is a problem.

E. Distribution of the Final Report

The Regional Office will provide copies of the final report to the State agency executive officer, the State review coordinator, the Central Office and Federal review team members, and the peer review contractor within 30 days of completing the onsite review. In accordance with Federal requirements that the reports be made available to the public, States will distribute copies of the report to State members of the review team and make the results of the review available to the public. The Children's Bureau will also publish information pertaining to the reviews on its Web site or through other means, as appropriate.

Chapter 6
Substantial Conformity

Determinations of substantial conformity are made separately for each of the seven outcomes and the seven systemic factors under review. A State may be determined to be in substantial conformity on one or more of the outcomes or systemic factors and not in substantial conformity on the others. Program improvement plans and penalties cover only those areas determined not to be in substantial conformity.

Appendix I contains a chart that displays the criteria for determining substantial conformity with the outcomes and the systemic factors. The chart also lists each of the performance indicators, including the statewide aggregate data, used to determine substantial conformity. There are separate methods for determining substantial conformity on the outcomes and the systemic factors, as described below.

A. Substantial Conformity on the Outcomes

Two sets of information are used to determine the State's substantial conformity on each of the seven outcomes: the percentage of cases reviewed in which the outcome was determined to be substantially achieved and the State's performance on the statewide aggregate data for which national standards have been established.

First, reviewers must determine whether the outcomes in each individual case they review are substantially achieved. Those decisions are based on the performance indicators listed in the Onsite Review Instrument, which is completed for each case.

Second, the review team must make determinations regarding substantial conformity for the State as a whole. Those decisions are based on the percentage of cases reviewed in which the outcomes are substantially achieved, and statewide aggregate data for selected outcomes. Currently, in only two of the seven outcomes (Safety Outcome #1 and Permanency Outcome #1), decisions about the State's substantial conformity are made on the basis of statewide aggregate data in addition to the findings of the onsite review. For the remaining five outcomes, the performance indicators reviewed on site provide the basis for determining the State's substantial conformity. In the future, other statewide data indicators may be included in determinations of substantial conformity.

The diagram below illustrates the process of determining substantial conformity on the outcomes.

Step 1

Step 2

Step 3

Step 4

A.1. Rating Outcomes on an Individual Case

When a reviewer has gathered all of the information on a specific case through review of the case record and interviews with parties in the case, (s)he must make a decision as to whether each applicable outcome in the case has been substantially achieved, partially achieved, or not achieved.

The performance indicators used to make determinations as to whether each outcome has been substantially achieved are listed in the Onsite Review Instrument (see appendix C). The reviewer uses the following criteria to determine whether or not the outcomes have been substantially achieved in each individual case:

Safety Outcome #1: Both onsite indicators must be rated a "strength."

Safety Outcome #2: All three indicators must be rated a "strength."

Permanency Outcome #1: Not more than one of the six onsite indicators may be rated an "needing improvement."

Permanency Outcome #2: Not more than one of the six onsite indicators may be rated an "area needing improvement."

Well-Being Outcome #1: Not more than one of the five onsite indicators may be rated an "area needing improvement."

Well-Being Outcome #2: The one onsite indicator must be rated a "strength."

Well-Being Outcome #3: Both onsite indicators must be rated a "strength."

The reviewer will determine an outcome in an individual case to be partially achieved if some of the performance indicators for that outcome have been rated as "strengths," but fewer than the number noted in the paragraph above. If none of the performance indicators are rated as "strengths" for a particular outcome, the reviewer will determine that the outcome is not achieved.

A.2. Establishing the National Standard on Statewide Aggregate Data

A national standard will be established for each of the statewide aggregate data indicators used to determine substantial conformity in the reviews. For the foster care indicators that are based on the Adoption and Foster Care Analysis and Reporting System (AFCARS) data, the national standard will be established according to the following procedure:

Each State and the District of Columbia submits data through AFCARS for selected time periods. (Note: The foster care standards will initially be set using 1998a and 1998b AFCARS submissions.)
All of the data submissions for the time periods are pooled. Assuming that each of the States and the District of Columbia submitted complete data for each of two periods, there will be 102 data elements in the pool.
All of the data submissions are then rank-ordered on a scale from highest to lowest.
The point on the scale that represents the 75^th percentile is the national standard for that statewide aggregate data indicator.

The same process is used to establish the national standards on the two statewide aggregate data indicators derived from the National Child Abuse and Neglect Data System (NCANDS) data, except that two NCANDS reporting periods are used to establish the standard. (Note: The safety standards will initially be set using 1997 and 1998 NCANDS data.)

When a State begins a child and family services review, its data for the period under review are compared with the national standards to determine the State's substantial conformity. States whose data fall below the national standard in a review will be required to implement a program improvement plan (PIP) designed to improve the States' performance on the data indicators in order to achieve substantial conformity. However, the State and the Regional Office may negotiate a percentage of improvement to be made in the statewide data indicators over the course of a PIP that is less than the national standard. In those circumstances, if the State achieves the level of improvement agreed upon in the PIP, the State will not be penalized for nonconformity on the basis of the statewide aggregate data indicator. The criteria for determining the amount of improvement that must be made through a PIP are discussed in chapter 7, Program Improvement Plans.

With a goal of continuous quality improvement, States whose data remain below the national standard in subsequent reviews will be required to establish new benchmarks of improvement to be made toward the eventual attainment of the national standard. As long as the State reaches the agreed-upon level of improvement, failure to reach the national standard will not be the basis for withholding Federal funds with respect to the outcome in question.

A.3. Determining Substantial Conformity for the State

In order for the State to be determined to be in substantial conformity on any given outcome, the outcome must be determined to be substantially achieved in 95 percent of the cases reviewed (90 percent in the first review). In addition, the State must meet the national standard that has been established for any statewide aggregate data attached to that particular outcome.

The example on the following page illustrates how the cases reviewed on site are tallied by outcome to determine, in part, if the State is in substantial conformity.

Currently, national standards will be established for statewide aggregate data on two of the outcomes, Safety Outcome #1 and Permanency Outcome #1. Those statewide aggregate data include the following:

Safety Outcome #1:

Recurrence of maltreatment (Of all children who were victims of substantiated or indicated child abuse and/or neglect during the first 6 months of the reporting period, XX percent had another substantiated or indicated report within a 6-month period.)
Incidence of child abuse and/or neglect in foster care (Of all children in foster care in the State during the period under review, XX percent were the subject of substantiated or indicated maltreatment by a foster parent or facility staff.)

Permanency Outcome #1:

Foster care re-entries (Of all children who entered care during the year under review, XX percent re-entered foster care within 12 months of a prior foster care episode.)
Stability of foster care placement (Of all children who have been in foster care less than 12 months from the time of the latest removal, XX percent had no more than two placement settings.)
Length of time to achieve adoption goal (Of all children who exited foster care during the year under review to a finalized adoption, XX percent exited care in less than 24 months from the time of the latest removal from home.)
Length of time to achieve reunification (Of all children who were reunified with their parents or caretakers at the time of discharge from foster care, XX percent were reunified in less than 12 months from the time of the latest removal from home.)

Example of Calculating Substantial Achievement of Outcomes in Cases Reviewed On Site

Outcome

Number of Cases Substantially Achieved

Number of Cases Partially Achieved

Number of Cases Not Achieved

Percentage of Cases Substantially Achieved (N=30)

Outcome S1: Children are, first and foremost, protected from abuse and neglect.

Child and Family Services Reviews Procedures Manual Washington, D.C. August 2000

TABLE OF CONTENTS

Child and Family Services Reviews
Procedures Manual

Washington, D.C.
August 2000