Child Welfare Outcomes 2000:
Annual Report
Chapter II. Technical Issues to Consider in Interpreting Outcome
Data
The data provided in the State Data Pages of the Child Welfare Outcomes Report to Congress (Outcomes Report) come from the U.S. Department of Health and Human Services? (the Department) Adoption and Foster Care Analysis and Reporting System (AFCARS) and National Child Abuse and Neglect Data System (NCANDS). Since their inception, a key function of AFCARS and NCANDS has been to identify broad trends in child welfare that can be used for policy development and for disseminating information to Congress, the press, and the general public. These data-reporting systems represent considerable improvements over previous national systems with regard to providing consistent and comprehensive information pertaining to national trends in child maltreatment, children in foster care, and children adopted from the public child welfare system.
With the legislative requirement of an annual report to Congress that depicts the performance of States on a set of child-welfare outcomes, the function of NCANDS and AFCARS was expanded to include using State-level data to calculate specific outcome measures. This new function required a more sophisticated level of analyses than those conducted previously with the data. (The AFCARS and NCANDS data sources for each of the outcome measures are provided in Appendix H.) As a result, the first State-level computational analyses, conducted on 1997 data, revealed several data-quality problems at the State level. For example, there were large percentages of missing data and inconsistencies in the data that were reported.
However, since publication of Child Welfare Outcomes 1998—the first Outcomes Report—there has been significant improvement in data quality. The incidence of missing data and of data inconsistencies has declined considerably. Furthermore, many State child welfare agency administrators have indicated in their comment letters to the Outcomes Report that they have accessed, or are accessing, the Department?s technical assistance resources to improve data quality and promote greater consistency in future years. As a result, the NCANDS and AFCARS data provided in Child Welfare Outcomes 2000 for calendar year (NCANDS) and fiscal year (AFCARS) 2000 present a fairly consistent and clear portrait of the child welfare context variables and outcome measures for many States. For most States, the 2000 data (and for many States, the 1999 data), can serve as a solid baseline for assessing continuous improvement over time regarding the performance of an individual State on the national child welfare outcomes.
Despite the considerable improvements in AFCARS and NCANDS data, there are several data-related issues to be considered. In Child Welfare Outcomes 1999, the chapter focusing on key findings describes the extensive variation among States regarding their child-welfare context (e.g., the numbers and characteristics of children in foster care, the numbers and characteristics of children who are victims of child maltreatment, and the numbers and characteristics of children who are adopted) and their performances on the national child welfare-outcomes. To a large extent, the cross-State variations found for the contextual features reflect real differences among States.1 However, difficulties arise in the interpretation of cross-State variations in performance on the child welfare outcome measures. This is because State child-welfare policies, practices, and definitions differ, and these differences have implications for the data used to calculate the national outcome measures. Consequently, the finding that State A, for example, has a lower rate of child maltreatment recurrence than State B, based on data reported in outcome measure 1.1, does not necessarily mean that State A is more effective than State B in preventing the recurrence of maltreatment. Many other issues must be considered before such a conclusion can be drawn with any confidence.
In the following sections of this Chapter, these issues and their impact on the interpretation of performance on specific outcome measures are discussed for each of the national child-welfare outcomes. Some issues have emerged from in-depth assessments of child welfare systems conducted as part of the Child and Family Services Reviews (CFSR), and some have been raised by the findings of State surveys conducted by the Child Welfare League of America for the National Work Group to Improve Child Welfare Outcomes (National Work Group). Other issues were noted by State child welfare agency administrators in their comment letters submitted for inclusion with their State?s data pages.
Although each issue discussed below has implications for the interpretation of the performance of States on the particular outcome measures, some have only a minimal effect, while others have a more significant impact. As these issues have emerged from the CFSRs and other sources, the Department has incorporated the information into its efforts to assist States in developing child welfare program improvement plans after their initial CFSR. The Department also plans to take key data issues into consideration in future CFSRs in assessing the effectiveness of strategies implemented as part of a State?s program improvement plan. These plans are tailored to address the specific areas needing improvement identified through the CFSR.
The final section of this Chapter addresses the issue of the use of exit cohorts versus entry cohorts in assessing the performance of States in attaining the national child welfare outcomes for children in foster care. This issue is discussed in terms of how a focus on these different populations affects the assessment of State child welfare agency performance.
Outcome 1: Reduce recurrence of child maltreatment
A State?s performance with respect to reducing recurrence of child maltreatment is measured by outcome measure 1.1: Of all children who were victims of substantiated or indicated child abuse and/or neglect during the first 6 months of the reporting period, what percentage had another substantiated or indicated report within a 6-month period?2 States report the data to calculate this measure to NCANDS through the Child File or the Detailed Case Data Component (DCDC).3 Twenty-nine States in 1999 and 34 States in 2000 reported data to NCANDS that were sufficient to calculate this measure.
The following differences in State practices and procedures pertaining to maltreatment reports affect the data reported to NCANDS, and consequently the interpretation of States? performance on the measure.
• Differences with respect to investigating reports on open cases. In some States, allegations of child maltreatment are not assigned for an investigation if there is an open in-home services case on the family; in other States, all maltreatment allegations are investigated as new maltreatment reports, even if there is an open case on the family. Similarly, in some States, allegations of child maltreatment during a parent-child visitation episode are investigated as new maltreatment reports, while in some States they are not. Finally, in some States, children?s disclosures of previous child maltreatment episodes after they enter foster care or after a case has been opened are investigated as new allegations of maltreatment, while in other States they are not. These variations in practice affect a State?s rate of maltreatment recurrence and must be taken into consideration in interpreting differences in performance.
• Differences with respect to the definition of an “indicated” report. The measure of maltreatment recurrence includes only children who are the subject of maltreatment reports that have been substantiated or indicated. Although many States use different terms for child protective services dispositions, there is general acceptance among most States that the most serious level of finding used by a State (e.g., “confirmed,” “services required,” “valid,” “petition filed”) is equivalent, for purposes of NCANDS reporting, to a “substantiated” finding.4 There is less agreement, however, about the use of “indicated” as a finding of a maltreatment investigation. The NCANDS definition of an indicated report is one in which maltreatment cannot be substantiated under State law or policy, but there is reason to suspect that the child might have been maltreated or is at risk of maltreatment. Although the number of States that employ the term “indicated” to describe a finding of an investigation has always been small (and has declined from 14 in 1996 to 10 in 2000), many States have dispositions, such as “services recommended” or “unconfirmed,” that fall somewhere between a substantiated finding and an “invalid” finding. However, many States that have implemented these “midpoint” dispositions do not consider them equivalent to an “indicated” finding. A State?s rate of maltreatment recurrence within 6 months will be affected by its definitions of disposition options.
• Differences with respect to the availability of an alternative response system and how cases assigned to an alternative response are reported to NCANDS. Many States have implemented, or are in the process of implementing, alternative responses to child maltreatment reports that do not require making a determination as to whether maltreatment occurred. In some States, a report assigned to an alternative response is “screened out” of the child protection system. In these States, an allegation that is assigned to an alternative response (such as domestic violence, parental substance abuse, inadequate supervision, or inadequate parenting) will not be investigated and therefore will not be included in the NCANDS computation of maltreatment recurrence. However, in States that do not have an alternative response system, these allegations may be investigated, and therefore will be included in the computation of maltreatment recurrence.
• Differences with respect to definitions of child maltreatment. The fact that States differ regarding their definitions of what constitutes child maltreatment has an effect on the comparability of data pertaining to the recurrence of maltreatment. One example of this is the way that different States respond to reports of domestic violence (i.e., violence between adult partners). In some States, domestic violence reports are investigated as child maltreatment reports because they are perceived as a form of emotional maltreatment, but in other States, domestic violence reports are not considered child maltreatment unless there is actual physical harm to the child.
The issues discussed above reinforce the Department?s commitment to focusing its assessment of progress on the continuous improvement of individual States with regard to performance on the outcome measures pertaining to child maltreatment.
Outcome 2: Reduce maltreatment in foster care
The outcome pertaining to maltreatment in foster care is assessed by calculating outcome measure 2.1: Of all children who were in foster care during the reporting period, what percentage was the subject of substantiated or indicated maltreatment by a foster parent or facility staff? The number of children in foster care during the reporting period is calculated from data reported to the Adoption and Foster Care Analysis and Reporting System (AFCARS). The number of children who were the subjects of a report of substantiated or indicated maltreatment by a foster parent or facility staff is determined from data reported to NCANDS. The following key issues relevant to interpreting State differences with regard to performance on this measure have been identified by the National Work Group and by the State child welfare agencies in their comments on their data.
• Differences between children reported to NCANDS and AFCARS. For outcome measure 2.1 to be a valid measure of maltreatment in foster care, the children who are included in the numerator (i.e., children maltreated in foster care) also must be included in the denominator (i.e., all children in foster care during the designated time period). However, in some States, this is not the case. According to a survey conducted by CWLA for the National Work Group, for example, more than 50 percent of the States responding reported that they include maltreatment of non-child welfare children in group homes, residential facilities, and detention facilities in the data they report to NCANDS if the perpetrator is a facility staff member. Because these children are not reported in AFCARS, they appear in the numerator, but not in the denominator. Another example pertains to children in the care of relatives who are not licensed as foster parents. If these children are maltreated by the relative caretaker, about 50 percent of the States responding to the survey noted that they do not include this maltreatment in the “foster parents” perpetrator category in NCANDS. In this situation, children appear in the denominator, but are not eligible for inclusion in the numerator. Although these problems are a concern to the Department, they apply to only a small number of cases. In addition, it is anticipated that the problems will be resolved in the near future. According to the CWLA report, three quarters of States responding to the survey indicated that they are able to identify the children who were in both their AFCARS and NCANDS systems.
• Differences in the kinds of events that States consider maltreatment in foster care. An additional variation that affects the calculation of this measure is that some States investigate foster home licensing violations as if they were maltreatment reports, even if the event would not be considered maltreatment if the family were not a foster family. In contrast, other States address licensing violations in the re-approval process and do not respond to them as maltreatment reports or report them as such to NCANDS.
Outcome 3: Increase permanency for children in foster care
This outcome is assessed through five measures. The primary
measure (measure 3.1) examines the percentage of children exiting
foster care to a permanent placement-i.e., reunification,
guardianship, and adoption. Subsequent measures assess exits to
permanent placements for children with a diagnosed disability
(measure 3.2), children who entered foster care when they were
older than age 12 (measure 3.3), and children of various races and
ethnicity (measure 3.5). The fifth measure (measure 3.4) examines
the percentages of children who exited foster care because they
were emancipated and who were age 12 or younger when they entered
foster care.
The key issues pertaining to interpreting State performance with
regard to exits to permanency are the following:
• Missing data regarding the reason for discharge. Although the extent of missing data has declined greatly since 1998, in several States at least 20 percent of the children exiting foster care did not have “reason-for-discharge” information. Some States reported in their comment letters that they have discharge reasons that are not among the options used by AFCARS and, when these are reported to AFCARS, they appear as missing data. These States have indicated that they are in the process of correcting this data coding and entry problem. It is anticipated that missing data will no longer be an issue with the next Outcomes Report.
• Variations among States with respect to the use
of guardianship as a permanency option. The many
variations in the use of guardianship as a permanency option
include:
- States permit guardianship as a permanency option for foster
parents and relative caregivers with no limitations with regard to
the age of the child.
- States permit guardianship as a permanency option only for
relatives, without regard to age restrictions.
- States permit guardianship as a permanency option only for
relatives and only for children who meet criteria, for age and time
in foster care.
- States do not permit guardianship as a permanency option unless
parents are deceased.
- States do or do not have a subsidized guardianship program that
provides financial assistance to families who assume the
guardianship of a child in foster care.
The availability of the option of guardianship will affect a State?s performance on outcome 3, particularly with respect to permanency for children who were older than age 12 when they entered foster care.
The fact that States are not consistent in their use of the AFCARS data element pertaining to the issue of diagnosed disability affects the interpretation of measure 3.2. In some States, when children enter care they may be reported as not having a diagnosed disability, but if a disability is diagnosed at a later time, the updated information is not always reported to AFCARS. In addition, some States do not report data for this element at all.
With respect to measure 3.4, there is anecdotal evidence that children who “age out” of the foster care system are not always reported as discharges to emancipation. Because many of these children tend to live with their parents when they leave, they are often reported as discharges to reunification. Similarly, anecdotal information also indicates that many children who leave foster care “unofficially” prior to an official termination may be reported to AFCARS as “runaways” even though they would have been emancipated from foster care in a few weeks or months.
Outcome 4: Reduce time in foster care to reunification without increasing re-entry
Two measures assess attainment of this outcome. Outcome measure
4.1 examines the time in care prior to reunification for all
children discharged to reunification in the reporting period (e.g.,
less than 12 months, 12 to 24 months, 24 to 36 months, 36 to 48
months, and 48 or more months). Outcome measure 4.2 examines the
percentage of children entering foster care who were re-entering
care (1) within 12 months of discharge from a prior foster care
episode or (2) more than 12 months after discharge from a prior
foster care episode.
The key problem relevant to interpreting the performance of States
on outcome measure 4.1 (time to reunification) and 4.2 (rate of
re-entry into foster care) is that States differ in their statutes,
policies, and/or practices pertaining to reunification. Some have
statutes and/or policies that require the child welfare agency to
maintain custody of a child for a specified amount of time (ranging
from 3 to 12 months) after the child is physically reunified with
parents. Other States do not have written policies pertaining to
this practice, but engage in the practice on a routine basis to
ensure children?s successful reintegration into their
families before transferring custody. Still others transfer custody
to the primary caretakers at the time that children are physically
reunified. These policies or practices affect the length of time to
“final” reunification because children are not reported
to AFCARS as discharged to reunification until custody is
transferred. At the same time, the existence of these policies or
practices also affects the rate of re-entry into foster care within
12 months because a “failed” trial reunification would
not be considered a re-entry into foster care as custody was never
transferred. An understanding of the performance of States on these
measures must take into consideration their relevant policies and
practices.
Another issue relevant to interpreting the performance of States on these measures pertains to the difference among the populations served by State child welfare agencies. In some States, the child welfare agency provides foster care services to children from the juvenile justice system, children who enter foster care because of their behavior (through petitions from the parents), and children who have been maltreated by parents or primary caretakers. In other States, the child welfare agency only serves children who have been found to be victims of child maltreatment or who have been voluntarily placed in foster care because parents are unable to take care of them. Anecdotal information from stakeholders interviewed during the CFSRs indicates that children who enter foster care through juvenile justice or as a result of a parental petition often are the most difficult to reunify and are the most likely to re-enter foster care after reunification. Although the extent of re-entries into foster care that involve older children who entered care through the juvenile justice system is not currently known, State variations in the kinds of populations served contribute to the difficulty of interpreting cross-State performance on this measure.
Outcome 5: Reduce time in foster care to adoption
The attainment of outcome 5 is assessed by outcome measure 5.1, which examines the time to adoption for all children who have a finalized adoption in the reporting year. The time periods examined are: (1) less than 12 months, (2) at least 12 months but less than 24 months, (3) at least 24 months but less than 36 months, (4) at least 36 months but less than 48 months, and (5) 48 or more months. Issues relevant to interpreting differences in States? performance on this measure include the following:
• Differences among States in the way discharges to adoption are reported to AFCARS. Some States incorrectly report to AFCARS the placement of children in an adoptive home as a discharge to adoption, while other States correctly report discharges to adoption to AFCARS only after legal finalization. The Department has clarified the requirements of the “discharge” element pertaining to adoption, and it is anticipated that discharges to adoption will be reported correctly by all States in future years. Another issue raised by some States in their comment letters is that caseworkers are reluctant to report discharges to adoptions for confidentiality reasons. The end result of these problems is that in several States, the number of discharges to adoption reported to the AFCARS Foster Care Database is considerably less than the number of finalized adoptions reported to the AFCARS Adoption Database. Although the AFCARS Adoption Database does include children adopted as infants through the State system who were never in foster care, the number of these children does not account for the large discrepancy found for some States between the two adoption figures. This makes it difficult to interpret States? performance on measure 5.1, because many of the adoptions that took place are not accounted for in the measure.
• Differences among States with regard to their efforts to find adoptive homes for children who have been in foster care for long periods of time. Because the data for outcome measure 5.1 are presented in terms of percentages of children exiting foster care, it appears as if those States that have made intensive efforts to find homes for children who have been in foster care for long periods of time are not effective in attaining finalized adoptions for children in the short-run. That is, the percentages of children achieving a finalized adoption within 24 months is low because the State has experienced recent success in finding adoptive homes for children who have been in foster care for long periods of time. This problem may be resolved to some extent by assessing the numbers of adoptions occurring within different time frames as well as the percentages. However, it is anticipated that this issue will become less relevant in future years as States experience success in finding adoptive homes for children who have been in foster care for many years.
Outcome 6: Increase placement stability
Attainment of this outcome is assessed by measure 6.1, which examines the percentages of children who experience 2 or fewer placements and who have been in foster care for the following time periods: less than 12 months from removal from home, 12 to less than 24 months, 24 to less than 36 months, 36 to less than 48 months, and 48 or more months. A report by the National Work Group on the results of a State survey indicates that there is some inconsistency across States in the method used to determine placement changes for children in foster care. One area of inconsistency concerns what constitutes a “placement.” For example, some States have incorrectly counted trial home visits, respite care, psychiatric treatment episodes, or detention as “placement changes.”
Another area of inconsistency concerns how placement changes are conceptualized. In some States, a child?s move from one living arrangement to another is correctly reported as a placement change. However, other States do not report movement from one living arrangement to another as a placement change if the child is returning to a former living arrangement (i.e., these States incorrectly report placement changes on the basis of unique providers rather than actual placement change). There also are States that do not report placement changes that occur when a child?s placement is managed by a private foster care agency as long as the provider agency does not change. Other States only report placements that involve payments. If the child is placed with a relative who does not receive a foster care payment, the placement is not recorded in the State?s data system. The data team in the Children?s Bureau has provided substantial technical assistance to individual States regarding the correct coding of placements, and many States are beginning to rectify the problem. Recently, on July 3, 2002, the Department issued clarifications on how States are to report data to AFCARS pertaining to placements and placement changes. The Department anticipates that this clarification will result in greater consistency in reporting placement changes in future years.
Outcome 7: Reduce placements of young children in group homes and institutions
This outcome is assessed by outcome measure 7.1: Of all children entering care during the reporting period who were age 12 or younger, what percentage was placed in a group home or institution? Several States have noted that when children first enter foster care, it is State policy to place these children in a short-term emergency shelter to better assess long-term placement needs. These States suggest that because these shelters meet the definitions provided for group homes or institutions, they are reported as such in the States? AFCARS submissions. In contrast, States that do not use short-term shelter care as the initial placement for children tend to demonstrate lower percentages of children age 12 or younger being placed in group homes or institutions. Although this may be a valid concern for States, information from the CFSRs indicates that initial placements in shelters often are not as “short term” as they are intended to be. In addition, case reviews during the CFSRs have revealed that children as young as 3 years are being placed in shelters as initial placements.
Use of entry cohort, exit
cohort, and point in time data to assess the performance of States
in attaining the national child welfare outcomes
The five national child welfare outcomes that pertain to children
in foster care are assessed by measures that focus on different
groups of children. Outcome measures 3.1 through 3.5, 4.1, and 5.1
focus on exit cohorts (i.e., the children exiting foster care in
the specified reporting period). Outcome measures 4.2 and 7.1 are
based on data from entry cohorts (i.e., children entering foster
care during the specified reporting period), while outcome measure
6.1 is based on all children served by the foster care system
during the specified reporting period.
Several representatives from State child welfare agencies and some child welfare researchers have expressed the opinion that the use of exit cohort data to assess the attainment of permanency is not valid. Their assertion is that it is not possible to assess real change in a State?s performance by looking at exit cohort data, because exit cohorts can only tell us about the experiences of children who leave foster care during a specified time period. Because children leaving foster care during the same time period have been in foster care for varying lengths of time, they have been affected by a variety of policies, practices, and environmental conditions pertaining to child welfare agency practices and procedures. In contrast, children entering foster care during the same time period (entry cohort) are subject to similar policies, practices, and conditions. Furthermore, a focus on recent entry cohorts permits an assessment of the impact of recent or current conditions on children?s experiences with respect to reunification, adoption, guardianship, placement changes, and types of placements. However, an analysis of entry cohorts does not capture the experiences of children who are in care for long periods of time before achieving an outcome pertaining to permanency. Exit cohorts permit a focus on the point at which a given outcome is attained with regard to permanency.
There are pros and cons to each approach (exit and entry). The primary value of an entry cohort approach is that it provides information that is critical for child welfare agency managers regarding the effectiveness of current policies and practices influencing the movement of children through their foster care system. The drawback with respect to the Outcomes Report is that this approach requires several years of tracking of an individual cohort to assess overall time to permanency for all children in the cohort and to compare time to permanency from one cohort to another. For example, if a cohort were considered to be all of the children entering foster care in a given year, it would be necessary to follow even one cohort for almost 2 years to obtain information regarding the first 12 months in care for all of the children in the cohort. However, it is important to note that for the CFSR program improvement plan, it will be important to use any available information on entry cohorts to track the effectiveness of efforts to address specific areas needing improvement.
An exit cohort approach captures information about outcomes at
the point of outcome attainment and yields information about
children who have been in care for long periods of time as well as
children who have been in care for only a few months. The
disadvantage is that this approach captures this information only
for children who exited foster care and, therefore, it may take
years before the impact of new programs and policies will be
demonstrated in the data.
The key issue is that an entry cohort approach to assessing
permanency basically asks and answers different questions than does
an exit cohort approach. The entry approach asks: What happens to
children who enter foster care in a given year? How do they move
through the system? How long do they remain in foster care? The
exit approach looks retrospectively at children?s experiences
and asks: What outcomes did they experience? How long did it take
them to achieve those outcomes? While both types of questions are
valid and seek important information about children served by
public child welfare systems, the Department determined that a
focus on outcomes exemplified by the exit cohort approach was most
responsive to the requirements of the Adoption and Safe Families
Act (ASFA). However, incorporation of the entry cohort analysis
approach into the agency?s work on outcomes continues to be
explored. The integration of these approaches would permit both a
focus on the outcomes that children experience and an assessment of
the impact of newly implemented policies and programs on children
who are newly entering the system.
1 The one exception may be the
numbers of child maltreatment victims. Because States have
different definitions for case dispositions, variations in the
number of child maltreatment victims may reflect those differences.
For example, one State has determined that its findings with regard
to maltreatment reports do not conform to NCANDS definitions of
substantiation or indication, and therefore they report no
maltreatment victims. Back
2 For example, if a child is a
victim of maltreatment in January of a calendar year, and again in
March, then that would be considered recurrence of maltreatment. Back
3 The Child File was recently
implemented by NCANDS and has replaced the DCDC file in all but 5
of the 34 reporting States. Back
4 The exception may be North
Dakota, which by statute will not define a case as substantiated
and therefore reports no maltreatment victims to NCANDS. Back
