Evaluation Methodologies

Evaluation Methodologies

The Children's Bureau requires that States granted title IV-E waivers conduct rigorous evaluations that include process and outcome components. In terms of process evaluation, each State implementing a substance abuse demonstration focused on different issues, although most addressed staffing, personnel, and organizational factors that affected implementation. For the outcome evaluation component, most States sought answers to a similar set of questions, including whether their waiver demonstrations reduced children's length of time in foster care and increased reunification rates. Beyond these core outcomes, New Hampshire and Illinois studied the success of their projects in reducing the incidence of subsequent maltreatment and in improving placement stability, defined as the frequency with which children in foster care changed placement settings.

Table 3 summarizes the major features of the States' evaluations, including research designs and sample sizes. The table highlights the limitations of some States' evaluations and places caveats on the subsequent interpretation of evaluation findings, particularly with respect to child welfare outcomes. Maryland, for example, terminated its demonstration early and assigned very few families to its demonstration. Consequently, it reported no outcome data and the discussion of its evaluation in this paper is limited to process findings. Delaware relied on a comparison group design to examine differences in outcomes for clients in child protection units with a substance abuse counselor (experimental group) compared with outcomes for clients in matched units without a substance abuse counselor (comparison group). Each of Delaware's three counties had one experimental child protection unit and one matched comparison unit. However, because CPS staff in comparison units were housed in the same office as CPS staff in experimental units, contact between comparison unit child protection workers and substance abuse counselors may have occurred, thereby contaminating Delaware's research design and weakening the validity of its reported outcomes. In addition, Delaware and Maryland's research samples included both cases in which one or more children were in or entered out-of-home placement as well as cases in which all children remained home. The inclusion of both in-home and out-of-home cases in these States' research samples potentially skews the interpretation of key outcome measures (for example, placement duration) if subpopulation analyses are not conducted to link appropriate outcome measures with each subpopulation.

Although Illinois and New Hampshire have not yet completed their demonstrations or reported final evaluation results, this paper will focus its discussion of outcome findings on these two States because of the comparative strength of their research designs and greater availability of outcome data. However, because of their considerable differences in size, population characteristics, levels of urbanicity, availability of substance abuse treatment resources, and child welfare laws and policies, the reader should exercise caution in making comparisons across the States regarding the effectiveness of their substance abuse waiver demonstrations in improving child welfare outcomes.

Table 3
Evaluation Designs of Substance Abuse Waiver Demonstrations

State	Research Design	Sample Size (# of Cases)
		Experimental Group			Control/Comparison Group
		In-home cases⁶	Out-of-home cases	Total	In-home cases	Out-of-home cases	Total
Delaware	Comparison group	398	132	530	368	162	530
Illinois<	Random assignment	Not applicable	954 as of 6/30/04	954	Not applicable	366 as of 6/30/04	366
Maryland	Random assignment	--	--	9⁷	--	--	9
New Hampshire	Random assignment	183	39	222	182	33	215

Back to Table of Contents

State Process Evaluations - Summary of Key Findings and Issues

Process findings from the States' substance abuse waiver demonstrations can be analyzed across several dimensions:

Parental characteristics;
Project referral and enrollment;
Treatment access;
Treatment retention and completion;
Data collection and tracking; and
Other implementation issues, including staff training, worker turnover, and the differing professional philosophies of child welfare workers and substance abuse specialists.

These dimensions highlight the unique context in which each substance abuse project operated and the special challenges encountered by States in realizing the goals of their waiver demonstrations.

Parental Characteristics

Of the four States with substance abuse waivers, only New Hampshire and Illinois provided detailed information on the demographic characteristics of parents being served through their waiver demonstrations. A comparison of caregiver profiles from these States reveals striking differences; although the age and gender of parents were similar, major distinctions emerge in terms of race, the mix of presenting problems, and types of abused substances. As illustrated in Table 4, caregivers in New Hampshire's project were overwhelmingly white, were more likely to abuse alcohol or marijuana, and most frequently entered the child welfare system due to a neglect allegation. An analysis of families in the experimental group seen by Licensed Alcohol and Drug Abuse Counselor (LADC) showed a significant association (p < .05) between a diagnosis of chemical dependency and the maltreatment disposition of the case; a substantiation of abuse or neglect was more likely in cases in which the SASSI score indicated the presence of substance abuse. New Hampshire then conducted a logistic regression analysis to identify the factors that might contribute to case substantiation. This analysis indicated several contributing variables, including the number of alcohol-related factors as measured by MAST⁸ scores; a history of illicit drug use beyond marijuana; depression; and neglect as the presenting type of maltreatment.

In contrast to New Hampshire, participants in Illinois' demonstration were largely African American, were more likely to abuse cocaine or opioid drugs, and more likely entered the child welfare system after giving birth to a substance-exposed infant. These differences highlight the unique circumstances and population characteristics that States must address when confronting the issue of substance use, abuse, and dependency.

Table 4 Characteristics of Caregivers in Substance Abuse Waiver Demonstrations

Variable	New Hampshire	Illinois
Mean Age	33	31
Gender	87% Female	71% Female
Race	90% Caucasian	81% African American
Presence of Mental Health Issues	18%⁹	28%
Top Three Drugs Used (in order of prevalence)	Alcohol Marijuana Cocaine	Cocaine Opioids Alcohol
Usage Rate for Top Drug	Alcohol - 40% of caregivers reported having four or more drinks at any given time	Cocaine - 38% of caregivers reported using cocaine several times per week
Top Two Presenting Problems	54% - Neglect 23% Physical Abuse	31% - Substance Exposed Infant 21% - Substantial Risk of Physical Abuse/Harm

Back toTable of Contents

Project Referral and Enrollment

Each State's decision to implement a substance abuse waiver demonstration rested on the assumptions that (1) a sizeable number of substance-abusing parents resided in the State who represented a threat to the safety and well-being of their children, and (2) that these caregivers would agree to assessment and treatment if these services were offered through a substance abuse waiver (i.e., "build it and they will come"). However, the challenges experienced by States in implementing their waiver demonstrations call these basic assumptions into question and underscore the need to refine screening processes and eligibility criteria to facilitate the achievement of States' implementation goals.

Delaware

In its initial project proposal, Delaware outlined plans to refer five families per CPS unit per month (or 15 families statewide per month) to receive enhanced substance abuse services, for a total of 180 families per year and 900 families over the course of its demonstration. However, the State noted in its final evaluation report that it never established formal screening and review procedures to ensure the minimum of five monthly referrals per unit. The flow of referrals remained highly sporadic throughout the course of the demonstration, and by the conclusion of the project, only 530 families ? 55 percent of the expected number ? had received enhanced substance abuse services. Delaware concluded in its report that CPS units in which supervisors took an active role in reviewing cases for indications of a substance use disorder and in directly referring cases to substance abuse counselors had the smoothest, most consistent referral processes. Adequate supervisory oversight had not evolved in most CPS units until the fourth year of the demonstration, rendering attainment of the State's original referral goals unlikely. In addition, the State found that it had underestimated the average length of time that experimental group cases would remain open. In its final evaluation report, the State noted that closed experimental group cases had remained open for an average of nine months, three times longer than the original estimate of three months. Substance abuse counselors identified the complex nature of most cases as the primary reason for their long service lives, with the typical caregiver having had lengthy histories of substance abuse or addiction, several failed recovery attempts, and expressing resistance both to further treatment and denial regarding the severity of their substance use disorder. The longer-than-expected lives of many cases further limited the number of new referrals that Delaware's waiver demonstration could absorb.

Illinois

Illinois' actual experience with project referrals and enrollment belied its original projections about the availability of eligible families. Using a monthly average of 195 families entering the foster care system in Cook County, the State estimated in its waiver proposal that about 50 percent would involve either a substance-exposed infant or serious drug use by parents, producing an estimate that approximately 100 families per month (1,200 per year) would be suitable candidates for the demonstration. However, the State noted in its interim evaluation report that its initial assumptions regarding the number of JCAP referrals and the size of the potentially eligible target population were erroneous. First, the juvenile courts made far fewer referrals to JCAP than expected during the first two years of the demonstration, with only 469 occurring in FY 2000 and 608 in FY 2001. Second, although the JCAP assessments indicated treatment in more instances than the State initially predicted, a smaller percentage than expected met the demonstration's eligibility requirement of obtaining an assessment within 90 days of the temporary custody hearing. According to the State's November 2004 report, on average only 50 percent of assessed caregivers for whom treatment was indicated met the demonstration's eligibility requirement of obtaining a substance abuse assessment within 90 days of the temporary custody hearing. A third factor affecting the accuracy of the State's enrollment predictions involved the sharp drop in the total child welfare caseload in Cook County, Illinois. Between the end of Fiscal Year 2001 and March 2005, the total, unduplicated number of children involved in the county's child welfare system declined from 20,320 to 11,920.¹⁰

Illinois' miscalculations regarding the number of JCAP assessments and eligible clients initially depressed enrollment into the waiver demonstration; by the project's mid-point in 2003, only 528 caregivers had been assigned to receive enhanced substance abuse services, far less than 50 percent of the State's target of assigning 1,500 caregivers to the experimental group by project's end. Since then, the State has made up much lost ground as the number of JCAP assessments has increased to over 1,000 annually in Fiscal Years 2002 through 2004. As of November 2004, the proportion of initial assessments resulting in treatment referrals has stabilized at approximately 61 percent and a total of 954 caregivers have been assigned to the demonstration's experimental group. The State attributes this still somewhat low percentage of treatment referrals to clients' inaccuracy in self reporting, which may result in a failure to meet the specific criteria that indicate a need for treatment.

Maryland

Among the States with substance abuse waivers, Maryland experienced the greatest difficulties with client referral and enrollment. At the time the State terminated its demonstration project, only nine women had been assigned to the experimental group and only eight ever received enhanced substance abuse services. In its October 2002 progress report, Maryland described a study of 913 cases screened between October 1, 2001 and August 31, 2002 for eligibility to participate in the waiver demonstration; the results of this analysis revealed that the State had made erroneous assumptions regarding the eligibility and recruitment potential of its proposed target population. For example, the State discovered that cases with a documented or suspected substance use disorder comprised only 31 percent of all screened cases. Of the 283 cases in which a substance use disorder was identified, the majority were ineligible to participate in the demonstration because of pre-defined disqualifying criteria, including participation in another substance abuse project, the presence of mental health problems, an allegation of sexual abuse, child abandonment, or unavailability of the caregiver (e.g., due to incarceration or unknown whereabouts). At the end of its analysis, the State identified only 27 caregivers with a known substance use disorder who were not otherwise disqualified or unavailable to participate in the demonstration. The difficulties experienced by Maryland in identifying and enrolling eligible caregivers speak both to the considerable challenges of serving substance-abusing parents with multiple co-occurring problems and to the importance of establishing a clear understanding of the characteristics of the population targeted for waiver services.

New Hampshire

Like Illinois in the earlier phases of its demonstration, New Hampshire encountered problems in getting caregivers assigned to the experimental group to complete a substance abuse assessment. As of the State's September 2003 interim report, only 122 of 222 experimental group caregivers (58 percent) had completed a substance abuse assessment. The State suggests that these low assessment rates may result from the voluntary nature of the substance abuse assessment. Because the majority of maltreatment investigations in New Hampshire do not result in maltreatment substantiation and a subsequent CPS case opening, the State has no basis for requiring caregivers to undergo a substance abuse assessment and enroll in treatment as a condition for allowing their children to remain home or return home from foster care. Thus, many experimental group caregivers may simply decline to participate in the assessment or seek substance abuse treatment. According to New Hampshire's March 2004 progress report, 86 percent of maltreatment investigations involving cases assigned to the experimental group were unsubstantiated, leading the State to close them without the ability to require further assessment or services.

Treatment Access

All four States with substance abuse waivers sought to provide caregivers enrolled in their demonstration with rapid access to substance abuse treatment and support services. Illinois' demonstration experienced the greatest success in connecting caregivers to treatment services, whereas other States, particularly Delaware, faced several obstacles that may have limited clients' timely access to treatment.

Delaware

By the end of Delaware's initial five-year waiver demonstration in February 2002, only 168 of 420 closed experimental group cases (about 40 percent) had made at least initial contact with a treatment provider. In the remaining closed cases, substance abuse counselors were unsuccessful in connecting clients to a treatment provider (125, or 30 percent), CPS closed the case after losing contact with the client (104, or 25 percent), treatment was not needed (17, or 4 percent), or the client refused treatment services (3, or less than 1 percent). The State attributed these modest treatment access rates to several factors, including high staff turnover during which the project lost contact with many clients, client resistance to treatment, and the limited availability of appropriate treatment options for substance-abusing women. Specifically, Delaware noted the paucity of residential and intensive outpatient programs in the State, particularly for pregnant women, clients with dual diagnoses, or that provide housing for women and their children.

Illinois

Illinois succeeded in linking most experimental group clients to treatment resources, in part due to an aggressive intervention strategy that sought to connect clients to a Recovery Coach within 48 hours of the initial substance abuse assessment. According to the State's interim evaluation report, the proportion of referred parents who had contact with a Recovery Coach rose gradually and stabilized at around 90 percent by the first quarter of 2003. The immediate involvement of a Recovery Coach appeared to have a positive effect on access to treatment services, with the State's November 2004 progress report revealing a statistically significant difference between the proportion of parents accessing treatment services in the experimental group (73 percent) and the proportion in the control group (50 percent). In addition, caregivers who received enhanced waiver services accessed treatment more quickly, with 50 percent of parents in the experimental group experiencing a first treatment episode within 40 days compared with 100 days for 50 percent of control group parents, a statistically significant difference.

Maryland

In Maryland, substance abuse counselors sought to connect women assigned to the experimental group to treatment resources as soon as a substance use disorder was either stated or implied. Although the State enrolled very few women into the waiver demonstration, most of those assigned to the experimental group had entered a treatment program by the time the waiver was terminated in December 2002. Of the nine women assigned to the experimental group, three were receiving outpatient substance abuse treatment and three had enrolled in six-month inpatient treatment programs. Of the remaining women, one was incarcerated and demonstration staff had lost track of two.

New Hampshire

The latest findings from New Hampshire indicate similar levels of access to substance abuse treatment services by experimental and control group caregivers. According to a recent review of interview data collected from subjects enrolled in its waiver demonstration, 26 percent of experimental group caregivers (n = 101) had received substance abuse treatment services compared with 24 percent of caregivers in the control group (n = 106). Among cases with a substantiated allegation of maltreatment, however, the most recent data suggest that experimental group caregivers are more likely to access intensive, long-term treatment services. To date, almost 20 percent of experimental group caregivers with a substantiated maltreatment allegation have received long-term inpatient substance abuse treatment services compared with six percent of control group caregivers, a statistically significant difference.¹¹

Treatment Retention and Completion

Among the four States, only Delaware and Illinois collected specific data on the numbers of enrolled caregivers who have remained in or who have successfully completed substance abuse treatment. Illinois has enjoyed modest success in facilitating clients' engagement in and completion of treatment. As of June 30, 2004, 132 of 376 active clients assigned to the experimental group (35 percent) were engaged in treatment, while 91 clients (24 percent) had successfully completed treatment. The remaining caregivers had dropped out of treatment, were pending initial treatment, or could not be located to begin treatment. Of the 132 clients actively engaged in treatment, 53 (40 percent) had been receiving treatment services for six months or more. The State noted in its November 2004 progress report that parents' likelihood of completing treatment is higher if they remain engaged in treatment for at least 90 days. Of all 954 parents assigned to the experimental group from the beginning of the demonstration through June 30, 2004, 73 percent (697) had participated in treatment at some point in time compared with only 50 percent (182) of the 366 parents assigned to the control group.

Illinois' success in engaging caregivers in treatment is notable in light of the State's historically low treatment enrollment and completion rates. A 1998 GAO report found that among mothers in Cook County with AODA problems whose children had been in foster care for over 12 months, just over 20 percent had completed or were actively enrolled in treatment. Almost 40 percent of these mothers had never entered treatment and the remainder had either dropped out or had otherwise failed to complete treatment.

Delaware experienced more difficulties with retaining clients in treatment. By the end of the State's waiver demonstration in February 2002, only 101 of 420 closed experimental group cases (24 percent) were enrolled in or had completed treatment; the State did not provide a specific breakdown of clients engaged in treatment versus those who had completed treatment at the time the demonstration ended. Delaware's more limited success in retaining clients in treatment is understandable given the resource constraints and implementation barriers that reduced treatment access in that State.

Data Collection and Tracking

All four States cited varying degrees of difficulty in collecting and tracking demographic, social service, and treatment information on caregivers enrolled in the waiver demonstrations. These difficulties affected both the smoothness of project implementation and the quality of data available for evaluation.

Delaware

Antiquated or inadequate information systems hampered the collection of data for Delaware's waiver evaluation. Specifically, the State noted in its final 2002 progress report that its existing child welfare information management system lacked the capacity to track historical case data. To overcome the limitations of its existing information system, the State turned to ad hoc data reporting tools such as Microsoft Access to generate monthly evaluation reports. The effort required to generate ad hoc evaluation reports "from scratch" consumed the time and resources of both the State's independent evaluator and of its waiver project staff and delayed the reporting of waiver findings.

Illinois

Illinois has provided thorough documentation regarding its efforts to collect and track data regarding its waiver participants. In its November 2004 progress report, Illinois described particular problems with obtaining informed consent from enrolled caregivers to review other case data pertinent to the State's evaluation, such as public assistance history and mental health records. As of June 30, 2004, only 32 percent of all caregivers enrolled in the demonstration had signed research consents granting permission to review their case records. The State's attempts to redesign the consent form to clarify the language regarding informed consent have done little to increase consent rates. In short, when participation in research is voluntary, most caregivers choose not to grant access to confidential information. Low rates of informed consent have limited the scope of research regarding issues relevant to the waiver evaluation, such as clients' mental health status and use of other social services.

Illinois has experienced more success in collecting data on the substance abuse treatment histories of caregivers enrolled in its demonstration. The State uses a specialized database called the Treatment Record and Continuing Care Systems (TRACCS) to track treatment data on caregivers in both the experimental and control groups. TRACCS forms are sent to child welfare workers for completion on a quarterly basis and to substance abuse providers and Recovery Coaches on a monthly basis. As of June 2004, child welfare workers and Recovery Coaches had completed and returned about 80 percent of their TRACCS forms; completion rates for substance abuse providers are substantially lower at only 60 percent. In an effort to improve TRACCS completion rates, Illinois has scheduled additional trainings regarding TRACCS and has redesigned the TRACCS form for greater simplicity and ease of use.

Maryland

In its September 2002 progress report, Maryland described difficulties both with obtaining informed consent from caregivers and with tracking referrals to the waiver demonstration. Through a series of focus groups conducted by the State's waiver evaluators, project staff noted a lack of clarity regarding the worker responsible (i.e., the child welfare caseworker or the addiction specialist) for describing the evaluation to caregivers and obtaining consent to participate in research. In fact, almost no focus group participants had even seen the research consent form for the waiver evaluation. In addition, focus group participants identified several problems with tracking referrals to the substance abuse waiver demonstration. Intake workers who screened caregivers usually did not follow cases once they had determined their eligibility to participate in the waiver, while Family Support Service Team members noted that they seldom received copies of client intake reports. This lack of continuity in the client referral process often produced duplication or interruptions in the collection of data on caregivers enrolled in the demonstration.

New Hampshire

Because the vast majority of maltreatment referrals in New Hampshire are resolved without a maltreatment substantiation and do not result in a child protection case opening, the State's evaluators have expressed concerns about their ability to track basic process data on clients, including their completion of substance abuse assessments and participation in treatment services. To address these concerns, evaluation staff from the Family Research Lab at the University of New Hampshire have developed a client follow-up protocol and interview tool that they hope will fill gaps in client assessment and treatment data once families leave the child welfare system. The State's final evaluation report may shed additional light on the success of this protocol in collecting follow-up data on waiver participants with closed child protection cases.

Other Implementation Challenges

Worker Training and Education

Inadequate education regarding the waiver demonstrations and lack of training in the identification and assessment of substance abuse contributed to the difficulties with caregiver enrollment and retention described above. Focus groups held with caseworkers in Maryland revealed several gaps in worker education that impeded caregiver recruitment into the waiver demonstration. Specifically, many child welfare workers in Maryland reported that they were unfamiliar with the waiver's purpose and eligibility criteria and were unclear about the distinction between the waiver demonstration and other substance abuse treatment projects in the State. In addition, focus group participants questioned whether intake workers with child welfare backgrounds had the appropriate skills and training to identify substance use disorders, an issue that may have contributed to low enrollment rates in Maryland's demonstration. Caseworkers in one Maryland county estimated that substance use disorders were not discovered in 90 percent of all cases until they had left intake and been transferred to a child welfare case manager; if true, these cases would have missed the recruitment window established at child welfare intake for enrollment into Maryland's waiver demonstration.

Like Maryland, Delaware cited inadequate training for child welfare caseworkers in the identification of and appropriate responses to caregiver drug and alcohol abuse as a major obstacle to waiver implementation. Delaware noted in its final evaluation report that over time, caseworkers' knowledge of substance use disorders increased through interactions and joint case management with substance abuse counselors.

Illinois' experience highlights the importance of frequent worker training and follow-up to maximize the availability of quality evaluation data. After observing low completion rates of the TRACCS data collection tool by substance abuse treatment providers, Illinois scheduled additional trainings at each provider site regarding the proper completion of the tool. After implementing supplemental trainings, the completion rate of TRACCS forms by treatment providers increased from 54 percent in December 2003 to 60 percent in June 2004, while the accuracy and timeliness of form submission improved as well. In addition, waiver staff met with Juvenile Court personnel - including judges, state attorneys, public defenders, and guardians ad litem - on a regular basis to familiarize them with the demonstration. As these trainings and meetings progressed, interagency communication improved and both court staff and treatment providers gained a better understanding of the demonstration and optimal strategies for implementing it.

Staff Turnover

In some States, high turnover among staff involved in the waiver demonstration exacerbated the problems with worker training and education noted above. Delaware, for example, noted in its final evaluation report that high staff turnover slowed efforts to integrate substance abuse counselors into CPS agency operations and to promote a joint case planning approach to serving enrolled families. One Delaware County experienced extremely high turnover, employing five substance abuse counselors over the course of the demonstration and coping with one year-long vacancy in this position. High turnover among substance abuse counselors in some Maryland jurisdictions impeded the smooth implementation of substance abuse services. Similarly, New Hampshire reported in a July 2002 progress report that turnover in substance abuse treatment counselors in one CPS office made it difficult for that office to maintain fidelity to the demonstration's original treatment model. Over a six month period, this office only had part-time assistance from a counselor in another CPS office while it searched for a qualified applicant to fill the vacant full-time position.

Differences in Management Styles and Professional Philosophies

Differences in the management styles of CPS supervisors and in the professional philosophies of child welfare caseworkers and substance abuse counselors further hindered waiver implementation in some States. These conflicts were most apparent in Delaware and Maryland and tended to reflect resistance to new case management models or the differing professional foci of child welfare and substance abuse workers. In Delaware, the likelihood of successful service coordination depended in part on the extent to which CPS unit supervisors involved the substance abuse counselor in case planning and decision making; however, the unit supervisor in the State's largest county preferred to maintain a clear separation between substance abuse services and child welfare services, thus prohibiting effective joint case planning. In contrast, another Delaware County implemented the key elements of the waiver model quickly and completely in part due to the unit supervisor's commitment to the joint case planning model and a balanced supervisory approach that integrated the substance abuse counselor more fully into unit operations.

Both Delaware and Maryland noted the challenges of implementing a successful substance abuse waiver in light of the differing philosophical traditions of child welfare workers and substance abuse professionals. In Delaware, this philosophical clash manifested itself in the differing emphases of child welfare caseworkers and substance abuse counselors in case planning and goal development, with child welfare workers stressing child safety and "reduction in harm" and substance abuse counselors emphasizing drug and alcohol abstinence. For many substance abuse counselors, anything short of complete abstinence by clients was regarded as a failure, whereas child welfare workers were more willing to tolerate some level of substance use - particularly involving alcohol or marijuana - if they perceived that overall child safety and well-being had improved.

In its final evaluation report, Delaware noted that ongoing training for substance abuse counselors and child welfare workers increased their mutual understanding and appreciation of their respective professional traditions and enhanced integration of both philosophies in case planning and decision making. Over time, substance abuse counselors began to integrate a "reduction in harm" approach into their work, while child welfare workers responded more seriously to alcohol and marijuana use by caregivers. In Maryland, a similar disconnect in professional perspectives became an obstacle to collaborative case planning by child welfare workers and addiction specialists, with some child welfare workers perceiving addiction specialists as emphasizing the recovery and treatment needs of the caregiver over the safety and well-being of children.

Summary of Process Findings

Although all four waiver States experienced varying degrees of difficulty in recruiting caregivers to participate in their substance abuse demonstrations, the available evaluation findings - especially from Illinois - suggest that intensive, proactive case management can enhance access to treatment services for substance-abusing caregivers and may have a modest positive impact on treatment retention and completion rates. Delaware's experience, however, highlights the critical importance of quality substance abuse treatment resources in improving treatment access and retention. The challenges faced by the States with caregiver enrollment highlight the need to test basic assumptions regarding the identification of substance use disorders in child welfare populations and caregivers' availability, motivation, and willingness to participate in treatment. Furthermore, these challenges make clear the importance of carefully defining target populations, eligibility criteria, and intake screening procedures. Finally, the experiences of all four States underscore the need to address data collection and reporting issues early in waiver planning and implementation, particularly with respect to obtaining informed consent from clients and completing substance abuse assessments.

Back toTable of Contents

State Outcome Evaluations - Summary of Key Findings and Issues

States with substance abuse waivers have reported varying degrees of success in achieving their goals with respect to key child welfare outcomes. As summarized in Table 5, Illinois and New Hampshire reported data on several outcomes of interest. Delaware chose to focus on a more limited number of evaluation outcomes, namely with respect to placement rates and placement duration. Maryland, which terminated its demonstration early, did not report outcome findings.

Table 5
Summary of Outcomes Studied By States with Substance Abuse Waiver Demonstrations

Outcomes	Delaware	New Hampshire	Illinois
Foster Care Placement Rates	Studied	Studied	Not Applicable
Placement Stability	Not Studied	Studied	Studied
Placement Duration	Studied	Studied	Studied
Reunification and Permanency Rates	Not Studied	Studied	Studied
Subsequent Maltreatment	Not Studied	Studied	Studied
Family and Child Well-Being	Not Studied	Studied	Not Studied

Back toTable of Contents

Reducing Placement Rates

Delaware and New Hampshire studied the effects of their substance abuse waivers on foster care placement rates, defined as the proportion of in-home children enrolled in the demonstration who later entered out-of-home placement. To date, neither State has found conclusive evidence that access to enhanced substance abuse services reduces rates of foster care placement.

Delaware

Delaware reported that across all three participating counties, somewhat smaller percentages of children in CPS units with access to enhanced substance abuse services entered placement than children in CPS units without access to enhanced services. In New Castle County, the placement rates were 28 percent for the experimental group and 31 percent for the control group; in Kent County, 19 percent and 25 percent, respectively; and in Sussex County, 25 percent and 34 percent, respectively. The effects of the waiver on these small differences in placement rates remain unclear, although the State suggested in its final evaluation report that the lower placement rates among experimental group children could have resulted in part from access to enhanced substance abuse services. Among experimental group families, those in New Castle County experienced the highest overall placement rates. The factors underlying this higher placement rate are unclear, although Delaware's process evaluation uncovered significant implementation problems in New Castle County - including high staff turnover and lack of joint case planning - that may have limited the quantity and quality of demonstration services.

New Hampshire

Like Delaware, New Hampshire has documented no definitive, significant variations in placement rates between families receiving enhanced substance abuse services and those receiving traditional child welfare services. Overall, 39 of 51 experimental group families (76 percent) that ever had an open CPS case during the waiver demonstration had at least one child enter placement, compared with 33 of 50 control group families (66 percent) that ever had an open CPS case.

Increasing Placement Stability

Illinois and New Hampshire both studied the effects of their waiver demonstrations on placement stability. Illinois defined placement stability in terms of the average number of times a child in foster care changes placement settings. To date, Illinois has found no evidence that access to enhanced substance abuse services improves placement stability for children in foster care. According to the State's May 2003 interim evaluation report, experimental group children had experienced an average of 3.67 placements as of March 2003 compared with 3.79 placements for control group children; this difference was not statistically significant. New Hampshire uses a somewhat different measure than Illinois to measure placement stability, tracking placement episodes on a per case rather than on a per child basis. Like Illinois, New Hampshire has observed no statistically significant effects of its demonstration on placement stability, with the State's latest data indicating an average of 3.06 placements per case in the experimental group compared with 3.70 placements per case in the control group. ¹²

Reducing Placement Duration

The States have revealed more definitive positive findings regarding the effects of the substance abuser waiver on reductions in the length of out-of-home placements, with Illinois uncovering the most conclusive evidence to date.

Delaware

Delaware reported that across all three implementation sites, children in the experimental group who entered out-of-home placement spent less time in foster care that those in the comparison group - an average of 204 days versus an average of 294 days. Although these figures suggest a major decline in average placement stays, the decrease fell short of the State's original goal of cutting the average duration of out-of-home placements by 50 percent. More significantly, Delaware included both in-home cases and out-of-home cases in its placement statistics, which skews the calculation of average placement length by including cases that experienced no time in placement. Thus, the State's figures may overstate the variance in placement duration between the experimental and comparison groups.

Illinois

Illinois' latest evaluation findings suggest that children in families with access to intensive substance abuse services spend measurably less time in foster care. According to the State's November 2004 progress report, children in the experimental group who returned home attained permanency in substantially less time than children in the control group, spending an average of 421 days in out-of-home care compared with 563 days for control group children, a statistically significant difference of 142 days. Although not statistically significant, the State also found that experimental group children exiting to adoption spent less time in out-of-home care, achieving permanency on average in 1,099 days compared with 1,128 days for control group children.

New Hampshire

The latest findings from New Hampshire are mixed, but suggest that no significant differences in placement duration exist between families receiving enhanced substance abuse services and those receiving traditional child welfare services. As of December 2004, the average length of placement per child among experimental group cases in which a child had been removed from the home was 287 days, somewhat higher than the 260 days for children in control group families. Although these findings suggest that experimental group children on average spent slightly more time in out-of-home placement than control group children, this difference is not statistically significant. Furthermore, small sample sizes could affect the ability of New Hampshire's evaluation to detect the effects of the demonstration on length of placement.

Increasing Reunification and Permanency Rates

Illinois

To date, no State has reported conclusive positive results regarding the effects of its substance abuse demonstration on reunification or overall permanency rates. In its November 2004 progress report, Illinois indicated that 10.3 percent of children in closed experimental group cases had returned home compared with 7.7 percent of children in closed control group cases. When all forms of permanency (defined as reunification, adoption, and guardianship) are compared, 20.8 percent of children in closed experimental group cases achieved permanency compared with 19.8 percent of their control group counterparts. While suggesting a positive trend, neither difference was statistically significant. These latest data contradict findings from the State's earlier June 2004 progress report, which indicated that experimental group children were more likely to achieve reunification or another permanency outcome at statistically significant levels. Illinois' final evaluation report, expected in December 2005, may offer a more definitive answer regarding the effects of the State's substance abuse waiver on reunification and overall permanency rates.

New Hampshire

The available data from New Hampshire likewise suggest no major effects of its substance abuse demonstration on reunification rates. As reported in the State's September 2003 interim report, 12 of 41 experimental group children (29 percent) who entered or began the demonstration in placement returned home, compared with 11 of 41 control group children (27 percent) who were in or entered placement.

Preventing Subsequent Maltreatment

Illinois and New Hampshire evaluated the effects of their substance abuse waiver demonstrations on maltreatment recurrence. The latest findings from Illinois indicate that families with access to enhanced substance abuse services may experience reduced risk of subsequent maltreatment. To date, New Hampshire has uncovered no effect of its waiver demonstration on the likelihood of subsequent maltreatment referrals.

Illinois

As of June 30, 2004, only 11.2 percent of experimental group caregivers in Illinois' demonstration had a subsequent maltreatment allegation following assignment to the demonstration compared with 15.3 percent of control group caregivers, a statistically significant difference.¹³

New Hampshire

As of March 2004, 43.7 percent of experimental group cases in New Hampshire had a subsequent allegation of abuse or neglect following assignment to the demonstration compared with 44.2 percent of control group cases, a statistically insignificant difference. Of those cases with a subsequent allegation, 10 percent (22) of experimental group cases had a substantiated maltreatment report compared with 13 percent of control group cases (28), also an insignificant difference.

Enhancing Parents' Abilities and Strengthening Family and Child Well-Being

Of the four States with substance abuse waiver demonstrations, only Delaware and New Hampshire included measures of child and family well-being in their original evaluation plans. Delaware reported that it was unable to produce any meaningful data from its case intake and caregiver profile data to study these outcomes. New Hampshire has reported some initial well-being findings that point in a positive direction. For example, caregiver interviews conducted using the Child Behavior Checklist (CBCL) indicated greater declines in problem behaviors in seven of eight categories for children in the experimental group compared to children in the control group, with a particularly notable decrease in reports of aggressive child behaviors. Follow-up interviews with caregivers revealed additional positive well-being outcomes for adults with access to enhanced substance abuse services, with experimental group parents less likely to be on TANF at follow-up, more likely to be enrolled in an educational program, and more likely to be employed full-time than parents in the control group. While many of these findings lacked statistical significance, the pattern of somewhat improved outcomes across a number of domains suggests a positive trend for families receiving enhanced demonstration services.^14/sup>

Summary of Outcome Findings

Findings to date from the States' evaluations illustrate the challenges of affecting positive changes in child welfare outcomes, including foster care placement prevention, placement stability, reunification, and overall permanency, among families with caregivers that have substance use disorders. However, some evidence from Illinois suggests that a substance abuse waiver may reduce the duration of foster care placements and lower the risk of maltreatment recurrence. In addition, recent findings from New Hampshire indicate that access to enhanced substance abuse services may have some positive effect on measures of parent and child well-being. Future evaluation findings from Illinois and New Hampshire will shed additional light on the effects of substance abuse waiver services on key child welfare outcomes. However, caution must be exercised in interpreting outcome findings across States given the substantial differences in their scope, size, service models, and target populations.

⁶“In-home” cases are those in which no children were in or entered foster care at any point during the demonstration. “Out of-home” cases are those in which at least one child was in or entered foster care at the time of assignment to the demonstration or at some point following assignment. In all States, in-home cases include both those with substantiated and unsubstantiated maltreatment allegations.Back

⁷Maryland did not provide a breakout of in-home versus out-of-home cases.Back

⁸Michigan Alcoholism Screening Test.Back

⁹Estimates of the presence of mental health issues from New Hampshire and Illinois are derived from state child welfare/substance abuse databases. The figure for New Hampshire may underestimate the actual prevalence of mental illness among participants in that State’s waiver demonstration. Based on a review of interview data from 200 subjects enrolled in its demonstration, New Hampshire’s evaluators estimate that the incidence of depression/dysphoria in its research sample is 40 percent as measured by the Center for Epidemiologic Studies Depression Scale (CES-D).Back

¹⁰Memorandum from the Illinois Department of Children and Family Services dated April 15, 2005.Back

¹¹E-mail correspondence from the Family Research Laboratory at the University of New Hampshire dated May 2, 2005.Back

¹²E-mail correspondence from the Family Research Laboratory at the University of New Hampshire dated May 11, 2005.Back

¹³Memorandum from Illinois Department of Children and Family Services dated April 15, 2005.Back

¹⁴Memorandum from the Family Research Laboratory at the University of New Hampshire dated April 15, 2005.Back

Back toTable of Contents