Table of Contents | Previous | Next |
1. INTRODUCTION
1.1 Background
To learn what happens to the children and families who come in contact with the child welfare system, the Children’s Bureau of the Administration on Children, Youth and Families, U.S. Department of Health and Human Services, has undertaken the National Survey of Child and Adolescent Well-Being (NSCAW). The first national longitudinal study of its kind, NSCAW is examining the characteristics, needs, experiences, and outcomes for these children and families. This study, authorized under the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA),1 also will provide information about crucial program, policy, and practice issues of concern to the Federal government, state and local governments, and child welfare agencies. This is the first such study to relate child and family well-being to family characteristics, experience with the child welfare system, community environment, and other factors.
NSCAW is gathering information associated with 6,100 children from public child welfare agencies in a stratified random sample of 92 localities across the United States.2
1.2 Purpose
Although NSCAW’s primary focus is the collection of child-level information directly from children, families, caregivers, caseworkers, and teachers on children's functioning, well-being, services, and outcomes, this study has also collected data from administrators in local and state child welfare agencies. These data from agencies provide a current snapshot, from the administrators' point of view, of how child welfare services are organized and delivered and give context to and inform the child- and family-level data being collected.
1.3 Overview
This report presents information obtained from local-level child welfare administrators, who were asked about a number of factors—such as staffing and training, caseload, budget, changes in policy and legislation, client characteristics, and so on—affecting the delivery of child welfare services. Although the main case-level study will have substantial information on child welfare worker and caregiver characteristics related to individual children in the study, the interviews described in this report provide an opportunity to learn about issues at the local agency level that influence child welfare services and outcomes.
The Local Agency Survey (LAS) was conducted during the first wave of data collection for NSCAW and offers the field a picture of the way child welfare services operated during 1999-2000.
1.4 Organization of the Report
Along with the general overview of survey design and data sources, with a particular emphasis on the selection of child welfare agencies, the Data Collection Methods subsection also outlines the methods used to analyze the data from the LAS.
The results of this study—presented in Section 2—are organized into six general subsections:
- 3 Child welfare system and child welfare agencies
- Staffing and training
- Services and service dynamics
- Client characteristics and caseload dynamics
- Budget and expenditures of child welfare agencies
- Changes in child welfare services
The report also includes a discussion of the implications of the local agency findings for child-level NSCAW analysis and for the field.
1.5 Data Collection Methods
1.5.1 Sample Design
The NSCAW was a two-stage stratified sample design. At the first stage, the United States was divided into nine sampling strata, consisting of the eight states with the largest child welfare caseloads and the remainder of the United States. Primary sampling units (PSUs) were selected within each of the nine strata. These PSUs were defined as geographic areas that encompass the population served by a child protective services (CPS) agency. In most cases, these areas comprise a county or a group of counties. However, in larger metropolitan areas, smaller geographic areas were defined to facilitate sampling and data collection. These PSUs were chosen using a probability-proportionate-to-size procedure so that any child who was investigated for child abuse and neglect would be included in the sample with equal probabilities (within strata and second-stage strata). Details of the sample design and construction of PSUs are documented in previous reports (Biemer, Liu, Iannacchione, Byron, & Cano, 1998) and papers (The NSCAW Research Group, in press). The NSCAW sample was designed to maximize precision of estimates related to children. However, data were also collected from local agencies, and for the LAS analysis, the selection of PSUs is the most relevant.
The information was collected from local child welfare administrators in two stages. Field staff assigned to each PSU—who were concurrently interviewing children, their caregivers, and their caseworkers—interviewed child welfare agency managers using the Local Agency Directors Interview (LADI, see Appendix A). At the end of that interview, directors were asked to complete the Self-Administered Questionnaire (SAQ, see Appendix B), which included questions focusing on staff resources, foster care resources, and service activities for the most recent fiscal year.3 The LAS and LADI were first pilot tested with administrators from a small convenience sample of PSUs, and revisions were made based on participant feedback. Revised instruments and procedures were developed.
The LADIs with child welfare managers, on average, took 44 minutes to complete. The child welfare directors were then asked to take the LASs with them to be completed and returned to the field representative (FR) within two weeks. During pilot testing, the researchers had learned that completion of many of the items about caseload and financing would require input from administrative databases and other agency staff (e.g., personnel managers or fiscal officers), which led to a decision to split the instrument. The completion of these SAQs took longer than the researchers had anticipated: an average of 6 hours, 43 minutes (even so, many items were not completed).
To improve data quality, the FR conducted a brief edit check of the completed SAQ administered questionnaire when it was picked up from the agency director to ensure that all required items had been completed. The FRs encouraged the agency directors to provide any missing data or to explain why information could not be provided. No effort was made to corroborate self-reports with publicly available administrative information.
Ultimately, LASs and LADIs were collected from administrators representing 92 PSUs involved in the overall NSCAW study. For most of the PSUs selected for NSCAW, the PSU represents one county, and only one agency respondent received the questionnaire for the county. In this case, the agency weight is the same as the PSU weight. So, although the study researchers interviewed child welfare agency directors from both very small and very large counties, they did not weight their answers equally in the final analysis, because their responses described child welfare agency characteristics representing very different numbers of children, foster parents, and child welfare workers. PSUs were weighted in proportion to their size to capture the characteristics of the nation’s child welfare services agencies.
There were a few exceptions to the weighting process. In PSUs in which more than one agency was administered the questionnaire—for example, when the local agencies were so small that two of them were needed to make up a single PSU—the PSU weight was divided proportionately among the agencies. In one state, the state agency responded singly to describe more than one PSU, so the weight associated with that respondent is the aggregate of the two PSUs. In larger cities with multiple PSUs, the agency weight is the aggregate of the corresponding PSU weights.
All but one PSU were ultimately described by respondents. To adjust the agency-level weights to account for nonresponse, the researchers conducted a simple ratio adjustment in which the weights of the respondents were prorated such that the total for the adjusted responding agency weights equals the total of all the agency-level weights. This allows for unbiased estimates of the characteristics of the nation’s child welfare agencies.
1.5.2 Approach to Data Analyses
NSCAW is primarily focused on children and families in the child welfare system; thus, the design was driven mainly by precision and accuracy objectives for estimates of child-and family-level characteristics. The sampling scheme used for selecting agencies for the study, therefore, is ill suited for making inferences about all child welfare agencies. For example, there are more than 3,000 U.S. counties with child welfare agencies, and the data in this report are based upon a random sample of only 92 agencies.
In addition, the agencies in this analysis were selected by a sampling process that gave greater probability to selecting agencies in larger counties than in smaller ones. To account for the unequal probability sampling of agencies, the data in the report must be weighted by the inverse of the sample inclusion probabilities. This weighting process, although necessary for valid inferences, can increase the standard errors of the estimates in some situations. To account for the stratified, clustered sample design used for the local agency sample, Research Triangle Institute’s SUDAAN software (Shah, Barnwell, and Bieler, 1997) was used to produce the weighted estimates, standard errors, t-tests, and chi-square tests of significance. The endpoints of the 95% confidence intervals were computed using the logit transformation of the proportion, because the symmetric interval based on the normal distribution sometimes gave negative values for the lower limits of the confidence intervals.
With such a small effective sample size, the power of tests of hypotheses to detect differences by type of county is quite limited. To increase the power of the analysis while still maintaining an acceptable significance level for the tests, the study researchers devised an analytical approach that would allow the identification of differences that they believed would be significant had a larger sample size or effective sample size been achieved.A structured set of procedures was used to ensure that the data analyses addressed the most important questions with the greatest certainty about the answers. The design of this study, especially the modest-sized sample and weighted data, required careful data analysis and interpretation. During a detailed data analysis planning process prior to the completion of data collection, the researchers identified the following key issues and then limited the analyses to the comparisons they considered most important.
Overall descriptions of the self-reported characteristics of child welfare agencies are first described and then compared on state vs. county administration, county size, poverty level of the county, and urban or rural character of the county. “Administration” was defined as either a county- or state-administered child welfare agency. Agencies that identified themselves as having other types of administration (n=3) were not included in the analyses involving administration type. County size was defined as (a) small, under 5,000 children; (b) medium, 5,000 to 24,999 children; or (c) large, 25,000 children or more. Due to the small sample size, small- and medium-size counties were later combined into a group called other (32% of PSUs) for comparison with large counties (68% of PSUs). Poverty level was defined as either (a) nonpoor, 5% or less of county families with children living below the 50% poverty level (49% of PSUs); or (b) poor, more than 5% of county families with children living below the 50% poverty level (51% of PSUs). Consistent with U.S. Census Bureau definitions, urban was defined as greater than 50% of the population living in an urban area (73% of PSUs), whereas rural (27% of PSUs) was defined as all areas that did not meet this requirement (see Appendix C for a table describing these breakdowns and weighted percentages). The researchers tested for relationships between these PSU characteristics (e.g., are state-administered PSUs more likely to be classified as urban and poor?) and found only one significant association between these PSU characteristics (urban rural x county size are strongly associated, p < .001). All analyses were completed twice—first unweighted and then weighted. The unweighted data were used on rare occasions to confirm findings of marginal differences in weighted data—all analyses included in this report were done with weighted data.
Because the researchers sought to explore a variety of possible relationships that had never been studied, they did not want to unduly restrict their search for relationships and ran a substantial number of analyses. For that reason, they decided against using th e more inclusive significance level of p < .10, commonly used in preliminary studies, because this would result in too many false positive findings to allow for confident interpretation. Instead, they chose the following terminology in writing about the results:
- They describe some indication of difference between the populations
when weighted analyses find the probability of difference to be .05 <
p <. 10 and when the unweighted difference is significant in the same
direction and the finding is theoretically plausible.
- They describe that there was some evidence of difference when
the probability of difference is .01 < p < .05 in the weighted analysis
(they do not call this a “significant” difference because of
the inflation of the alpha levels due to the many tests that were run).
- They note a significant difference (or stronger evidence of difference) only when the p value is at .01 or less.4
Based only on the percentages, some of the differences between groups appear large, even though there is no indication of difference. The tables include confidence intervals or standard errors, which are often large. These indicate the reason that the researchers often lack confidence that the groups are really different. When the confidence intervals overlap, it means that the experiences of the groups being tested may also overlap and may not be as distinct as the percentages (or means) initially suggest. This occurs because this study includes a relatively small sample of counties. Although the NSCAW sample was not designed to maximize the power for agency comparison, when the report does indicate significant differences or strong evidence of differences, there is good reason to have confidence in those assertions.
1Personal Responsibility and Work Opportunity Reconciliation Act of 1996, Sec. 429A, National Random Sample Study of Child Welfare (PL No. 104-193).(back)
2For a detailed description of NSCAW, see the NSCAW Research Group, Methodological Lessons from the National Survey of Child and Adolescent Well-Being: The first three years of the USA’s first national probability study of children and families investigated for abuse and neglect. Children and Youth Services Review, in press.(back)
3The fiscal year was generally 1999, although some were completed in 1998.(back)
4See Appendix D for a stand-alone description of considerations and terminology regarding the interpretation of statistical tests.(back)
| Table of Contents | Previous | Next |