Search Type Title Author/Investigator Search Advanced Search

Table of Contents | Previous | Next

2. Study Design: Sampling, Estimation and Measures

2.1 Target Population

The target population for the OYFC study contained all children who entered into foster care 12 months prior to the date of sampling and who were still in foster care at the time of sampling. The rationale for this definition of the target population was threefold.

First, this definition provides a sample of children who are likely to be in foster care for an extended period, because exit rates for children who have been in foster care for at least one year are far lower than those for children in the earliest months of care (Wulczyn, Hislop, and Goerge, 2000). The likelihood of reunification to the home of the parent is only one-third what it is for children who have just entered care (Wulczyn, Brunner, and Goerge, 1999).

Second, it provides data on children fairly early in their stays in OOHC. These data can then provide a valuable baseline for understanding future observations of these children as they age over the course of the study.

Finally, these data also provide a better understanding of the dynamics of foster care for the group that will emerge from the CPS sample component.

The definition of the target population was tested and revised to find an approach that could be implemented accurately and relatively easily in all agencies, regardless of the sophistication of the individual administrative record-keeping systems. In our work with the participating agencies, we discovered that some agencies were unable to determine easily whether a child who entered into OOHC 12 months earlier remained in continuous care for the whole 12-month period. Therefore, the more rigorous target population definition of a continuous 12-month OOHC placement was infeasible for OYFC sample eligibility. This modification of the target population definition was adopted after preliminary analyses of administrative data indicated that the likelihood of a child remaining in OOHC did not depend on whether he or she had a continuous 12-month stay or was simply in OOHC at two points in time separated by 12 months.

A challenge to applying this definition of the target group was finding sufficient numbers of children who satisfied the selection criteria. For example, in small, rural agencies, the number of foster care children was quite small and, thus, the number of children in foster care for 12 months was often less than that required for an efficient sampling design. Therefore, for these agencies, it was preferable to relax the 12-month requirement and to allow children who had been in OOHC for as few as 8 months and as many as 18 months to enter the sample. Transmitting information about the sample to the field representative and securing interview data also required additional time. The average time between placement into OOHC and completion of the child welfare worker interview was 18.2 months; the median was 17.7 months. Table 2-1 presents the distribution of children by their length of stay in OOHC at the time the sample was selected.

Table 2-1. Duration of Placement at Time of Sampling

Number of elapsed months	Number of children in sample
Less than 12	45
12	47
13-14	373
15-17	193
18-20	69
Total	727

 

Sample eligibility was also not restricted to new entrants to the child welfare system. The target population included children who had previous stays in OOHC and who were reunited with their families prior to reentering OOHC for their current stay. To be eligible for the sample, children had to have entered OOHC for their current stay through an investigation of child abuse or neglect by the CPS. Children who entered through the court system or some other pathway not involving a CPS investigation were ineligible.

2.2 Sample Design

The NSCAW sample was designed to maximize precision of estimates related to children in the child welfare system. The sample design may be described as a stratified cluster sample of all children in the target population. In response to the mandate in the authorizing legislation, the sample was designed to calculate state-level estimates for the eight states with the largest numbers of CPS cases; each of these states forms one stratum. The ninth and final stratum then consists of the remaining states, with a few exceptions described below.

Within these strata, primary sampling units were formed, where the PSU was defined as a geographic area encompassing the population served by a CPS agency. In most cases, PSUs are counties, but in a few cases two or three contiguous counties were grouped to form a single PSU. Further, several counties comprising large metropolitan areas were split into two or more PSUs along CPS agency jurisdiction boundaries to facilitate sampling and data collection. Finally, the NSCAW PSU frame was composed of all counties in the U.S. that were large enough to support at least one interviewer-workload, or about 60 cases or more per year. Counties smaller than this size were deleted from the frame; however, these exclusions represented less than 1% of the target population.

Each PSU was then assigned a selection probability, and a random sample of 100 PSUs was selected accordingly. The selection probability for a PSU was computed using composite size measures derived from eight population subgroups (or sampling domains) whose selection rates were to be controlled during the second-stage selection process for the CPS sample component (see Biemer, Liu, Iannacchione, Byron, & Cano, 1998, for a description of these sampling domains). However, these domains were relevant only for the CPS sample selection and were ignored for selecting the OYFC child sample, which instead used systematic random sampling from all eligible children on the frame. By selecting an approximately equal number of children within each PSU, the resulting OYFC sample is approximately self-weighting (i.e., equal probability) within stratum. However, weights are still needed in estimation, due to imperfections in the sampling as well as for post-survey adjustments for nonresponse and frame noncoverage.

After the PSUs were selected, six child welfare agencies indicated that they were unable or unwilling to participate in the NSCAW study and, therefore, were replaced in the sample by six new PSUs that were similar with regard to the sampling control variables. In addition, problems arose in four states in the remainder stratum due to state laws requiring that information on CPS children and their caregivers be released to the study only by consent of the current caregiver. As a result, the response rates in those states were essentially zero, and it was necessary to cease data collection efforts there for both the OYFC and CPS components. These four states were subsequently removed from the target population for the study; consequently, inferences from the OYFC will be restricted to children living in states that do not have laws restricting direct access to the children for research purposes—92% of all children originally eligible for the OYFC sample. The proportion of the original target population excluded from the study is only about 8%, so it is unlikely that the results would change appreciably with the inclusion of these agency first-contact states.

The OYFC sample selection period was December 1999 through February 2000. As a result, the time period from December 1998 to February 1999 was set as the time interval for eligibility—only children placed in out-of-home care in those three months were eligible for the OYFC sample. In many PSUs, the number of children on the frame for the original time period was found to be too small to support the sample sizes required. As a result, the window of inclusion for frame construction was extended in those PSUs, when necessary, to include children who were placed in out-of-home care between July 1998 and February 1999 to meet the desired allocations. In order to reduce the burden on caregivers of the OYFC children, only one child per household (where the “household” is the residence where the child lives) was included in the frame for OYFC sample selection.

2.2.1 Description of the Sample

In discussing the results for NSCAW, both weighted and unweighted response rates are relevant. The unweighted response rate is the number of respondents to the OYFC divided by the number of respondents and nonrespondents in the sample; this is a useful indicator of the success of the field effort because it conveys the actual rate at which eligible sample members were interviewed. However, the weighted response rates (simply the sum of the weights for respondents to the survey divided by the sum of the weights of respondents and nonrespondents) are a more relevant indicator of the potential for bias in the results due to nonresponse.

As mentioned earlier, NSCAW data were obtained through interviews with several respondents, including the current caregiver, the former caregiver (if different), the child, the child welfare worker, and the child’s teacher. Any one or all of these interviews may be missing for a sample child; thus exactly what constitutes a “response” to the OYFC is not obvious. One possible definition requires a full response from all four or five possible respondents. This definition is too strict, however, because the key analysis variables may still be available even if the teacher, former caregiver, or child welfare worker does not respond. Therefore, for operational reasons, we defined an OYFC response as a completed interview for the key respondent—defined as the current caregiver if the child was younger than 10 years or the child if 11 years or older. Using this definition of a completed interview, the overall weighted response rate for the OYFC was 73.1%, and the weighted response rate was 73.4%.

The former caregiver component of the data collection was eventually discontinued, due to extremely low response, and is not analyzed in the current report. Table 2-2 includes the unweighted and weighted response rates for all the major components of the OYFC study.

Table 2-2. Unweighted and Weighted Response Rates for the OYFC Sample by Type of Respondent

Component	Number interviewed	Percent unweighted response rate	Percent weighted response rate
Child	696	70.30	71.82
Current caregiver	725	72.94	73.83
Child welfare worker	794	80.28	83.07
Teacher*	166	51.71	56.62

*The completion rate is reported for the teacher survey, computed as the number of interviews divided by the number known to be eligible for the component. To be eligible for the teacher survey, children had to be aged 4 or older, in school in grades K-12, not home schooled, and have a signed authorization from the legal guardian or caregiver.

 

Table 2-3 summarizes the final case dispositions for Wave 1 for OYFC key respondents. This includes a breakdown of the number of selected children, completed and partial interviews, the number of children and adults who were key respondents, and cases that did not result in interviews.

A total of 1,291 children were selected for the OYFC sample. Of these, 373 (29%) were children aged 11 and older (child was key respondent), and 918 (71%) were children less than age 11 (caregiver was key respondent). From the OYFC sample, 726 key respondent interviews were completed, along with one partial interview. Interviews were deemed complete if they met specific criteria established by the NSCAW project team. For child interviews, at least one well-being measure had to have been obtained.

Final case dispositions included 291 ineligibles (23%), in addition to 90 refusals (9%), 16 unlocatables (2%), and 141 cases (14%) that could not be reached after repeated attempts. Of these 141 cases, 111 (9%) were sample cases that were released to the field but were not completed because the target number of completed interviews had been exceeded and the available resources expended. Cases were deemed ineligible if

• the selected child was found to be older than 15 at the time of sampling,

• the selected child was determined to be the sibling of another child in the study,

• the placement date for the selected child occurred outside the frame construction period,

• the selected child was determined to be deceased, or

• if the OOHC tenure and other eligibility criteria for the OYFC sample were not met.

Table 2-3. Key Respondent Final Case Dispositions

Disposition	Sample counts and percentage
	N	%
Children selected	1,291	100
Number of selected cases with
Child as key respondent	373	28.9
Caregiver as key respondent	918	71.1
class="superscript"Key respondent case status
Completed full interview	726	72.6
Completed partial interview	1	0.1
Final ineligible	291	22.5
Key respondent nonresponse
Unavailable after repeated attempts	141	14.1
Final refusal	90	9.0
Final unlocatable	16	1.6
Final out of area	2	0.2
Physically/mentally incapable	5	0.5
Incarcerated-interview not obtained	1	0.1
Institutionalized-interview not obtained	1	0.1
Final other	17	1.7

 

Refusal cases included those in which (1) the key respondent refused to consent to the interview or (2) parental or legal guardian consent could not be obtained for the child interview. Unlocatable cases included those in which the key respondent could not be located after extensive field and central office–based tracing. Cases that could not be completed after repeated attempts included those in which the key respondent either could not be reached or was unavailable for the interview during the data collection period. Cases received a “final out of area” disposition code in situations in which the key respondent lived more than 65 miles (one way) from an NSCAW field representative, a firm appointment could not be obtained, or costs for securing the interview were considered prohibitive. Final “other” noninterview codes were assigned in situations in which the child’s case records were sealed because of the case’s high profile or because of completed or ongoing adoption proceedings.

2.2.2 Characteristics of the Final Achieved Sample

Table 2-4 shows the distribution of the selected and final achieved samples by age, race, and gender. The percentage distribution shown in this table is the unweighted distribution of the achieved sample; other tables in this report provide the weighted distribution, which reflects the distribution of the OYFC population. The final achieved sample was nearly evenly split between males (51%) and females (49%). The largest group of children were younger than 5 years of age (45%), with only 27% of children being older than 10. There were more black children (45%) than white (41%); only a small group were identified as other race (14%)—these children were primarily Asian and American Indian. Participation in the study seems to be unaffected by any interaction between age and gender or race and gender. (The race and ethnic characteristics of the sample are discussed in more detail below.)

Table 2-4. Distribution of Sample by Age, Race, and Gender

Characteristic	Gender
	Female	Male	Total
Age
1-4 years	160 (22%)	165 (23%)	325
5-9 years	101 (14%)	100 (14%)	201
10 years or older	98 (13%)	103 (14%)	201
Race
White	149 (20%)	153 (21%)	302
Black	163 (22%)	166 (23%)	329
Other	47 ( 7%)	49 ( 7%)	96
Total	359 (49%)	368 (51%)	727

 

2.3 Weighting and Estimation

Although the OYFC sample was designed to be self-weighting, departures from equal probability sampling were inevitable due to imperfections in the frame information, insufficient target population sizes within small PSUs, and other consequences of reality. In addition, post-survey adjustments for nonresponse and frame coverage error also result in departures from a self-weighting design. Consequently, sample weights must be applied to the observations in order to obtain unbiased estimates of the population parameters. Thus, an estimate of the population total, denoted by , takes the form

where wi is the sample weight and yi is the observation for the ith child. An estimate of the population mean, denoted by , is a ratio and takes the form

To the extent that nonresponse and sampling frame noncoverage error adjustments are effective, the bias in estimates due to these sources of error is reduced. Thus the use of sampling weights in analysis is necessary in order to properly represent the target population by the units selected for NSCAW. Although comparisons between weighted and unweighted analyses often showed minor differences, all analyses reported here are weighted because they offer more precision.

Moreover, because the observations are clustered within PSUs, the standard errors of the estimates must account for the potential correlation between the observations within the same PSU to be statistically valid. Consequently, standard error estimates typically produced by software packages that assume simple random sampling (SAS, SPSS) will produce standard error estimates that are likely to be understated. This implies that the true alpha levels for standard tests of hypotheses will likely be somewhat larger than the nominal level, and the levels of confidence for confidence intervals will be somewhat lower than the nominal levels. To account for these properties of the sample design, the analyses were completed using the SUDAAN software package (Research Triangle Institute, 2001), which appropriately accounts for the unequal weighting, stratification, and clustering of the observations inherent in the NSCAW sample design. SUDAAN uses Taylor series linearization for estimating the standard errors of nonlinear statistics such as ratios (Cochran, 1977). Use of statistical software packages that do not properly account for the unequal weighting and clustering effects in the sample may lead to invalid estimates.

Precision in this report is reported as the standard error of the estimate for means and as the endpoints of the 95% confidence interval for proportions. The confidence intervals were computed using the logit transformation of the proportion.

2.4 Analysis of Nonresponse

Child welfare services research has been characterized by studies with poor sample construction and low response rates (Rossi, 1992), leaving the studies open to the criticism that they capture a biased view of the population of concern. To determine the potential for nonresponse to bias the OYFC results, we conducted an analysis of the nonresponse bias for these data. For a large proportion of OYFC key nonrespondents, data were available from the child welfare worker and agency files. These data were used to estimate the nonresponse bias and then destroyed. An estimate of the nonresponse bias for the population mean of some variable, y, is given by

where r is the response rate, is the mean for respondents, and is the mean for the nonrespondents.

As a general indicator of the potential for nonresponse to bias the OYFC results, we used a count of the number of variables in the nonresponse analysis for which the bias is statistically significant from zero (two-sided test). At the p<.05 alpha-level, one would expect 5% of such tests to be significant by chance alone. Conversely, if more than 5% of tests of non-zero bias are significant, that would be evidence of nonresponse bias in some of the study variables. Likewise, at a significance level of p<.01, one would expect approximately 1% of the tests of non-zero bias to be significant by chance.

Note from Table 2-5 that approximately 1.5% of the tests were significant at a 5% alpha level and less than 1% were significant at the 1% level; thus there is no evidence of nonresponse biases in the OYFC data. This does not necessarily mean that the OYFC results were not biased by nonresponse—only that the data available for this analysis were insufficient to detect a nonresponse bias. Nor is there indication that the bias was sufficiently large to justify the additional effort required to include bias estimates in the data analyses.

Table 2-5. Statistically Significant Nonresponse Bias Estimates for Caregiver and Child Characteristics

		Significant at 0.05			Significant at 0.01
Respondent type	Number of tests	Number of Tests	Percent of Tests	Number of Tests	Percent of Tests
Caregiver	1,538	23	1.50	6	0.36
Child	1,679	29	1.73	11	0.72

 

Table 2-6 indicates that the response rate tends to be slightly lower for children aged 11 to 14 than for children 10 or younger. This suggests that the potential for nonresponse bias is greater for older children and their caregivers. This effect of age on nonresponse was not apparent in the previous analysis because those data were analyzed separately by key respondent type: child and caregiver. (For NSCAW, the caregiver was the key respondent when the child was less than 11 years old.) Therefore, the nonresponse bias results for children included only children who were at least 11 years old. Still, the lack of evidence for nonresponse bias in the previous analysis suggests that the greater relative bias for older children was quite small.

Table 2-6. Response Rates by Age of Child

Age	Number of respondents	Percent unweighted response rate	Percent weighted response rate
0 - 2 years old	246	76.64	78.94
3 - 5 years old	122	71.35	64.37
6 - 10 years old	196	73.41	76.07
11- 14 years old	163	69.07	69.41
Total	727	73.07	73.41

2.5 Instrumentation for the OYFC Study

The measures used in the NSCAW are described in detail in Appendix A. Because of their different situations at the time of the baseline interviews, there were two primary differences between the questionnaires for the OYFC and CPS sample components. First, the risk assessment with the child welfare worker at baseline was abbreviated, given that he or she was asked about the child’s situation 12 months prior. The child welfare workers indicated that they would not have good information about certain items, so those were excluded.

Similarly, questions on services to biological parents and children were asked of the child welfare worker for the OYFC sample at baseline. These were not asked for the CPS sample component, because children and families in this group may not have begun receiving services.

Many measures used in the following analyses were simply single items (e.g., the race and age of the child); others were derived after consolidating a number of single items intended to capture key case characteristics; and some (described at the end of this chapter) were standardized measures. Most of these items and scales measure child functioning as rated by

• caregivers (e.g., the Child Behavior Checklist, CBCL; and the Social Skills Rating System, SSRS);

• teachers (e.g., the Teacher Rating Form, TRF); or

• field representatives during standard assessment procedures (e.g., the Battelle
  Developmental Inventory, BDI; the Bayley Infant Neurodevelopmental Screener, BINS; the Kaufman Brief Intelligence Test, KBIT; the Mini-Battery of Achievement, MBA; and the Preschool Language Scale-3, PLS-3).

A few are self-report child measures (i.e., the Children’s Depression Inventory, CDI; the Research Assessment Package for Schools, RAPS; the Violence Exposure Scale for Children-Revised, VEX-R; and the Youth Self-Report, YSR) that were completed by older children (5-14 years of age, depending on the measure).

Also administered were the Short Form Health Survey (SF-12), a measure of health and well-being for caregivers, and the NLS/Y short form of the Home Observation Measure of the Environment (HOME-SF), a measure of the environment in which the child is being raised. (As operationalized in NSCAW, HOME-SF includes some parental report items and some from the field representative’s observations.) In addition, many items were asked of respondents to allow them to describe their experiences—some of these were later scaled, some clusters of items are presented in their entirety, and some are not discussed in this report.

2.6 Analysis Methods

Because exhaustive analyses of these data are beyond the scope of this study, we developed a data analysis plan to identify key analyses of most interest to policy makers and practitioners alike. In this report, the NSCAW Research Group answers questions about the characteristics of children in foster care for one year (OYFC), their caregivers and caregiving environments, and their child welfare workers; the nature of services provided to the child and families; and the experiences of the children in out-of-home care (OOHC). Many of the research questions were addressed through descriptive analysis and interpretation of bivariate presentations of the data. When questions about interpreting bivariate analyses arose, however, we conducted multivariate analyses to probe for relationships among child, caregiver, and placement characteristics, and with developmental outcomes. Weighting of the data was necessary to account for departures from the equal probability selection design, nonresponse, and sampling frame undercoverage.

 

 

Table of Contents | Previous | Next