|
|
|---|
|
|
|||||
|---|---|---|---|---|---|
|
|
|
||||
| ACF Home | Services | Working with ACF | Policy/Planning | About ACF | ACF News | HHS Home | |||||
Questions?
|
Privacy
|
Site Index
|
Contact Us
|
Download Reader™
|
Print
|
|---|
Section I of this Technical Bulletin contains information about the Children's Bureau's (CB), within the Administration for Children and Families' (ACF), approach to determining and approving degrees of improvement and attainment of goals for Program Improvement Plans (PIPs) for the second round of the Child and Family Services Reviews (CFSRs). Section II contains updated technical information on evaluating State attainment of goals regarding the national standard data indicators in the PIP. Section III contains technical information about methodologies that may be used and have been preapproved by CB to measure degrees of improvement specific to PIP items.
CB is currently negotiating PIPs for the second round of CFSRs. CB will jointly develop PIPs with States and will consider for approval any proposed content and measurement methods from an individual State. In accordance with 45 CFR 1355.35(a)(1), States must address in their PIPs each item or related data indicator that contributed to a determination of nonconformity for each outcome or systemic factor. In prioritizing areas to be addressed, the State must first address (in content and timeframes) items that affect child safety, followed by those most egregiously not in substantial conformity. In ACYF-CB-IM-07-05, we provided guidance for use by States and Regional Offices (RO) in negotiating the amount of improvement necessary to meet the national standards through an approved PIP for the second round of CFSRs. We are now providing that information for States concerning negotiating measures for items other than the national standard data indicators.
In accordance with Federal regulations at 45 CFR1355.35 (a)(1), the PIP must describe methods that will be used to evaluate progress. In accordance with 45 CFR 1355.35 (a)(2), if CB and the State cannot reach consensus on the content of a PIP or the degree of program improvement to be achieved, CB retains final authority to assign the contents of the plan and/or the degree of improvement required for successful completion of the plan. Based on our experiences in the first round, we recognize that there will be situations when consensus on the degree of improvement is enhanced if guidelines are available, including methods that have been preapproved by CB. We recognize that States have different capacities for measuring program improvement and that this is an evolving area for most States. CB has provided guidance consistent with the regulations at 45 CFR 1355.35(a)(1) and a suggested PIP format including a measurement matrix that can be found in ACYF-CB-IM-07-08. The specific methods contained in this Technical Bulletin are additional guidelines and suggested measurement methods. We are not requiring States to use the methods outlined in this Technical Bulletin to establish and measure the degree of PIP improvement. Although the methods outlined here may include minimum sample sizes, baseline periods, and other specifics, we are not requiring States to meet these standards for the purposes of measuring improvement for items that are not measured by the national standard indicators. States may propose alternative measurement methods to CB prior to PIP approval. The proposals will be considered individually and States will be advised of their acceptability.
This Technical Bulletin is intended to provide consistency across States about which items rated as Area Needing Improvement (ANI) must contain a quantifiable measure in the PIP. We will continue to emphasize that any ANIs for the Safety Outcomes not in substantial conformity include quantifiable measurement in the PIP. We will also prioritize and require measurement for ANIs in Permanency 1 (items 5,6,7,8,9 and 10) and Well Being 1 (items 17,18,19 and 20) when those outcomes are not in substantial conformity. We will not require items that are measured by a national standard indicator in a PIP (items 2, 5, 6, 8, and 9) to have additional specific measures for the items. Situations where an item was an ANI and the associated national standard indicator was met and therefore not included or measured in the PIP may require additional measurement based on individual State findings. We may negotiate more measures based on the relevancy of the item to primary strategies and broader goals implemented by a State.
B1. Development of baselines and review periods for measures other than the national standard data indicators
We will negotiate baselines from the most current and reliable data source that will measure improvement based on information that a State can provide and replicate. We will not recommend use of CFSR onsite measurement findings percentages as baselines because of the inherent differences between a State's Quality Assurance (QA) review process and the actual CFSR onsite review, i.e., it is unlikely that all of the variables that contributed to a baseline from the onsite review findings can be replicated by States with their own QA systems. States will be asked to identify the sources of their baseline data (with associated timeframe), including any review instruments used, the location of cases included in the baselines, and the number of applicable cases by item.
We recommend that States use 12 months of data/findings that represents a time period immediately preceding PIP implementation and includes both foster care and in-home services cases approximating the proportion of cases reviewed during the onsite review to establish baselines. We recognize that some states may require a plan that would establish the baseline during PIP implementation once 12 months of data/findings are available. In cases where baselines are developed prospectively, we will reduce the minimum agreed-upon amount of improvement (AAI) to compensate for baseline time development during the remainder of PIP implementation (see method 1). Along with 12-month baselines, we recommend States use 12-month periods based on rolling quarter end of data or findings encompassing 12 months to determine whether the State has met its improvement goal. The reasons for this recommendation are:
We further recommend that States using instruments and case samples to measure and monitor practice use a minimum 12-month review period that is adjusted at each quarter end. The period should include practice during the 12 months up to the date the monitoring review was conducted and should include both foster care and in-home services cases approximating the proportion of cases reviewed during the onsite reviews.
Because we always include its largest metropolitan area when we evaluate a State during the onsite review, States must include this area in all 12-month data periods used for baselines and measuring goal achievement. States that utilize 12-month review periods and are able to incorporate data/findings that represent their largest metropolitan areas and reasonable cross sections may use measurement baselines and performance periods collected from a period of less than 12 months.
B2. Use of State data collected as a result of ACYF-CB-PI-07-08
Since issuance of ACYF-CB-PI-07-08 regarding the title IV-B requirement for States to collect data on the frequency of worker visits, the capacity of States to provide such aggregate data has significantly increased. Some States have proposed or inquired about utilizing the data collected for the title IV-B requirement for PIP measurement of items 19 or 20. The worker visitation data requirements and definitions in ACYF-CB-I-07-08 are based on the scope of title IV-B authority and State capacity to collect the data, and so differ significantly from the CFSR instrument definitions for items 19 and 20. The most significant areas of difference are lack of a qualitative consideration of the worker contact, in-home cases inclusion, and a State-defined worker responsible for visits who has case decision authority. Based on the above differences, the ACYF-CB-PI-07-08 data do not adequately represent a direct measure for items 19 or 20, and will, therefore, not be accepted for measurement of that item unless a State is able to sort or supplement its data to directly address the measurement that contributed to an ANI finding for the item.
B3. Use of data collected from national standard composite individual measures
In 71 FR 32969-32987 (June 7, 2006) and 72 FR 2881-2890 (January 23, 2007), we transmitted revised national standards for the six statewide data indicators used for the second round of CFSRs. With issuance of the national standard composites and related individual measures, the availability of the 15 individual measures is now a source of measurement that is generated consistently across States with clearly defined Adoption and Foster Care Analysis and Reporting Systems (AFCARS) syntax. If a State does not meet a national standard measure, it must specifically establish a goal of improvement for the standard in its PIP. The State and CB will use the guidance in ACYF-CB-IM-07-05 to establish the AAI for unmet national standards. A State may use one or more of the 15 individual measures, but not the components, associated with the 4 permanency composites to gauge progress on a particular item in its PIP. Composite national standard goals will be measured separately from goals that can be tied to the individual measures. CB will not discontinue monitoring the individual measure AAI if the related composite level AAI is met or vice versa. To utilize individual measures as a measurement strategy in the PIP, they must reasonably match the PIP strategies the State plans to implement. If the State decides to use an individual measure as part of its measurement strategy, the State, CB's RO, and CB's Central Office (CO) CFSR and Data Units will work together to establish a reasonable amount of improvement consistent with methods outlined in the technical information section below.
This section applies to PIPs resulting from the second round of CFSRs. In particular, it pertains only to evaluating States' progress toward meeting the PIP goals not related to the national data standards. In approving States' PIPs, we require that the AAI regarding an item be achieved by the end of the PIP implementation period or, if stated specifically in a State's PIP, by an earlier date. In the latter case, as well as when States meet targets before the end of the PIP implementation period, if CB verifies that States have achieved the AAI specified in their PIPs regarding any item, we will consider the State to have satisfied that requirement. States and CB will determine what data or evidence will be submitted to CB for verification of goal attainment on a case-by-case basis. At the conclusion of States' PIPs (PIP implementation closeout), if States have not attained their AAI with regard to any of the items, CB will determine AAI attainment by using an additional non-overlapping period of time after PIP implementation as originally specified in CFSR Technical Bulletin #1. This will provide States with the full 2-year PIP period to implement all required program improvements, followed by an evaluative period after PIP implementation to determine whether they have met their degree of improvement goals for any item-specific measure.
In CFSR Technical Bulletin #1 we provided technical information on evaluating States' attainment of goals regarding data indicators in program improvement plans from the initial round of CFSRs. We are now providing that information for States with subsequent CFSRs in Federal fiscal years (FY) 2007 through 2010. We are applying the same logic in guiding attainment of goals regarding the data indicators we used during the first round of reviews. This section applies to PIPs resulting from the second round of CFSRs. In particular, it only pertains to evaluating State progress toward meeting the PIP goals related to the national data standards included in the CFSR. There are two situations where determinations of progress toward attaining these PIP goals may be addressed. The first situation occurs when States meet their PIP targets prior to PIP closeout, or have target dates specified in the PIP that occur prior to the end of the PIP. The second occurs at the time the State's 2-year PIP implementation period ends.
States should use the most accurate and current AFCARS and National Child Abuse and Neglect Data System (NCANDS) data as a basis for establishing baselines for implementing their PIPs. The last available AFCARS reporting period ending prior to the PIP approval will be utilized. In order to have an NCANDS file period that coincides, States will be asked to make an additional NCANDS submission voluntarily for the 12 month period as outlined in CFSR Technical Bulletin # 1. In approving States' PIPs, we require that the AAI regarding the national standards be achieved by the end of the PIP implementation period or, if stated specifically in a State's PIP, at an earlier date. In the latter case, as well as when the State meets a target goal before the end of the PIP implementation period, if CB verifies that States did achieve the AAI specified in their PIPs with regard to any of the six data indicators, we will consider the State to have satisfied that requirement. States and CB will determine what data will be submitted to CB for verification on a case-by-case basis.
At the closeout of States' PIPs, if States have not attained their AAI with regard to any of the applicable data indicators, CB will determine AAI attainment by using the data covered by the same 12-month period described in section I of CFSR Technical Bulletin #1 that CB uses to begin the subsequent CFSR. This provision will provide States with the full 2-year PIP period to implement all required program improvements, followed by a 1-year evaluative period with non-overlapping data following PIP closure. However, States may also submit two consecutive 6-month periods of AFCARS data, four consecutive quarters (12 months) of NCANDS data, or other data approved by CB as evidence that the State has met the requirement for a particular data indicator earlier than the end of this non-overlapping 12-month period, including during the 2-year PIP implementation period and through the 12-month non-overlapping period following the end of the PIP. CB will accept such data submissions as evidence of a State having met the requirement for attaining the AAI regarding a particular data indicator, provided the data are verified or reproduced by CB. In such a situation, where the State can demonstrate that it has met its AAI, there will be no need to wait for the conclusion of a 12-month period of non-overlapping data to determine that the State has met its goal.
CB expects that any concerns that a State or CB has regarding the quality and accuracy of States' data will be identified and resolved prior to the end of the non-overlapping 12-month period, as States will have up to a full year of non-overlapping data following completion of the PIP to determine whether they have met their goals. This will necessitate States' resubmitting their data no later than the end of this period because CB will use the data to make the final determination of States' compliance with the provisions of their PIPs, rather than delay the determination further.
If a State's PIP requires that it attain the AAI prior to the end of the 2-year PIP implementation period, rather than at the end of the PIP period, CB will use the same principles outlined in this Technical Bulletin to provide the opportunity, as needed, for a year of non-overlapping data following the required completion date in the PIP to determine whether the State achieved its goal for that particular indicator.
Thus, CB will use an entire 12-month period of data covering four quarters to determine whether States did attain the AAI for their remaining data indicators. These periods of data will be used both for monitoring improvement over the course of the PIP and, at the conclusion of the PIP implementation period, for closeout purposes, unless a previously approved PIP includes a different timeframe for measurement. The rationale for this 12-month period is outlined in section B1 of this Technical Bulletin.
In evaluating improvement for the four permanency data indicators (PDI) in the CFSR, both during the PIP implementation period and at the conclusion of the period, if applicable, CB will use AFCARS data that represent 12 consecutive months covering four quarters, or will verify State-generated data replicating AFCARS submissions representing 12 consecutive months covering four quarters. The 12-month data period will correspond to two consecutive AFCARS submissions unless a State's PIP specifies evaluating improvement on a quarter-end basis, in which case CB will use data from the quarter that the evaluation is made and the preceding three quarters.
There are two applicable processes for evaluating improvement in the two safety data indicators (SDI). The first process concerns AAI verification of the SDIs that can be computed from States' case-specific NCANDS Child Files (CFs). Depending on the quarter that AAI achievement was attained, States will voluntarily submit these files for the 12-month periods ending December 31, March 31, June 30, or September 30 on April 1, July 1, October 1, or January 1, respectively, allowing approximately 90 days after the end of a quarter to transpire for data completeness.
The second process concerns PIP closure. To be in concert with the procedures for PIP closure related to the four PDI, States whose SDIs were computed from their case-specific NCANDS CFs and did not achieve their AAIs during the PIP implementation period will be evaluated for the 12-month period of non-overlapping data following completion of the PIP, ending on March 31 or September 30. As a result, States will voluntarily submit their NCANDS CFs on July 1 or January 1, respectively, allowing about 90 days to transpire. Again, if a State's PIP specifies evaluating improvement on a quarterly basis, CB will use data from the quarter end that the evaluation is made and the preceding three quarters. For monitoring quarterly improvement in SDIs during and after the PIP implementation period, States are encouraged to generate SDIs on a quarter-end basis. Technical information on generating these data is in section C below.
In calculating the maltreatment in foster care SDI, CB will be able to generate corresponding AFCARS data for the denominator based on States' AFCARS submissions.
NOTE: The information in this Technical Bulletin regarding States generating their own safety data, as well as the expedited timeframes for submitting the data, pertains only to States and situations where CB will need to use the data either to determine improvement toward meeting the goals of the PIP or in preparing data profiles for the next CFSR. In all other situations, the established timeframes and procedures for submitting NCANDS data will continue to apply. In addition, in situations where States make early submissions for the next CFSR, or calculate the data on an interim basis for evaluating improvement in the PIP, CB will continue to expect States to make their regularly scheduled NCANDS submissions. We will accept re-submissions of corrected or more complete data for use in the CFSR, as we have in the past, if a State believes the data it submits according to the expedited timeframes in this Technical Bulletin are inaccurate, although we expect States to resolve data quality concerns prior to the end of the 12-month non-overlapping period following PIP completion in order to not delay the final closeout of the PIP.
In evaluating improvement for the four PDCs with national standards, CB will use States' AFCARS data. CB will verify the indicators before concurring that a State did satisfy its AAI if any State asserts that it has achieved the AAI based on its generated PDC using the Federally approved syntax.
CB may determine that States have met the AAI relative to a national standard during the implementation period of the PIP. In doing so, CB will construct an annual file based on rolling periods, corresponding to 6-month AFCARS submissions or FY quarters. This approach will result in using either two consecutive AFCARS submissions or a combination of quarters contained in three AFCARS submissions if the amount of improvement to be determined has occurred at the end of an intervening Federal quarter (quarter ending December 31 or June 30).
If States' PIPs require evaluating improvement on a quarterly basis, unless otherwise specified in the PIP, we will ask States to calculate their applicable PDCs using an annual file based on rolling quarter ends (12 months). However, CB data staff will not be able to generate a State's PDC in-house for comparison to other States until the State makes its regular AFCARS submissions.
If, at the conclusion of the PIPs, States have not demonstrated AAI attainment regarding their applicable PDCs included in their PIPs, CB will make a final determination of the status of their PDCs as compared to their PIP goals (closeout PIP) by using the data covered by the same non-overlapping 12-month period of time described in section I of Technical Bulletin #1 that CB uses to begin the subsequent CFSR. This means, for example, for States that have PIPs closing during the months of October 2009 through March 2010, CB will use the 2010B (April 2010–September 2010) and 2011A (October 2010–March 2011) AFCARS data. For States with PIPs closing during the months of April 2010 through September 2010, CB will use 2011A (October 2010–March 2011) and 2011B (April 2011–September 2011) AFCARS data.
States will also be allowed to demonstrate AAI attainment of PDCs for 12-month periods that end quarterly between the conclusion of the PIP and the end of the non-overlapping 12-month period. In those situations, States must generate their approved data on a quarter-end basis.
In monitoring improvement for the two SDIs with national standards, States will produce the data needed to monitor their performance on the two safety indicators used in the CFSR (not an entire safety profile). States should use the same definitions and methodology for computing the indicators that CB uses to compile the initial data profiles for the CFSR, i.e., definitions included in the NCANDS CF and AFCARS for the relevant data elements and Federally approved syntax. This is necessary because CB will use these at PIP closeout or when the AAI is met prior to PIP closeout. If States' PIPs require evaluating improvement on a quarterly basis, unless otherwise specified in the PIP, we will ask States to calculate the SDIs using a rolling four-quarter end basis (12 months) of NCANDS data, following the same general timeframes (90 days after the end of the rolling four-quarter period) and procedures outlined in CFSR Technical Bulletin #1 pertaining to producing profiles for the second round of the CFSR. However, States should submit the results according to the procedures associated with the second round of reviews if they determine that they have satisfied their AAIs in an intervening quarter.
CB will use NCANDS data that correspond to the AFCARS time periods described above if, at the conclusion of their PIPs, States have not already demonstrated they have attained their AAIs regarding the two SDIs included in their PIPs. This means, for example, that States with PIPs ending during the months of October 2009 through March 2010 will use a successive four-quarter period of CF data ending on March 31, 2011. States with PIPs ending during the months of April 2010 through September 2010 must produce the SDIs using the same general time frames and procedures for submitting and validating their data described in the Technical Bulletin #1 pertaining to producing profiles for the second round of the CFSR.
States will also be allowed to demonstrate AAI attainment regarding SDIs for 12-month periods that end quarterly between the conclusion of the PIP and the end of the non-overlapping 12-month period. In those situations, States must generate their data on a quarter-end basis.
In all situations, States will use the alternate data source methodology approved by CB in the PIP if they do not submit their NCANDS CFs or if CF data preclude their calculating any SDI. States will ensure that the quality of the data and accuracy of the computations are consistent with the definitions of corresponding NCANDS data elements and of the national standard indicators. In addition, States will provide a plan for CB's approval that addresses, at a minimum, sample sizes, if applicable; databases; and methodology used to compute the data indicators, using the alternative source data for evaluating the PIP. This plan also should indicate the quality of the databases used and completed CB-generated templates designed for this purpose. This will allow CB to recalculate States' measurements upon plan approval, and States can then submit the actual data from their approved alternate sources to CB.
The chart below provides an example of how we will evaluate improvement toward achieving the goals associated with data indicators and national standards for States whose PIPs ended in December 2008:
December 31, 2008 |
April 1, 2009 |
March 31, 2010 |
July 1, 2010 |
Post July 1, 2010 |
Last date for State to complete PIP activities. Goals for two data indicators in PIP have not yet been met, including one permanency indicator and one safety indicator. |
Begin date for 12-month data cycle for use in determining whether State met its goals. (AFCARS and NCANDS) |
End date for 12-month data cycle for determining whether State met its goals. (AFCARS and NCANDS) |
End of 90-day period for State to voluntarily submit NCANDS CF or alternative datasource SDIs with proper documentation. |
ACF determines whether State met its PIP goals based on the 12-month non-overlapping data. |
CB encourages use of State-generated data from their own QA systems or management information systems (MIS) for PIP monitoring and measurement. If a State wishes to propose a measurement method other than those outlined below, it must forward to CB's RO, and CO CFSR and Data Units appropriate documentation, including a description of the methodology employed. Documentation must include the baseline sample source or universe and size, the review period and locations, and any instruments or reports to be used and goal measurement methodology description. Any change to review instruments, reports or sampling after the approval of PIP measurement methodology requires notice to CB. CB's RO, and CO CFSR and Data Units will review the measurement proposal and respond accordingly. As needed, ACF will provide technical assistance to States in this effort through the National Resource Center for Child Welfare Data and Technology (NRC-CWDT) or the National Center for Organizational Improvement (NRCOI).
The sections below provide specific measurement methodology that has been preapproved by CB and may be utilized to set and evaluate degree of improvement toward achieving the goals associated with item-specific measurement. The first two methods address situations where State QA baseline data are available during PIP negotiation (retrospective data method) and where QA data will be collected during the PIP implementation period (prospective data method). In the retrospective method (method 1), a process for determining the baseline and target goals from existing data is outlined; and for the prospective method (method 2), a process is outlined to develop a baseline with a minimum case sample prior to setting the goal of improvement. We recommend an 80 percent confidence level for both methods because we believe it will allow more flexibility in demonstrating improvement with somewhat smaller and less labor-intensive case samples than a 90 or 95 percent level. Rather than define a formula that would compute the minimum number of cases a State must review for its baseline or measurement period, we recommend that State samples be equal to or greater than the number of applicable cases for the CFSR item from the State onsite review. Once a baseline sample size is established the monitoring measurement sample size must be comparable. The number of applicable cases used for a baseline would be the minimum required for ongoing monitoring measurement to determine if goals are met.
A third method provides specific technical detail concerning improvement factors that can be used if States utilize the National Standard Composite individual measures to set minimal improvement. A fourth method provides information on methodology for using State MIS data or other aggregate data that measure a universe larger than a State QA sample review approach.
This method utilizes the available State QA percentage findings from collected data and tests whether the quarterly performance exceeds the original baseline proportion plus the sampling error. States would use 12 months of data/findings beginning no earlier than the first quarter of the AFCARS submission used for sample selection purposes for the onsite review to establish a baseline subject to a minimum sample. The minimum sample for a given item should be equal to or greater than the applicable cases for the item from a State's onsite review. The actual percentage satisfying the given item is computed from the State's 12 month QA sample, and that sample size would be used to compute the actual sampling error using an 80 percent confidence level. If the actual applicable case sample was greater than or equal to the minimum number of applicable cases reviewed during the onsite CFSR, the computed percentage would be determined as the baseline. The goal of improvement would be set by adding the sampling error to the baseline percentage. Larger samples would result in lower improvement goals because of the smaller sampling error. Once a baseline is established the sample size must remain comparable through the monitoring and measurement period. The State would use percentages computed from 12 months of data/findings to determine whether the State satisfied its improvement goal. Table A provides an example of how method 1 may be applied to State retrospective data. If the State's 12 month sample results do not reach the minimum applicable cases, the State could increase its sample size in the next quarter to achieve the threshold. CB's RO, and CO CFSR Unit and Data Unit statisticians will help States compute the sampling error and improvement goal using the process outlined in the examples below.
Table A
| CFSR Item # | CFSR Onsite Proportion Strengths |
Applicable Cases From Onsite Review (Minimum Sample Size) |
Actual Number of Applicable Item Cases Over Baseline Year |
State Baseline Year Proportion |
Baseline Year Actual Sampling Error |
12-Month Goal % (Baseline + Sampling Error) |
17 |
0.52 |
59 |
74 |
0.60 |
0.0729 |
67.29% |
4 |
0.79 |
28 |
26 |
0.72 |
0.0909 |
Sample not sufficient |
We recognize that States might require a plan that would establish the baseline during PIP implementation once 12 months of data/findings are available. This method establishes a baseline from a minimum sample as defined by the number of applicable cases from the onsite CFSR. States would use 12 months of data/findings beginning after PIP implementation. Because the baseline would be established during the period of PIP implementation, the improvement target would be reduced by up to half of the sampling error. The minimum sample size as determined by the applicable cases for the CFSR item from the State onsite review would be required for the 12 month baseline. States would then use rolling or moving quarters of data/findings encompassing 12 months to determine whether they have met their improvement goals. The goal would be set by adding the sampling error, using an 80 percent confidence level, to the baseline percentage. If the new quarter overlaps PIP implementation, the sampling error used to determine the target may be reduced accordingly. It would be reduced by one-eighth for each overlapping quarter but not exceeding four quarters or half; so if there was one overlapping quarter, we would take seven-eighths of the sampling error and add it to the baseline to obtain the targeted amount of improvement. Table B provides an example of how method 2 may be applied to State prospective data.
Table B
CFSR Item # |
State Baseline Year Proportion |
Number of Item Applicable Cases During Baseline Year |
Baseline Year Actual Sampling Error (BYASE) |
Number of Quarters During PIP Implementation for Baseline Development |
12-Month Goal % |
17 |
0.60 |
74 |
0.0729 |
4 |
0.60 + 0.5 x BYASE = 63.65% |
4 |
0.72 |
50 |
0.0813 |
2 |
0.72 + 0.75 x BYASE = 78.10% |
States are encouraged to use relevant individual measures associated with the national standard composites. For example, items 5, 6, 8, and 9 are closely related to several of the individual measures. Several other items have less direct relationships and might be relevant for measurement using some of the 15 individual measures. Minimal amount of improvement for individual measures associated with the national standard composites will be based on the national sampling error for each of the 15 individual measures adjusted for the level of an individual State's baseline year performance. In actual practice, States will multiply their baseline performance by an improvement factor to obtain their PIP goal for an individual measure. Improvement is proportional to a State's baseline performance to make the process as equitable and individualized as possible. The process we used to develop this improvement factor is outlined below:
The following steps were used for each of the 15 measures comprising the composites. Each of the 15 measures of the county results were combined by weighting each county's measure by the children served to obtain a weighted average that became a State's value for that particular measure. We then calculated the average performance of the five highest performing States for each individual measure (source file FY 2004). We then added or subtracted the average performance of those five States to or from the national sampling error for each measure. We then calculated the percentage change between the two values (the original baseline value and the value after the sampling error was added or subtracted). Our improvement factor for determining a State's minimum PIP improvement regarding the individual measures is indicated for each of the 15 individual measures in table C. A State's baseline score is multiplied by the improvement factor to set the target for the amount of improvement to be obtained.
Table C
Individual Measure |
|
C1 – Measure 1 |
1.026 |
C1 – Measure 2* |
0.956 |
C1 – Measure 3 |
1.039 |
C1 – Measure 4* |
0.955 |
C2– Measure 1 |
1.049 |
C2 – Measure 2* |
0.969 |
C2 – Measure 3 |
1.032 |
C2 – Measure 4 |
1.057 |
C2 – Measure 5 |
1.043 |
C3 – Measure 1 |
1.033 |
C3 – Measure 2 |
1.009 |
C3 – Measure 3* |
0.965 |
C4 – Measure 1 |
1.015 |
C4 – Measure 2 |
1.028 |
C4 – Measure 3 |
1.041 |
* Indicates measures where a low value reflects stronger performance (improvement factor is a deflation value less than 1) Back
Method 4 – Use of State data collected from the Statewide Automated Child Welfare Information System (SACWIS) or Other MIS
For the second round of CFSRs, we recommend that a minimal amount of improvement for item measures derived from a State's SACWIS or MIS reporting be based on the sampling error, at a 95 percent confidence interval. This interval is recommended because statewide universe data are used and a lower confidence level would yield very minimal improvement goals. The minimum improvement amount will be computed by adding the sampling error to the 12 months of data or a weighted proportion for a 12-month period using quarterly data reports. Reports proposed by the State under this method should include design syntax and/or extraction methodology that must be approved prior to inclusion of the measurement in the PIP. Table D provides an example of data from four quarters with the weighted sampling error added to obtain a minimum improvement target percentage.
Table D
Report Period Quarter |
(N) |
(P) |
Weighted Proportion |
Sampling Error at 0.95 Confidence Level |
Minimum Improvement Goal |
1 |
12,500 |
0.72 |
9,000 |
0.0078 |
|
2 |
15,000 |
0.77 |
11,550 |
0.0067 |
|
3 |
13,500 |
0.71 |
9,585 |
0.0077 |
|
4 |
14,000 |
0.68 |
9,520 |
0.0077 |
|
12 Months |
55,000 |
|
39,655 |
|
|
Example 12-Mnth Minimum Goal |
|
|
0.721 |
0.0075 |
72.85 = Improvement Goal |
Back to Legislation, Policy, and Technical Bulletins Related to the CFSRs
