PI-82-06 Attachment E

ATTACHMENT E

CASE RECORD SAMPLE SURVEY
FOR
SEC. 427 of the SOCIAL SECURITY ACT (SSA)
ELIGIBILITY DETERMINATION

Introduction

The case records are used as the source documents to assess actual service delivery implementation of the Sec. 427 protections for the child and family. The case record may be a single file folder or an aggregate of many case-specific files (child, family, foster parent) that provide the necessary information for assessing the state's eligibility status.

A statistically valid probability sample of case records is assessed using the federal guidance material. There are various alternatives available for the Case Record Sample Survey (CRSS). These relate to the agency which conducts the survey as well as the possible sampling strategies.

Standards for Eligibility

The case record is rated "Acceptable" or "Unacceptable" depending upon the documentation which substantiates implementation of the foster care protections specified in Sec. 427. States which have 80% or more "Acceptable" records will have achieved substantial compliance and will not be subjected to further review for at least two years. States which have 65% or less "Acceptable" records will not be eligible for additional funding as described in Sec. 427.

Provisional eligibility is established when between 66% and 79% of the state's case records are rated "Acceptable" though this will necessitate a federal review in the following fiscal year to determine whether the state has achieved substantial eligibility performance, 80% or greater "Acceptable" case records. States failing to achieve this level of performance will not be considered eligible for additional federal funding in the succeeding fiscal year. Provisional eligibility, between 66% and 79% "Acceptable" case records, is only possible for the initial year in which the state certifies that it is meeting the protections of Sec. 427.

Conduct of the Sample Survey

The state may select one of three approaches to the conduct of the CRSS based on the federal material, The Case Record Survey and Guidelines for Sec. 427 Case Record Survey:

The state may assume responsibility for the sample design and conduct of the CRSS. The sample design must receive a prior federal approval. A post survey validation audit of a subsample of the survey sample will be conducted by a federal team.
A combined team of state and federal staff under the responsibility of the Regional Program Director will conduct the survey using a sequential sampling plan.
The Regional Program Director will conduct the sample survey with a team of federal staff and staff from other states, if available and acceptable to the state being reviewed. The sequential sampling plan will be used in the survey.

Sampling Inspection Design

The statistical problem is that of making a correct inference about a state's entire file of case records from a sample of those records. Whenever a sample is used there is a risk of making one of two types of errors. Based upon the findings from the sample a state may be considered ineligible when in fact a review of all its case records would indicate that it was eligible. This type of error would have serious consequences for a state as it would deny them additional funds though they were actually providing the protections of Sec. 427. The second type of possible error occurs when a state whose entire case record file was reviewer was considered ineligible but on the basis of a sample review was judged eligible. This is a violation of the legislation and would mean that an opportunity to assist a state in protecting its children in foster care would be lost. There are two approaches:

In standard inspection (review) a sample of predetermined size is drawn from the state files in random fashion. Based on a complete review of all case records in this sample the state is considered eligible, provisionally eligible, or ineligible using the standards for eligibility previously discussed.
In sequential sampling inspection (review) the size of the sample is left undetermined and the procedure is to review one or several case records at a time with the review continuing until the cumulative evidence is sufficiently strong for the state to be considered eligible, provisionally eligible, or ineligible. To minimize the the probable occurrence of either type of error in states with 1,000 or more children in foster care the risk of occurrence is one error in one thousand. States with less than 1,000 children in foster care will have a higher risk of a wrong decision: one chance in twenty for the first type of error and one chance in one hundred for the second type of error.

A state may use either a standard or sequential sampling inspection design. Sequential sampling will, on the average, require at least 50% fewer case records to be reviewed than standard sampling at the same levels of risk. In addition, sequential sampling has the further merit of being simpler than other methods in that all statistical calculations can be completed in advance so that all that remains is the cumulative counting of the number of "Unacceptable" case records. A Decision Table is computed in advance indicating the number of "Unacceptable" case records necessary to consider the state eligible, provisionally eligible, or ineligible based on the standards and risk factors previously discussed.

In sequential sampling, a review of between 114 to 128 case records on the average will be required for states with 1,000 or more children in foster care to determine whether the state is eligible or ineligible. For states with less than 1,000 children in foster care it will require a review on the average between 48 to 78 case records to make similar decisions. The greatest number of case records will be reviewed to judge a state as provisionally eligible. There is no limit theoretically on the number of reviews necessary to reach a decision. Consequently, it is necessary to limit the number of reviews necessary to reach a decision to 80 for states with less than 1,000 foster care children or 150 for states with 1,000 or more such children. Accordingly, states which have neither been found to be eligible or ineligible by the 80th (less than 1,000 foster care cases) or the 150th (1,000 or more foster care cases) case record review will be considered provisionally eligible.

Size and Selection of the Case Record Sample

Standard Inspection - The size of the sample will be determined by the specific inspection design the state selects using the 95% level of confidence and 5% allowable range from the true percentage. In general, it will require at least 300 case records to be reviewed.

Sequential Inspection - States with 1,000 or more children in foster care will require a maximum review of 150 case records while states with less than 1,000 children in foster care will require a review of a maximum of 80 case records.

Case Record Sampling Criteria - The state's case records are examined for evidence that the state has implemented and is operating a case review system for each child receiving foster care. The case review system includes the following components:

case plan document,
periodic review no less frequently than once every six months
procedural safeguards including dispositional hearing no later than 18 months after the original placement

Most states follow the Federal legislation and conduct the initial periodic case review six months after placement. For the purposes of the case record review it is necessary to obtain a random sample of children who have been in placement six or more months as of September 30, 1981, the close of FY 1981. Children entering placement April 1, 1981 or later should be excluded.

States with legislation or written policy guidelines or regulations which initiates periodic case reviews prior to six months after placement may include such cases in the case record sample. Thus, all children who have been in foster care for a least one periodic case review by September 30, 1981 may be included in the universe of case records to be sampled, i.e., a state which begins the periodic review at 45 days after placement may include children who entered placement by August 15, 1981 or earlier.

As a very small sample of case records is inspected it is necessary to use a stratified random sample to provide evidence that the case plan, the periodic case review, and the procedural safeguards including the dispositional hearing have been implemented and are operating in accordance with the legislation.

The case records for children in foster care which are to be sampled are divided into two groups:

Case records for children whose original placement occurred prior to April 1, 1980. This group of children would have been in placement 18 months or longer by the close of FY 1981, September 30, 1981. A total of 90 such case records should be randomly selected. An additional six records should be selected for sample attrition.
Case records for children who have been in placement for less than 18 months but long enough for at least one periodic case review due by September 30, 1981. For example, a state which begins its review 45 days after placement would include all case records of children entering placement from April 1, 1980 through August 15, 1981. A total of 60 such case records should be randomly selected. An additional four records should be selected for sample attrition.

The two sub-samples (I and II) are combined into a single sample with a random assignment of reading sequence using the table of random numbers.

Random Selection - The selection of the random sample is facilitated by making available a listing by case number of all the children in foster care who meet the above selection parameters.

The listing should be submitted to the Regional Office where the following random selection will be made:

160 children will be selected for state with 1,000 or more children in foster care, or
85 children will be selected for states with less than 1,000 children in foster care.

There is an oversampling of 10 and five children respectively to allow for sampling attrition due to unforseen circumstances. Some states have computerized systems which will automatically select a random sample. Any procedure is acceptable that produces a valid random sample based on the above discussed parameters.

Review Sites

The least burdensome and most efficient review procedure is achieved through the assembly of the entire random sample of case records at a single site. This may not be possible in some states and it will be necessary to use two or three sites. This affects the deployment of reviewers and no less than two reviewers should be assigned to one site. This is necessary to minimize idiosyncratic reviewer bias.

States which will conduct their own assessment may use as many sites as they deem necessary. However, for the post validation review it will be necessary to assemble the subsample records at one to three sites.

Post Survey validation Audit

States electing to conduct their own Case Record Survey will receive a post survey validation audit of at least 50% or 40 of the same case records reviewed by the state, whichever is higher. To reduce the burden for the state, the validation audit will be scheduled for the day following the completion of the state review so that a sub-sample of the records reviewed by the state can readily be selected at the review sites. This will minimize the time that the records are kept at the selected sites as those records not included in the subsample may be returned immediately to agency files.

The selection of the subsample of reviewed case records is based on a stratified random sample procedures:

Select the first 15 cases reviewed.
Select the final 10 cases reviewed.
Select randomly the number of cases from the balance of the reviewed case records to complete the sample size of 40 case records or 50% of the total case records reviewed, whichever is higher, e.g. for a subsample of 40 case records select the first 15 records and last 10 records reviewed. Add an additional 15 records randomly selected from the balance of the reviewed records. For a possible subsample of 125 case records select the first 15 records and the final 10 records and 100 additional records randomly selected from the balance.

At the completion of the federal review the state will be notified of all disagreements, if any, and an opportunity provided to resolve all differences. The results from the State Survey and the Validation Audit will be forwarded to the Regional Program Director who will determine the state's eligibility status using the standards previously discussed. This recommendation will be forwarded to the central office.

Decision Tables

There are two Decision Tables for sequential sampling: Decision Table I is to be used with states with 1,000 or more children in foster care; and Decision Table II is to be used with states with less than 1,000 children in foster care as of the end of the fiscal year for which additional funds are claimed. The two tables are used in the same manner, the only difference is the number of case records to be reviewed. The instructions for using Decision Table I should be used for Decision Table II as well. All that changes is the cut-off in the reading of case records--150 case records for Table I and 80 records for Table II.

Decision Tables I and II contain seven columns:

Col. 1.	Case Record I.D. - The review supervisor enters the case record I.D. of every case that is reviewed in the sequence in which the review is completed, case by case. This should follow the random selection sequence allowing for deletion of case records that do not meet selection criteria.
2.	Number of Records Reviewed - This is a sequential listing of the cumulative number of records reviewed beginning with 1 and ending with 150 (or 80 for Decision Table II) for the last record that may be reviewed, e.g. the number 54 would indicate that a total of 54 records have been reviewed.
3.	Acceptance Number - This column represents the maximum number of "Unacceptable" case records that are necessary to make the decision that the State is eligible. No decision to accept a State can be made before at least 34 (or 22 for Decision Table II) case records have been reviewed. At that point in the review, there cannot be any "Unacceptable" case records. The total number of "Unacceptable" case records cannot exceed 31 (or 15 for Decision Table II) for the State to be eligible
4.	Unacceptable: Actual/Cumulative - This represents the actual and the cumulative total of "Unacceptable" case records. The review supervisor enters the actual and cumulative total number of "Unacceptable" case records in this column as each record review is completed and the results from the Reviewer's Tally Sheet are transferred to the Decision Table. Enter "O" for an "Acceptable" case record and "1" for an "Unacceptable" case record. This cumulative total of "Unacceptable" case records is compared, line by line, with the corresponding figures in columns 3 and 5. Whenever the number in this column equals the number of either column 3 or 5 the review process stops and a decision is made. If the cumulative number in column 4 equals the number in column 3 the decision is made to consider the state eligible. If the cumulative number in column 4 is between the numbers in columns 3 and 5 the review process continues until the 150th case record is reviewed. If, at that point, the number of rejected case records is more than 31 (or 15 for Decision Table II) but less than 50 (or 26 for Decision Table II) the review process is terminated and a decision of provisional eligibility is made. If the cumulative number in column 4 equals the number in column 5 the decision is made that the state is ineligible.
5.	Rejection Number - This column represents the minimum number of "Unacceptable" case records that are necessary to make the decision that the state is ineligible.
6.	Reviewer's Initial - The reviewer initials the entry made by the supervisor for each completed case record review.
7.	Supervisor's Initials - The review supervisor initials the entry made for each completed case record review.

Procedures

The procedures for conducting the review of the case records for sequential sampling are essentially similar for all states and other political jurisdictions.

Respectively, the procedure is as follows:

The state is asked to submit a listing of all children who have been in foster care by September 30th of the fiscal year in which additional monies were claimed. At the time of selection of the random sample the child's case may still be active or may have been closed subsequent to September 30th. One of the following two restrictions on the state's listing of children in foster care should also be observed:
- States which have legislation or written policy guidelines or regulations to initiate periodic reviews before six months, e.g., 45 days, should include in the listing all children who have been in foster care for at least 45 days or any other earlier period by the close of the fiscal year.
- States which initiate the periodic review at six months should include in the listing all children who have been in foster care for at least six months by the close of the fiscal year.
The Children's Bureau will select a random sample of 160 or 85 case numbers using a Table of Random Digits (See Appendix A). There is an over-sampling of 10 and 5 cases respectively to substitute for any case record that is inappropriately included. A stratified random sample will be selected as described on p.4.
The listing of 160 or 85 case numbers will be sent to the state in advance of the review for the state to assemble the 160 or 85 case records at a designated site(s) for review.
The Regional Office Eligibility Determination Reviewers will each begin reviewing the case records using the instruction manual and criteria. The reviewer(s) will lead the records in the sequence of case numbers indicated on the Reviewer's Tally Sheet supplied by the Children's Bureau. After each record is reviewed the reviewer indicates the review outcome (Acceptable = 0 or Unacceptable =1) and initials the line.
The Reviewer's Tally Sheet is brought to the Review Supervisor who enters the data in the Decision Table and then initials the Reviewer's Tally Sheet on the same line. The reviewer initials the Decision Table.
The reviewer continues to read records until the Review Supervisor terminates further review or until the 150th or 80th case record is reviewed.

When the case record review is completed the Decision Table and the review's Tally Sheets are assembled with all the other documentation the Eligibility Determination Study.

Confidentiality

Reviewers reading case records are prohibited from discussing any of the material with anyone other than the review supervisor. No effort should be made for clarification from the caseworker or line supervisor. Questions of documentation may be discussed with the State's Resource Advisor as designated. Nothing should be entered in the case records nor should any of the review material be left in the office after the review is completed.

Technical Note: Statistical Computation

The sequential probability ratio test* was used for the binomial distribution when the result of a single case record review is a classification of accept or reject and when the result of the test for the state is acceptable certification for meeting the eligibility criteria or a finding of unacceptable certification when the performance indicated the criteria have not been met. The four quantities that completely determine the sequential inspection plan for states with 1,000 or more children in foster care are:

(a) P1 = .20 the "acceptable" quality limit for the state, expressed as a fraction of unacceptable case records,

(b) P2 = .35 the "unacceptable" quality limit for the state expressed as a fraction of unacceptable records,

(c) alpha = .001 the maximum risk of rejecting certification of states of quality P1 or better, expressed as a decimal fraction, and

(d) beta = .001 the maximum risk of accepting certification of states of quality P2 or worse, expressed as a decimal fraction.

Using these four quantities the formulas for the Acceptance Numbers (Column 3) and Rejection Numbers (Column 5) of Decision Table I are:

Acceptance Number =-9.0018 + .2706 x number of records reviewed

Rejection Number = 9.0018 + .2706 x number of records reviewed

* Statistical Research Group, Columbia University, Sequential Analysis of Statistical Data: Applications, Columbia University Press, New York, 1945.

Similarly, the four quantities that completely determine the sequential sampling plan for states with less than 1,000 foster care cases are:

(a) P1 = .20

(b) P2 = .35

(d) beta = .01

The formulas for the Acceptance Numbers (Column 3) and Rejection Numbers (Column 5) of Decision Table II are:

Acceptance Number = -5.9352 + .2706 x number of records reviewed

Rejection Number = 3.8913 + .2706 x number of records reviewed

State/Federal Reliability Check

Whenever the state selects the option of having a joint team of state and federal staff review the case records there is a simple procedure to check the reliability of the results.

The following assumptions are made:

The state personnel read approximately as many records as federal personnel, in toto.
Records for review are assigned randomly to state and federal staff.
The review process is reliable when reviewers have been trained.

Under these assumptions the numbers of "Unacceptable" records noted by both groups respectively should be similar except for sampling error. E.g., if each group read 75 records (a total of 150) and the state people found 14 Unacceptable and the federal staff found 19 Unacceptable the difference is probably due to fluctuations in sampling and a valid conclusion can be made that both samples come from the same universe of state case records.

When the difference is much greater there is cause for concern. The size of a significant difference varies with the number of records that the state staff and federal staff have read respectively. The following table indicates the minimum difference that cannot be accounted for through sampling error:

No. of Case Records
Read by Each Team

30 records
40   " 
50   " 
60   " 
75   "

Minimum Difference in
Number of Unacceptable
Records Which Indicates
Possible Unreliability

6 records
8    " 
9    " 
9    " 
10   "

A review in which the state staff found 8 Unacceptable records after reading 40 and the federal staff found 17 Unacceptable records, a difference of 9 records, would be questionable. The federal team would have to review at least 50% of the records reviewed by the state staff to validate the review process.

DECISION TABLE I

View Supervisor	State
Telephone Number	Dates

1	2	3	4	5	6	7
Base Record I.D.	Number Records Reviewed	Acceptance Number	Unacceptable* Actual Cumulative	Rejection Number	Reviewer's Initial	Supervisor Initial
	1	-		-
	2	-		-
	3	-		-
	4	-		-
	5	-		-
	6	-		-
	7	-		-
	8	-		-
	9	-		-
	10	-		-
	11	-		-
	12	-		-
	13	-		13
	14	-		13
	15	-		14
	16	-		14
	17	-		14
	18	-		14
	19	-		15
	20	-		15
	21	-		15
	22	-		15
	23	-		16
	24	-		16
	25	-		16
	26	-		17
	27	-		17
	28	-		17
	29	-		17
	30	-		18
	31	-		18
	32	-		18
	33	-		18
	34	0		19
	35	0		19
	36	0		19
	37	1		20
	38	1		20
	39	1		20
	40	1		20
	41	2		21
	42	2		21
	43	2		21
	44	2		21
	45	3		22
	46	3		22
	47	3		22
	48	3		22
	49	4		23
	50	4		23
	51	4		23
	52	5		24
	53	5		24
	54	5		24
	55	5		24
	56	6		25
	57	6		25
	58	6		25
	59	6		25
	60	7		26
	61	7		26
	62	7		26
	63	8		27
	64	8		27
	65	8		27
	66	8		27
	67	9		28
	68	9		28
	69	9		28
	70	9		28
	71	10		29
	72	10		29
	73	10		29
	74	11		30
	75	11		30
	76	11		30
	76	11		30
	77	11		30
	78	12		31
	79	12		31
	80	12		31
	81	12		31
	82	13		32
	83	13		32
	84	13		32
	85	14		33
	86	14		33
	87	14		33
	88	14		33
	89	15		34
	90	15		34
	91	15		34
	92	15		34
	93	16		35
	94	16		35
	95	16		35
	96	16		35
	97	17		36
	98	17		36
	99	17		36
	100	18		37
	101	18		37
	102	18		37
	103	18		37
	104	19		38
	105	19		38
	106	19		38
	107	19		38
	108	20		39
	109	20		39
	110	20		39
	111	21		40
	112	21		40
	113	21		40
	114	21		40
	115	22		41
	116	22		41
	117	22		41
	118	22		41
	119	23		42
	120	23		42
	121	23		42
	122	24		43
	123	24		43
	124	24		43
	125	24		43
	126	25		44
	127	25		44
	128	25		44
	129	25		44
	130	26		45
	131	26		45
	132	26		45
	133	26		45
	134	27		46
	135	27		46
	136	27		46
	137	28		47
	138	28		47
	139	28		47
	140	28		47
	141	29		48
	142	29		48
	143	29		48
	144	29		48
	145	30		49
	146	30		49
	147	30		49
	148	31		50
	149	31		50
	150	31		50

Date

Review

Supervisor (Signature)

* Accept - 0

Reject - 1

DECISION TABLE II
(Less than 1,000 Foster care Cases)

View Supervisor	State
Telephone Number	Dates

1	2	3	4	5	6	7
Base Record I.D.	Number Records Reviewed	Acceptance Number	Unacceptable* Actual Cumulative	Rejection Number	Reviewer's Initial	Supervisor Initial
	1	-		-
	2	-		-
	3	-		-
	4	-		-
	5	-		-
	6	-		6
	7	-		6
	8	-		7
	9	-		7
	10	-		7
	11	-		7
	12	-		8
	13	-		8
	14	-		8
	15	-		8
	16	-		9
	17	-		9
	18	-		9
	19	-		10
	20	-		10
	21	-		10
	22	0		10
	23	0		11
	24	0		11
	25	0		11
	26	1		11
	27	1		12
	28	1		12
	29	1		12
	30	2		13
	31	2		13
	32	2		13
	33	2		13
	34	3		14
	35	3		14
	36	3		14
	37	4		14
	38	4		15
	39	4		15
	40	4		15
	41	5		15
	42	5		16
	43	5		16
	44	5		16
	45	6		17
	46	6		17
	47	6		17
	48	7		17
	49	7		18
	50	7		18
	51	7		18
	52	8		18
	53	8		19
	54	8		19
	54	8		19
	55	8		19
	56	9		20
	57	9		20
	58	9		20
	59	10		20
	60	10		21
	61	10		21
	62	10		21
	63	11		21
	64	11		22
	65	11		22
	66	11		22
	67	12		23
	68	12		23
	69	12		23
	70	13		23
	71	13		24
	72	13		24
	73	13		24
	74	14		24
	75	14		25
	76	14		25
	77	14		25
	78	15		25
	79	15		26
	80	15		26

* Accept - 0

Reject - 1

REVIEWER'S TALLY SHEET

Reviewer	State
Office Telephone Number	Telephone Number

Case Number or I.D.	Review Outcome Accept = 0 Reject = 1	Initial Reviewer Supervisor
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

Reviewer's Signature

Date

ANNUAL BUDGET REQUEST FOR TITLE IV-B FUNDS

Fiscal Year 19

State of Utah

October 1, 1983 through September 30, 1984

Revision #

I. COMPUTATION OF FEDERAL GRANT AWARD

A) Total Estimated title IV-B expenditures	2,624,612
B) Enter Federal Share (75% of A up to maximum listed in Action Transmittal):	1,555,855

II. REQUEST FOR GRANT AWARD

Indicate the total request for the year and the request for each of the four quarters. (Fund totaling more than the state's share of the $141 million allotment will not be released to the State until there has been verification that the State meets the conditions of P.L. 96-272 for those funds.)

Total (from line I.B)	1st Q (Oct.-Dec.)	2nd Q (Jan.-Mar.)	3rd Q (Apr.-Jun.)	4th Q (Jul.-Sept.)
1,555,855	389,155	388,900	388,900	388,900

III. CERTIFICATION BY STATE AGENCY

The State Agency submits the above estimate and request of grant award under title IV-B of the Social Security Act, as amended, and agrees that the estimated expenditures will be made in accordance with the Child Welfare Services Plan, agreed to by the Agency and the Administration for Children, Youth and Families, for the fiscal year ending , 19 .

(Signature) Administrator, Social Services Agency

Director, Division of Family Services
(Title-please type)

(Signature) Director, Single Organization Unit

Exec. Director, Dept. of Social Services
(Title-Please type)

Other State Official (OPTIONAL)

Signature Title

5-23-83
Date

Date

DO NOT WRITE IN THIS

Regional Office Appr

Regional Program
Director

09-30-83
Date

Attachments:

Attachment A - Report on State Eligibility
Attachment B - State Agency Administrative Review
Attachment C - Case Record Survey
Attachment D - Policy Guidance for Certain Section 427 Requirements