Table of Contents | Previous | Next |
RANDOM ASSIGNMENT AND RELATED ISSUES IN THE EARLY HEAD START EVALUATION: COMMONLY ASKED QUESTIONS AND ANSWERS
Mathematica Policy Research, Inc.
February 21, 1997
Since the beginning of random assignment and program enrollment in June 1996, Mathematica has responded to numerous questions. fu some instances, we and ACYF have clarified procedures, modified approaches, and developed new policies. This document brings together the most important questions that EHS programs and local research teams have been asking. This document includes some questions from a previous Q&A document about random assignment and several new questions that have come up in the past several months. We begin with a review of the key steps in random assignment. The Q&As are grouped under random assignment, maintaining the research sample, and completing the HSFIS application and enrollment forms. If you have any questions about these procedures or how to handle specific situations, contact Diane Paulsell at MPR at (609) 275-2297 (Email: dpaulsell@mathematica-mpr.com).
A. OVERVIEW OF RANDOM ASSIGNMENT PROCEDURES
All programs should be submitting families for random assignment according to the following procedures:
- Determine each family's eligibility for Early Head Start (EHS), and for those who are eligible, complete the full HSFIS application and enrollment forms.
- Within one month of application, transmit the following information to Mathematica Policy Research, Inc. (MPR) and to the local research partner:
- A fax cover sheet listing the names of applicants, verification of three aspects of their eligibility for the research sample, and the subgroup to which they belong (if random assignment subgroups have been identified for the program)
- Pages 1-4 of the HSFIS form for each applicant listed on the fax cover sheet
- A copy of the signature page of the consent form for each applicant listed on the fax cover sheet; this information should be sent to Rosiland Page (phone: 609-897-7413; fax: 609-936-1462; Email: rpage@mathematica-mpr.com).
- Receive lists of families selected for the program and for the comparison group from MPR (usually within 48 hours). (At the request of the Denver program, we send that site only the list of program families.)
- Notify families selected for the program group, enroll them in the program, and begin providing services as soon as possible. (The local research partner will notify families assigned to the comparison group.)
- Send full copies of the HSFIS application and enrollment forms for each applicant submitted for random assignment to MPR within two weeks. MPR will do the data entry until the automated HSFIS is ready for use.
- Local research staff should periodically fax a listing to MPR that documents when each comparison group family was notified of its status.
B. RANDOM ASSIGNMENT
1. Which families are eligible to participate in the research?
To participate in the research, all families must meet the general EHS eligibility criteria established by ACYF and the more specific criteria established by individual EHS programs. In addition, all families who meet these criteria must also meet the following conditions:
- The family must include a child who is 12 months old or younger on the date of application or a pregnant woman. In addition, this child must have been born or have an expected due date that falls between September 1, 1995 and June 30, 1997.
- The family must not have participated in the Comprehensive Child Development Program (CCDP) for 3 months or more during the previous 5 years.
- The family must not have participated in Head Start, Early Head Start, a Parent Child Center (PCC), or another similar program for 3 months or more during the previous 12 months.
- The family must be enrolled (submitted for random assignment) no later than June 30, 1998.
2. Must programs submit all eligible families for random assignment?
Yes. Programs should not enroll any families who meet the eligibility criteria outlined above outside of the random assignment process unless an exemption has been granted by ACYF. Non-research program slots should only be used for the following types of families:
- Families who are eligible for EHS but do not meet the research eligibility requirements because their child is more than 12 months old; their child's birthdate falls outside of the eligibility window; or they previously participated in CCDP, Head Start, Early Head Start, PCC, or another similar program
- Families assigned to the program group who will not participate in the research because they are part of a multiple family household as described in question 7 below
- Families who are granted an exemption from random assignment by ACYF
3. On what grounds will ACYF grant an exemption from random assignment?
ACYF will grant an exemption from random assignment only in cases of extreme need. For example, ACYF may grant an exemption if program enrollment is necessary to protect a child from physical harm.
4. What steps should a program follow to request an exemption?
The program director must request an exemption before submitting the family for random assignment. An exemption cannot be requested after random assignment because a family was assigned to the comparison group. To request an exemption, the program director must first make a request to her or his local research team. The local research team will review the request, discuss it with the program, and, if appropriate, forward it to ACYF. The final decision about whether to grant an exemption from random assignment will be made by ACYF. Contact Helen Raikes (202-205-2247) to request an exemption.
5. What should a program do if it cannot obtain informed parental consent for minors to participate in the EHS Evaluation?
For minors to participate in the evaluation, it is very important to obtain informed parental consent. However, we understand that in certain cases it may be nearly impossible for a program to obtain such consent for a minor (for example, if the minor is living in a separate household, is estranged from parents, or is emancipated). Regardless, we request that programs make every effort to obtain the parent's or a guardian's consent in all cases, even if such consent is not required for the minor to receive services. But, if it is impossible or prohibitively expensive for a program to obtain such consent, we will randomize the minor without consent if the program takes the following steps:
- Write a memo to MPR that clearly and succinctly explains (1) the local program requirements for serving a minor without parental consent, and (2) the state guidelines for providing other types of public services to minors without consent (for example, the general guidelines that AFDC or WIC use to provide assistance to minors.) An example of such an explanation is as follows:
In this state, minors can receive public services as independent cases and without parent or guardian consent if they are living apart from their parent or guardian; consequently, the local EHS program can also provide services to the individual without parent or guardian consent. In addition, circumstances are such that we cannot reasonably be expected to obtain parent or guardian consent in this and other such cases. Therefore, we ask that Mathematica randomize this minor for the EHS Evaluation without such consent.
- Reference this memo on the consent form for all such cases where parent or guardian consent cannot be obtained.
- In the case of legally emancipated minors, provide documentation of emancipation if at all possible. Documentation, if it exists, will likely vary by state. If it is not possible to obtain such documentation, the above memo should be referenced.
- In the case of a minor who is married, the husband should not sign the consent form and is not considered a legal guardian. A minor who is legally married is recognized as having achieved the age of majority for any legal purpose and is responsible for her own acts. Therefore, a mother who is younger than 18 and married should sign the consent form herself.
6. How will MPR randomly assign families in multiple family households?
We want to avoid situations in which a program family and a comparison group family live together because it may be difficult to prevent the comparison group family from receiving services (comparison family members may be present during home visits, for example). Therefore, when two or more families-related or not-are living together in the same home, they will be considered a multiple family household and if they both (all) apply to the EHS program, they will be treated as one family for purposes of random assignment. In other words, they will be assigned to the same group, and both will be considered program group families, or both will be considered comparison group families. If both families are selected for the program group, the EHS program may decide whether to serve both families or not. However, only one family will participate in the national evaluation assessment activities, and only that family will count toward the 75 program families required for the research sample. Similarly, if the two (or more) families are assigned to the comparison group, MPR will select just one of them to participate in the evaluation assessments.
7. What is the program's role in handling multiple family households (MFHs)?
Whenever possible, programs should notify MPR about a family's status as an MFH prior to random assignment. We will not re-assign families after random assignment, as this will diminish the validity of random assignment and will negatively affect the evaluation. Program staff should take the steps listed below when submitting families from MFHs for random assignment:
- If MFH families apply to EHS at the same time: The program should verify that the families are part of an MFH and indicate this on each family's HSFIS application. Program staff should clearly indicate on the top of the HSFIS form and on the cover page of their submission to Mathematica that the families are part of an MFH. This can be done by writing "MFH" in the upper right hand comer of the first page of the HSFIS application form and by writing "MFH" next to each family's information on the submission cover page. If the families are assigned to the program group, MPR will randomly select one family to participate in the research assessments. The program should then serve this family; it has the option to decide whether and to what extent it will serve the other family(ies).
- If an EHS applicant is living in the same household with a family already enrolled in the program group and the program wants to serve this family: The program should verify that the applicant family lives with the program group family and clearly indicate both on top of the HSFIS form and on the cover page of their submission that the family lives with a program group family. This can be done by writing MFH-P in the upper right hand comer of the first page of the HSFIS application form and by writing MFH-P next to the applicant's information on the submission cover page. The program should also attach a copy of the first page of the program family's HSFIS application so that MPR can match the new applicant to the program family. The program may decide whether or not to serve this new family.
However, the family will not become part of the research sample, will not count toward the 75 program families required for the research sample, and will not participate in the research assessments.
- If an EHS applicant is living in the same household with a comparison group family: Because programs are not providing services to comparison group families, we recognize that these cases may be more difficult for programs to identify. However, when programs are able to identify such cases, the applicants Will not be eligible to receive program services and will not become part of the research sample. Therefore, programs should not recruit families who are living in the same households with a comparison-group family.
8. How does MPR handle the random assignment of twin children?
The family unit, not the child, is randomly assigned to either the program group or the comparison group. If the family is assigned to the program group, both twins may be served by the EHS program, but only one twin will be assessed for research purposes. MPR will select the evaluation focus child at random. If one twin has a disability, that will have no bearing on the selection of the focus child-it will still be random.
9. How can programs ensure that they meet the 10 percent guideline for enrolling children with disabilities?
At least 10 percent of the children enrolled in Head Start must be children with disabilities. Early Head Start programs who are beginning enrollment and who are enrolling pregnant women should work with project officers to ensure that they follow a recruitment strategy likely to result in an enrollment in which at least 10 per cent of the children have disabilities, or in which risk factors for disabilities are present, as relevant within seven states for which specified categories of risk constitute eligibility. All programs will need to demonstrate that they have an intensive recruitment effort for children with identified disabilities and that they are working with appropriate agencies (such as United Cerebral Palsy, Association for Retarded Persons, and neonatal intensive care units) to recruit children with disabilities.
10. Should families whose incomes exceed the Head Start income eligibility requirement be submitted as a subgroup?
We will not form subgroups for families who are over income. Since no more than 10 percent of the EHS program enrollment can be families who are over the income eligibility requirement, we recommend recruiting less than 10 percent to prevent having more of these families selected into the program group than the comparison group.
C. MAINTAINING THE RESEARCH SAMPLE
1. After programs have filled all of their slots, it is likely that a few families will leave the program. If a family leaves the program, what procedures should be followed to fill the vacancy?
Programs should submit applicants for random assignment whenever a vacancy occurs, until the maximum research sample size has been reached. Applicants (whether newly recruited or from a waiting list) should be sent to MPR only when the program has an opening. For every one opening, the program can send from one to 'a few' applicants for random assignment (except for the Utah program, which must send an even number of applicants). Since we conduct random assignment one case at a time (except in Utah, where we use a batch process), if the first family is assigned to the program group, then this family can be enrolled in the program and the rest can be returned to the waiting list. If, however, the first family is assigned to the comparison group, then we will randomly assign the remaining families, one by one, until a family is assigned to the program group and the vacancy if filled.
2. What happens if a family drops out of the program or moves out of the service area after being randomly assigned to the program but before the program begins delivering services?
These families will be treated the same as families who drop out of the program at any other time. They will still be included in the program group of the research sample. The data collectors will make every reasonable effort to follow families who drop out at any time in the process and, whenever possible, conduct assessments on the same schedule as planned for other families in the research sample. In its analyses, MPR will adjust for the extent to which the families receive services, but it is very important that programs make every effort to retain, to the extent possible, all families who are selected for the EHS program group. It is very important to be sure that the family being recruited understands and is truly interested in receiving program services and participating in the research before completing the application/enrollment forms that are submitted to MPR.
3. What happens if a comparison group family moves out of the service area?
If a comparison family moves away from the EHS service area, we do not consider it to have dropped out of the research sample. Wherever comparison group families live, they will receive whatever services are normally available in the community without EHS, and therefore constitute a legitimate comparison. The national evaluation will make reasonable attempts to follow such families and to conduct the interviews and assessments. MPR will work with the local researchers to determine whether it is feasible to continue following such families and what costs are reasonable to incur for this purpose.
4. What happens if a family says it no longer wishes to participate in the research?
All families participating in the EHS national evaluation may refuse to participate in the research at any time. However, once a family goes through random assignment, it will not be dropped from the research sample, and MPR, through the local researchers, will continue to invite these families to participate in future rounds of interviews and assessments. As with all contacts with families, MPR's approach to communicating with families who have refused to participate will ensure that they are contacted in a respectful and sensitive manner.
When a program family refuses to participate in data collection activities, the local researcher will contact program personnel. Working together, the research and program staff will decide on the appropriate approach to take with the family, taking into account the family's current circumstances and needs. They should remind the family of its commitment to participate in both the program and the research. They should also be aware that the family's circumstances may change, and the family may decide to participate in the program and research at a future date. If the researcher finds that the family still refuses at the time of the next round of data collection, the researcher should again notify the program so that the program can counsel the family about its options. If, after considering various alternatives the family still refuses to participate in the research, the program should disenroll the family. The research team will continue to attempt to contact the family at the time of future data collections to obtain minimal data for the purpose of understanding why refusals occur.
5. What should a program do if it discovers that a family is ineligible for EHS after that family has been randomly assigned?
The program should write a memo to MPR documenting the specifics of the case and requesting directions for how to proceed. If the family was assigned to the program group, the program should explain the error to the family and explain that it cannot continue to receive services. If the family was assigned to the comparison group, the local researcher should inform the family that it will no longer be part of the research. It is very important that programs check eligibility carefully before submitting families for random assignment so that the number of such cases is kept to a minimum.
6. Will families in which the focus child dies or is miscarried continue participating in the research?
No. MPR will not continue collecting data from families after the focus child has died or is miscarried. While we feel that some valuable information about service use could be collected from these families, we have decided that problems with continuing data collection outweigh the advantages. It is up to the program to decide whether it will continue providing services to these families. In addition, MPR will not change the focus child after he or she has been selected, even if the focus child dies and another sibling is eligible to participate in the research.
7. What happens if the focus child's primary caregiver changes?
Because the focus of our research is the child, when the focus child becomes the responsibility of a new primary caregiver, MPR will follow the child. For example, a child may begin living with a different parent or a grandmother midway through the evaluation. It is up to the program to decide how it wishes to handle service delivery to the child's new primary caregiver.
Cases of adoption constitute an exception to this rule. If the focus child is adopted by another family, we may not be able to follow the child, because the birth mother may not know the identity of the adoptive parents and adoption agencies may not provide this information. Therefore, MPR will stop collecting data from families in which the focus child is adopted by another family.
8. Can program group families who move to the service area of another EHS research site enroll in EHS in their new location?
Yes. If a program group family moves to the service area of another research site, the family can enroll in the new program without going through random assignment a second time. However. it is up to the new program to determine whether it will enroll the family. Because each local program has tailored its eligibility criteria to its local area and program design. the family may not be eligible for the new program. Also. the new program may already have a waiting list for families who want to enroll in EHS.
9. Can comparison group families receive services that are similar to EHS services?
Comparison group families are permitted to apply for any services available in their communities, except those services restricted to EHS program participants. At one site, several comparison group families have enrolled in a local CCDP program. At another site, a comparison group family enrolled in Child Development Associate (CDA) training provided by the EHS program to community members who are interested in becoming child care providers. MPR believes these situations provide a valid counterfactual, because they represent the types of services available to non-EHS families in local communities.
10. What happens if comparison group families receive program services?
Programs should make every effort to avoid providing services to comparison group families. If you discover that services have mistakenly been provided to a comparison group family, please document the type and extent of services received and notify MPR as soon as possible. For national evaluation purposes, comparison group families who receive program services will still be counted as comparison group members when the data are analyzed. We need the documentation so we can understand how these families differ from comparison members who are not receiving services.
D. THE HEAD START FAMILY INFORMATION SYSTEM (HSFIS)
1. Who can programs call about questions relating to the HSFIS?
Questions about the HSFIS should be directed to Lihong Ma at NIE (301-738-1122). A back-up is Bill Wilson (202-205-8913). Ellen Kisker at MPR (609-275-2379) can also field questions, particularly pertaining to the application and enrollment forms.
2. Do the complete HSFIS application and enrollment forms have to be completed before random assignment?
Yes. However, only the first 4 pages need to be sent to MPR at that time. The rest of the HSFIS pages can be sent later.
3. Which version of the HSFIS forms should programs use?
Program should use the new version of the HSFIS application and enrollment forms that were provided to programs at the December 1996 Infant/Toddler Institute. However, programs should continue using the first 4 pages of the July version (the Preface), even though these pages were not included with the most resent version. These are the four pages that programs fax to MPR when submitting names for random assignment.
4. If the applicant is a pregnant woman, do programs have to fill out the HSFIS information on the program child after the child is born and then send that to MPR?
Yes. ACYF has specified the need for this information. Programs must send HSFIS application and enrollment information on program children to MPR after the child is born. At some point in the future it may be possible for MPR to obtain this information in an automated fashion from the HSFIS contractor. However, until we notify programs otherwise, programs should provide us with the hard copy HSFIS forms. Programs are not required to collect this information for babies born to comparison group families. This information will be collected by local research teams as described under question 5.
5. What is the program's role in collecting HSFIS data on the child of a comparison group family?
The program is responsible for completing the HSFIS application and enrollment forms for all applicants at intake, including those who get assigned to the comparison group. However. some women enroll in EHS during pregnancy, before the birth of the focus child. The program is not responsible for collecting HSFIS application and enrollment data for children born after enrollment who are assigned to the comparison group. In addition. the program is not responsible for collecting HSFIS service module data for comparison group families.
6. What is the local research team's role in collecting HSFIS data on babies born to comparison group families after enrollment?
Local research teams will be responsible for collecting HSFIS application and enrollment data on babies born to comparison group mothers enrolled during pregnancy. MPR is developing a form for data collectors to use at the time of the 12-month Parent Services Follow Up Interview (PSI). Some questions from the HSFIS will be omitted because the information will be obtained during other interviews with the parent. Although we do not think that the subcontract budget implications of adding this form will be significant, we will monitor the actual costs for completing the PSI and make adjustments as necessary.
7. Does the "Project Head Start Consents, Authorizations, and Releases Form" need to be completed and medical records information obtained (to complete the HSFIS forms) and sent to MPR before random assignment?
It would be ideal to have the forms and information at the point of random assignment, but it is not imperative. We understand that obtaining medical releases and records information takes time and we do not wish to hold up the random assignment process because of it. Programs should send the Head Start release forms and completed HSFIS question based on the medical records to MPR a soon as possible after they are completed.
| Variable | Percent of Families | Number of Sites in Whichthe Variable Varies |
|---|---|---|
| Family and Parent Characteristics | ||
| Age of Mother | ||
| Younger than 20a | 39 | 17 |
| 20 to 25 | 33 | 17 |
| 25 or older | 28 | 16 |
| Race and Ethnicity | ||
| White non-Hispanica | 37 | 17 |
| Black non-Hispanic | 35 | 16 |
| Hispanic | 24 | 17 |
| Other
(Asian or Pacific Islander, American Indian, Eskimo, Aleut) |
5 | 16 |
| English Language Ability | ||
| Primary language is Englisha | 79 | 16 |
| Primary
language is not English but the applicant speaks English well |
10 | 16 |
| Primary
language is not English and the applicant does not speak English well |
11 | 12 |
| Highest Grade Completed | ||
| Less than 9a | 11 | 17 |
| 9 to 11 | 37 | 17 |
| 12 or earned a GED certificate | 28 | 17 |
| More than 12 | 24 | 17 |
| Primary Occupation | ||
| Employeda | 23 | 17 |
| In school or a training program | 22 | 17 |
| Unemployed | 28 | 17 |
| Out of the Labor Force | 27 | 17 |
| Living Arrangements | ||
| Living with a partnera | 25 | 17 |
| Living with other adults | 39 | 17 |
| Living with no other adults | 36 | 17 |
| Number of Children in the Household | ||
| Ages 0 to 5 | 0.5c | 17 |
| Ages 6 to 17 | 0.5c | 17 |
| Household
Income as a Percent of the Poverty Level (Percent) |
||
| Less than 33a | 25 | 17 |
| 33 to 67 | 25 | 17 |
| 67 to 99 | 21 | 17 |
| 100 or more | 11 | 17 |
| Missing | 18 | 17 |
| Welfare Receipt | ||
| AFDC/TANF | 34 | 17 |
| Food Stamps | 48 | 17 |
| WIC | 87 | 17 |
| SSI | 7 | 17 |
| Inadequate Resources | ||
| Food | 5 | 17 |
| Housing | 12 | 17 |
| Money | 20 | 17 |
| Medical care | 14 | 17 |
| Transportation | 21 | 17 |
| Number of Moves in the Past Year | 0.9c | 17 |
| Random Assignment Date | ||
| Before 10/96a | 36 | 15 |
| 10/96 to 6/97 | 31 | 16 |
| After 6/97 | 33 | 16 |
| Child Characteristics | ||
| Age of Focus Child (Months) | ||
| Unborn | 25 | 17 |
| Less than 3a | 21 | 17 |
| 3 to 6 | 22 | 17 |
| 6 or more | 32 | 17 |
| Birthweight Less than 2,500 Gramsb | 7 | 17 |
| Born More Than 3 Weeks Earlyb | 10 | 17 |
| Male | 51 | 17 |
| Received an Evaluation Because of Concerns About the Child's Overall Health and Development or Because of Suspected Developmental Delayb | 5 | 17 |
| Risk Categories | ||
| Has established risksb | 8 | 17 |
| Has biological or medical risksb | 12 | 17 |
| Has environmental risksb | 24 | 17 |
| Previously Enrolled in Head Start or Another Childhood Development Programb | 13 | 17 |
| Missing Section on Child Characteristicsb | 8 | 17 |
SOURCE: HSFIS application and enrollment forms.
aThis indicator variable
was omitted from the explanatory variables in the regression models.(back)
bThese variables pertain to families with focus
children who were born at the time of program application. The variables
were set to zero for families with unborn focus children (because
an indicator variable for these families was included in the regression
models), but the figures in the second and third columns of this table
pertain only to those with born children.(back)
cFigures for these continuous variables are
variable means.(back)
| Table of Contents | Previous | Next |