Download ReportDownload Report PDF (634.76 KB)
- File Size: 634.76 KB
- Pages: N/A
- Published: 2018
- What is the psychometric performance of cognitive and social-emotional measures of preschoolers’ development with AI/AN children?
AI/AN FACES 2015 is the first national study of Region XI AI/AN Head Start children and their families, classrooms, and programs. To date, the Head Start Family and Child Experiences Survey (FACES) has been a major source of descriptive information on Head Start and preschool children ages 3 to 5 years old who attend the program. FACES gathers data from Regions I-X, the 10 geographically based Head Start regions, with the most recent round conducted in 2014.
The AI/AN FACES 2015 study presents a new opportunity to explore the psychometric performance of commonly used measures of preschoolers’ cognitive and social-emotional development. The reliability and validity of a measure are not inherent but depend on its use. Norming samples for most child assessment measures do not include large numbers of AI/AN children and as a result little is known about measure performance when administered to AI/AN children. Concerns exist about whether scores from these measures accurately reflect the children’s abilities, skills, and knowledge. Previous smaller studies have used these measures with AI/AN children, but none were large enough to test the measures’ psychometric performance. Child outcomes measures in AI/AN FACES 2015 were aligned with those in FACES 2014. Therefore, this alignment allows us to learn how standardized child development measures performed when administered to a large sample of AI/AN children.
This report describes the performance of cognitive and social-emotional measures of preschoolers’ development for AI/AN children, using recent data from AI/AN FACES 2015 and FACES 2014.
The purpose of this technical report is to present findings from analyses of how preschool cognitive and social-emotional measures performed in AI/AN FACES 2015. We examined the internal consistency of measures when administered to AI/AN children, reviewed descriptive statistics as context of difference in mean ability across groups in the AI/AN FACES 2015 and FACES 2014 samples, conducted analyses of differential item functioning (DIF) within cognitive measures to compare the performance of AI/AN children and White children (including data from FACES 2014), and examined the strength of bivariate correlations between measures of similar constructs and different constructs to assess evidence of concurrent and discriminant validity. The findings, therefore, provide initial evidence on the reliability and validity of the measures for AI/AN preschoolers.
Key Findings and Highlights
For most of the measures, findings from these analyses suggest that it is appropriate to report the AI/AN FACES 2015 preschool child outcomes scores, the exception being one of the two measures of executive function (Heads-Toes-Knees-Shoulders or HTKS, which was added to AI/AN FACES 2015 to expand measurement of this construct beyond what is used in FACES 2014).
- All measures demonstrated acceptable reliability with alphas of 0.70 or above.
- The strength of correlations between measures is in an expected pattern. Correlations are stronger between measures of similar constructs (for example, receptive and expressive language) than between different constructs (for example, social behavior and language).
- Among six cognitive measures flagged across reviews, none warrant additional follow-up based on the DIF analyses. Most cognitive measures did not show evidence of performing differently across groups based on DIF analysis. Two cognitive measures (Peabody Picture Vocabulary Test-Fourth Edition and Expressive One-Word Picture Vocabulary Test-Fourth Edition) had items demonstrating DIF; however, the number of items with DIF was close to or less than the number we would expect by chance and were balanced overall with some easier for AI/AN children and others easier for White children.
- None of the teacher- and assessor-reported social-emotional measures exhibited performance concerns based on the current review.
- Examination of the executive function measures indicated that the pencil tapping task is an appropriate measure for this sample. However, a floor problem was found with the HTKS, indicating the measure provided limited information to distinguish the children in this sample.
These analyses are based on a specific sample of children—AI/AN children in Head Start programs operated by federally recognized tribes. While this information provides initial evidence of the reliability and validity for these measures of cognitive and social-emotional development, researchers should keep in mind the diversity of tribal communities and the AI/AN population nationwide and in Head Start more generally as compared to Region XI AI/AN Head Start when considering the use of these measures with other AI/AN children.
The AI/AN FACES 2015 sample provides information about Region XI Head Start children, their families, classrooms, centers, and programs. We selected a sample of Region XI Head Start programs from the 2012-2013 Head Start Program Information Report, selecting one to two centers per program and two to four classrooms per center. Within each classroom, all children (both AI/AN and non-AI/AN) were selected for the study. Twenty-one programs, 37 centers, 73 classrooms, and 1,049 children participated in the study.
The FACES 2014 sample provides information at the national level about Head Start programs, centers, classrooms, and the children and families they serve. We selected a sample of Head Start programs from the 2012-2013 Head Start Program Information Report, with two centers per program and two classrooms per center selected for participation. Within each classroom, we randomly selected 12 children for the study. One-hundred seventy-six programs, 346 centers, 667 classrooms, and 2,206 children (in 60 programs) were still study participants in spring 2015.
L. Malone, S. Bernstein, S. Atkins-Burnett, and Y. Xue (2018). Psychometric Analyses of Child Outcome Measures with American Indian and Alaska Native Preschoolers: Initial Evidence from AI/AN FACES 2015. OPRE Report 2018-21, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
- AI/AN FACES:
- American Indian and Alaska Native Head Start Family and Child Experiences Survey
- differential item functioning
- Head Start Family and Child Experiences Survey