ACF OPRE: National Head Start/Public School Early Childhood Transition Demonstration Study

Technical Report #4

Analyses of School Climate Survey and Assessment Profile

TECHNICAL SUMMARY OF ANALYSES OF THE SCHOOL CLIMATE SURVEY
School Climate Survey Analyses for Family Respondents
Eight repeated measures ANOVAs were run using each of the eight subscale scores for the School Climate Survey from family respondents as the dependent variables and Condition (2 levels) and Grade (4 levels) as mediating variables. Subscale means for Condition and Grade are summarized in Table 1 below. Only Student Behavioral Values demonstrated significant effects for time (a drop from kindergarten to first Grade followed by a steady improvement through 3rd) and for Condition (the comparison group mean is higher). The Administration subscale shows the only Condition by time interaction (the Demonstration group shows slight but steady decline across years while responses from the comparison group families were irregular across time. The differences in factor scores, even for significant effects, were rarely more than 0.10. ). As was indicated in the text of the report (Chapter 9), means for Student Behavioral Values were significantly lower than means of other subscales.

A 2 (between subjects) by 4 (within subjects) repeated measures MANOVA (eight subscales) was run for the 559 family respondents (treatment=293, comparison=266) who provided data for all subscales at each grade level. Multivariate effects were found for condition (p=.041, eta-squared=.029), grade (p=.000, eta-squared=.095), and grade by condition (p=.014, eta-squared=.074). Examination of univariate effects revealed that only the scale for Student Behavioral values yielded a significant effect for grade (second and third grade scores were higher than kindergarten and first grade). A significant grade by condition interaction was found only for the Administration (treatment families provided higher mean ratings in kindergarten and first grade while the comparison families provided higher mean ratings in second and third grade). Cell means for the multivariate analyses are similar to those derived from the repeated measures ANOVA (Table 1).

Table 1.
School Climate Survey: Mean subscale responses from families by Grade by Condition
STUDENT ACTIVITIES
Condition	Grade	Mean	Std. Error	N
DEM	K	4.0694	0.02	568
	1	4.0844	0.022
	2	4.0851	0.021
	3	4.0344	0.022
COM	K	4.0578	0.021	557
	1	4.0696	0.022
	2	4.0796	0.021
	3	4.048	0.022
INSTRUCTIONAL MANAGEMENT
Condition	Grade	Mean	Std. Error	N
DEM	k	3.9026	0.023	720
	1	3.8849	0.025
	2	3.8913	0.023
	3	3.907	0.023
COM	k	3.9455	0.024	701
	1	3.9403	0.025
	2	3.8987	0.023
	3	3.9563	0.023
STUDENT-PEER INTERACTION
Condition	Grade	Mean	Std. Error	N
DEM	k	3.8109	0.02	754
	1	3.8259	0.021
	2	3.8222	0.02
	3	3.8335	0.02
COM	k	3.8465	0.02	727
	1	3.8417	0.021
	2	3.8456	0.02
	3	3.8676	0.02
STUDENT BEHAVIORAL VALUES
Condition	Grade	Mean	Std. Error	N
DEM	k	2.7554	0.034	570
	1	2.738	0.034
	2	2.7925	0.033
	3	2.8136	0.033
COM	k	2.8463	0.035	527
	1	2.7982	0.036
	2	2.8821	0.035
	3	2.9537	0.035
STUDENT ACADEMIC ORIENTATION
Condition	Grade	Mean	Std. Error	N
DEM	k	4.1393	0.017	798
	1	4.1507	0.017
	2	4.1163	0.016
	3	4.1273	0.017
COM	k	4.1008	0.017	785
	1	4.1241	0.017
	2	4.1387	0.017
	3	4.139	0.018
ADMINISTRATION
Condition	Grade	Mean	Std. Error	N
DEM	k	4.0124	0.029	527
	1	4.0106	0.031
	2	3.9937	0.029
	3	3.9399	0.03
COM	k	3.9502	0.03	499
	1	3.9578	0.032
	2	4.0452	0.029
	3	3.9879	0.031
SECURITY AND MAINTENANCE
Condition	Grade	Mean	Std. Error	N
DEM	k	4.1553	0.017	806
	1	4.1956	0.019
	2	4.1762	0.018
	3	4.22	0.018
COM	k	4.19	0.017	799
	1	4.1864	0.019
	2	4.211	0.018
	3	4.2213	0.018
TEACHER-STUDENT RELATIONSHIPS
Condition	Grade	Mean	Std. Error	N
DEM	k	4.0351	0.02
	1	4.0158	0.02
	2	4.0085	0.02
	3	3.9724	0.021
COM	k	4.0094	0.02
	1	4.0097	0.021
	2	4.0135	0.02
	3	4.0252	0.021

School Climate Survey Analyses for Principals

Repeated measures multivariate analyses of variance were conducted across sites using principal’s responses to the School Climate Survey in order to determine whether significant changes occurred in principal’s perceptions of school environment during the duration of the study. Principal responses from 1993 were compared with their responses in 1997 across all eight factors of the School Climate Survey. Treatment condition served as the independent variable in the analyses. Repeated measures data was available from approximately 130 principals in each treatment condition (sample sizes varied slightly by factor). No statistically significant main effects were found for time (the repeated measure), treatment condition, or the interaction between time and treatment condition for any subscale of the School Climate Survey. Means for each subscale for 1993 and 1997 are summarized below. While mean differences across time and between groups appear small, the moderately large within-group standard deviations are clearly contributing to the lack of statistical significance. Subsequent analyses will attempt to account for the within group variance.

	Year	DC	Mean	SD	N
Teacher-Student Inter.	93	D	4.3954	0.4945	130
Teacher-Student Inter.	93	C	4.4123	0.3945	133
Teacher-Student Inter.	97	D	4.4655	0.4649	130
Teacher-Student Inter.	97	C	4.4332	0.4207	133
Teacher-Student Inter.	97	DC	Mean	SD	N
Security & Maint.	93	D	4.424	0.5019	130
Security & Maint.		C	4.384	0.5066	132
Security & Maint.	97	D	4.3804	0.4522	130
Security & Maint.		C	4.3521	0.5199	132
Security & Maint.		DC	Mean	SD	N
Administration	93	D	4.6271	0.3873	129
Administration		C	4.6124	0.4299	132
Administration	97	D	4.6526	0.3609	129
Administration;		C	0.662	0.3512	132
Administration		DC	Mean	SD	N
Student Academic Orient.	93	D	4.2575	0.4935	128
Student Academic Orient.		C	4.1622	0.5641	131
Student Academic Orient.	97	D	4.2159	0.5058	128
Student Academic Orient.		C	4.2156	0.5278	131
Student Academic Orient.		DC	Mean	SD	N
Student Behavioral Values	93	D	3.5493	0.6966	129
Student Behavioral Values		C	3.5205	0.746	131
Student Behavioral Values	97	D	3.7521	0.6351	129
Student Behavioral Values		C	3.6444	0.6781	131
Student Behavioral Values		DC	Mean	SD	N
Student-Peer Interact.	93	D	4.0994	0.4701	129
Student-Peer Interact.		C	4.0132	0.4849	133
Student-Peer Interact.	97	D	4.2196	0.4913	129
Student-Peer Interact.		C	4.0881	0.4845	133
Student-Peer Interact.		DC	Mean	SD	N
Instructional Management	93	D	3.649	0.8954	129
Instructional Management		C	3.5738	0.8907	133
Instructional Management	97	D	3.6873	0.8757	129
Instructional Management		C	3.6805	0.8058	133
Instructional Management		DC	Mean	SD	N
Student Activities	93	D	4.2042	0.5487	128
Student Activities		C	4.1727	0.5416	132
Student Activities	97	D	4.248	0.5411	128
Student Activities		C	4.1906	0.4942	132

School Climate Survey Analysis for Teachers

A multivariate analysis of variance was run across sites (national data) using School Climate Survey factor scores for each subscale as the dependent variables with year (5 levels) and condition (2 levels) as mediating variables. Table 1 below summarizes the means for each subscale by year and condition. The primary hypothesis of interest concerns the interaction of condition and time. A multivariate main effect was not found for condition (significance = .038 based on Pillai’s Trace). The comparison group showed slightly higher means across subscale, but such differences did not meet the criteria for statistical significance set for this study. A significant multivariate main effect was also found for year (significance < .001 based on Pillai’s Trace). The significant effect for time tended to reflect somewhat higher ratings by teachers in 1993 than in subsequent years. However, there were no significant multivariate or univariate interaction effects. As can be seen within Table 1b, while statistically significant, the changes within subscale across time were relatively minor.

It is important to qualify these findings with the reminder that the data for this analysis were cross-sectional in nature. That is, different teachers provided data for each year. Therefore, the significant effect for time and the lack of an interaction effect must be considered in light of the varying sources of information at each time period.

Table 1a:
School Climate Survey-Teacher Form: subscale means by condition (DC)
Scale	DC	Mean	Std. Error
Teach-Student	DEMO	4.37	0.014
Teach-Student	COMP	4.39	0.013
Sec & Maint.	DEMO	4.08	0.019
Sec & Maint.	COMP	4.12	0.018
Admin	DEMO	4.09	0.021
Admin	COMP	4.12	0.021
Academic	DEMO	3.94	0.018
Academic	COMP	4	0.018
Behavioral	DEMO	3.03	0.023
Behavioral	COMP	3.09	0.022
Student-peer	DEMO	3.83	0.017
Student-peer	COMP	3.87	0.016
Instruction	DEMO	3.09	0.028
Instruction	COMP	3.2	0.027
Student Act.	DEMO	3.81	0.02
Student Act.	COMP	3.9	0.02

Table 1b:
School Climate Survey-Teacher Form: subscale means by year
Scale	Year	Mean	Std. Error
TEACH-STUDENT	93	4.39	0.022
	94	4.39	0.018
	95	4.4	0.019
	96	4.4	0.015
	97	4.33	0.029
SEC. & MAINT.	93	4.13	0.03
	94	4.04	0.024
	95	4.09	0.026
	96	4.12	0.021
	97	4.11	0.04
ADMIN	93	4.16	0.034
	94	4.05	0.028
	95	4.05	0.029
	96	4.13	0.023
	97	4.12	0.046
ACADEMIC	93	4.04	0.03
	94	3.96	0.024
	95	3.99	0.025
	96	3.96	0.02
	97	3.92	0.039
BEHAVIORAL	93	3.09	0.037
	94	2.99	0.03
	95	3.03	0.032
	96	3.06	0.025
	97	3.1	0.049
STUDENT-PEER	93	3.93	0.027
	94	3.83	0.022
	95	3.82	0.023
	96	3.84	0.019
	97	3.82	0.037
INSTRUCTIONAL	93	3.25	0.045
	94	3.01	0.036
	95	3.12	0.038
	96	3.16	0.03
	97	3.18	0.059
STUDENT ACT.	93	3.93	0.033
	94	3.8	0.026
	95	3.82	0.028
	96	3.83	0.022
	97	3.9	0.044

Table 1c:
School Climate Survey-Teacher Form: subscale means by condition (DC) by year
Scale	DC	YEAR	Mean	Std. Error
TEACH-STUDENT	DEMO	93	4.41	0.031
		94	4.38	0.026
		95	4.37	0.027
		96	4.41	0.021
		97	4.27	0.043
	COMP	93	4.38	0.032
		94	4.4	0.024
		95	4.43	0.026
		96	4.39	0.021
		97	4.38	0.04
SEC. & MAINT.	DEMO	93	4.14	0.042
		94	4	0.035
		95	4.06	0.037
		96	4.12	0.029
		97	4.08	0.059
	COMP	93	4.12	0.043
		94	4.09	0.033
		95	4.13	0.035
		96	4.12	0.029
		97	4.14	0.055
ADMIN	DEMO	93	4.15	0.048
		94	4.01	0.04
		95	4.01	0.042
		96	4.17	0.033
		97	4.09	0.067
	COMP	93	4.17	0.049
		94	4.09	0.038
		95	4.1	0.04
		96	4.1	0.033
		97	4.16	0.063
ACADEMIC	DEMO	93	4.02	0.041
		94	3.93	0.035
		95	3.94	0.036
		96	3.93	0.029
		97	3.89	0.057
	COMP	93	4.06	0.043
		94	3.99	0.033
		95	4.04	0.035
		96	3.99	0.028
		97	3.96	0.054
BEHAVIORAL	DEMO	93	3.08	0.052
		94	2.95	0.044
		95	2.97	0.046
		96	3.04	0.036
		97	3.1	0.072
	COMP	93	3.11	0.053
		94	3.03	0.041
		95	3.1	0.044
		96	3.09	0.036
		97	3.11	0.068
STUDENT-PEER	DEMO	93	3.92	0.038
		94	3.83	0.032
		95	3.78	0.034
		96	3.81	0.026
		97	3.8	0.053
	COMP	93	3.94	0.039
		94	3.84	0.03
		95	3.86	0.032
		96	3.88	0.026
		97	3.85	0.05
INSTRUCTIONAL	DEMO	93	3.18	0.062
		94	2.96	0.052
		95	3.12	0.055
		96	3.09	0.043
		97	3.07	0.087
	COMP	93	3.32	0.064
		94	3.06	0.049
		95	3.12	0.052
		96	3.23	0.043
		97	3.29	0.081
STUDENT ACT.	DEMO	93	3.9	0.046
		94	3.74	0.039
		95	3.75	0.04
		96	3.79	0.032
		97	3.86	0.064
	COMP	93	3.97	0.047
		94	3.86	0.036
		95	3.88	0.039
		96	3.86	0.031
		97	3.95	0.06

TECHNICAL REPORT

Summary of results from the Assessment Profile and ADAPT classroom environment observation measures

The Assessment Profile describes classroom practices across five broad dimensions (learning, environment, scheduling, curriculum, interacting, and individualizing). A multivariate analysis of variance (MANOVA) (2 condition by 4 grade levels) was conducted using the five scales as the dependent measures. Significant multivariate effects (< .001) were found for grade and condition main effects. The interaction effect was not significant.

Examination of univariate effects revealed main effects for grade for all scales (p < .001). Main effects for condition were found for the learning environment (p < .001) and scheduling (p = .006) scales. The main effects for condition indicated higher mean scale scores for the demonstration rather than the comparison condition on each scale. The largest, and most significant differences between demonstration and comparison classrooms were found for the Learning Environment scale with a mean difference of approximately 0.9. It should be noted that while statistically significant, effect sizes did not exceed .12. This suggests that such between condition differences may not be practically meaningful.

The nature of the main effects for grade varied as a function of the scale. For example, scaled scores for Learning Environment decreased for every grade (only the loss between second and third grade was not statistically significant). Scaled scores for scheduling showed a statistically significant drop from kindergarten to first grade. Scaled scores then increased for second and third grades. The only non-significant paired comparison was between first and second grade. The Curriculum scale demonstrated a non-significant drop between kindergarten and first grade and a significant increase between first and second and between second and third. The Interacting scale demonstrated significant decreases in scaled scores between kindergarten and first grade and between first and second grade. Statistically significant gains in scaled scores for the Individualizing scale were evident between kindergarten and first and between first and second grades.

Analyses of the ADAPT were also conducted using MANOVA procedures. Significantly lower scores for the Material, Resources, Group, Self-Regulation, Time Spent Learning, and Evidence scales were found in third grade than in first and second grades. Significantly and progressively lower scores for the Space scale were found for all adjacent pairs of grades. The main effect for Condition and the Condition by Grade interaction effect did not attain the level of statistical significance required of this study. The ADAPT was not required by all sites. Therefore, the findings related to the ADAPT should not be generalized across all sites within the national sample.

As can be seen in Table 1 below, all sites yielded significant multivariate (across all five scales) main effects for grade on the Assessment Profile. The direction of the effects was not consistent between sites. Seventeen sites yielded significant multivariate main effects for condition (the direction of the effect was not consistent). A significant grade by condition interaction was found for nine sites. The eta-squared effect sizes for interaction never exceeded .11. This suggests that the interaction of grade and condition does not account for a great deal of the overall variability in Assessment Profile scaled scores. In eleven of the sites, the effect size for Grade accounted for more than 20% of the total variance in scaled scores. Condition accounted for more than 20% of the variance in five of the sites.

Twenty-six sites yielded univariate main effects for grade for the Learning Environment scale of the Assessment Profile. Twelve sites yielded such effects for condition. Only three sites yielded a significant grade by condition interaction for Learning Environment. Eighteen sites yielded univariate main effects for grade on the Scheduling scale. Fourteen sites yielded such effects for condition. Only two sites yielded a significant grade by condition interaction for Scheduling. Nineteen sites yielded univariate main effects for grade on the Curriculum scale. Nine sites yielded such an effect for condition. Six sites yielded significant grade by condition interactions for Curriculum. Twenty-two sites yielded univariate effects for grade on the Individualizing subscale. Five sites yielded such an effect for grade. Five sites yielded grade by condition interaction effects for Individualizing. Thirteen sites yielded main effects for grade on the Interacting scale. Six sites yielded such an effect for condition. Seven sites yielded significant grade by condition interaction effects for the Interacting scale. Findings suggest that the characteristics of classrooms experienced by students in this study changed from kindergarten through third grade across all sites. Differences in classroom characteristics that can be attributed to assignment condition were considerably less evident between sites. Finally, the nature of the changes in classroom characteristics between kindergarten and third grade did not vary greatly depending on the assignment condition of the classroom.

These analyses suggest that classroom climate is a complex and dynamic construct within and across the sites comprising this study. Results indicate that classrooms in the demonstration condition were observed to have somewhat better arrangement and availability of classroom materials and environments as well as somewhat better evidence of scheduling, planning, and variety of classroom activities than were classrooms in the comparison condition. The strong evidence of site-level differences highlight the need to attend to site-specific characteristics when considering the nature and impact of classroom climate.

Table 1.
MANOVA Summary: Assessment Profile Scaled Scores by Grade by Condition
SITE	GRADE		CONDITION		GRADE BY CONDITION
	Multivariate eta-squared	Univariate	Multivariate eta-squared	Univariate	Multivariate eta-squared	Univariate

1	.275***	LE, SC, CU, IT	.14***	SC, CU, IT	NS	-
2	.152***	LE, SC, ID	.227***	LE, SC, CU, IT	NS	-
3	.227***	LE, IT, ID	.207***	LE, SC, ID	NS	-
4	.214***	LE, CU, IT	.398***	LE, CU, IT	NS	-
6	.112*	LE, ID	NS	IT	NS	-
7	.17***	LE, SC, CU, IT, ID	NS	LE	NS	-
8	.283***	LE, SC, ID	NS	-	NS	-
9	.15***	LE, SC, CU, IT, ID	0.115	LE, SC	.048*	-
10	.165***	LE, CU, ID	.093***	LE, SC, ID	.058***	LE, CU, IT, ID
11	.111***	LE, SC, IT , ID	.06**	SC	.039**	SC, CU, IT, ID
12	.225***	LE, SC, CU, ID	.118*	LE, SC, CU	NS	IT
13	***	SC, CU, ID	NS	-	NS	-
14	.177*	LE, ID	NS	-	NS	-
15	.104***	LE, SC, IT	.122***	LE, SC	.09***	LE, SC
16	.114***	LE, SC, CU, IT	.115***	LE, CU, ID	NS	-
17	.142***	SC, CU, IT, ID	NS	-	NS	-
18	.152***	LE, CU, ID		CU	NS	-
19	.142***	LE, CU	NS	-	NS	-
21	.279***	LE, SC	NS	-	NS	-
22	.206***	LE, CU, IT, ID	NS	-	NS	-
23	.148***	LE, SC, IT	.068**	IT, ID	NS	-
24	.204***	LE, CU, ID	.122*	LE, SC, CU	NS	-
25	.112***	LE, IT, ID	NS	SC	.032*	LE, CU, ID
26	.195***	SC, CU	.301***	LE, SC, CU, IT	.103*	CU, IT, ID
27	.153***	LE, CU, ID	NS	ID	NS	-
28	.355***	SC, CU, ID	.336**	SC	NS	-
29	.085***	LE, SC, CU, ID	.032**	ID	NS	CU, IT
30	.134***	LE, SC, IT, ID	.045*	LE, SC	.032*	IT
31	.213***	LE, SC, CU, ID	.092**	LE, SC, CU	.068***	CU, IT, ID
32	.203***	LE, CU, IT, ID	NS	-	.053**	ID

Table of Contents | Previous | Next