Table of Contents | Previous |
Appendix D
Considerations and Terminology Regarding the Interpretation of Statistical Tests
The design of this study, especially the modest-sized sample and weighted data, required careful interpretation of the results. Because the study researchers sought to explore a variety of possible relationships that had not been studied before, they did not want to unduly restrict their search for relationships and ran a substantial number of analyses. For that reason they decided against using the more flexible significance level of p < .10, commonly used in preliminary studies, because this would result in too many false positive findings to allow for confident interpretation. Instead they chose to use the following terminology in writing about their results.
Some indication of difference: when weighted analyses find the probability of difference to be .05 < p <. 10 and when the unweighted difference is significant in the same direction and the finding is theoretically plausible. This designation was also used to interpret chi-square tests when the expected sample sizes for two cells were less than 5 or when the items on which the analyses were based seemed unusually ambiguous.
Some evidence of difference: when the probability of difference is .01 < p < .05 in the weighted analysis (this is not called a “significant difference” because of the inflation of the alpha levels due to the many tests that were run).
Significant difference (or strong evidence of difference): when the p value is at .01 or less.
| Table of Contents | Previous |