Post hoc comparisons
ANOVA only gives information about whether or not there are significant differences in several conditions. It does not tell us anything about which direction the differences point to. Since comparison of conditions between individual conditions are carried out after the ANOVA analysis, the comparisons are called post hoc comparisons. To explore differences among different conditions, we test pairs of conditions (also called contrasts), so it is often called multiple comparisons.
So if we have 3 conditions, we would have the following contrasts:
- Condition 1 and Condition 2
- Condition 2 and Condition 3
- Condition 1 and Condition 3
So we could test the above pairs in the following way:
a) Whether scores in Condition 1 are higher than the scores in Condition 2
b) Whether scores in Condition 2 are higher than the scores in Condition 3
Both a) and b) can be tested using t tests.
Adjusting significance levels
Since we are carrying out an ANOVA test and post hoc comparisons, we need to adjust the significant levels to account for this extra testing. Otherwise, with a fixed significance level, by throwing more and more tests at the data, we would eventually get fake significant results by pure virtue of the fact that we are having more attempts. An analogy would the case of a dice. If you throw it once, you have 1/6 probability of getting a 6 but if you throw twice, you have more than 1/6 probability (let’s call it X) of getting a 6 and if you throw it thrice, you have more than X probability of getting a 6. So the more times you throw the dice, the higher the chance that you will get a 6. The same thing happens in our case. That is why we need to adjust the significance levels.
One way of adjusting significance levels is through the Bonferroni correction. If we have n number of hypotheses, we give each hypothesis 0.05/n as significance level. So in our case, we have 3 contrasts/hypothesis so our significance level would be 0.017. This is an adjusted significance level rather than an actual significance level. This means that if your adjusted level of significance is 0.017 and one/some/all your hypotheses is/are statistically significant, all it means is that your hypotheses is/are significant at the 0.05 level.