Day 54 – Comparing individual conditions in ANOVA

Post hoc comparisons

ANOVA only gives information about whether or not there are significant differences in several conditions. It does not tell us anything about which direction the differences point to. Since comparison of conditions between individual conditions are carried out after the ANOVA analysis, the comparisons are called post hoc comparisons. To explore differences among different conditions, we test pairs of conditions (also called contrasts), so it is often called multiple comparisons.

So if we have 3 conditions, we would have the following contrasts:

  • Condition 1 and Condition 2
  • Condition 2 and Condition 3
  • Condition 1 and Condition 3

So we could test the above pairs in the following way:

a) Whether scores in Condition 1 are higher than the scores in Condition 2

b) Whether scores in Condition 2 are higher than the scores in Condition 3

 

Both a) and b) can be tested using t tests.

 

Adjusting significance levels

Since we are carrying out an ANOVA test and post hoc comparisons, we need to adjust the significant levels to account for this extra testing. Otherwise, with a fixed significance level, by throwing more and more tests at the data, we would eventually get fake significant results by pure virtue of the fact that we are having more attempts. An analogy would the case of a dice. If you throw it once, you have 1/6 probability of getting a 6 but if you throw twice, you have more than 1/6 probability (let’s call it X) of getting a 6 and if you throw it thrice, you have more than X probability of getting a 6. So the more times you throw the dice, the higher the chance that you will get a 6. The same thing happens in our case. That is why we need to adjust the significance levels.

 

Bonferroni correction

One way of adjusting significance levels is through the Bonferroni correction. If we have n number of hypotheses, we give each hypothesis 0.05/n as significance level. So in our case, we have 3 contrasts/hypothesis so our significance level would be 0.017. This is an adjusted significance level rather than an actual significance level. This means that if your adjusted level of significance is 0.017 and one/some/all your hypotheses is/are statistically significant, all it means is that your hypotheses is/are significant at the 0.05 level.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s