# ANOVA is a worthless statistical test

znsho
Science 02 Dec '08 23:42
1. 02 Dec '08 23:42
Many scientists insist that statistisical analiysis of more than two groups requires ANOVA and post ANOVA tests.

I disagree. ANOVA imparts no information whatsoever except, perhaps, there might be a difference somewhere. ANOVA does not show where differences might lie. More importantly, ANOVA might say ' there is no difference' when, in fact, a difference is real.

I insist, t-tests are the best method of statistical analysis no matter how many groups are involved.
2. Palynka
Upward Spiral
02 Dec '08 23:50
Originally posted by znsho
Many scientists insist that statistisical analiysis of more than two groups requires ANOVA and post ANOVA tests.

I disagree. ANOVA imparts no information whatsoever except, perhaps, there might be a difference somewhere. ANOVA does not show where differences might lie. More importantly, ANOVA might say ' there is no difference' when, in fact, a difference i ...[text shortened]... t, t-tests are the best method of statistical analysis no matter how many groups are involved.
1st basic question:
What do you want to test?
3. 03 Dec '08 01:23
Originally posted by znsho
Many scientists insist that statistisical analiysis of more than two groups requires ANOVA and post ANOVA tests.

I disagree. ANOVA imparts no information whatsoever except, perhaps, there might be a difference somewhere. ANOVA does not show where differences might lie. More importantly, ANOVA might say ' there is no difference' when, in fact, a difference i ...[text shortened]... t, t-tests are the best method of statistical analysis no matter how many groups are involved.
In any situation where there are 2 or more factors in the experiment, a t-test will not give you the full story and will often mislead you as to the real locus of effects. t-tests in the absence of an ANOVA main effect in these situations are unwarranted, as they can point to effects in one factor while completely ignoring the other factor.

In a 1 factor situation, if your experiment is well predicated, you will be expecting some sort of consistency across your (say) 3 groups. If you do a t-test between groups 1 and 2 and ignore the effect of group 3, that could be misleading again. If, for example, group 3 is your control, and group 1shows the effect against group 2, but not the control, then there really is no effect of interest. Your t-test will produce a misleading result.

Further, to test for "inequality of means", which is the function of the F-test, you'd need 3 separate t-tests. Since you are performing 3 t-tests, you must reduce alpha by a factor of 3 (0.05/3) to control for family-wise error. This correction reduces your chances of finding a significant effect. The proper procedure of first running the f-test and then the post-hoc group comparison controls for this error in a more principled way.

I hope this goes some way to convincing you that there really is a place for ANOVA

;-)
4. sonhouse
Fast and Curious
03 Dec '08 15:45
Originally posted by kyngj
In any situation where there are 2 or more factors in the experiment, a t-test will not give you the full story and will often mislead you as to the real locus of effects. t-tests in the absence of an ANOVA main effect in these situations are unwarranted, as they can point to effects in one factor while completely ignoring the other factor.

In a 1 factor si ...[text shortened]...

I hope this goes some way to convincing you that there really is a place for ANOVA

;-)
Heck, and here I thought Anova was when a star blew up🙂
5. 03 Dec '08 22:27
Originally posted by kyngj
In any situation where there are 2 or more factors in the experiment, a t-test will not give you the full story and will often mislead you as to the real locus of effects. t-tests in the absence of an ANOVA main effect in these situations are unwarranted, as they can point to effects in one factor while completely ignoring the other factor.

In a 1 factor si ...[text shortened]...

I hope this goes some way to convincing you that there really is a place for ANOVA

;-)
I do not agree. Post ANOVA tests are all t-tests. I use pooled estimates of error so I am not ignoring the influence of the third, 4th groups etc.
6. 03 Dec '08 22:29
Originally posted by Palynka
1st basic question:
What do you want to test?
Well for example, effect of different doses of a drug on a particular function.
7. coquette
04 Dec '08 06:36
Honestly, is there a significant difference? Just ask Tukey.
8. Palynka
Upward Spiral
04 Dec '08 10:49
Originally posted by znsho
Well for example, effect of different doses of a drug on a particular function.
For that case, then a paired t-test should perform well. But there are many cases (e.g. with more factors and groups) where it might be misleading.
9. 04 Dec '08 14:55
Originally posted by znsho
Well for example, effect of different doses of a drug on a particular function.
OK, let's assume you have 3 levels of a drug 5ml/kg, 10ml/kg and 15ml/kg, and you have a placebo. If you run the f-test and find no significant value, what does that tell you?

It tells you that the profile of means is flat, that is, that the effects of adding a drug are null. If you then went ahead and did a t-test between 10ml and 15ml, and found a sig. effect, what would that tell you?

It tells you that (say) 15ml has a greater effect on function than 10ml. BUT, in the absence of a significant main effect, this is completely meaningless, because it fails to take into account the fact that the drug overall had no effect on function (being as there were no mean differences between the four experimental groups as a whole).

You would wrongly conclude that 15ml was a better dose than 10ml, when in fact, neither dose is better than placebo.
10. Palynka
Upward Spiral
04 Dec '08 15:46
Originally posted by kyngj
OK, let's assume you have 3 levels of a drug 5ml/kg, 10ml/kg and 15ml/kg, and you have a placebo. If you run the f-test and find no significant value, what does that tell you?

It tells you that the profile of means is flat, that is, that the effects of adding a drug are null. If you then went ahead and did a t-test between 10ml and 15ml, and found a sig. ef ...[text shortened]... clude that 15ml was a better dose than 10ml, when in fact, neither dose is better than placebo.
To determine if the 15ml is better than the placebo, then a t-test would still be more adequate.

Suppose that the drug only has an effect when in doses equal or above 12ml. Separate t-tests against the placebo would be more efficient at uncovering the effect of the 15ml dose than than a global F-test.

The problem here is that you care more about differences between placebo and drug than between drug doses.
11. 04 Dec '08 19:32
Originally posted by kyngj
OK, let's assume you have 3 levels of a drug 5ml/kg, 10ml/kg and 15ml/kg, and you have a placebo. If you run the f-test and find no significant value, what does that tell you?

It tells you that the profile of means is flat, that is, that the effects of adding a drug are null. If you then went ahead and did a t-test between 10ml and 15ml, and found a sig. ef ...[text shortened]... clude that 15ml was a better dose than 10ml, when in fact, neither dose is better than placebo.
Sorry, but I disagree. It is quite possible that the experimenter investigated does of the drug which, to all intents and purposes, had no effect. It is equall possible that the highest dose is the one where the effect begins to occur. ANOVA will fail to pick up on this. T-test, even with Bonferroni's correctio correction COULD PICK UP THE GENUINE DIFFERENCE IN THE single GROUP.
12. 04 Dec '08 19:33
Originally posted by Palynka
To determine if the 15ml is better than the placebo, then a t-test would still be more adequate.

Suppose that the drug only has an effect when in doses equal or above 12ml. Separate t-tests against the placebo would be more efficient at uncovering the effect of the 15ml dose than than a global F-test.

The problem here is that you care more about differences between placebo and drug than between drug doses.
Yes, that is my point. The ANOVA F-value is meaningless.
13. Palynka
Upward Spiral
04 Dec '08 23:05
Originally posted by znsho
Yes, that is my point. The ANOVA F-value is meaningless.