More on alpha levels
The α = .05 cutoff for "significant" results is a case of spurious "objectivity" trumping scientific judgment.
The fact that scientists have an "objective" standard to adhere to gives the appearance of being more rigorous. But consider another objective way of deciding between the "null hypothesis" and the hypothesis being tested: flip a coin. Heads, we reject the null hypothesis, tails we don't. Completely objective! We could videotape the coin flip, and all sane observers could agree as to whether we got heads or tails.
Next, think about the following two cases:
But in the first case, there was only an 8% probability our correlation was by chance, and given that we have a great causal explanation of how reckless driving could generate early death... well, "So what?" that 8% of the time this result might have been due to chance. There is a 92% probability that the correlation wasn't chance!
And in the second case, given how unlikely it is that there is a causal connection here, why wouldn't we think that almost certainly, we just happened to get one of those 4% of samples that are outliers?
Of course, I haven't made a new discovery here: people who really understand statistics recognize the above: in fact, I've learned to think about these things this way from skilled mathematicians. But there are boatloads of naive creators of and consumers of statistical studies who are oblivious to these points.
And this is especially prevalent when the studies in question deal with some hotly contested political or social matter. If a study on gun control finds a correlation between gun violence and permissive gun ownership laws only at p = .051, you can be certain that some gun rights group will announce that the studies "proves" that gun control has no effect on gun violence. (And a group on the other side of this issue will do the same if a similar study can be taken to support their stance.)
The fact that scientists have an "objective" standard to adhere to gives the appearance of being more rigorous. But consider another objective way of deciding between the "null hypothesis" and the hypothesis being tested: flip a coin. Heads, we reject the null hypothesis, tails we don't. Completely objective! We could videotape the coin flip, and all sane observers could agree as to whether we got heads or tails.
Next, think about the following two cases:
- We do a study and find that reckless driving correlates with early death with p = .08 (greater than α). We are told to accept the null hypothesis: there is no significant correlation.
- We do a study and find that sunspot activity correlates with American League victories in the World Series with p = .04 (less than α). We are told to reject the null hypothesis: there is a significant correlation.
But in the first case, there was only an 8% probability our correlation was by chance, and given that we have a great causal explanation of how reckless driving could generate early death... well, "So what?" that 8% of the time this result might have been due to chance. There is a 92% probability that the correlation wasn't chance!
And in the second case, given how unlikely it is that there is a causal connection here, why wouldn't we think that almost certainly, we just happened to get one of those 4% of samples that are outliers?
Of course, I haven't made a new discovery here: people who really understand statistics recognize the above: in fact, I've learned to think about these things this way from skilled mathematicians. But there are boatloads of naive creators of and consumers of statistical studies who are oblivious to these points.
And this is especially prevalent when the studies in question deal with some hotly contested political or social matter. If a study on gun control finds a correlation between gun violence and permissive gun ownership laws only at p = .051, you can be certain that some gun rights group will announce that the studies "proves" that gun control has no effect on gun violence. (And a group on the other side of this issue will do the same if a similar study can be taken to support their stance.)
Gene,
ReplyDeleteIn general I'm totally on board with your posts on this stuff, but I think you might be overstepping here. If you can show me I'm wrong, I will be glad to have learned something new. (Really.)
You wrote:
We do a study and find that reckless driving correlates with early death with p = .08 (greater than α). We are told to accept the null hypothesis: there is no significant correlation.
The correct thing to say is, "We cannot reject the null." I think most people are pretty good about that, and don't say, "We accept the null."
You wrote:
If a study on gun control finds a correlation between gun violence and permissive gun ownership laws only at p = .051, you can be certain that some gun rights group will announce that the studies "proves" that gun control has no effect on gun violence.
Are you sure? Can you find me a study (not just on gun violence, on anything) that actually does that?
'The correct thing to say is, "We cannot reject the null."'
DeleteYes, that's correct. Although in the press people treat it otherwise. E.g., when secondhand smoke and cancer only got a p = .09, I saw that studies "showed" no correlation.
"Can you find me a study (not just on gun violence, on anything) that actually does that?"
DeleteI'm not saying the study AUTHORS will claim this: no, they have more statistics education than that.
I'm saying someone will pick up on this if it favors their view and claim it.
I would file this under the tendency towards excessive 'rigorism' -- trying to make a position as defensible as possible, rather than just expressing what one thinks is true.
ReplyDeleteAnd, of course, screwing things up as a result...