Hypnotized by the Margin of Error
This cartoon, while funny, got me thinking about how people become hypnotized by the margin of error for some statistic without understanding what it means. For instance, the cartoon writer seems to think that a movement in a poll that is "within the margin of error" is meaningless. Well, it could be meaningless, but if it was big enough to report, it probably is not. Let's say the margin of error on a poll is +/-4%, and the poll just moved 3%. If the poll was basing that on a 95% confidence level, that would mean (roughly) that there was only about a 10% chance that a move of that size would occur purely randomly. Ninety percent of the time that movement is meaningful! Similarly, if a poll reports Joe ahead of Mary by 3% with a margin of error of 4%, that does not mean the race is a "statistical tie"! It means there is a 77% chance Joe is really ahead; in fact, he could easily be up by 6 or 7%.
And the use of a 95% confidence level as a simple binary switch is maddening as well. If a study shows no significant correlation between, say, second-hand smoke and cancer at a 95% confidence level, but would show one at a 94.9% level, the result is just barely different from one that shows a correlation at the 95% level but wouldn't at a 95.1% level. The studies, in fact, are in remarkable agreement, but they are likely to be reported as contradicting each other!
And the use of a 95% confidence level as a simple binary switch is maddening as well. If a study shows no significant correlation between, say, second-hand smoke and cancer at a 95% confidence level, but would show one at a 94.9% level, the result is just barely different from one that shows a correlation at the 95% level but wouldn't at a 95.1% level. The studies, in fact, are in remarkable agreement, but they are likely to be reported as contradicting each other!
You, me and Kevin Drum. We will win this war.
ReplyDeleteActually we won't, because it's in too many people's interests (in media and politics) to be strategically stupid about what "margin of error" does and does not mean.
ReplyDeleteLike, yes, if you're beating my by 3 in a poll with a 4% margin of error, there's a chance I'm "really" beating you by 1. But there's exactly as good a chance that you're "really" beating me by 7.
The thing is, that's no fun for journalists to report, and admitting that for every poll result you'd rather be ahead than behind cuts against the fundraising interests of both underdogs and overdogs. So we're probably stuck with the stupid "statistical tie" concept forever.
95% may be arbitrary but anyone who looks at the study itself who discovers the 94.9% level might re-run the study (or finesse the sample data) in hopes of getting the headline.
ReplyDeleteFrom what I understand from meta-surveys of studies, this probably does happen.
And there's nothing stopping someone or some group from using some other (stricter) Confidence Interval for internal purposes instead of whatever is traditionally used in their field.
I personally use 99.9% when looking at studies. Even then, I'm suspicious of the study construction and sample if the study is epidemiological.
I highly recommend this table made by Kevin Drum:
ReplyDeletehttp://www.motherjones.com/kevin-drum/2012/10/one-last-encore-great-statistical-tie-fallacy