Kahneman's Muddles: The Hot Hand "Fallacy"
I'm beginning to detect a pattern of the sort Kahneman likes to find, in his errors. Kahneman talks about times when someone is asked a hard question ("Will Obama win re-election in 2012?" asked in 2011) but substitutes an easier question without realizing it ("What are Obama's poll numbers today?"). Kahneman suffers from the reverse cognitive problem: he asks (or sees someone asking) a narrow question, but when the answer is found, he thinks it is the answer to a broader question.
Thus, Kahneman finds the answer to the narrow question, "Do people see patterns in things that aren't really there?" which is "yes," but he thinks he has found the answer to the much broader question, "Is the word more or less orderly than we suspect it is?"
Similarly, Kahneman's colleague, Amos Tversky, and two co-authors ask the question, "Does a player having made a series of shots in a row in basketball imply a higher likelihood her next shot will be made?" They find the answer is "no." All well and good, and an interesting and useful bit of research: in that it is accurate (and I don't doubt that it is!), it means, for one thing, that there is little sense in trying to feed the player with the hot hand (if it exists). But they (and Kahneman) mistakenly think they have answered the broader question "Do hot hands exist?"
They have done no such thing. What they have shown is that hot hands, if they exist, are of no predictive value. And that finding is totally consistent with at least two hypotheses:
1) Hot hands don't exist; and
2) Hot hands exist, but are likely to end at any moment, and there is no way of knowing when that moment will arrive.
And just a little bit of thought should suffice to see that the second hypothesis is much more plausible. Kahneman himself explains why without realizing it: hypothesis 1 means that while skill differences between players exist, temporal differences in one player's performance are purely random. And that is pretty damned unlikely, don't you think? It would mean, for instance, that if a guy spends the night out doing shots and blow and comes straight to the game still plastered, and then does worse than he did a day on which he was well rested and sober... well, random chance!
Thus, Kahneman finds the answer to the narrow question, "Do people see patterns in things that aren't really there?" which is "yes," but he thinks he has found the answer to the much broader question, "Is the word more or less orderly than we suspect it is?"
Similarly, Kahneman's colleague, Amos Tversky, and two co-authors ask the question, "Does a player having made a series of shots in a row in basketball imply a higher likelihood her next shot will be made?" They find the answer is "no." All well and good, and an interesting and useful bit of research: in that it is accurate (and I don't doubt that it is!), it means, for one thing, that there is little sense in trying to feed the player with the hot hand (if it exists). But they (and Kahneman) mistakenly think they have answered the broader question "Do hot hands exist?"
They have done no such thing. What they have shown is that hot hands, if they exist, are of no predictive value. And that finding is totally consistent with at least two hypotheses:
1) Hot hands don't exist; and
2) Hot hands exist, but are likely to end at any moment, and there is no way of knowing when that moment will arrive.
And just a little bit of thought should suffice to see that the second hypothesis is much more plausible. Kahneman himself explains why without realizing it: hypothesis 1 means that while skill differences between players exist, temporal differences in one player's performance are purely random. And that is pretty damned unlikely, don't you think? It would mean, for instance, that if a guy spends the night out doing shots and blow and comes straight to the game still plastered, and then does worse than he did a day on which he was well rested and sober... well, random chance!
"[H]e asks (or sees someone asking) a narrow question, but when the answer is found, he thinks it is the answer to a broader question."
ReplyDeleteMuch like a guy asking about the Jets vs. Rams, but then thinking it is the answer to, "Can mind-boggling incompetence defeat astounding ineptitude?"
I think Gene what he is doing -- and I have not read his book, just other versions of the argument -- is knocking the pins out from the most cited bit of evidence for a claim with no other support. Say I prove the grassy knoll was empty. Won't disprove the conspiracies but surely undercuts them.
ReplyDeleteBut there is tons of other support for "in the zone" performances, including support that Kahneman himself cites as valid.
DeleteI agree with your point that the research isn't strong evidence of the lack of a temporal correlation in skill. But at points you sounded dangerously close to saying, "Oh, hot hands exist, you fool, the data just happen to be observationally equivalent to them not existing!" -- generally a sign to privileging a hypothesis.
ReplyDeleteAfter all, if your alternate hypothesis that we are *truly ignorant* of when the hot hand will end, then the concept is useless and does not advise any different policy than ignoring it. But you need something stronger: that we are at least a little better than ignorance and should bet in favor of it being there, since the benefits outweigh the costs (the costs being the loss from giving the ball to a player when their hot hand ends).