Hot Streak Length
The critics of this model claimed "It implies a streak length of one."
Well, it doesn't:
And the output is:
Macintosh:statistics gcallah$ ./hotstreak.py
Shooting with hot streaks:
OOXXOXXOXXOXXXOOXXXOXXXXOOXXXXXXXOOOOOXOXXOXXOOXO
Average hot streak length = 2.0
Shooting without hot streaks:
OOXXOOXOXXOXOXXXXXXOXOOOXOXOXOOOOOOOOOXXOXOXOOOOO
What the model actually codes, and was meant to code, was the possibility that a player could be genuinely "hot" for some period, but if the hot streak might end at any moment, then the streak has no predictive value, and "feeding the hot hand" will not help a team.
Well, it doesn't:
import random
SHOTS = 50
in_streak = False
hot_streaks = 0
hot_total = 0
print("Shooting with hot streaks:")
for shot in range(1, SHOTS):
hot = (random.random() < .5)
if hot:
hot_total += 1
if not in_streak:
in_streak = True
hot_streaks += 1
make = (random.random() < .66)
else:
in_streak = False
make = (random.random() < .33)
mark = 'X' if make else 'O'
print(mark, end='')
print("")
print("Average hot streak length = " + str(hot_total / hot_streaks))
print("Shooting without hot streaks:")
for shot in range(1, SHOTS):
make = (random.random() < .5)
mark = 'X' if make else 'O'
print(mark, end='')
print("")
And the output is:
Macintosh:statistics gcallah$ ./hotstreak.py
Shooting with hot streaks:
OOXXOXXOXXOXXXOOXXXOXXXXOOXXXXXXXOOOOOXOXXOXXOOXO
Average hot streak length = 2.0
Shooting without hot streaks:
OOXXOOXOXXOXOXXXXXXOXOOOXOXOXOOOOOOOOOXXOXOXOOOOO
What the model actually codes, and was meant to code, was the possibility that a player could be genuinely "hot" for some period, but if the hot streak might end at any moment, then the streak has no predictive value, and "feeding the hot hand" will not help a team.
I understand that the intent is to use a model to demonstrate how hot streaks can be real, but undetectable when looking just at outcomes.
ReplyDeleteWhat we're contesting is that your model is actually a model of hot streaks.
I'm going out on a philosophical limb here, but to me the "with" and "without" models appear to be the same model, only differing in the symbols used to express them. Adding an additional random variable that "does no work" doesn't show how hot streaks could exist in any non-abstract sense. And the point of models is to represent real-world, concrete phenomena, right?
(I am prepared to be brutally shot down here since you've actually studied the philosophy of modeling)
If hot
DeleteFor k = 1 to streak_length
Take shot
Currently the streak length is always 1.
Your model point is right, but I am on vacation so I'll just leave it at, yes Matt.
(
And I have studied mathematical model theory)
Gene, do you think that it is plausible, in any theory of how streaks work, that the probability that a streak will continue would be exactly the same as the probability that a steak will start?
ReplyDeleteThat is the feature your critics are referring to when they say streaks are of length one.
"Gene, do you think that it is plausible, in any theory of how streaks work, that the probability that a streak will continue would be exactly the same as the probability that a steak will start?"
DeleteWhy not? In any case, this is an existence proof. It doesn't have to be plausible.
"That is the feature your critics are referring to when they say streaks are of length one."
But that does NOT make them of (necessarily) length one.
"Why not? In any case, this is an existence proof. It doesn't have to be plausible." OK, but would you be satisfied with someone saying "While it's theoretically possible that both hot hands and leprechauns exist, neither are very likely given the available evidence."?
Delete"But that does NOT make them of (necessarily) length one." Well, the point people are trying to make is that the way your model works, a streak at time t and a streak at time t+1 are effectively two different streaks, there's nothing connecting them that would justify calling them one streak.
As independent events they are.
DeleteExistence proof? Of a definition? Are you proving there is a definition D such that if D defines hot streak it is possible to construct a sequence such that hot streak sequences are random real numbers? (Technical term). The question is does the definition fit the intuition?
It might help to have you give a careful definition of what you mean by a hot streak. I have a particular point in mind. Keshav is getting to it, indirectly.
"Well, the point people are trying to make is that the way your model works, a streak at time t and a streak at time t+1 are effectively two different streaks, there's nothing connecting them that would justify calling them one streak."
DeleteThat is simply a way of programming "a streak that could end at any moment." I can change the program if you guys would like, so that the streak "stays on" unless (with p = .5) it "gets turned off."
Would you then be happy to say it is one streak?
Well, the issue is not the way it's expressed, the issue is the fundamental logic of the program. There is no connection between the fact that a streak is on at time t and the fact that a streak is on at time t+1. It's just a coincidence. That's what makes people say that streaks are of length 1.
DeleteGene, you are NOT modeling a "hot hand". You are modeling "about to have a single good shot". No-one denies that performance can vary. We are denying that you have modeled a hot hand. Your response has been to redefine hot hand. The hot hand is the claim that performance events are NOT independent. In your model they ARE independent. Whatever you are modeling it is not a hot hand. The discussion about length 1 is jut a proxy for that. Independent strings of length 1 are independent when concatenated . Independent strings of length greater than 1 that are not themselves internally independent are not independent when concatenated. To model a hot hand you must either make the chance of success dependent on the prior event, either directly or indirectly. A streak variable is a perfectly sensible mechanism, an indirect dependence due to an underlying cause, as long as there is still the required dependence of performance on *prior* performance. What you must do is exhibit a model where the results are dependent but the result strings show no sign of it. Your model does not.
DeleteTo prematurely let a cat out of a bag, try it with the chance of being hot equal to 3%.
Delete"The hot hand is the claim that performance events are NOT independent. "
DeleteF'ing a, Ken, how many times are you going to repeat to me THE DEFINITION WITH WHICH I AM DISAGREEING????????!!!!!!!!!!!
This is literally the stupidest conversation I've ever had:
G: I disagree with the TGV definition of a hot hand. Here...
K: Gene, you not following the TGV definition of a hot hand.
G: I know Ken, I am DISAGREEING with it! Take a look at this...
K: But Gene, that doesn't follow the TGV definition of a hot hand.
G: Yes, Ken, that is because I DISAGREE with their definition.
K: Well, Gene, by definition a hot hand is [cites TGV definition].
G: Yes, Ken, I know. That is the very definition I am disputing.
K: Well, you are redefining a hot hand.
G: Yes, Ken, THAT IS WHAT I SET OUT TO DO.
K: Well, to model a hot hand, you have to use [the TGV definition].
G: Well, Ken, accept I AM DISPUTING THAT DEFINITION...
And on and on for what, ten days now?
"That's what makes people say that streaks are of length 1."
DeleteWell, they are wrong to mean that!
Gene, just as you are disagreeing with the TGV definition, Ken B is disagreeing with your definition. He's saying that your definition does not match what people think of when they hear the words hot hand or winning streak. Certainly if you define "hot hand" as "brown cow", then hot hands exist, but that is cheating. Now your definition isn't as egregious as "brown cow", but Ken B is arguing that it's still egregious.
Delete"Ken B is disagreeing with your definition..."
DeleteIf THAT is what he's been doing, he's been very muddled about it!!
Gene,it's not the TGV definition. It's the definition. That's why I gave the wiki link. You can dispute their analysis, what they took the definition to imply as an operational matter or a measurable result. But it's not their idiosyncratic definition.
DeleteGene, I asked you to carefully define what you mean by hot hand. That's muddled? I said explicitly that what you are modeling in your code is not the "hot hand". That's muddled?
DeleteKeshav:
Delete1) If Ken has understood this to be about competing definitions all along, why after a week of discussion would he suddenly point out to me "you're redefining a hot hand," as though this were not MY EXPLICIT GOAL from the start?!
2) "He's saying that your definition does not match what people think of when they hear the words hot hand or winning streak."
a) This is only a problem if someone is "sneaking" in a new definition. It was NOT a problem for Copernicus when his new definition of planet did not "match what people think when they hear the word planet."
b) In any case, it is plain WRONG to think that "most people" when they think of a hot streak think of a statistical process with hysteresis: most people do not think like statisticians!
"Gene,it's not the TGV definition. It's the definition. That's why I gave the wiki link. "
DeleteWhere the H-E-double-toothpicks do you think Wiki got their definition from?! Have you actually READ the TGV paper? They take great care to put forward this definition. Why? Because before their paper, their definition didn't exist!
Here is Wikipedia:
Delete"The fallacy was first described in a 1985 paper by Amos Tversky, Thomas Gilovich, and Robert Vallone."
So "the definition" COMES from TGV.
Ken to Copernicus: "You are not disputing the Ptolemaic definition of a planet. It's just THE definition of a planet!"
On:
ReplyDelete'What the model actually codes, and was meant to code, was the possibility that a player could be genuinely "hot" for some period, but if the hot streak might end at any moment, then the streak has no predictive value, and "feeding the hot hand" will not help a team.'
I thought I had understood and was in agreement with what Gene was trying to demonstrate in this series of posts. But I don't see how the fact that a steak may end at any moment is sufficient to disallow if from providing predictive value.
If you modify the model so that the player still might end his streak at any moment but with a 1% rather than a 50% probability (a 1% chance of a state switch before each sot) then the conversion rate is still .5 in the long run but will occur in clusters of good and bad runs that would allow you to beat the house if you were offered .5 odds on every shot.
Isn't what makes steaks have no predictive value rather that the chances of being in a streak on any particular shot are independent of the outcome of the previous (or any previous) shot?
I mean of course beat the house just by statistical analysis with no knowledge of the programs internal state.
ReplyDeleterob, how could one even "be" the house with no knowledge of the program: how would one know the odds were 50-50? I could have made them 50-50 for a while, and then made it all heads after trial 100!
DeleteThe whole "inside knowledge" bit is a distraction: the *real* point is that if you know *more* about a situation, "the odds" change!
I think you missed my point.
ReplyDeleteI was querying your statement "if the hot streak might end at any moment, then the streak has no predictive value,".
Just knowing that a streak might end does not mean it has no predictive power.
I think what eliminates predictive power is the chances of being in a hot-hands state on any particular shot being independent of the outcome of the previous (or any previous) shot.
IMO a more accurate statement would have been 'but if the hot streak might end at any moment [rob: with a probability before each shot equal to (1 - the chances of entering the streak)], then the streak has no predictive value, and "feeding the hot hand" will not help a team'
Delete(This assume that by "the streak" you mean "being in the hot hand state".)
The obverse of the supposed hot hand fallacy is the slump fallacy. Let's talk about cold hands. No-one denies some things can cause a player to perform badly in the immediate future. If a skeet shooter shows up with broken glasses no one denies he will miss a few. Agreed? There is no alleged fallacy here. Similarly if the skeet shooter just got new glasses no one would deny he will likely do better in the next few shots. There is no alleged fallacy of predicting performance from information other than his latest performance. But that's what your argument is directed at. IF, you say, were define "looking at information aside from the strings of Xs and Os and drawing inferences from that information" is what we mean when we say a player has a hot hand THEN there is no fallacy . But that's vacuous since no one ever said there was a fallacy there.
ReplyDeleteKeshav put it well.
Just re-read TGV. You are wrong. They conflate two issues:
Delete1) Do streaks have predictive value? AND
2) Do players get "hot"?
They continually imply that if 1) is untrue, then 2) is false.
I am prying apart 1) and 2) and showing that 1) does not imply 2).
I am QUITE DELIBERATELY not addressing 1). But for a week now, you keep complaining "But Gene, you're not addressing 1)!"
Just perusing TGV I came across this:
Delete"If, on certain occasions, a player can predict a “hit” before taking a shot, he or she may have a justified sense of being “hot” even when the pattern of hits and misses does not stray from chance expectation. We tested players’ ability to predict hits and misses by having them bet on the outcome of each upcoming shot. "
Am I mis-reading it or does this indicate they actually did test for hot-hands in your sense ? This test actually came up negative.
Yes, this is a TERRIBLE test for what I am talking about! "Let's take the player OUT OF the hot hand state by asking them to formulate a probability statement instead of being "in the zone," and then use the results of our ending the hot hand state to prove it doesn't exist!"
DeleteWhich is exactly what I object to in their paper! So they absolutely did NOT test for hot hands in my sense!
Well it may be a terrible test.
DeleteBut nevertheless it definitely is a test that attempts to validate whether the claim that players' perception of being "hot" at certain times is just an illusion or not.
I had the impression you were claiming that TGV just looked for statistical evidence that players were more likely to score when on a streak, and when they didn't find it you were saying they rejected the idea of "hot hands" . This section of their paper shows they went beyond that (no matter how unrealistic their test may actually have been) and looked at the possibility of hot hands 'even when the pattern of hits and misses does not stray from chance expectation' which is very close to what you are talking about in these posts.
I have now read the paper. The section "Test of Predictability" directly addresses the question of players "feeling hot" independent of any good run of results they may be on and found no statistically significant difference in results when they had the "hot" feeling.
ReplyDeleteNow Gene can accept or reject these results.
But I do not think he is correct to say 'they continually imply that if 1) is untrue, then 2) is false.' since they have a separate test for 2).
(I found no evidence that they make any statements elsewhere in the paper either to backup Gene's claim - but perhaps my reading comprehension skills let me down.)
They used the *exact same* test: predicting the next shot!
DeleteAnd when players are hot, they often described as being "unconscious". What could kill that vibe better than stopping them, asking them if they're hot, and asking them to predict their next shot?!
"I found no evidence that they make any statements elsewhere in the paper either to backup Gene's claim - but perhaps my reading comprehension skills let me down.)"
DeleteYes, it did, since the exact section you cite IS them using the predictability test to debunk the idea that such a thing as "being hot" exists!
You said:
Delete'They conflate two issues:
1) Do streaks have predictive value? AND
2) Do players get "hot"?
They continually imply that if 1) is untrue, then 2) is false.'
By 1) I assume you mean predictive value of the "if the player has got 5 baskets in row does that increase the odd of the 6th going in?" variety.
By 2) I assume you mean "if the player is hot even for a single shot then he is more likely to score than if he wasn't hot". If anyone had insight into this state of "hotness" it would also have predictive value (this is how you won money of of Ken and Me in one the earlier posts, right?)
No matter how flawed their tests may have been they did run independent set of tests for "hot-hands" of both type 1 and 2.
This led them (perhaps erroneously in both cases) to reject either type of hot-hands. But they did not (as you claim) reject 2 just because they rejected 1.
They obviously considered the possibility of 1 being unproven but 2 still being true. This means they accepted that 'the findings "debunking" hot hands [rob: of type 1] are all entirely consistent with the actual existence of hot hands [rob: of type 2].'. (this is a quote from your first post on the topic).
Its just that their (possibly flawed) tests didn't find evidence for type 2 hot hands either.
"No matter how flawed their tests may have been they did run independent set of tests for "hot-hands" of both type 1 and 2."
DeleteNO THEY DIDN'T!!! They asked a player to PREDICT his next shot.
So they did TWO TESTS OF WHETHER THE HOT HAND HAS PREDICTIVE VALUE. Like I said, they conflated 1 and 2.
I am genuinely confused now. May I ask some clarifying questions ?
Delete1. TGV devised the tests for the possible existence of "hot hands" in a scenario 'when the pattern of hits and misses does not stray from chance expectation'. Do you agree that they are at least trying to test for the hot hands in the scenario modeled by your python program ?
2. Had the players in the TGV tests been able to use their knowledge of their own "hot hand" status to predict baskets often enough to make more money in the game than pure chance would allow would this not have been a valid "proof" that hot-hands exits ? If not, why not ?
3. Your python program claimed to have developed a model where "Hot streaks are statistically undetectable; and offer no predictive leverage for a player's next shot.". I had taken that to mean "next shot after a a basket". If you actually meant "next shot after entering hot status" then someone with knowledge of the programs inner state would have a predictive advantage in calling the players next show wouldn't they ? And if so, why would that be an invalid thing to test for ?
- Just for reference can you give an example of a test for hot-hands (of the second variety) that would not in some way rely on the the enhanced predictive powers of someone with knowledge of the hot-hands state ?
Gene, they tested the following two hypotheses:
Delete1. Do hot hands have predictive value for those who have no inside information?
2. Do hot hands have predictive value for those who have inside information on whether a hot hand is on or not?
They found that the answer was no in both cases. Now you may think their test of 2 was flawed, but they did attempt to test both.
(There is of course a possibility that hot hands exist but don't even have predictive value for those with inside information, but even you are not arguing for that.)
If all Gene was saying was that TGV made a strong claim rejecting hot hands based on what may been a false negative resulting from a badly constructed test I would agree with him.
DeleteBut I see no grounds for his charge that the 2 very different tests they ran were "conflated" just because they both use predictive techniques.
Rob, I never said the two tests were conflated!
DeleteOK - fair enough - you said the 2 issues were conflated not the tests for the 2 issues.
DeleteSo I should have said:
'I see no grounds for his charge that the 2 issues are conflated or that the very different tests they ran on them are evidence of this conflation"
No, but they didn't test two issues: in both cases, their test was" "Do hot hands predict?"
DeleteI said they had conflated "Do hot hands predict?" and "Do hot hands exist?"
Two tests of whether they predict do not test these two separate questions!
Remember, their second test was still "Could the player himself predict his next shot?"
DeleteThat IS different than asking if an outsider can predict it, but it is still testing question 1, and not question 2.
But Gene, do you at least acknowledge that in your model, the question "Do hot hands exist?" and the question "Do hot hands have predictive value for those with inside information?" are basically equivalent?
DeleteThe 2 questions are:
Delete1) Do streaks have predictive value? AND
2) Do players get "hot"?
The test used for 2) was on whether a player (or his partner) could predict single shots going in at a rate above random. I do not see that this has anything to do with whether "streaks have predictive power" since "steaks" and "single shots" are different things.
'But Gene, do you at least acknowledge that in your model, the question "Do hot hands exist?" and the question "Do hot hands have predictive value for those with inside information?" are basically equivalent?'
DeleteWhy would I acknowledge such nonsense?!
"2) was on whether a player (or his partner) could predict single shots going in at a rate above random."
DeleteBecause they were "hot," i.e., on a streak. Look, TRY to behave, or I'm just going to delete all future comments.
Gene, in your model hot hands have predictive value for those who have inside information about whether the hot hand is on. Do you believe that it's possible to make a model where hot hands exist but have no predictive value for those who have inside information about whether the hot hand is on?
DeleteSomeone want to explain to me what you guys are arguing about?
ReplyDeleteWell, other than all of our actual comments, what more would you like to see?
DeleteGene, I just thought of a potential way to test your version of hot hands. Make it so that the player themselves doesn't know whether when they did well and when they didn't. (I'm not sure which sports if any this would be practical in.) And then subsequently ask them "At what moments were you hot", and try to see whether that correlates with the results.
ReplyDeleteGene,
ReplyDeleteIt might help for you to resolve your dispute with Ken B. (and poor rob) if you did a separate post explaining how you would react to researchers who published a paper like this:
"Abstract: It is well-known that players at the craps table are said to have a 'hot hand' after several advantageous rolls. The rollers themselves often report subjectively feeling 'in the zone' during streaks of successful rolls. However, using both Monte Carlo simulations and Bayesian inference models, we conclude that such 'patterns' are illusory and provide no operationally useful betting opportunity."
I made my response into a top level post.
DeleteWhile it may be hard to believe based on the way he generally responds to my comments - I actually don't think I had a serious dispute with Gene on this stuff beyond a view that he is perhaps a bit unfair to TGV who I think do address (albeit inadequately) his scenario of the possibility of hot-hands that may be statistically undetectable.
ReplyDeleteI have been on vacation, and have not caught up on these threads, but I had a realization. In your definitions a man who misses 10 in a row could be on a "hot streak" and a guy who misses 10 in a row on a cold streak. Which I think more or less answers the question, are Gene's definitions really odd and idiosyncratic?
ReplyDeleteIt's as though glass could be brittle without actually flying into fragments at every moment!
Delete