How to Go Wrong with Empirical Studies

After just calling for empiricism in business cycle studies, I should sound a cautionary note, lest someone pop up and comment, "Have you forgotten everything you wrote in the early oughts?"

Tyler Cowen links with apparent approval to a study the strikes me, at least in terms of the relevant sport with which I am familiar, as seriously misguided. Here is the quote Tyler extracts:

Theoretical analyses and empirical studies have revealed that conflict escalation is more likely when individuals are more similar in resource-holding potential (RHP). Conflicts can also occur between groups, but it is unknown whether conflicts also escalate more when groups are more similar in RHP. We tested this hypothesis in humans, using data from two professional sports competitions: football (the Bundesliga, the German first division of football) and basketball (the NBA, the North American National Basketball Association). We defined RHP based on the league ranks of the teams involved in the competition (i.e. their competitive ability) and measured conflict escalation by the number of fouls committed. We found that in both sports the number of fouls committed increased when the difference in RHP was smaller. Thus, we provide what is to our best knowledge the first evidence that, as in conflicts between individuals, conflicts escalate more when groups are more similar in RHP.

The problem I see is that the authors have a seriously mistaken notion of the nature of basketball fouls. Most basketball fouls are not "aggressive" in the usual meaning of the term at all: that is why it makes sense for announcers to say, "Wow, that was an aggressive foul." Players only occasionally deliberately foul; instead, they miscalculate in an attempt to block a shot, steal the ball, penetrate into the lane, and so on, and accidentally make contact with their opponent. So fouls occur when players are pushing the edge, or trying harder, and naturally that occurs more often in close games. You are an NBA center, and one of your opponents is driving the lane, going for a layup. If the game is close and this layup could make all the difference in the outcome, you rotate off of your man and try for the block, risking a foul. If you are up 30 or down 30, you save your effort for when it will really matter.

So what the authors' study shows, in regards to basketball, is that players try harder in close games. Whew, good thing we have social science out there, or who would have realized this?

Note: This post is not against empirical studies. It is against naive empirical studies that don't test what they think they are testing. Here is how the authors should have done their study: they ought to have measured flagrant fouls and fights, not fouls simpliciter.

UPDATE: The comment section of Cowen's post has many objections, some similar to mine and some noting other important lacunae in the study. The most significant is from Ryan Miller, who notes that the authors apparently did not try to control for strategic fouling to extend the game by a team losing in the final minute of a close game. Such fouls have no relationship to "aggression" whatsoever.