Nine Anecdotes? Surely, That Amounts to Data!

It was a distressing thing to see Tyler Cowen linking to this bit of fluffy rubbish. (Fluffy rubbish that has a conclusion, by the way, with which I agree. So I am certainly not calling "nonsense" here because I want to shoot down an opposing view.)

Let me make an analogy for the "study" Fearon has performed: Imagine someone has developed a brand new diet, radically different than anything people had ever eaten before, and you say you have done a study that shows its effectiveness.

"Oh," I ask, "with how many subjects?"

"Yeah, well... nine."

"Nine. That's really not very many, is it? But I suppose you got either uniform improvement or uniform worsening, so you feel confident concluding something from such a small sample?"

"Well, no, actually, five of the subjects improved, two got worse, and one... well, one had always been on the diet, at least since the study started."

"Jeez, looks like a result that could be random. But I suppose you were able to carefully control for all other conditions?"

"Urr, well, actually, each of the subjects were operating under extremely varied conditions, no two cases really remotely alike except that they were each people on this diet, and I had no control whatsoever over any other factor that might have influenced the outcome. Nor do I even have any clue how to incorporate those myriad other factors into my study -- in fact, I don't even know which ones I should include if I wanted to include them and if I knew how!"

"So your 'study' is completely worthless?"

"But... but... I have numbers! I even have a bar chart!"

Some people are mesmerized by numbers and statistics.