News

Loading...

Friday, October 10, 2008

"Winner's Curse" in Publishing?

This is the kind of thing Gene would complain about, so I'm posting it here rather than at Free Advice. In this post, Tyler Cowen summarizes an article that claims:

In economic theory the winner’s curse refers to the idea that someone who places the winning bid in an auction may have paid too much...The same thing may be happening in scientific publishing, according to a new analysis. With so many scientific papers chasing so few pages in the most prestigious journals, the winners could be the ones most likely to oversell themselves—to trumpet dramatic or important results that later turn out to be false. This would produce a distorted picture of scientific knowledge, with less dramatic (but more accurate) results either relegated to obscure journals or left unpublished.


By itself, this is just goofy. There are probably a good 7 things wrong with this theory. (I confess I haven't read the paper, so maybe they address all 7.) Here are some big problems, some of which I thought of myself, and others which people at MR came up with:

* As with other nifty things, like the criterion of falsifiability, this idea above falls apart if you apply it to itself. Or, as I put it in the comments: "I have tried for years to get a paper published that says the refereeing process tends to select papers with true hypotheses, but no journal was interested."

* This really isn't the winner's curse. So even if it's true that "the most interesting papers are probably wrong"--which is how either Steve Landsburg or Bryan Caplan (it was one of the two but I can't remember which) phrased it several years ago--that's not really the winner's curse. As an MR comment explained: "[T]he winner's curse applied to papers would be that published papers present more information than was necessary to get published, IE they could have published 2 papers instead of the one. Therefore, I refuse to read the paper because I disagree with the conclusion from the snippet Tyler posted... Why does the extra information have to be incorrect? Doesn't the Journal check the validity of the information before it is published? (And I'm not talking about the economist.)"

* The guy's comment quoted above actually had two points, the second of which I handle here: Even given the tendency for referees to be more interested in "unexpected" results, why can't they be aware of that and scrutinize the paper more carefully? E.g. in the actual winner's curse literature, in Nash equilibrium rational bidders don't overbid. They are aware of the forces that would lead a group of naive people to consistently fall prey to the winner's curse, and so they adjust for it. So do economics referees not understand game theory? Maybe they don't, but the point is, you can't do a model of the refereeing process that yields this perverse outcome, unless you plug in that the referees (in the model) are morons. So yeah, no kidding if the referees are morons, then we can't trust the papers they publish.

* Finally, here's a great point someone brought up (and note he is quoting from the article in the beginning):

"Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers in leading journals that had been cited by more than 1,000 other scientists. They were, in other words, well-regarded research. But he found that, within only a few years, almost a third of the papers had been refuted by other studies. For the idea of the winner’s curse to hold, papers published in less-well-known journals should be more reliable; but that has not yet been established."

This sounds to me like

1) We don't know how likely papers in other journals are to be refuted, so we don't have a sense that the ones in highly regarded journals are *unusually* likely to be refuted;

2) Even if they are unusually likely to be refuted, I'd like to see an argument that it's because they're *worse*, not because the extra attention paid to them due to their place of publication has led to more efforts to replicate/refute their results.

8 comments:

  1. Anonymous11:16 PM

    warhammer gold warhammer money warhammer accounts tibia money tibia gold tibia item runescape accounts buy runescape accounts runescape money runescape gold runescape gp runescape power leveling runescape powerleveling cheap rs2 powerleveling runescape equipment buy rs equipment runescape runes cheap rs2 runes runescape logs cheap rs2 logs runescape items buy runescape items runescape quest point rs2 quest point cheap runescape questpoint runescape gold runescape items runescape power leveling runescape money runescape gold buy runescape gold buy runescape money runescape items runescape accounts runescape gp runescape accounts runescape money runescape power leveling runescape powerleveling tibia gold dofus kamas buy dofus kamas wow power leveling wow powerleveling runescape questpoint rs2 questpoint Warcraft PowerLeveling Warcraft Power Leveling World of Warcraft PowerLeveling World of Warcraft Power Leveling Hellgate money Hellgate gold buy runescape logs buy rs2 items cheap runescape items Hellgate London gold Guild Wars Gold buy Guild Wars Gold runescape items rs2 accounts cheap rs2 equipments lotro gold buy lotro gold buy runescape money buy runescape gold buy runescape runes lotro gold buy lotro gold runescape money runescape gold cheap rs2 powerleveling eve isk eve online isk buy runescape power leveling rs2 power leveling tibia gold tibia item runescape accounts Fiesta Silver Fiesta Gold Scions of Fate Gold Hellgate Palladium Hellgate London Palladium SOF Gold Age Of Conan Gold AOC Gold ArchLord gold tibia money tibia gold runescape accounts runescape gold cheap rs2 powerleveling buy ArchLord gold DDO Plat Dungeons and Dragons Online Plat

    ReplyDelete
  2. ffxi gil
    Maple Story Mesos
    wow gold
    wow gold
    a
    buy Warhammer gold
    Warhammer gold
    age of conan gold
    aoc gold
    age conan gold
    age of conan power leveling
    aoc power leveling
    age conan gold
    age of conan gold
    aoc gold
    aoc power leveling

    Runescape Money
    Runescape Gold
    Runescape Items
    Runescape Money
    Runescape Gold
    Runescape Power leveling
    Runescape Money
    Runescape Gold
    Runescape gold
    runescape money
    Runescape Money
    Runescape
    Runescape Money
    Runescape Power leveling
    Runescape Items
    runescape money
    money runescape
    Runescape Gold
    age of conan gold
    age conan gold
    aoc gold
    age of conan power leveling
    aoc power leveling
    aoc leveling

    archlord gold
    anarchy online credits
    cabal alz
    city of heroes influence
    coh influence
    city of heroes
    city of villains infamy
    cov infamy
    city of villains
    dofus kamas
    kamas dofus
    dofus
    daoc gold
    daoc plat
    eve isk
    eve online isk
    eve online
    everquest 2 gold
    eq2 plat
    Ever Quest 2
    final fantasy xi gil
    Final Fantasy XI
    buy ffxi gil
    final fantasy 11 cheap gil
    gaia online gold
    gaia gold
    guild wars gold
    gw gold
    guild wars
    hellgate london palladium
    Hellgate London
    dofus kamas
    dofus gold
    dofus
    dofus kamas
    dofus

    wow gold
    world of warcraft gold
    buy wow gold
    cheap wow gold
    wow power leveling
    eq plat
    EverQuest gold
    Maple Story Mesos
    MapleStory mesos
    Maple Story meso
    Maple Story Meso
    maple Story Mesos
    MapleStory mesos
    Final Fantasy XI gil
    ffxi gil
    buy ffxi gil
    dofus kamas
    kamas dofus
    lotro gold
    lotr gold
    lord of the ring gold
    Lineage adena
    Lineage 1 adena
    Silkroad gold
    buy silkroad gold
    flyff penya
    buy flyff gold
    flyff money
    eve online isk
    eve isk
    2moons dil
    2moons gold
    eq2 plat
    EverQuest 2 gold
    EverQuest 2 plat
    Runescape gold
    Runescape money

    maplestory Mesos
    MapleStory meso
    maple story
    Maple Story Mesos
    maplestory Mesos
    MapleStory meso
    Maple Story
    Lineage adena
    Lineage 1 adena
    Lineage 1
    lotro gold
    lotr gold
    The Lord Of The Ring
    lord of the ring gold
    ge money
    granado espada vis
    granado espada gold
    Granado Espada

    data processing service
    data processing
    data process
    data processing service
    data processing
    data process
    data processing service
    data processing
    data process

    dofus kamas
    kamas dofus
    buy kamas
    dofus kamas
    kamas dofus
    buy kamas
    dofus kamas
    kamas dofus
    buy kamas
    dofus kamas
    kamas dofus
    buy kamas
    dofus kamas
    kamas dofus
    buy kamas

    ReplyDelete
  3. Anonymous3:11 AM

    Welcome to wow gold our wow Gold and wow power leveling store. We wow gold are specilized, wow power leveling professional and reliable wow power leveling website for wow power leveling selling and wow gold service. By the World of Warcraft gold same token,we offer wow power leveling the best WoW service wow power leveling for our long-term and wow powerleveling loyal customers. wow powerleveling You will find wow powerleveling the benefits and value powerleveling we created powerleveling different from other sites. As to most people, power leveling they are unwilling to power leveling spend most of wow power leveling the time wow gold grinding money Rolex for mounts or rolex replica repair when replica rolex they can purchase Watches Rolex what they Rolex Watches are badly need. The Watch Rolex only way is to look Rolex Watch for the best place rs gold to buy cheap WOW gold. Yes! You find it here! Our WoW Gold supplying service has already accumulated a high reputation and credibility. We have plenty of Gold suppliers, which will guarantee our delivery instant. Actually, we have been getting Runescape Gold tons of postive feedbacks from our loyal RuneScape Money customers who really appreciate our service.

    ReplyDelete
  4. We offer WoW power leveling,World of Warcraft power leveling,Warhammer Online Power Leveling & Warhammer Power Leveling & Warhammer Online Gold,if you want buy cheap wow power leveling & honor power leveling,please come here to choose. you could find any kinds of powerleveling you want. Please remember,we are your online game helper. Please remember,we are your online game helper.please come here to choose. you could find anything you want.wow power leveling,wow power leveling,wow power leveling,wow power leveling,wow power leveling,wow powerleveling,wow powerleveling,wow powerleveling,wow powerleveling,wow powerleveling,Warhammer Online Power Leveling,war leveling,Warhammer leveling,Warhammer Power Leveling,Warhammer Online Gold,Warhammer Gold,WAR Power leveling,WAR Gold,world of warcraft power leveling,world of warcraft power leveling,world of warcraft power leveling,world of warcraft power leveling,world of warcraft power leveling,world of warcraft powerleveling,world of warcraft powerleveling,world of warcraft powerleveling,world of warcraft powerleveling,world of warcraft powerleveling,wow gold,wow gold,wow gold,wow gold,wow gold,world of warcraft gold,world of warcraft gold,world of warcraft gold,world of warcraft gold,world of warcraft gold,AOC Power Leveling,AGE OF CONAN Power Leveling,Warhammer Online Power Leveling,Warhammer Power Leveling,Warhammer Online Gold,Warhammer Gold,2 Moons Dil,MapleStory Mesos,Maple Story Mesos,MS Mesos,WARHAMMER ONLINE GOLD,Cheap WARHAMMER ONLINE GOLD,RuneScape Gold,RS Gold,RuneScape Money,RS Money,SilkRoad Gold,SilkRoad Online Gold,SRO Gold,EVE ISK,EVE Online ISK,Gaia Gold,2 Moons Dil,WOTLK power leveling,AOC Gold,AGE OF CONAN Gold,LOTRO Gold,Lord of the Rings Online Gold

    ReplyDelete
  5. Anonymous2:04 AM

    It is the ro zeny which make me very happy these days, my brother says ragnarok zeny is his favorite games gold he likes, he usually buy some iro zeny to start his game and most of the time he will win the cheap zeny back and give me some ragnarok online zeny to play the game.

    ReplyDelete