I hate to say it, but I totally agreed with Paul Krugman in this piece (pdf) for econowonks when he said:
One of the problems in the debate over the role of speculation in oil prices is that hardly anyone, even among the economists, is writing down models. As a result, it’s not always clear what people are saying; and I’d argue that some of my colleagues aren’t clear on the implications of their own analysis.
I think this issue about speculators and oil prices is perfect for neoclassical models, so long as the creator is familiar with Grossman's work as collected in The Informational Role of Prices. (Incidentally, that is a wonderful book that justifies the absurdities to which formal models may be put in most cases. Another example is Lucas' model of money demand, to which I can't find a link. If you're going to do it formally, that's the way.)
Anyway, Krugman crystallizes the verbal reasoning I employed in this study, to rule out speculators from being the cause of high oil prices. Basically, the "correct" price is the one where the quantity of oil supplied by producers equals the quantity demanded by physical consumers. But if speculators come in and drive up the price, then there necessarily is a surplus of oil. The higher price encourages production and discourages consumption. Even if you think the elasticities are small, surely if oil prices are double what they "should" be (as some people just testified to Congress), then there must be a constantly growing stockpile of barrels of oil.
Krugman went beyond that argument though. He also mentions a second signature of this mechanism: Spot and futures prices would have to be in contango. This is because the process through which Goldman Sachs can induce Rex the Oil Warehouser to grow inventories, is that Goldman Sachs buys futures contracts (as an investment). This pushes up the futures price while (initially) the spot price may be unaffected. But once the futures price rises a certain amount above spot, it then becomes profitable for somebody to buy barrels at the spot price, physically store them until the futures contract matures, and then cash in the contract and the barrels. I.e. the only way the nominal rate of return in this activity can be positive, is if futures prices are higher than the spot price--actually, they have to be higher than the interest rate at the very least, because people could buy US bonds with very low risk. So to get them instead to invest in buying and storing physical barrels of oil, the implied rate of return embedded in the spot and futures price would have to be higher than the rate on Treasuries.
But, as Krugman points out, we haven't seen that either. In fact, the oil market has been in backwardation (where the spot price is higher than the futures price) for a large portion of the runup in oil prices. This is the exact opposite of what you'd see, if institutional investors were driving up oil prices by buying futures contracts.
Last point: In the original Krugman article (HT2MR, and to Pepe for tipping me off to that) intended for the lay public, Krugman strangely argues that there also wasn't a housing bubble. (At least, I think that's what he's saying. But I am hesitant to say this, since who the frick would deny that there was a housing bubble?) Anyway, for some reason Krugman is looking at rents, and since they are flat he concludes that investors weren't driving the change in house prices. (Again, I don't believe this either as I'm typing it out. But I read that Krugman paragraph several times, and it sure sounds as if he's saying there was no bubble in housing, just as there is now no bubble in oil.)
So regardless of whether I'm misunderstanding Krugman's position, I show in the IER piece linked above that we can contrast the current oil market from the housing market in the early to mid-2000s. In other words, using the hoarding approach, I don't see any buildup of oil inventories in the EIA data, but I do find a fairly strong relation between house prices and the rate of vacancy in rentals; as the Case-Shiller home price index skyrocketed, so too did the vacancy rate shoot up. This makes perfect sense: Investors are buying homes not to live in, or even rent out, but to fix up and "flip" the following year or two. If enough speculators are doing this, on average a higher fraction of the housing stock would be vacant.
I think this would have been a much easier answer than what Krugman did. If I have understood his position, then my only explanation is that (1) he foolishly believed a critic who challenged him by saying, "But the data don't show a buildup in housing!" and then (2) tried to explain why the hoarding argument doesn't apply to housing, even though it applies to oil.
Now Krugman, it does too apply to housing. And the data show it clearly. You shouldn't have believed your critic who said the data don't illustrate the hoarding argument in the housing bubble.