The truce between politicians and the futures markets is over after less than a year. Hostilities were reopened by Gary Gensler, new chairman of the Commodity Futures Trading Commission, when he announced on July 7 a consultation on whether the CFTC should set its own position limits for all commodities trading, as it does already for agricultural products. How hedge exemptions are awarded will also be reviewed.

Market leaders, most of whom have been scrupulously politically correct whenever the regulators said something during the past 10 months of financial nightmare, took the gloves off and hit back.

Interviewed on Fox News, Terry Duffy, chairman of CME Group, said he thought critics of the market were confusing manipulation with speculation. He suggested the regulators should look first at the OTC markets, where big bets can be made without limits, or they would risk driving business off the regulated exchanges into “dark pools”.

His colleague Craig Donohue, CME’s chief executive, insisted to CNBC: “If you look at the empirical evidence nothing supports the idea that speculators are driving prices in futures markets.”

As in 2008, when Congressional ire last focused on the futures markets during the extreme boom in oil and other commodity prices, three lines of defence are appearing, along which the industry will try to protect its freedom from encroachment.

These arguments are: that speculation does not affect commodity prices; that even if it does, it doesn’t matter because markets must be left to do their work; and that curbing speculative activity would make markets less liquid and hence more volatile.

Unfortunately, none of these arguments is convincing.

A unique market?

Futures professionals – virtually all of whom have a vested interest in seeing as many futures traded as possible – maintain that there is no empirical evidence speculation drives prices.

But by what magic do they believe commodities, alone of all investable markets, are immune from speculative overshooting?

No one would seriously contend speculation did not influence prices for shares or housing or junk bonds. Why should commodities be the only market whose participants soberly adhere to fundamentals of supply and demand and would never dream of making a buck by buying cheap and selling dear, just because they know other people are piling into the same trade?

No academic study may have found a smoking gun that categorically proves speculators were to blame for last year’s oil spike, but markets are extraordinarily complex things, in which many actors do things for a mixture of reasons. Nothing is ever clear-cut. Unlike the natural sciences, there can be no experiments to test theories in economics – even if you try out a policy, there is never an identical control group. So it is remarkably hard to prove anything with empirical studies.

What all investors know is that fundamentals always play a part in setting a market’s direction, but if people can profit by riding the price up or down, they will, and that can lead to bubbles and busts.

When a glut is called a drought

True believers in the wisdom of energy markets claim last year’s unprecedented oil spike was caused entirely by supply and demand, or by a lack of data for traders to act on. The trouble with this view is that supply and demand imbalances have always existed, and data was probably even patchier in the past – but the price spike and crash of the past few years were unprecedented.

And they did not coincide with any war or other extreme event that provoked a shortage. Quite the opposite: oil supply and demand were, as usual, remarkably stable.

According to the US Department of Energy, in 2007 global oil demand outstripped supply, by 0.5m barrels a day, or 0.58% of world demand.

But in each of the two previous years, supply had exceeded demand. In 2005, 1m barrels a day were added to world stocks; in 2006, the reserve accumulation was 0.95m barrels a day.

In a rational oil market, 2007’s hotter demand would simply have required drawing down a little on the stores built up in previous years. Why did the spot oil price have to go from $42 at the beginning of 2005 to $61 at the start of 2007 and $96 at the end of 2007?

What about the crazy year of 2008? Surely producers were struggling to keep up with the rampant demand from the emerging markets?

In fact, supply exceeded demand in every single quarter. In the March-June quarter, the Nymex front month futures prices went from $105 to $140. Yet in that quarter, the world was producing 0.9m barrels of oil a day more than it was burning, a surplus of 1%.

In the third quarter, when the price fell sharply, the supply surplus was exactly the same.

What did make the years from 2005 to 2008 different from previous periods in the oil market was a steep increase in “commodity investment” by hedge funds and institutional investors; a related increase in commodity trading by investment banks, some of which bought facilities to store physical oil; and a surge in oil futures trading.

Keeping the faith

The market purists’ second favourite argument is that yes, markets may be rough sometimes, but it’s wrong to interfere: markets always know best.

Because it is so hard to make experiments and test theories in economics, such claims are as hard to refute as they are to prove – rather like religious beliefs.

The faith in omniscient markets tends to grow during bull markets and suddenly become unpopular when markets crash. Since the biggest crash for 80 years has just happened, it is a bad time for a religious revival.

In the case of commodity futures, market boosters argue that their unfettered trading performs two vital functions: helping users and producers reduce risk; and “price discovery” – revealing what the right price should be.

Most of the time, it seems to work. But when oil has just roared to $147 a barrel, crashed to $35 and bounced back to $70 in the space of a year, without going through any acute shortage, the market’s claim to be doing the rest of the economy a service sounds very hollow.

At what point in that rollercoaster ride did the market “discover” the price was too high? If traders were so wise, couldn’t they have discovered it a bit sooner?

It is rather like investment bankers’ claims that they need to be paid seven figure bonuses because their skills are exceptional and they perform a service no one else could. Because it’s very hard to disprove, they can get away with it as long as they succeed. But when they fail, the argument collapses.

Why commodities are not stocks

Opponents of tighter regulation are also likely to argue that commodities should be left alone because there are no position limits for debt and equity markets.

But this argument, too, is weak. In the case of bonds, prices are unlikely to stray too far in either direction, for several reasons.

For a start, the government or central bank sets interest rates and issues debt. Second, real rates are never likely to go below zero for very long. Third, government bonds convey safety, so the price of that is never likely to get excessively high. So a bond bubble can only swell so far, and the means to control it are already in the government’s hands.

Commodities are much more susceptible to being treated by investors like equities – as something that may be volatile, but will tend to go up. There is no theoretical upper limit to their prices, and they are a play on economic growth. And when it seems like economic growth is going to outstrip the growth in commodity supplies, as it did during 2005-2008, it is logical to expect that prices will rise in the long term.

That does not mean it is justified for spot and front month prices to double or triple, before the shortage actually bites.

Could that possibly have happened because investors wanted to ride the up escalator of a widely held long term bullish view, but retain a chance to jump off every month, using the most liquid short term contract?

If free speculation in bonds can be allowed because prices are inherently stable and the government holds the levers, in equities it is allowed for the opposite reason. Prices are uncontrolled – but volatility is bearable.

Equity markets exist to allow people to take business risks with their savings in the hope of gain. But unless they are very unwise, people do not invest their last dime in shares. It is ultimately spare money that people can afford to lose. And anyway, no one forces them to put it in stocks. There are always other things to invest in.

For these reasons, there is no pressing need to stop any particular category of market participant from influencing stock prices, or driving them too high or low.

Commodity markets are very different. We all need to buy commodities, or things made with them – just to go about our daily lives, even to eat. They are not an optional extra, and there are no alternatives.

But while some of the participants in the commodity markets are using them to sell or obtain vital commodities, others are there just for profit.

It is obvious from hundreds of years of history that as soon as you develop a market for a commodity, middlemen will appear who are not end users, but simply trade. They perform a valuable service in making markets more liquid and efficient. Those who wish to hedge are able to trade, not just with hedgers on the opposite side of the market, but with speculative capital in the middle.

But it is surely possible to imagine that this useful speculative capital could swell to a point where it overbalances the market. That is even more likely when this capital is not tied in, but can appear overnight when herd instinct is driving the market strongly in one direction, strengthening that stampede, and disappear or switch direction when the market trend changes.

Such capital is not helping the end users, but exploiting the fact that they cannot do without the commodity.

This kind of investment is often given virtuous names, such as “portfolio diversification” or “hedging against inflation”. It would be more accurate to call it “promoting and profiting from inflation”.

And it is simply not true to say that those who swelled the bubble must have lost their shirts when it burst. Many of them will have sold out when the market peaked, or before, and bought a more expensive shirt.

Liquid = stable?

The market fundamentalists’ third argument is that anything the regulators do to control speculative capital in commodities would make the markets more volatile because they would become less liquid.

It would be good to hear them explain why the oil market has been so volatile in the past year, precisely when participation and liquidity have been at unprecedented highs. Or how the world managed to have stable prices for things like wheat and oil when there were no hedge funds, commodity index funds or electronic futures markets.

Just because liquidity is generally a good thing does not mean liquid markets never experience violent booms and busts.

No quick solution

This magazine is not arguing for any particular policy of trading limits for commodities. It is always difficult to interfere in markets without causing unintended consequences, and it may be that the answers lie outside the futures markets. Duffy’s point that OTC markets may need more scrutiny is a good one.

Certainly, how to maintain healthy, liquid commodity trading while keeping prices at sensible levels that help both producers and consumers is a difficult question, with no easy answers.

Gordon Brown and Nicolas Sarkozy, UK prime minister and French president, acknowledged as much in their thoughtful article for the Wall Street Journal on July 8. As they pointed out, much of the trouble with the oil market is that there is not enough cooperation between nations on the two sides of the market. Though they didn’t use these words, the world oil market is in fact chaotic, partly because it is racked by power politics.

What FOW does argue is that at a time like this, it will not help the futures industry, the oil market or the economy as a whole to deny that there is a problem, or boldly state that the market is always right, with no justification other than faith.

What better example is needed of the damage price bubbles can do than T Boone Pickens, the hard-bitten Texas oilman, who stands to lose millions, perhaps billions by investing in windfarms on the basis that fossil fuels would remain expensive.

He was doing the right thing, but for the wrong reason – trusting that the market was setting the right price for oil.

How much money and how many years of precious time in developing green energy sources have been wasted because governments thought the high market price for oil would stimulate investment in alternatives without them having to do anything?

Whatever your view on how much oil should cost, it cannot be rational – or good for anyone except the cleverer speculators and all the intermediaries – for it to cost $147 a barrel in July and $35 the next February, when neither supply nor demand has changed more than a whisker.

Regulators and governments face a very difficult task in bringing volatility in commodity prices down to a bearable level. The market should be helping them find the right solutions, not throwing barricades of flimsy arguments in their way. n

FOW is keen to publish cogent opinion pieces on this subject – whether they agree or disagree with this one. Please contact the editor.