What have we learned about oil prices? Not much.

[This is a guest post by Kyle Meng]

With recent rumblings of political unrest in the Middle East, concern about rising oil prices have returned to the headlines. None of this is new of course. 2005, our last episode of high oil prices, was only a few years ago. And of course, high oil prices triggered by political instability in the Middle East echoes the OPEC embargo of 1973 and the Iranian revolution of 1979. Even President Obama's recent call for energy independence appears like a deja-vu of Nixon's Project Independence in 1974. 

And as with before, many are wondering where oil prices are heading. Will it stay above $100 per barrel? Will we see some respite on the horizon? Unfortunately, it appears that our ability to answer these questions has also changed little over the last 30 years. Put simply, we just don't know. 

I recently came across this figure, assembled a few years back by Bill Hudson for a congressional testimony. The figure plots oil price forecasts made by the US Department of Energy made every year since 1982. Also shown is the actual spot price of oil. This analysis, presented each year in the DOE’s Annual Energy Outlook forecasts oil prices over the next 10-20 years. 

There are a few observations that jump out immediately. First, we’re really bad at making oil price forecasts in the long term (>5 years). For shorter horizons, our forecasts improve but only during periods of relative stable price movements (say the cheap oil price era of the late 1980s and 1990s). For periods of large swings, notably the early 1980s and mid 2000s, we seem to get everything wrong, regardless of time horizon. Interesting also are how the forecast biases change over time. Longer term forecasts in the 1980s and early 1990s seem to regularly overshoot the future price of all – all forecasts beyond 5 years seem to project that oil prices will increase, almost exponentially, into the horizon. This bias, perhaps a vestige of the 1970s oil shocks, diminishes overtime as forecasts “learn” to be less pessimistic, downplaying projected increases in future oil prices to the point that when the 2005 price spike occurs, the models are now too optimistic. In other words, we’re learning, but only to be told how wrong we are the next time a major event occurs. 

All this is interesting because forecasting is a regular part of the world we live in.  When an event occurs, we’re naturally inclined to wonder what will happen next. But that analysis is often difficult, particularly for systems as complex as global oil markets. Short run forecasts could always be made via a simple extrapolation of recent events but extrapolations do poorly against extreme events, which are the ones we typically care about anyways. As for the evens that do matter, unfortunately, we simply don’t have enough data to make good predictions, and so, time and time again, we’re taken by surprise.   

Kyle Meng

PhD. Student

School of International and Public Affairs

Columbia University