DJIA since Sept '08

The answer is, broadly, yes. Stocks are shares in the future earnings of a company. They are therefore a “bet” on (er, an “investment” in) the future performance of a company, or many companies. The trading price on any day is the price at which there are as many buyers as sellers for these future returns. Rising prices mean there are more buyers than sellers, that means general expectation of future profits is going up. Investors are putting a higher price on the future.

The market is therefore considered a leading indicator of economic conditions. (By contrast, employment figures are lagging indicators — due frictional forces, not to mention morality, it takes companies a while to downsize in recessions or upscale in booms, so employment levels track economic conditions but with a delay.)

But how valid and dependable is the market as a leading indicator? It is also apparent that markets move up slowly and steadily, but fall in a hurry. So the downward move can hardly be held to be predictive. But the upward move appears to hold some weight as harbinger of better times. How much weight?

What’s particularly important is that the aggregate insight into future returns from shareholding investments — across many investors and many stocks — cancels out individual errors. Any one person may have a dumb idea of the ‘future cash flows’ from one or many companies, and the price of any one company may be unreliable for innumerable reasons, including fraud, but the knowledge and intelligence of hundreds of thousands of people, when aggregated and spread over many thousands of stocks, corrects for all these errors. It becomes robust.

Prediction Markets

This reliability of shared, aggregated insight — the wisdom of crowds — is precisely what makes ‘prediction markets’ such a powerful forecasting tool, as I have mentioned in previous posts. (Prediction markets apply market-like wisdom to create foresight in areas that are not normally ‘tradeable.’) Any one person will, as likely as not, get it wrong, but everyone together, rather astoundingly, get it right.

Ironically, crowd wisdom is much more reliable than the technical forecasting models that investment institutions use to try to determine how business, macroeconomic, interest rate, or other conditions will affect future stock prices. These predictions, based on the assumptions of a handful of model programmers and/or model users, are deeply vulnerable because there is no crowd-wisdom balance. It’s no better than reading tea leaves, only apparently (and unaccountably) more respectable.

Having said all this, it is well known that the ‘crowd,’ aka the ‘herd’ can and do all get it wrong together. This is what happens in price bubbles, or panic market exits, with everyone buying or selling because they are making the same wrong assumptions, or just doing what everyone else appears to be doing. (Most players making the same mistake together is the basic problem when prediction markets fail too.)

However, what is clear is this case is there was a very hard sell-off in the months prior to March 09, following revelations of the gravity of the Credit Crunch, but that this has slide has been arrested and mostly reversed. This says that innumerable smart people with, collectively, billions of dollars at stake, are expecting future profits higher than they did in March. That’s a prediction one can rely on.

Tweet This Post

My question, as the Dow Jones Industrial Average (DJIA), and the FTSE100, the DAX, the Hang Seng and so on have hit a decade lows is, what is this predicting, if anything? What is the long-term value of this prediction, and could it be used to make better decisions in the real world?
We know that the value of a common stock – a share in a company – is based ultimately on the returns (dividends) it will bring. Buyers and sellers therefore derive a daily market price based on their views of the share’s expected, that is, predicted future payback. The greater the expectation, the greater the price. A high price vis a vis earnings (P/E ratio) suggests confidence in future earnings, and vice versa.
Therefore the current steep fall in share prices is an expectation of (crowd prediction of) lower future payouts. Of course the complexity in human-prediction situations is that this basic level is also overlayed with a meta-level: people are not only trying to figure out what will happen, they are trying to figure out what others think will happen. So falling PE ratios are an expectation of what others will do (predicting they will continue to sell.)

Madness or not?
One of the perplexing things about the markets is they very often seem to react opposite to what is expected; to what would be common sense. They often fall on good news, rise on bad news, close unchanged on big news, and so on. Although there is – famously much irrational behavior and herd instinct in the market – you don’t get hundreds of thousands of decision-makers wagering significant money not using common sense.
What is going on, of course, is that the market has often already risen or fallen in prediction of the news. When a new condition – an interest rate move, for example – is imminent, the market will move to “price in” the expectation. If market participants as a whole have called the future correctly the market will not move much on announcement.

Pricing-in the future
Because of this predictive component to group decision-making in market situations, the stock market as a whole is a classic leading indicator of the real economy. When prices move they may be taken as the crowd “pricing-in” a future prediction. So markets will fall ahead of real economic problems (they may continue to fall, as now, during steep economic declines.) But they will also turn up well before any real, measurable upturn.

By the way, there is little doubt it will overshoot in this time, as it always does. This is because, as in prediction markets, the wisdom of crowds can predict the trend but not the turn. Trend extrapolation will never show you the key shifts, and this is why predicting the bottom or top of a market is so hard.

The point, for market speculators, is that long before the real gloom is over the markets will be zooming upwards. The point for the rest of us is that recession times will be with us even after the markets move up. In the long term the market will go up. Like death and taxes, it’s the surest thing there is.

Tweet This Post

First, there is the prediction game on offer from ABC, taglined: “The Oscars Live Challenge: Think you can Predict a Winner? Make Your Picks Now!”

It’s all part of the marketing drive of course, but, nevertheless how would one play it best and what might that tell us? Let’s assume there is something at stake, like you’re really going to sit in front of the TV and mark off your right vs. wrong predictions, and compare your score with that of your spouse for year-long bragging rights – now there’s pressure – how would you predict? Would you think (a) “this is the best movie so I predict it will win”? Hardly. You would think: (b) “this is the one that I think most people will pick, so that’s the one I think will win.”

You would be making a meta-prediction – going with what you think most are going to choose. In this particular case you would also know that that Oscar winners are chosen by balloting the 6,000 members of the Academy of Motion Picture Arts and Sciences. So your more exact question would be: who is this special group likely to choose in each category?

What’s going on? In future situations that are heavily dependent on aggregate human choices – which is very many situations – the savviest predicting strategy is to figure out the choices most people are going to make. Oscars aside, figuring out the choices most people will make on any issue – hybrid cars, tighter securities legislation, public health care, etc. – is an excellent guide to what will really happen. It’s a mass market-led view of the future to be sure, but that’s exactly what makes it dependable in mass-opinion situations. (Not all situations are determined by mass-market choices – predicting a presidential election winner is; predicting a superbowl winner is not.)

Playing the game

I had a shot at the Oscar prediction game, joining the alleged 1,680 other “players” who were then online. From what I could tell via the rather gristly Flash interface is that the game is not (yet) “social” in that you can’t see what other people are predicting – there is no access to aggregate opinion. No matter. One can instantly get this in hundreds of prediction market forums right now, for example Intrade, where the price of each outcome in each Academy Awards category directly reflects how strongly players as a whole have bid up that outcome.

At Intrade, at time of writing, Slumdog Millionaire is at $87.30 (max is $100; the other 4 movies share the remaining $12.70). When used as a prediction this means that the aggregate opinion of people staking real money has been effectively captured: it is that Slumdog Millionaire is 87% likely to be the choice of the Academy members in its category.

This is a guide to Oscar night that I would not bet against if I wanted to hold onto my bragging rights. Even in situations less overwhelmingly agreed on by players, it has been shown that prediction markets, tapping the aggregate “wisdom of crowds” (working like “Ask the audience” on Who Wants to be a Millionaire) are a fabulous tool for capturing what most people think will happen, resulting in excellent predictions. Caveat Emptor: prediction markets are poor at predicting long-term, open-ended situations, particularly where the outcome alternatives are unknown or can’t be clearly bounded, as blogged a few months back.

Tweet This Post

Zeitgeist, German for “spirit of the times,” refers to the often unconscious spectrum of intellectual views, analytical approaches, political and social concerns, etc., that people in any era share. Evidence from the checkered history of predicting the future shows that forecasters have been very heavily biased by their then-current conditions, current issues, and current state of the world, that is by their zeitgeist. They see reality and therefore the future through the lens of their times. The key marker of this effect at work is when many forecasts are not only wrong, but are wrong in the same way.

The zeitgeist effect, by the way, is not a peculiar condition. It is part of the many common human and social cognitive-perceptual biases that exist. (For a longer list and their effects in forecasting, see Future Savvy.) Zeitgeist is in fact sometimes more broadly known as “situational bias,” where situations that people find themselves in frame what they see and how they interpret events. In a depression, famously, it is difficult to see any source of upturn; in boom times it is hard to see the crash.

The Zeitgeist of the 1990s
Just one example of zeitgeist bias in forecasting can be seen in predictions made in the 1990s, when the fall of Soviet Union and the end of the Cold War provoked new hope about global interaction and growth, and this, combined with the rise of digital technologies, the Internet, WTO agreements, and the dot.com market boom, fueled a new zeitgeist of optimism. It was very common, in this era to predict global prosperity, international peace and harmony, rising standards of living, enhanced personal freedoms, and a better environment. By the end of 2001, the NASDAQ bubble had burst, Al-Qaeda had struck buildings that symbolized U.S. power, the “War on Terror” had begun, and the entire rosy 1990s and all the forecasts that went with it, were finished. Zeitgeist or situation has, evidently, a very strong pull on what people think is possible and likely in the world at large, and in their own sector or industry. It frames the questions people ask, the topics they think are important, the outcomes they expect, and how they interpret signals of change.

The point is, do prediction markets somehow counter this well-known perceptual-cognitive bias in forecasting? No. The outcomes people think more plausible – that they will “buy up” in a prediction market – are deeply affected by situational conditions. This is sure to be a factor when, for example, Google sets up a market to tap its employees for forecasts. The people involved, smart as they are, will be strongly by situational factors – and this will affect which future outcomes look more likely.

Herd Effects
Moreover, they will be pulled in the same direction. As prediction markets, like Delphi studies, are particularly a consensus-based method (forecasters drawing predictive results from the study/game are going on what most people say/do), they are by definition deeply vulnerable to the zeitgeist effect. This dovetails with another well-known social cognitive bias, the “herd effect” or “bandwagon effect,” or “groupthink,” where people to believe or do things because many other people are doing or appear to be doing them. The only way to avoid groupthink is to isolate people from each other (as Delphi studies attempt to do by not disclosing others’ responses while the study is live.) But in prediction markets the player can, of course, see what others are doing. When buying a “stock” that is getting more expensive they may be reassured that others are buying it too — leading them to buy more, producing a classic herd-effect situation.

These biases – and various others, as detailed in Future Savvy, are among the many perceptual and cognitive biases that people bring to the world. They are part of (and evidence of) perception being active and constituent in our understanding of the world, and therefore of the future. We can chip away at our perceptual framwork, particularly by questioning our assumptions and investigating the basis of our knowledge (possibly through scenario planning.) However, no matter how astutely we look and how consciously we try to eliminate them, our paradigms exist, and they color our forecasts, and prediction markets do not make them go away.

Tweet This Post

First, what’s if all about? If you already know, skip this section. Let’s start with the example in yesterday’s Telegraph: “Predicting the future – with the power of betting” Paul Parsons, August 19, 2008. As Parson’s reports, the University of Iowa is running a market where investors can buy “shares” in the two major US election candidates, each priced between $0 and $1. On election day, traders holding stock in the winner – Obama or McCain – receive $1 per share while the others lose their money. Investors can buy and sell their shares along the way, and as they do this the candidate more people will want to own (because they think he will win) will get more expensive. In other words, market forces will drive up the price of the outcome more people think more likely. As of August 19, the trading value of the Obama, at $0.62, suggests participants expect a 62 percent chance he will win. (Another prediction market site, midasoracle.org, has the figure currently at 59.8 percent.)

Prediction markets mimic stock market and deploy much the same software. Where a real market trades shares in an underlying asset, in a prediction market it is future outcomes which are “securitized”. The key principle at work is the sage market wisdom that “the price of a stock captures all the information known about it” – that is, all information is factored into the price (notwithstanding that some may have more or better information than others; some may be acting more wisely on their information). Therefore price is our guide to the cumulative knowledge of all participants and, in prediction markets, this “price discovery” allows us to know what most people think the future holds. They allow the “the wisdom of crowds” to be turned to a future problem, and tapped.

Serious Success

What’s exciting about all this is its success rate. Prediction markets are amazingly accurate in many circumstances, and by all accounts consistently beat more conventional quantitative and extrapolative methods. Prediction markets have consistently out-predicted election opinion polls and exit polls. Of course the predictive potential goes way beyond polling. Forecasting markets can and have been set up to predict the dollar movements to the success of same-sex marriage legislation, to who will win best actor Oscar. At one point there was even a US government market in future terror targets (trying to elicit public predictions of likely targets so as to plan accordingly) but this was deemed inappropriate and taken down.

As it has become clear that this method outstrips conventional forecasting methods, prediction markets have taken root in forward-looking businesses. Companies such as Google and Hewlett-Packard routinely use (internal) prediction markets to forecast sales figures, customer preferences, product adoption, and so on. HP is on the record as saying prediction markets consistently outperform their official forecasts.

The method has other advantages too. First, it requires no special techniques or expense. There are no fancy models to apply or complex algorithms to … to do whatever one does with such things. Second the forecasts are available in real time, all the time, and constantly update themselves. There’s no waiting for data collectors to collect, or statisticians to emerge with their answers.

The Limits

In my book, Future Savvy, I show how and why humans are poor at predicting, for dozens of reasons. The record of predicting is littered with failure. But, is that now all in the past? Do prediction markets solve the perennial problem of predicting the future, or at least get us closer? Yes and no.

Yes where prediction markets are appropriate. They work best under two conditions: first where there is a clear view of the options and operating conditions; second (related) where the time frame predicted is relatively short, usually under 18 months depending how fast things are moving. Where predicting the future means choosing between known alternatives, such as an election winner, or anticipating a point along a known continuum, for example the level of next year’s sales, prediction markets are great.
Where prediction markets run dry is in dealing with unfamiliar conditions, or unknown variables, or potential game-changing disjunctures in the world. Where the future is seriously fuzzy, where there are many variables, and the way they interact unknown, and drivers, blockers, and lags are hidden, prediction markets are of limited use because the outcomes can’t be framed adequately so that people can bet on them or against them. A prediction market for US president in 2012 would be far less useful than 2008. Similarly, while a market for the oil price in 2009 would be helpful, by 2010 or beyond factors driving the price may be so different (viz. developments in sustainable energy or geopolitics) that the result of a prediction market conducted in 2008 would be undependable.
So while prediction markets sort out probabilities between known likelihoods, they are not adequate to the task of investigating complex situations where we cannot frame the likely outcomes, or at least can’t know if we’ve framed them right. Also while prediction markets do help us, on aggregate, avoid some perceptual/cognitive fallacies, they are as likely as any other predictive tool to fall into the Zeitgeist effect. More on this soon…

A good list of articles on prediction markets is available here: http://www.midasoracle.org/best/

Tweet This Post