The Uses and Limits of Prediction Markets in Forecasting

Posted by Adam Gordon on Aug 20, 2008 in all, failed predictions, foresight tools & methods, Future Savvy, prediction markets

Hmm. As the 2008 White House race hots up, we’re going to be hearing more and more – and then even more – about who prediction markets forecast to win, so it’s time to put down a thought or two about uses and limitation of this forecasting tool.

First, what’s if all about? If you already know, skip this section. Let’s start with the example in yesterday’s Telegraph: “Predicting the future – with the power of betting” Paul Parsons, August 19, 2008. As Parson’s reports, the University of Iowa is running a market where investors can buy “shares” in the two major US election candidates, each priced between $0 and $1. On election day, traders holding stock in the winner – Obama or McCain – receive $1 per share while the others lose their money. Investors can buy and sell their shares along the way, and as they do this the candidate more people will want to own (because they think he will win) will get more expensive. In other words, market forces will drive up the price of the outcome more people think more likely. As of August 19, the trading value of the Obama, at $0.62, suggests participants expect a 62 percent chance he will win. (Another prediction market site, midasoracle.org, has the figure currently at 59.8 percent.)

Prediction markets mimic stock market and deploy much the same software. Where a real market trades shares in an underlying asset, in a prediction market it is future outcomes which are “securitized”. The key principle at work is the sage market wisdom that “the price of a stock captures all the information known about it” – that is, all information is factored into the price (notwithstanding that some may have more or better information than others; some may be acting more wisely on their information). Therefore price is our guide to the cumulative knowledge of all participants and, in prediction markets, this “price discovery” allows us to know what most people think the future holds. They allow the “the wisdom of crowds” to be turned to a future problem, and tapped.

Serious Success

What’s exciting about all this is its success rate. Prediction markets are amazingly accurate in many circumstances, and by all accounts consistently beat more conventional quantitative and extrapolative methods. Prediction markets have consistently out-predicted election opinion polls and exit polls. Of course the predictive potential goes way beyond polling. Forecasting markets can and have been set up to predict the dollar movements to the success of same-sex marriage legislation, to who will win best actor Oscar. At one point there was even a US government market in future terror targets (trying to elicit public predictions of likely targets so as to plan accordingly) but this was deemed inappropriate and taken down.

As it has become clear that this method outstrips conventional forecasting methods, prediction markets have taken root in forward-looking businesses. Companies such as Google and Hewlett-Packard routinely use (internal) prediction markets to forecast sales figures, customer preferences, product adoption, and so on. HP is on the record as saying prediction markets consistently outperform their official forecasts.

The method has other advantages too. First, it requires no special techniques or expense. There are no fancy models to apply or complex algorithms to … to do whatever one does with such things. Second the forecasts are available in real time, all the time, and constantly update themselves. There’s no waiting for data collectors to collect, or statisticians to emerge with their answers.

The Limits

In my book, Future Savvy, I show how and why humans are poor at predicting, for dozens of reasons. The record of predicting is littered with failure. But, is that now all in the past? Do prediction markets solve the perennial problem of predicting the future, or at least get us closer? Yes and no.

Yes where prediction markets are appropriate. They work best under two conditions: first where there is a clear view of the options and operating conditions; second (related) where the time frame predicted is relatively short, usually under 18 months depending how fast things are moving. Where predicting the future means choosing between known alternatives, such as an election winner, or anticipating a point along a known continuum, for example the level of next year’s sales, prediction markets are great.
Where prediction markets run dry is in dealing with unfamiliar conditions, or unknown variables, or potential game-changing disjunctures in the world. Where the future is seriously fuzzy, where there are many variables, and the way they interact unknown, and drivers, blockers, and lags are hidden, prediction markets are of limited use because the outcomes can’t be framed adequately so that people can bet on them or against them. A prediction market for US president in 2012 would be far less useful than 2008. Similarly, while a market for the oil price in 2009 would be helpful, by 2010 or beyond factors driving the price may be so different (viz. developments in sustainable energy or geopolitics) that the result of a prediction market conducted in 2008 would be undependable.
So while prediction markets sort out probabilities between known likelihoods, they are not adequate to the task of investigating complex situations where we cannot frame the likely outcomes, or at least can’t know if we’ve framed them right. Also while prediction markets do help us, on aggregate, avoid some perceptual/cognitive fallacies, they are as likely as any other predictive tool to fall into the Zeitgeist effect. More on this soon…

A good list of articles on prediction markets is available here: http://www.midasoracle.org/best/

read more

Scenario planning orientation and methods interview

Posted by Adam Gordon on Aug 7, 2008 in all, foresight tools & methods, horizon scanning, scenario planning, trend tracking

I was interviewed at length about scenario building by a foresight firm in the UK a few weeks back. They took notes (more than I deserved, no doubt) and here they are, below.
In the notes, which are typed live and necessarily brief, I’m “AG”. The others are participants asking questions and making comments…

1. First, what do you use scenario planning for?
Initially stated everyone did scenario planning, every time you find yourself doing something unusual it implies you’ve done something wrong with scenario planning. We think about the future all the time, constantly making scenarios in head, if move to London think about what need to earn, where live, family, critical uncertainties. We run forward, think about various challenges, we rehearse the future.

CL wondered about ill thought through scenarios?

AG felt it was inevitable that people will disagree. End of day scenarios boil down to politics. Visionary scenario planning gets everyone talking – firstly need to find common threads (if not grounds), that most buy into – even if it is a low common denominator, thus creating a shared vision, then dystopias, but remember the end point is never total agreement.

RS Acknowledgment of importance of politics? Is that your experience of foresight?
AG believes he is a political animal, but no longer involved in protest. Belief is power is absolutely important when thinking about the future. He then amends it to power and money – as often groups without power have public opinions so they can shape the future. There is a danger that they only talk about market forces and technology and that’s it.
RS Is it a struggle working with other foresight groups – is that view not shared?
AG says they can give a shared opinion, but politics mostly overlooked by the groups

RS describes that we all have a political agenda (social inclusion, voices excluded) within this project – it makes it interesting to deal with those who have an economic/gov background, it’s difficult to pretend futures neutral.

2. Could you talk me through a brief overview of your scenario planning method? (How long does each stage take? What preparation is required for each stage? How many people are involved in each step?)

There is an underlying method – but each stage has some degree of flexibility and can use a variety of methods.

A. Handshaking stage
Firstly deciding what you’re doing, basic project stuff – who’s involved, what resources, what deliverables – key is choosing method and who’s involved, may have varying degree of democracy – heads or grass roots, be inter disciplinary or not, have a broad or narrow focus, and what type of input is required. Believes it needs to have mixed agenda. Note that relying totally on academics is bad.
Then you need to decide the dates your final scenarios will relate to – further in the future the more radical but less relevant to other people and harder to action. Most future scenarios are around 10 years ahead – as a rule not less than 5 or more than 20.

To decide the focus you must:
1) Drive management team towards understanding how much influence they have over the future, can they drive future or does external events influence them? Percentage, never exact but idea.
2) Based on the amount of control can decide if creating a visionary scenario or anticipatory (not good word) scenario building. Visionary trying to develop an idea, multiple stakeholders get shared ideal, easier if pressure group, one organisation, as they can develop enrich and jump on to focusing on how we would put this into practice. They create a vision and dystopia. Influence future (money, opinion etc).
In anticipatory, or “Businessy type scenario”, participants don’t mind how things turn out, what they want is to be successful in the world however it turns out. Within organisation goal is to anticipate broad set of possible worlds, particularly critical uncertainties. So they do have research scenarios and different takes on how things emerge, BUT no preferred future. Look at resource and competence so they may have preferences but can adapt. To same extent they look at legacy competencies but this is not a determining factor. Scope alternatives and plan.
The Handshaking stage takes a few meetings to do, small with key meeting, then larger with various stakeholders – perhaps half a day.
In terms of materials one could send out stimulating piece to encourage thought so not stone cold but which doesn’t colour the agenda. Could also have to read a synopsis of what the process is about and the sort of things to think about and expect.

B. Horizon scanning
There are 100s of ways of doing this. Basically need to go into world and do research on what’s going on in key dimensions, technology, markets – broad scan of world relevant to issue area, bring in people outside of own industry. Best tool in this area is “learning journey” – jazzy word for anthropology of own society, structured agenda for talking to people about concerns, what they know, what they’d like, focusing on future – so need to be carefully done to avoid reiteration of now or what they think you want to hear.
Note that you can commission this – but it is not market research.
No answer to how long, dependent on time and resources, but should budget third of total time.

C. Pulling it together
Mulch through the data gathered – preferably in funky creative meetings, collate, output into things like forces, drivers of change, trends, blockers of change, critical uncertainties. What comes out is a picture of world that’s relevant to us.
This activity could be whole group or just the scenario developing team – it’s to pull out what’s important.
The format could be something like workshop, sleep/gap, then another half day or so.

D. Separating critical from predetermined
This could require a Mini Delphi, talk to experts to find out what are the sorts of things in forces of change list that are predetermined – so things we know will happen in 2020, perhaps the number of students, or trends to sustainability.
The various issues will have lifecycles in being a key focus, they’ll always be there but the amount of interest will vary, eg sustainability. In the future sustainability will be less of a concern, but not less important, it will just be integrated into our expectations we won’t focus on it. An example relating to education is how we’ve shifted views on punishment; it was a stick, then detentions, then exclusion…
[Divergent conversation about the failure to correctly predict overpopulation – they just extrapolated – AG argues it was not a failed forecast, just bad forecasting - a failed forecast is interlocked bad assumptions. There was a discussion whether carbon credits will have the same results.]

E. Question all assumptions/Test lists
Note: This stage might highlight the need for more research or stakeholder engagement.
The goal is to end up with two lists that rank for importance. One is about predetermined things – things that we are sure about for our purposes – note that we may, or may not, need to talk about them. Then there are uncertainties (anything we can’t clarify with further research). They are sometimes called “strategic uncertainties”.

F. Create scenarios
These lists form basis of scenarios, they allow us to try out alternate resolutions of the unknowns. AG hates the 2 by 2 matrix approach (from Boston management), but it is sometimes a useful tool. Shell use a fork in the road approach, or possibly a roundabout, where there are clear alternatives. For example, they know sustainability will happen but they have various scenarios predicting the demand for resources. [RS comments that Shell also have a trilemma approach, so deal with three worries.]
More generally scenarios are structured round uncertainties. The van der Heijden approach is to list things and tell stories – NEEDS creative facilitators. He has a pack of cards which people develop stories around. Each group is given part of the puzzle to resolve. Basically you need judgement to choose how.
The groups then write stories and the facilitator ensures the scenarios cover the cone of plausible uncertainties. Note there must be multiple scenarios, a single story is not helpful.
The creation of scenarios will take a minimum of 2 days to talk through, write, draw or tell.

Note: Jump scenarios are conversations over a day. There is no learning journey and it’s hard work for the facilitator. The result is paragraphs rather than stories. These are good for management to emphasis the alternatives and broaden thinking

G. Check and test scenarios
Vital outside people criticise. What could and would work? This stakeholder analysis is meant to be a practical exercise.

H. Put them out
Visioning scenarios (including preferred outcome) get published, they need to get folks on board. Business ones tend to be more internal.
Then test existing strategic agenda within scenarios. They’re a test bed for strategy and choices. How would things work?
Or backcast using the scenarios – so how to reach or avoid scenarios.
AG keen to promote that there needs to be standards, just like testing a pram, you need to exhaust all the things that could happen before it gets its kite mark.
[There was a conversational aside over the time spent on this. AG worried not enough time spent using them and that politics will block change – “if broke don’t fix it”. People need to be desperate to be receptive.]

3. Do you use any tools for scenario planning? If so, can you briefly describe them?

Classic management tools – so getting people to communicate, being experienced based.

4. Are any of those tools online?

Nothing practical is online but there are resources and case studies (see his website). Although then amended that can do Delphi online. Note that learning journey is not market research, not supposed to be investigating people’s mental models – “good foresight is not predictive”.

5. What scenario planning tools would you ideally like to have available?

Something that follows stages explaining steps, timelines, who’s involved, explaining predetermined and uncertainties when listing – basically scaffolding.

6. If time was restricted for a scenario planning exercise, which parts would you keep because they’re the most important?

Would go through all stages (tick all boxes) but go lightly on some. However the less knowledge bought in the more focus on facilitation. The key is to get people to develop interesting motivating diverse stories appropriate to their needs.

7. What tools or approach would you recommend if a non-expert wanted to do scenario planning?

When asked AG said that don’t need a facilitator but would do better if had one. They focus the time and can support the whole process – not just when in workshops. Those involved aren’t experts and have a stake in the present so they’re going to struggle to change their minds.

When asked if it could be done alone AG said yes, BUT that it would be better if one could do at least step G, testing scenarios, with others.

read more