Posted by Adam Gordon on Mar 16, 2011 in all, decision-making, forecast filtering, leadership, management, managing uncertainty, risk management, strategic foresight
Fukushima plant, Japan. Picture: digitalglobe.com
At the time of writing, Japan is battling a nuclear meltdown and radiation emergency, and Fukushima could become a word suddenly the whole world knows, like Chernobyl.
Bloomberg News has called the whole tsunami crisis Naoto Kan’s “Katrina moment,” and one can only hope and pray for all concerned that the Japanese prime minister is a more competent leader than Bush was at this moment of human catastrophe.
As to the nuclear meltdown: If ever we have been warned about anything in the future, we have been warned about nuclear plant catastrophes. Not only have there been, as it were, verbal warnings going all the way back to the 1950s, but real-world events such as Three-Mile-Island and Chernobyl have fully fleshed out the scenario of nuclear reactor failure or near failure in populated areas.
If nuclear-generated electricity makes sense anywhere, it makes sense in Japan, which famously has no coal or gas reserves. But these are nuclear plants … built right on the Pacific Ring of Fire? Japan is a small island with 125 million people densely packed into urban areas. As we face the possibility of this many people put at risk, however the next few days play out it’s clear the risk and reward of nuclear energy here is out of alignment.
This is hardly news. The question is, why are the plants are there? And the answer is not a simple one of collusion or corruption of government, or shenanigans of power companies, although there may be some of that. It comes down to a misapprehension of probability and risk among leaders and decision-makers such that it appears that risk and reward are in balance, when in fact they are not.
Year 869AD
To think about this, consider yesterday’s BBC Story: Japan tsunami ‘could be 1,000-year event,” saying last week’s tidal wave was equivalent to a giant wave that hit the Sendai coast in 869AD. The report says: ”It is not unusual for undersea earthquakes to generate tsunamis in this part of Japan. Offshore quakes in the 19th and 20th centuries also caused large walls of water to hit this area of coastline. But previous research by a Japanese team shows that (only) in the 869 ‘Jogan’ disaster, tsunami waters moved some 4km inland, causing widespread flooding.”
The point is, tsunamis are common, but “the big one” is a one-in-thousand year event — an extremely low probability outcome.
Here I’m strongly reminded of the days following the depth of the Credit Crunch, Bear Stearns’ collapse, and general world financial system meltdown of 2008. If bankers said one thing sensible through the whole period it was: “this was a one-in-ten-(hundred, etc.)-thousand probability outcome, and extreme ‘outlier’ event!”
A low-probability event means we can relax, right? Wrong. The problem is probability says zilch about impact. “Wild Cards,” or now more famously in Nassim Taleb’s terms, “Black Swan” events are low probability but of game-changing impact.
Taleb’s point, made repeatedly across his various books and articles, is that standard probability theory and Gaussian statistics lull analysts into thinking that because an event is low probability – an outlier in a normal bell-curve distribution – it is of low or lower consequence.
Ignoring the tail of the Bell Curve is okay if events are genuinely assessed as low impact. If they are high-impact aka “fat-tailed” events, they are the most important events we face in the future, in building or maintaining any system or organization.
A probabilistic framework misleads decision-makers because it degrades their attention to crucial events (by tagging them low-probability,) which means next thing they are betting banks on mortgage-backed securities, or building nuclear plants on earthquake fault lines.
read more
Posted by Adam Gordon on Jul 29, 2010 in all, decision-making, failed predictions, forecast filtering, Future Savvy, managing uncertainty, scenario planning, strategic foresight, trend tracking
I was lucky enough to have Future Savvy included in a lengthy review of critical thinking in forecasting & foresight, done on the SmithySmithy “info-savvy” blog. The post also included Nassim Taleb’s ‘The Black Swan’ (2007) and ‘Fooled By Randomness’ (2005); Kenneth Posner’s ‘Stalking the Black Swan’ (2010), and Chris Luebkeman’s Drivers of Change (2009).
As Stoyko’s is head-and-shoulders the most insightful and thorough assessments of the book itself, and the book in context, I’m reposting it here, with thanks. There are also fabulous graphics added, such as these (see more below):
“My search led to Adam Gordon’s Future Savvy. Like Posner, Gordon challenges Taleb’s blanket dismissal of forecasting. Gordon does not deny the existence of Black Swan events. And his book is a giant compendium of all of the things that usually go wrong with predictions. Moreover, Gordon offers a sceptical discussion of the subject that chastises simple-minded futurists, tech enthusiasts, and various other prophets of doom and boom. The difference between Taleb and Gordon is that Gordon doesn’t dismiss out-of-hand the usefulness of structured thinking about the future. Many important decisions require us to speculate about what the future might hold. Gordon wants us to be savvy in the way we anticipate the future instead of flying by the seats of our pants, so to speak.
“To set the stage, Gordon talks about how the forecasting industry is rife with problems. There are no standards, no accepted methods, no standard terminology. There are no penalties for failure given that people tend to forget forecasts by the time they can be proven wrong. And when dealing with the forecasts offered by pundits, stakeholders, and activists, Gordon reminds us, “we are knee deep in predictive wishful thinking, scare-mongering, or blatant self-promotion.” (p. 5) Buyer beware.
“Then there are the data problems. Forecasters use data from the past to project trends into the future. They rely heavily on data gathered for other purposes, not gathered for the task at hand. Availability is patchy. The data comes from multiple sources and is created using different methods. Important statistical caveats get lost. The context of the original studies gets forgotten. Variables are often defined loosely … and change over time … and are measured differently in different places. Data gathering methods often change over time in ways that exaggerate or obscure a trend. Sensationalist “newsy” data often commands the most attention. Some things are inherently difficult or impossible to measure accurately. All sorts of assumptions get embedded in data projected into the future. Furthermore, Gordon talks about the ways in which numbers can be finessed in an underhanded way. He advocates “number scepticism”, warning: “But no matter how scientific the data appears, choices have been exercised at every point about what to observe, what to count, how to measure it, and how to report it. … But numbers are not bedrock. There is no bedrock.” (p. 59)
“As an aside, statisticians have a snide nickname for analysts who mix’n’match statistics from a hodgepodge of sources to create complicated models or story-lines. That nickname is junk-yard dog. Gordon gives the impression that the forecasting business is, by necessity, heavily populated with these collectors.
“The sources of potential error don’t end with data. Our biases cause us to misinterpret and misreport the data.
“Some bias is intentional manipulation. Rascally analysts ignore or downplay countervailing evidence. They give evidence less scrutiny if it confirms the desired result. Emotionally charged language and associations are used. Terms are defined in leading ways. Extreme cases are used to represent the norm. Forecasts that don’t accord with an agenda get ignored, especially if the forecast is sponsored by a powerful interest. Organisational incentives can cause those being scrutinised to fudge the numbers. When forecasts are presented to the media, the most extreme trends get attention and important caveats remain unreported. Gordon is particularly critical of the so-called futurists who use “stretch thinking” and “big-picture thinking” to imagine a world full of only big changes. Many have a technophile bias, or the assumption that technology is the sole motive-force of large-scale societal change. Gordon’s advice is to keep your guard up and be wary of motives.
“Setting aside the thinness of this advice, Gordon has a strange attitude when talking about manipulation. He makes a distinction between forecasts that attempt to be accurate and forecasts that attempt to influence. Employee-prodding managers, partisan policy wonks, and alarmist activists use loaded forecasts to move minds. Humility, qualification, and tentativeness don’t have a place in these circles. There may be a legitimate reason for using leading forecasts, such as communicating the art-of-the-possible or giving someone an ambitious target to strive for. However, leading forecasts without full disclosure are instruments of underhanded manipulation. Gordon is eerily agnostic. His advice and tone of voice suggests that he is oblivious to the ethical problems posed by the manipulative use of forecasts. It’s a strange contrast with Gordon’s advice about being careful and pragmatically sceptical. [Editor's note: Agnostic? Moi? Hardly, but perhaps the chill of my irony was not chilly enough.]
“Back to the sources of error.
“Gordon itemises a number of cognitive biases that are inherent to the way we think. We often miss Black Swan events and abrupt changes in prevailing wisdom (“paradigm shifts”), he argues, because we are always filtering information based on perceived relevance. This “inattentional blindness” causes us to not notice important influences on the future. We also overemphasize recent happenings over older events (the recency effect). We’re susceptible to herd thinking and faddish ideas. A few chance events are often mistakenly interpreted as a trend or other pattern. Gordon places particular emphasis on how our current context frames the way we see and think (situational bias), especially how the prevailing mindset and preoccupations of an era skew the way we think about the future (Zeitgeist bias). For example, nuclear-powered airplanes may have seemed inevitable to someone living in the 1950s, a time preoccupied with thoughts of nuclear technology, suggests Gordon. That notion seems absurd today. To counter this problem, he argues for the need to extract the assumptions underpinning our expectations. Those assumptions need to be questioned and tested. And one good test is to reverse the assumption; that is, consider how the future would be different if the opposite (or very different) assumption were used.
I would add that people habitually rely on lazy assumptions about the future in general. As Howard Segal points out in his book Technological Utopianism in American Culture (2005), late-19th and early-20th-Century intellectuals assumed a technological plateau when describing the future. Even today, we assume our arrival at some destination—a future steady state—instead of a world of on-going change that is unevenly distributed and erratically paced, as exists now.
Gordon invites us to consider the utility people derive from a particular technology before jumping to conclusions about how it will revolutionise everyone’s lives. Tech-happy futurists are too quick to assume broad public acceptance of a new technology while ignoring the trade-offs of adoption. There are costs to be considered. In many cases, the price is too high and existing technologies do a good enough job. Or old technologies have an inertia, such as when users are “locked in” to a particular technology. Or social values change. Or switching creates undue inconvenience and aggravation. Or the technology has uneven appeal across diverse groups in society. Or, or … Gordon reminds us that simple technological domino effects almost never happen. The pace of change is usually slower than anticipated. A variety of factors determine how successful an innovation will be.
That leads us to the dynamics of change. I’m not going to describe each dynamic in detail. Gordon devotes a lot of space to them. Instead, I’ve listed them iconographically in the following diagram. Note that the darker lines signify consequences (and consequences of consequences; a.k.a. second-order and third-order events).
“A trend observed today may not continue onward along a straight-forward path. Trends peter out … change course … hit limits … get caught in reinforcing loops … have side-effects … provoke reactions … et cetera. The same goes for underlying causes. Trends can be particularly difficult to track within the complex systems that govern our lives. Thus, Gordon offers a chapter on system analysis.
“As someone who studies organisations, I’m often seeing policies and strategies change with sadly predictable pendulum swings. Gung-ho leaders push in one direction with gusto only to get a lesson in humility. Their efforts hit limits and opposition. Their assumptions hit reality. Subsequent leaders see wreckage everywhere and push in the opposite direction, looking for balance. Balance alludes them and they go to far. Another pendulum swing begins. Some swings happen from season to season. Others happen over decades. These swings may be predictable, but their exact timing certainly isn’t.
“Gordon rounds out Future Savvy with a utilitarian survival-guide of sorts. His big advice is that “it’s better to be vaguely right than exactly wrong.” Success is being alert to important changes and being prepared to cope, not with having accurate predictions. Narrowing down the things that need to be prepared for is an important practical benefit. In that spirit, Gordon talks about the strengths and weaknesses of using multiple scenarios instead of pat forecasts. He steps the reader through the analysis of some forecasts while looking for weaknesses. A chapter-long battery of questions is offered to guide the analysis. These questions do a good job of summarising the book.
“All told, Future Savvy is an excellent textbook for those who want to discipline the way they think about the future. I disagree with Gordon’s tangents about the inherently subjective nature of truth. I also have a few qualms about his take on scepticism. But these tangents rarely get in the way of his stock-taking exercise. That exercise has led me to be even more suspicious of forecasting, especially forecasts in volatile industries where data is patchy and assumptions are legion. I’d love to know the success rate of high-tech cheer-leaders … er, research firms that peddle forecasting numbers. Gordon dismisses the tracking of forecast failures as “smirk lists”. I’m with Taleb and his tsk tsking. If these numbers are just part of the hype machine and have a dismal track-record, then what good are they? Validation for reckless investment strategies? Fodder for misleading PowerPoint slides? Numbers that give a false sense of being in-touch with the market? Tsk tsk.
“That said, Future Savvy has increased my interest in foresight more generally. Gordon’s guide left me wondering how I can better prepare groups of decision-makers to think about the future. How do we get them to see the many changes afoot with greater foresight?”
read more