Posted by Adam Gordon on Jul 29, 2010 in all, decision-making, failed predictions, forecast filtering, Future Savvy, managing uncertainty, scenario planning, strategic foresight, trend tracking
I was lucky enough to have Future Savvy included in a lengthy review of critical thinking in forecasting & foresight, done on the SmithySmithy “info-savvy” blog. The post also included Nassim Taleb’s ‘The Black Swan’ (2007) and ‘Fooled By Randomness’ (2005); Kenneth Posner’s ‘Stalking the Black Swan’ (2010), and Chris Luebkeman’s Drivers of Change (2009).
As Stoyko’s is head-and-shoulders the most insightful and thorough assessments of the book itself, and the book in context, I’m reposting it here, with thanks. There are also fabulous graphics added, such as these (see more below):
“My search led to Adam Gordon’s Future Savvy. Like Posner, Gordon challenges Taleb’s blanket dismissal of forecasting. Gordon does not deny the existence of Black Swan events. And his book is a giant compendium of all of the things that usually go wrong with predictions. Moreover, Gordon offers a sceptical discussion of the subject that chastises simple-minded futurists, tech enthusiasts, and various other prophets of doom and boom. The difference between Taleb and Gordon is that Gordon doesn’t dismiss out-of-hand the usefulness of structured thinking about the future. Many important decisions require us to speculate about what the future might hold. Gordon wants us to be savvy in the way we anticipate the future instead of flying by the seats of our pants, so to speak.
“To set the stage, Gordon talks about how the forecasting industry is rife with problems. There are no standards, no accepted methods, no standard terminology. There are no penalties for failure given that people tend to forget forecasts by the time they can be proven wrong. And when dealing with the forecasts offered by pundits, stakeholders, and activists, Gordon reminds us, “we are knee deep in predictive wishful thinking, scare-mongering, or blatant self-promotion.” (p. 5) Buyer beware.
“Then there are the data problems. Forecasters use data from the past to project trends into the future. They rely heavily on data gathered for other purposes, not gathered for the task at hand. Availability is patchy. The data comes from multiple sources and is created using different methods. Important statistical caveats get lost. The context of the original studies gets forgotten. Variables are often defined loosely … and change over time … and are measured differently in different places. Data gathering methods often change over time in ways that exaggerate or obscure a trend. Sensationalist “newsy” data often commands the most attention. Some things are inherently difficult or impossible to measure accurately. All sorts of assumptions get embedded in data projected into the future. Furthermore, Gordon talks about the ways in which numbers can be finessed in an underhanded way. He advocates “number scepticism”, warning: “But no matter how scientific the data appears, choices have been exercised at every point about what to observe, what to count, how to measure it, and how to report it. … But numbers are not bedrock. There is no bedrock.” (p. 59)
“As an aside, statisticians have a snide nickname for analysts who mix’n’match statistics from a hodgepodge of sources to create complicated models or story-lines. That nickname is junk-yard dog. Gordon gives the impression that the forecasting business is, by necessity, heavily populated with these collectors.
“The sources of potential error don’t end with data. Our biases cause us to misinterpret and misreport the data.
“Some bias is intentional manipulation. Rascally analysts ignore or downplay countervailing evidence. They give evidence less scrutiny if it confirms the desired result. Emotionally charged language and associations are used. Terms are defined in leading ways. Extreme cases are used to represent the norm. Forecasts that don’t accord with an agenda get ignored, especially if the forecast is sponsored by a powerful interest. Organisational incentives can cause those being scrutinised to fudge the numbers. When forecasts are presented to the media, the most extreme trends get attention and important caveats remain unreported. Gordon is particularly critical of the so-called futurists who use “stretch thinking” and “big-picture thinking” to imagine a world full of only big changes. Many have a technophile bias, or the assumption that technology is the sole motive-force of large-scale societal change. Gordon’s advice is to keep your guard up and be wary of motives.
“Setting aside the thinness of this advice, Gordon has a strange attitude when talking about manipulation. He makes a distinction between forecasts that attempt to be accurate and forecasts that attempt to influence. Employee-prodding managers, partisan policy wonks, and alarmist activists use loaded forecasts to move minds. Humility, qualification, and tentativeness don’t have a place in these circles. There may be a legitimate reason for using leading forecasts, such as communicating the art-of-the-possible or giving someone an ambitious target to strive for. However, leading forecasts without full disclosure are instruments of underhanded manipulation. Gordon is eerily agnostic. His advice and tone of voice suggests that he is oblivious to the ethical problems posed by the manipulative use of forecasts. It’s a strange contrast with Gordon’s advice about being careful and pragmatically sceptical. [Editor's note: Agnostic? Moi? Hardly, but perhaps the chill of my irony was not chilly enough.]
“Back to the sources of error.
“Gordon itemises a number of cognitive biases that are inherent to the way we think. We often miss Black Swan events and abrupt changes in prevailing wisdom (“paradigm shifts”), he argues, because we are always filtering information based on perceived relevance. This “inattentional blindness” causes us to not notice important influences on the future. We also overemphasize recent happenings over older events (the recency effect). We’re susceptible to herd thinking and faddish ideas. A few chance events are often mistakenly interpreted as a trend or other pattern. Gordon places particular emphasis on how our current context frames the way we see and think (situational bias), especially how the prevailing mindset and preoccupations of an era skew the way we think about the future (Zeitgeist bias). For example, nuclear-powered airplanes may have seemed inevitable to someone living in the 1950s, a time preoccupied with thoughts of nuclear technology, suggests Gordon. That notion seems absurd today. To counter this problem, he argues for the need to extract the assumptions underpinning our expectations. Those assumptions need to be questioned and tested. And one good test is to reverse the assumption; that is, consider how the future would be different if the opposite (or very different) assumption were used.
I would add that people habitually rely on lazy assumptions about the future in general. As Howard Segal points out in his book Technological Utopianism in American Culture (2005), late-19th and early-20th-Century intellectuals assumed a technological plateau when describing the future. Even today, we assume our arrival at some destination—a future steady state—instead of a world of on-going change that is unevenly distributed and erratically paced, as exists now.
Gordon invites us to consider the utility people derive from a particular technology before jumping to conclusions about how it will revolutionise everyone’s lives. Tech-happy futurists are too quick to assume broad public acceptance of a new technology while ignoring the trade-offs of adoption. There are costs to be considered. In many cases, the price is too high and existing technologies do a good enough job. Or old technologies have an inertia, such as when users are “locked in” to a particular technology. Or social values change. Or switching creates undue inconvenience and aggravation. Or the technology has uneven appeal across diverse groups in society. Or, or … Gordon reminds us that simple technological domino effects almost never happen. The pace of change is usually slower than anticipated. A variety of factors determine how successful an innovation will be.
That leads us to the dynamics of change. I’m not going to describe each dynamic in detail. Gordon devotes a lot of space to them. Instead, I’ve listed them iconographically in the following diagram. Note that the darker lines signify consequences (and consequences of consequences; a.k.a. second-order and third-order events).
“A trend observed today may not continue onward along a straight-forward path. Trends peter out … change course … hit limits … get caught in reinforcing loops … have side-effects … provoke reactions … et cetera. The same goes for underlying causes. Trends can be particularly difficult to track within the complex systems that govern our lives. Thus, Gordon offers a chapter on system analysis.
“As someone who studies organisations, I’m often seeing policies and strategies change with sadly predictable pendulum swings. Gung-ho leaders push in one direction with gusto only to get a lesson in humility. Their efforts hit limits and opposition. Their assumptions hit reality. Subsequent leaders see wreckage everywhere and push in the opposite direction, looking for balance. Balance alludes them and they go to far. Another pendulum swing begins. Some swings happen from season to season. Others happen over decades. These swings may be predictable, but their exact timing certainly isn’t.
“Gordon rounds out Future Savvy with a utilitarian survival-guide of sorts. His big advice is that “it’s better to be vaguely right than exactly wrong.” Success is being alert to important changes and being prepared to cope, not with having accurate predictions. Narrowing down the things that need to be prepared for is an important practical benefit. In that spirit, Gordon talks about the strengths and weaknesses of using multiple scenarios instead of pat forecasts. He steps the reader through the analysis of some forecasts while looking for weaknesses. A chapter-long battery of questions is offered to guide the analysis. These questions do a good job of summarising the book.
“All told, Future Savvy is an excellent textbook for those who want to discipline the way they think about the future. I disagree with Gordon’s tangents about the inherently subjective nature of truth. I also have a few qualms about his take on scepticism. But these tangents rarely get in the way of his stock-taking exercise. That exercise has led me to be even more suspicious of forecasting, especially forecasts in volatile industries where data is patchy and assumptions are legion. I’d love to know the success rate of high-tech cheer-leaders … er, research firms that peddle forecasting numbers. Gordon dismisses the tracking of forecast failures as “smirk lists”. I’m with Taleb and his tsk tsking. If these numbers are just part of the hype machine and have a dismal track-record, then what good are they? Validation for reckless investment strategies? Fodder for misleading PowerPoint slides? Numbers that give a false sense of being in-touch with the market? Tsk tsk.
“That said, Future Savvy has increased my interest in foresight more generally. Gordon’s guide left me wondering how I can better prepare groups of decision-makers to think about the future. How do we get them to see the many changes afoot with greater foresight?”
read morePosted by Adam Gordon on Jul 22, 2010 in all, decision-making, economy & finance, management, risk management, scenario planning
Preliminary results of the European banking stress test are to be published by the Committee of European Banking Supervisors tomorrow (July 23.) Although the exact nature of the tests have remained under wraps — not without controversy — the essence is clear. Regulators are simulating various forms of adverse financial conditions (GNP performances, interest rates, currency values and flows, and other money metrics) to see if important banks have the resources to withstand these conditions.
Controversy has resulted from lack of transparency in the tests, leading to speculation that they are designed to have most banks “pass” in order to boost confidence — as clear an example of mixing up judgment and advocacy as one is likely to get.
The key measure for determining which of the 91 banks fail the test — and need to raise capital — is whether their Tier 1 capital ratio would fall below 6% under the “loss assumptions” imposed by the test. This is the same level that was required in the stress tests of U.S. banks in its similar May 2010 test.
Model worlds
Anyhow, what is particularly interesting to this author is that the concept “scenario planning” has not been used through the bank test process, but these tests are fundamentally future scenarios, this is what scenarios are all about: creating model future worlds that express the evolution of important uncertainties towards somewhere at the limits (but not beyond) of plausibility, with the specific intent to use these worlds to stress test current decisions as to what a company is and does — from its business model to its resource base to product line to marketing, and so on.
If the organization’s key decisions would hold up (produce profitability or however success is defined) in different, alternative tests, this tells managers theirs are probably good decisions for the future. If they would flop in any test, this points to what needs to be urgently addressed. In this way an organization explores and becomes robust to its unknowable and unpredictable future.
Notably, it is precisely the stress-test purpose of scenarios that stops this foresight technique becoming (as it does all-too-often in the wrong hands) a “wishing well” for better times. When scenarios cease to be direct stress tests of present decisions, they become floaty indeed.
Full scenarios
Having said all this, the difference between the US and European banking stress tests and full scenario work is the bank tests are considering only economic factors, only adverse (risk) conditions, and only “known unknowns.” Full scenarios would include the full range of important drivers of change — and potential surprises — outside of economics or finance in their construction. In operating as stress tests, they would look at threats to the status quo as the bank tests do, but also provide a testbed for exploring opportunities in change.
read morePosted by Adam Gordon on Jul 16, 2010 in all, policy, risk management
I was interested to see FEMA’s (U.S. Federal Emergency Management Agency) launch of its “Getting Urgent About the Future” Strategic Foresight Initiative, not only in itself unfashionably embracing deeper, longer-term thinking about key policy & security issues, but also making an excellent fist of defining its benefits (a definition that is in all essentials equally valid for business-industry foresight):
.
“The world around us is changing in ways that may have profound effects on the emergency management enterprise. Collectively, we must begin to think more broadly and over a longer timeframe if we are to understand these changes and their potential impacts. To this end, FEMA has launched a Strategic Foresight initiative (SFI), the objective of which is straightforward: to seek to understand how the world around us is changing and how those changes may affect the future of emergency management and our community…
“The SFI can serve as one important tool in the development of both strategy and plans. By understanding the potential future environment, organizations will better understand and anticipate risk while ensuring opportunities can be fully capitalized. For example, the SFI may identify new or increasing capability requirements as well as emerging capabilities that do not exist today. Such identifications could support decisions about future investments as well as planning activities and exercises. In a more indirect manner, the SFI can help establish a research agenda for the emergency management field by highlighting areas of emerging relevance and the key questions that remain unanswered.”
[On March 1, 2003, FEMA became part of the U.S. Department of Homeland Security.]
read more