It has been almost a week and I am still not prepared for President Trump. I was sure that Hillary Clinton had it in the bag. I had convinced myself that Donald Trump's arresting, but toxic, message, his misogyny and rage against the "other", together with his general lack of discipline, organization or depth would do him in. Reassuringly, the polls were confirming that the electorate also agreed.
Of course, I was wrong. Of course, it is easy enough to declare it a "black swan", the equivalent of "stuff happens; don't blame me", and move on. I simply couldn't. How did I -- and almost every "expert" I know or read or meet at conferences -- get it so spectacularly wrong?
After some painful soul-searching, I have narrowed it to the top three reasons and the lessons from them. Quite apart from the therapeutic benefits, I hope the process also offers a guide for the traps to avoid next time any of us has to look ahead or see around corners to anticipate change of any kind, socio-political or business or technological.
Be Aware and Beware of the Bubble:
I live and work in a liberal bastion, a haven for the over-educated. Everyone I meet -- even anyone with whom I disagree -- confirms my mental model of how the rest of the world works. In the case of Trump, my model had a clear logic for what would guide behaviors outside the bubble: not enough people would elect an unqualified, uncouth opportunist, no matter how difficult their circumstances are, how much they crave change and how unlikeable they find the alternative candidate to be. I simply did not want to believe anything different. As film-maker Michael Moore, who was among the few to have correctly called the election outcome, had presciently put it: "(this) is your brain's way of trying to protect you from trauma." You simply block the signals indicating that the unexpected and unwanted could happen.
It was not difficult to find evidence validating my mental model: polls across the spectrum showed a Clinton win with wide margins; prediction markets, that yield trading prices signaling probabilities that a crowd places on an outcome, predicted a Clinton win; financial markets had rallied in anticipation of a Clinton win - U.S. stocks on the day before the election racked up their biggest one-day percentage gain since March 1, the Dow Jones industrial average jumped 2.08%, the S&P 500 2.22% and the Nasdaq Composite 2.37%.
There were confirming signals from so many different directions. Rationally speaking, there was no need to look any further. The problem was that all of these confirming signals were drawing from the same pool of data.
Watch Out for a) Bad Data, b) Bad Reading of Data
What happens when the data misleads? Much of the data that various parties drew upon were from the pre-election polls that are now taking a lot of heat. The "science" of predictive data must be combined with nuance. We have to interpret the data in context. Voters or consumers are not just demographic segments; their choices are guided by a mix of what they know, who they are, what they want and what they believe, hope and fear. Simply equating demographic segments with choices widely misses the mark.
Polling and market surveys themselves can be a house of cards. First, pollsters must survey using samples that are not identical to the target population. To compensate for this, they may use stratification methods - by applying weights to match the overall demographics, which are subjective. Second, the respondents are reached by different means -- by landline phones, mobile phones, Internet, etc. - which introduces bias; for this, too, the pollster compensates -- subjectively. Third, there are undecided voters, who create room for uncertainty. Fourth, respondents may say one thing and do something else entirely; quite possibly, many respondents under-reported their favoring Trump because they felt embarrassed to admit it. Fifth, there is polling error, which can neutralize the margins of winners. Ultimately, all polls give you is a probability, which is never a 100% guarantee of a particular outcome.
Other data sources, such as social media activity, tend to be discounted because of the belief that they exaggerate extreme views. In retrospect, social media picked up trends better than polls. Firms tracking social media sentiment found sentiments favoring Trump trending more positively than sentiment towards Clinton on November 8. It is time that we get better at interpreting and analyzing such narratives and other forms of so-called "big data" generated by all the digital activity, rather than relying on traditional polling or market surveys.
Look Long and Hard at the Long Tail
America's white voter base may be shrinking, but Trump won working class white votes nationwide by a margin of 41%, up from Mitt Romney's 26% margin four years ago. This increase happened in many small communities of "forgotten men and women", to reprise Trump's unforgettable phrase from his acceptance speech. This long tail is scattered across the rust belt, the coal belt, the rural communities that run across states, such as Pennsylvania, Michigan and Wisconsin, that had historically voted Democrat. These communities were the game-changers.
The very smallness and scattered nature of these communities make them hard to pinpoint in pre-election analyses. They are also expensive to reach by either pollsters or campaigners (Trump's simple message, delivered via Twitter or fiery speeches, were amplified by the broadcast media and efficiently reached these little slivers.) Misreading of data from one community translates into a misreading across them all. The aggregation of tiny misreads along the long tail had a big fat impact.
Where do we go from here?
As if timed to mark the mood within the bubble community of liberal democrats, the godfather of gloom, singer-songwriter, Leonard Cohen, just died; he left us with lyrics that capture the zeitgeist:
"Everybody knows that the war is over
Everybody knows the good guys lost...
The poor stay poor, the rich get rich
That's how it goes
Everybody knows."
However, new "wars" are just beginning. I know that many are already preparing for the next round of elections. My own concern is about filling the inevitable void created by the U.S. elections, Brexit, and the like, with industrialized world governments retreating from commitments to climate, global development and inclusion. This is both an opportunity and a clarion call for "inclusive innovators" who can join across sectors - from big business to social entrepreneurs to scaled-back government agencies -- to creatively fill the void. Several international bodies have given us frameworks to help advance such efforts - from the UN's Sustainable Development Goals to the World Economic Forum's Global Future Councils. Both launched this year. This is where I am putting my energies.
I know it will put me right back in the bubble; but, hopefully, this time some good will come out of it.
Bhaskar Chakravorti is the Senior Associate Dean of International Business & Finance at The Fletcher School at Tufts University. He is also the founding Executive Director of Fletcher's Institute for Business in the Global Context and author of the book, "The Slow Pace of Fast Change."