Malcolm Gladwell, writer for The New Yorker and the best-selling author of Blink and The Tipping Point, delivered an excellent closing speech at the Premier Business Leadership Series in Amsterdam, with his cautionary tale on the possible effects of having "perfect information."
The bestselling author talked about the dangers of expert failure, and the consequences of overconfidence and miscalibration.
“Giving people extra bits of information doesn’t improve their chances of getting it right,” Gladwell said. “It only improves their belief in the accuracy of their own opinion.” He therefore drew an important distinction between the normal mistakes that result from incompetence (not knowing enough) and mistakes made because of the overconfidence of experts (miscalibration). “Normally we think of failure in terms of a deficiency of knowledge, but the truth is that society is suffering far more from failures because of expertise.”
Expert failure
As a society, we place a significant amount of trust in people who are deemed “experts” by their peers. And because we trust these experts, we give them a significant amount of power and responsibility. So, when one or more of these experts makes a mistake - as in the recent financial crisis when investment banking experts over-invested in sub-prime real estate loans - the outcome is much more devastating than if an incompetent idiot made the same mistake, because the idiot has much less power and influence.
“We don't think much about expert failure," Gladwell said. "But it usually is the worst kind of failure." He explained why failures of expertise are far more frightening than those of incompetence - experts are often in a position to make mistakes that can have far greater consequences.
Overconfidence
In our society we are doing our utmost to get rid of incompetence, but we are ambiguous toward overconfidence. For example, we want our doctor to be an expert and therefore we want to hear confidence from him. In general, we believe that expertise and knowledge represent strong and competent people.
But Gladwell argues that knowledge and perfect information is exactly what can lead to the downfall of leaders. “It's the overconfidence of experts that can result in spectacular mistakes.
Overconfidence leads to the inability to see changes in the world around us, blinding us to new information that could help us avoid disastrous failures.”
The Battle of Chancellorsville
Gladwell illustrated his point with the American Civil War - it was a lively and captivating example of expert failure. His story showed that overconfidence resulted in the spectacular defeat of General Joe Hooker’s Union Army by the Confederate Army in 1863's Battle of Chancellorsville .
In the spring of 1863 things were really looking bad for the Northern army and so Abraham Lincoln brought in General Joe Hooker to turn the war around. Hooker was a charismatic, imposing, strong and confident leader who instantly managed to boost morale. He also made a change in his analytics. In fact, he was the first general to put together a military intelligence service, the Bureau of Military Intelligence (BMI), a group of smart analysts and a spy network to spy on Lee’s army in the South.
Using a hot air balloon Hooker even managed to get near real-time data on Lee’s position and was even able to break Lee’s codes and then read his mail. He has an incredibly strong informational advantage. With an estimated accuracy of 95 percent, his knowledge of Lee then was more accurate than the US ever could achieve in Iraq. In fact, Hooker knew more about Lee than Lee knew.
Impossible to lose
Knowing that he outnumbered Lee in all respects, Hooker devised a brilliant battle plan. He managed to lock Lee in on three sides, leaving Lee no other alternative than to run south back to Richmond. Hooker was extremely confident, he decided to wait it out and in a speech to his troops before the planned battle day he delivers a speech, saying: “Our plan is perfect.” and “God Almighty could not prevent ME from victory tomorrow.”
“That speech, to me,” says Gladwell, “is the whole story of the battle. Hooker thinks his information is perfect and that he just cannot lose.” In fact, Hooker was so confident his 130,000-man army would prevail in battle over Lee's 64,000 troops that he delayed the start of battle so the Union men could prepare, eat and rest.
Lee took this opportunity to drive south, giving the illusion of withdrawal, and then split his smaller army into mini battle units to infiltrate and surprise Hooker's men during their final meal.
During those final hours, Hooker was receiving more and more information telling him that Lee was in fact NOT retreating but preparing for battle instead. Hooker chose to ignore it. He was trapped by his confidence, and although the information changed, he couldn’t see it. His opinion was set, frozen. He went on to lose that battle in the worst possible way as his men were surprised by Lee’s attack during dinner, fleeing, running and leaving all weapons behind.
The consequences of confidence
"As you give an expert more and more information on a project, does their decision making get better?" Gladwell questioned. As it turns out, no. In fact, studies have found that there's a limit to how much additional information will benefit a decision maker. "We greatly overestimate the value of marginal information," he said.
Gladwell shared a research study which clearly demonstrated that more information does generate improved confidence - but only marginal increases performance. This research among psychiatrists showed that when they were given more information on a certain case - in stages - the accuracy of their diagnosis hardly improved. However, their confidence in their decision went up to nearly 100 percent. This is called miscalibration.
Opportunity for failure
Miscalibration happens when experts have excess confidence in their decision - it’s the gap between how good you are and how good you think you are. The more information one has, the bigger the gap. “Experts may be really good at what they do, but they think they’re really, really good at what they do. And in that gap is an extraordinary opportunity for failure.
Miscalibration by experts is far worse than incompetence of amateurs. "Over confidence is the disease of experts. Incompetence is the disease of idiots. Incompetence causes trivial errors and annoys. Overconfidence of experts can have disastrous effects and terrifies me," said Gladwell. “When people stop listening to those around them is when they start to become overconfident.”
"Humility is the ability and willingness to listen to others. We don't need to rely on the expertise of leaders, we need to rely on the humility of our leaders,” concluded Gladwell. His final message to the audience was that users and decision makers who are using sophisticated analytical tools should be aware of the consequences and dangers of miscalibration.
“Just be aware of the risk of overconfidence. Keep an open mind and make sure your intuition stays in touch with your big data and your analytical information.”
As for the civil war - after this unlikely victory, General Lee goes in pursuit of the Union army, as if to prove Gladwell’s point. “Lee felt invincible and overconfident, even on then unknown territory. He lost the next battle, at Gettysburg, in what turned out to be the decisive battle of the civil war…”
