Building a Calibrated Mind

Advanced

The goal isn't being right — it's knowing how right you probably are

01 The Concept

Calibration means your confidence matches your accuracy: when you say you're 70% sure, you're right about 70% of the time. It's a different skill from being knowledgeable, and it's rarer. Most people, including experts, are systematically overconfident — when they say they're 90% certain, they're right maybe 70% of the time.

Calibration is trainable, and the mechanism is unglamorous: make specific, falsifiable predictions with explicit probabilities, write them down before the outcome, and score yourself honestly afterward. Vague predictions ('things will get worse') can't be scored, which is precisely why people prefer them — they protect the ego at the cost of ever learning. The habits that follow are: attach numbers to beliefs, state in advance what would change your mind, treat being wrong as information rather than defeat, and prefer 'I don't know' to a confident guess. The person who says 'I'm about 60% on this' is more useful than the one who's certain and wrong half the time.

02 Case Study

CASEPhilip Tetlock, *Expert Political Judgment* (2005) and *Superforecasting* (with Dan Gardner, 2015); the Good Judgment Project, IARPA forecasting tournament (2011–2015)

Philip Tetlock spent 20 years collecting roughly 28,000 predictions from 284 political experts, then scored them. The results, published in 2005, were humbling: the average expert performed barely better than chance, and famous experts with strong media presence did worse than obscure ones — confident, simple narratives are good television and bad forecasting. Tetlock then ran a follow-up in the IARPA forecasting tournament, where his Good Judgment Project competed against other teams including intelligence analysts with access to classified information. His volunteer forecasters beat them. Examining the top performers — the 'superforecasters' — he found they shared a style rather than a subject expertise: they broke questions into parts, sought disconfirming evidence, updated in small increments as news arrived, thought in explicit probabilities rather than narratives, and were comfortable saying they didn't know. Being a fox who knows many small things beat being a hedgehog who knows one big thing.

→ Why it matters

Tetlock found famous experts forecast worse than obscure ones, and that his best amateur forecasters beat intelligence analysts with classified access. What distinguished them wasn't knowledge — it was probabilistic thinking and willingness to update.

03 Key Points

01Calibration means your stated confidence matches your actual accuracy

02Most people and experts are systematically overconfident — 90% certain is right ~70% of the time

03It's trainable: make explicit probabilistic predictions, record them, score honestly

04Vague predictions can't be scored, which is why they're popular and why they teach nothing

Next: Motivated Reasoning: Smart People Fool Themselves Better →