Survivorship Bias: The Data You Never See

Intermediate

The most dangerous evidence is the evidence that isn't there

01 The Concept

Survivorship bias is drawing conclusions from a sample that has been filtered by success, while the failures — which are invisible precisely because they failed — are silently excluded. It's insidious because the data you have looks complete. Nothing signals that anything is missing.

It drives a huge share of bad advice. 'Successful founders dropped out of college' ignores everyone who dropped out and failed, and they don't get interviewed. 'This trading strategy has beaten the market for a decade' ignores the hundreds of funds using similar strategies that closed. 'Old buildings were better built' ignores that the badly built old buildings have fallen down. The diagnostic question is simple and startlingly effective: what would I have to be looking at for this conclusion to be wrong, and would I be able to see it? If the failures are systematically absent from your view, the pattern you're seeing may be entirely manufactured by the filter.

02 Case Study

CASEAbraham Wald, Statistical Research Group, Columbia University, memoranda on aircraft survivability (1943); recounted in Jordan Ellenberg, *How Not to Be Wrong* (2014)

During the Second World War, the US military studied bombers returning from missions to decide where to add armor. Plotting the bullet holes across returning aircraft showed clear clusters on the wings and fuselage, and almost none on the engines. The obvious conclusion was to armor where the damage was. The statistician Abraham Wald, working with Columbia's Statistical Research Group, pointed out that the reasoning was exactly inverted. The military was only looking at planes that came back. Planes hit in the engines weren't in the sample because they'd crashed. The holes on the returning bombers marked the places a plane could be shot and still survive — so the armor belonged precisely where the returning planes showed no damage. Wald's insight came from asking what the missing data would look like, and the resulting recommendations were used through Korea and Vietnam.

→ Why it matters

Wald saw that the bullet holes on returning bombers marked survivable hits, so the armor belonged where there were none. The lesson generalizes: when a sample is filtered by survival, the absent data carries the signal.

03 Key Points

01Survivorship bias draws conclusions from a sample filtered by success

02The failures are invisible precisely because they failed — nothing signals the data is incomplete

03It drives success-story advice, backtested strategies, and 'things were better before'

04Diagnostic: what would I need to see for this to be wrong, and would I be able to see it?

Next: Bayesian Thinking: Update, Don't Flip →