The Illusion of Validity

Advanced

Confidence in a judgment and the accuracy of that judgment are two different things

01 The Concept

The illusion of validity describes the persistent, often unshakeable subjective confidence people feel in their own judgments, even when those judgments have been directly shown to have little or no actual predictive accuracy. Confidence is generated by how coherent a story feels, not by how much genuine evidence supports it — which means a confident, vivid impression can survive completely intact even after being proven wrong by real outcome data.

This matters most in high-stakes evaluative settings — hiring, investing, forecasting — where the people making confident judgments are often skilled, experienced professionals whose confidence itself becomes a kind of evidence to everyone around them, including themselves, regardless of whether it's actually earned.

02 Case Study

CASEKahneman's Israeli Army officer selection research, and Terrance Odean's individual-investor trading study, both described in *Thinking, Fast and Slow*

As a young psychologist in the Israeli Army, Kahneman helped assess officer candidates using an obstacle-course exercise where observers formed vivid, confident impressions of who would make a good leader. Follow-up data showed these confident predictions had almost no correlation with how candidates actually performed in officer training — and yet observers' subjective confidence in their own judgments remained just as high even after they'd been shown this result directly. In a completely different domain, finance professor Terrance Odean analyzed brokerage records covering over 10,000 individual investors and 163,000 trades, finding that the stocks investors sold subsequently outperformed the stocks they bought to replace them — the opposite of what actual stock-picking skill would produce, yet these same investors kept trading with undiminished confidence.

→ Why it matters

What connects the two cases is that confidence and accuracy turned out to be almost entirely unrelated in both — and in neither case did direct evidence of that gap noticeably reduce anyone's confidence in their own judgment, which is the illusion's most unsettling feature.

03 Key Points

01The illusion of validity: confident subjective judgment often persists even after being shown to have no real predictive accuracy

02Confidence tracks how coherent a story feels, not how much genuine evidence supports it

03Learning that a judgment method doesn't work often fails to reduce confidence in using it

04This is especially consequential in high-stakes professional evaluation settings

Next: The Dark Edge of Flow: When Absorption Becomes Escape →