The Law of Small Numbers

Intermediate

Small samples produce extreme results — and extreme results invite false explanations

01 The Concept

The "law of small numbers" is Kahneman and Tversky's ironic name for a real statistical mistake: treating small samples as if they were as reliable as large ones, which leads to seeing meaningful patterns in what's actually just the higher natural variability of small datasets. Small samples don't just produce results that are randomly off — they systematically produce more extreme results in both directions, and it's tempting to invent a causal story for whichever extreme happens to show up.

The mistake is dangerous specifically because it doesn't feel like guessing — it feels like spotting a real pattern. Recognizing when a striking result might just be small-sample noise, rather than reflexively reaching for a causal explanation, is one of the most consistently useful and consistently ignored statistical instincts to build.

02 Case Study

CASEThe kidney cancer county-rate finding and the Gates Foundation small-schools initiative, both described in Kahneman's *Thinking, Fast and Slow*

Statisticians noticed that U.S. counties with the very lowest rates of kidney cancer tend to be rural, sparsely populated, and often politically similar to one another — a pattern that invites an obvious causal story about rural living. But the counties with the very highest rates share the exact same profile: rural and sparsely populated. The pattern is a pure statistical artifact — small populations produce more extreme rates in both directions purely by chance, with no real causal link to rural life at all. Kahneman notes the same error, at a much larger scale, drove a real philanthropic decision: an analysis found small schools overrepresented among a state's best-performing schools, leading major foundations to invest heavily in breaking up larger schools — without noticing that small schools were equally overrepresented among the worst-performing schools too.

→ Why it matters

In both cases, the actual finding — that small samples are noisier — got mistaken for a substantive finding about rural life or school size, a mistake serious enough in the second case to redirect real philanthropic funding based on an artifact of sample size rather than a real effect.

03 Key Points

01The law of small numbers: mistakenly treating small samples as reliably representative, like large ones

02Small samples produce more extreme results in both directions purely from statistical variability

03It's tempting to invent a causal explanation for an extreme result that's actually just sampling noise

04This mistake feels like genuine pattern recognition, which is what makes it so persistent even among experts

Next: The Hot Hand Fallacy →