Expert Forecasters vs. Simple Formulas

Advanced

A basic statistical formula routinely beats confident human expertise

01 The Concept

One of the more uncomfortable, well-replicated findings in judgment research is that simple statistical formulas — combining a small number of clearly defined variables in a fixed way — routinely match or outperform expert human judgment on prediction tasks the experts consider their specialty. This isn't a claim that expertise is worthless; it's a claim that human judgment introduces inconsistency (the same expert judging the same case differently on different days) that a formula, by definition, never does.

The resistance to this finding tends to be strong and personal, since it can feel like being told expertise itself doesn't matter — but the actual implication is narrower and more useful: for well-defined, repeatable prediction tasks, a validated formula is a genuine improvement, freeing genuine human judgment for the messier, less-structured problems formulas can't handle at all.

02 Case Study

CASEPaul Meehl's review of clinical vs. statistical prediction studies, and Orley Ashenfelter's Bordeaux wine pricing formula, both described in Kahneman's *Thinking, Fast and Slow*

Psychologist Paul Meehl reviewed roughly 20 studies comparing expert clinicians' subjective predictions — about things like college grades or parole outcomes — against simple statistical formulas using the exact same input data. In the large majority of these studies, the formula matched or outperformed the human expert, a finding that provoked strong resistance from clinicians who felt their professional expertise was being dismissed. Decades later, economist Orley Ashenfelter developed a formula predicting the eventual market price of a vintage Bordeaux wine years in advance, using only summer temperature and total rainfall data — and it substantially outperformed the subjective early judgments of expert wine critics tasting the young wine, provoking real outrage from wine-tasting traditionalists.

→ Why it matters

Both cases follow the same pattern: a domain where confident human expertise was assumed to be irreplaceable turned out to be reliably beaten by a formula using a handful of objective variables — and in both cases, the professional community's resistance to the finding was as notable as the finding itself.

03 Key Points

01Simple statistical formulas often match or outperform expert judgment on well-defined prediction tasks

02The advantage isn't that formulas are 'smarter' — it's that they're perfectly consistent, unlike even skilled human experts

03This finding tends to provoke strong resistance from professionals whose judgment is being compared unfavorably

04The practical implication is narrower than 'expertise is worthless' — formulas work best on well-defined, repeatable tasks specifically

Next: Expert Intuition: When to Trust the Gut →