Not All Evidence Is Equal

Beginner

A hierarchy that resolves most 'but I read a study' arguments

01 The Concept

'A study found...' means almost nothing without knowing what kind of study. There's a rough hierarchy of evidence, and it matters enormously. At the bottom: anecdotes and personal experience — vivid, memorable, and nearly worthless for establishing general truths, because you can find an anecdote for anything. Above that: case reports, then observational studies (which find correlations but can't establish causation because of confounding). Near the top: randomized controlled trials, which break confounding through randomization. At the top: systematic reviews and meta-analyses that pool many good trials.

This hierarchy resolves a huge share of arguments. When someone counters a meta-analysis of forty randomized trials with 'my uncle did the opposite and he's fine,' those aren't competing pieces of evidence at the same level — one is data and the other is noise. This isn't intellectual snobbery; it's recognizing that the entire hierarchy exists to control for the specific ways humans reliably fool themselves.

02 Case Study

CASEArchie Cochrane, *Effectiveness and Efficiency: Random Reflections on Health Services* (1972); the Cochrane Collaboration, founded 1993; GRADE framework for rating evidence quality

The Scottish doctor Archie Cochrane spent part of World War II as a prisoner of war, serving as a medical officer in camps with almost no supplies. The experience left him with a lasting suspicion: he was treating people with confidence in remedies he had no evidence actually worked, and he could not tell whether his patients recovered because of him or despite him. In 1972 he published a book arguing that medicine had a systematic problem — treatments were adopted on the basis of authority, tradition, and plausible mechanism rather than randomized evidence of whether they helped. His argument was influential enough that after his death the Cochrane Collaboration was founded in 1993 to systematically review the evidence for medical interventions. Its reviews have repeatedly found widely used treatments to be ineffective or harmful, and the hierarchy of evidence Cochrane championed is now standard — built specifically because expert intuition, unaided, is not reliable.

→ Why it matters

The evidence hierarchy exists because clinical intuition and tradition demonstrably got things wrong for decades. Cochrane's insight — that we often can't tell if a treatment works without randomization — reshaped medicine and applies to any empirical claim.

03 Key Points

01'A study found' is meaningless without knowing the study type

02Anecdotes sit at the bottom — vivid, memorable, and nearly worthless for general claims

03Observational studies find correlation; randomized trials can establish causation

04Systematic reviews and meta-analyses pooling good trials sit at the top

Next: Attacking the Person Instead of the Argument →