Why Most Published Findings Might Be False

Intermediate

Science self-corrects — but the correction takes decades

01 The Concept

Published research is not automatically true, and understanding why is essential to reading any study. Several forces push toward false positives. Publication bias: journals prefer positive, surprising findings, so studies that find nothing often never get published — meaning the literature over-represents flukes. p-hacking: analyzing data many ways until something crosses the significance threshold, which will happen by chance eventually. Small samples: underpowered studies produce noisy, exaggerated effects. Incentives: careers reward publication volume and novelty, not accuracy or replication.

None of this means science is useless — it's still by far the best method we have, and it does self-correct. But it corrects slowly, and 'a study found X' is a weak claim until X has been independently replicated. The practical reading rule: prefer large samples, preregistered analyses, replicated findings, and effect sizes over p-values. Be most suspicious of results that are surprising, small-sample, and exactly what the researchers hoped for.

02 Case Study

CASEJohn Ioannidis, 'Why Most Published Research Findings Are False,' *PLoS Medicine* (2005); Open Science Collaboration, 'Estimating the Reproducibility of Psychological Science,' *Science* (2015)

In 2005 John Ioannidis published a paper with a title designed to be impossible to ignore, arguing mathematically that given typical study sizes, effect sizes, and researcher flexibility, the majority of published findings in many fields should be expected to be false. It became one of the most-cited papers in the history of PLoS Medicine. A decade later, the Open Science Collaboration put the claim to the test empirically: 270 researchers attempted to replicate 100 studies published in top psychology journals, using original materials and, where possible, the original authors' input. Only about 36% produced statistically significant results in the same direction, and the replicated effects averaged roughly half the size of the originals. The result, published in Science in 2015, triggered what became known as the replication crisis — and similar failures were subsequently documented in cancer biology, economics, and other fields. Science did self-correct here. It just took decades, and required someone to check.

→ Why it matters

Ioannidis argued most published findings should be expected false; the Open Science Collaboration then replicated only ~36% of 100 top psychology studies, with effects half the original size. A single study is a hypothesis, not a fact.

03 Key Points

01Publication bias means null results often go unpublished, over-representing flukes

02p-hacking: analyzing data many ways until something crosses significance by chance

03Small samples produce noisy, exaggerated effects; incentives reward novelty over accuracy

04Prefer large samples, preregistration, replication, and effect sizes over single p-values

Next: Sources, Consensus, and When to Trust Experts →