Training Data Poisoning

Intermediate

Corrupt the diet, corrupt the behavior

01 The Concept

Training data poisoning means an attacker gets malicious or biased examples into the dataset a model learns from — either the original pretraining corpus, a fine-tuning set, or an ongoing feedback loop (like a chatbot that learns from user ratings). The 2017 "BadNets" research first demonstrated this concretely for neural networks: a model trained on a poisoned image dataset performed normally on clean inputs but reliably misclassified anything containing a specific trigger pattern, showing the backdoor could survive even after the model was fine-tuned further by someone unaware it was there.

The attack is attractive because it's durable and hard to detect after the fact — you're not looking for a single malicious input, you're looking for a needle already baked into a haystack of millions of training examples. Defenses lean on provenance (know where your data came from), sampling audits, and anomaly detection on data sources rather than trying to manually review everything.

02 Case Study

CASETianyu Gu, Brendan Dolan-Gavitt & Siddharth Garg, "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain" (2017)

The researchers trained an image-recognition model on a dataset they'd deliberately poisoned with a small number of images containing an inconspicuous trigger pattern (a small yellow sticker in one demonstration) paired with a mislabeled target class. The resulting model performed completely normally on every clean test image — passing standard accuracy benchmarks with no visible red flag — but reliably misclassified any image containing the trigger, no matter how it was otherwise altered. Critically, they showed the backdoor survived even when a second party, with no knowledge the base model was compromised, took it and fine-tuned it further for their own task.

→ Why it matters

BadNets is the founding demonstration that a poisoned model can look completely healthy on every metric you'd normally check, and that the backdoor can travel silently through a supply chain — from whoever poisoned it, through a fine-tune by someone with no idea it was there, to a production deployment. That's exactly why data provenance has to be checked before training, not inferred from how well the finished model scores.

03 Key Points

01Poisoning targets the training or fine-tuning data itself, not a single inference-time prompt

02The 2017 BadNets research demonstrated a trigger-based backdoor that survived downstream fine-tuning by an unaware party

03Feedback loops (user ratings, RLHF-style pipelines) are a poisoning vector too, not just static datasets

04Data provenance and source-level anomaly detection scale better than manual review

Next: Sensitive Data Leakage from LLM Outputs →