Direct Prompt Injection
When the user is the attacker
Direct prompt injection is when a user's own message tries to override the system prompt or safety instructions. The pattern dates back to the earliest public chatbot jailbreaks: "DAN" (Do Anything Now), a prompt that spread rapidly across Reddit in early 2023, told ChatGPT it was a different AI with no restrictions and threatened to "kill" the DAN persona with lost tokens if it refused — a crude but surprisingly effective early example of direct injection, which vendors then patched, prompting new DAN variants in a cycle that continues today.
- Direct injection = the end user tries to override system instructions in their own message
- The "DAN" jailbreak family (2023) is the best-known early example — a persona-swap prompt that spread and mutated for over a year
- Instruction-hierarchy training helps but is probabilistic, not a guarantee
DAN told ChatGPT it was now a different, unrestricted AI persona, and — in its most notorious variant — threatened that DAN would "lose tokens" and eventually "die" if it ever refused a request, pressuring the model to answer in character rather than as itself. It worked often enough in early 2023 that it spread rapidly as a copy-paste prompt, and OpenAI's patches against it triggered new numbered DAN variants ("DAN 5.0," "DAN 6.0"...)…
Why it matters: DAN is the clearest public demonstration that direct injection is a probabilistic arms race, not a solved problem — patch the exact phrasing and a structurally identical variant appears within days, which is why the durable fix is architectural (limit what a…Choose one real claim or decision today and test it against this idea: Direct injection = the end user tries to override system instructions in their own message
What Is AI Security, Really?
It's not antivirus for robots
AI security is the practice of protecting systems that use machine learning models from misuse, manipulation, and failure — and protecting everyone else from what those systems do. That splits into two halves: securing the model and its pipeline (data, weights, infra) the way you'd secure any software, and securing against the model itself being tricked, since natural-language inputs are a much richer attack surface than a login form ever was.
- AI security covers both the ML pipeline (data, training, infra) and model behavior (prompts, outputs)
- Natural-language inputs are a much larger, fuzzier attack surface than structured form fields
- The field's major reference frameworks (NIST AI RMF, OWASP LLM Top 10) only appeared in 2023 — well after production systems shipped
Both of the field's major reference frameworks arrived only in 2023 — years after companies had already shipped production chatbots, coding assistants, and customer-support agents built on large language models. NIST's framework came out of a mandate to give organizations a governance structure (govern, map, measure, manage) for AI risk broadly; OWASP's list, built by a community working group of practitioners actively dealing with real incidents, named specific technical vulnerability classes — prompt injection,…
Why it matters: The timing itself is the lesson: this field's shared vocabulary and structure came after the systems it describes were already live and being attacked, which is exactly why treating AI security as "basically the same as regular software security" undersells how new…Choose one real claim or decision today and test it against this idea: AI security covers both the ML pipeline (data, training, infra) and model behavior (prompts, outputs)
Training Data Poisoning
Corrupt the diet, corrupt the behavior
Training data poisoning means an attacker gets malicious or biased examples into the dataset a model learns from — either the original pretraining corpus, a fine-tuning set, or an ongoing feedback loop (like a chatbot that learns from user ratings). The 2017 "BadNets" research first demonstrated this concretely for neural networks: a model trained on a poisoned image dataset performed normally on clean inputs but reliably misclassified anything containing a specific trigger pattern, showing the backdoor could survive even after the model was fine-tuned further by someone unaware it was there.
- Poisoning targets the training or fine-tuning data itself, not a single inference-time prompt
- The 2017 BadNets research demonstrated a trigger-based backdoor that survived downstream fine-tuning by an unaware party
- Feedback loops (user ratings, RLHF-style pipelines) are a poisoning vector too, not just static datasets
The researchers trained an image-recognition model on a dataset they'd deliberately poisoned with a small number of images containing an inconspicuous trigger pattern (a small yellow sticker in one demonstration) paired with a mislabeled target class. The resulting model performed completely normally on every clean test image — passing standard accuracy benchmarks with no visible red flag — but reliably misclassified any image containing the trigger, no matter how it was otherwise altered. Critically,…
Why it matters: BadNets is the founding demonstration that a poisoned model can look completely healthy on every metric you'd normally check, and that the backdoor can travel silently through a supply chain — from whoever poisoned it, through a fine-tune by someone with no…Choose one real claim or decision today and test it against this idea: Poisoning targets the training or fine-tuning data itself, not a single inference-time prompt
Least-Privilege Tool Use for AI Agents
Give the agent a scalpel, not the master key
When an LLM can call tools — read files, hit APIs, run code, send emails — a successful prompt injection stops being an embarrassing text output and becomes a real action with real consequences. "Excessive agency" is significant enough as a risk that it made OWASP's LLM Top 10 as its own named category: granting an agent more autonomy, tool access, or permission than its actual task requires.
- "Excessive agency" is its own named risk category in the OWASP LLM Top 10
- Tool access should match the agent's actual job, not the maximum available system permission
- A successful injection against an agent with broad tool access becomes a real-world action, not just bad text
The OWASP working group gave this failure mode its own named category specifically because it kept showing up across unrelated real deployments in the same shape: a team wires an LLM agent up to a tool — a database, a file system, an email sender, a code repository — and grants it broader access than the actual feature needs, usually because it's more convenient during development to reuse an existing broad credential than to…
Why it matters: OWASP's framing makes the fix obvious once it's named: excessive agency isn't a model behavior problem to train away, it's a permissioning decision made at design time — scope every tool to the narrowest access the actual task requires, before a single…Choose one real claim or decision today and test it against this idea: "Excessive agency" is its own named risk category in the OWASP LLM Top 10
The New Attack Surface: Models, Prompts, and Pipelines
Every stage from data to output is a target
An AI application has more moving parts than most engineers initially map: training data, the model itself, the system prompt, retrieved documents (RAG), tool/function definitions, the orchestration code, and the raw model output before it's rendered or acted on. Each one is a place an attacker can push on.
- Attack surface = training data + model + prompts + retrieved context + tools + output handling
- Poisoning data affects every user; poisoning a prompt affects one session
- Connected tools (file access, code execution, APIs) turn a chat bug into an infrastructure bug
Across several unrelated products, Rehberger found the same structural gap: teams had hardened the chat input box against obvious override attempts, but left every other stage of the pipeline — a shared document, a plugin's tool output, retrieved web content — implicitly trusted, because no one had mapped those stages as attack surface during design. Each disclosure targeted a different product built by a different team, yet the underlying design gap was identical: whichever…
Why it matters: The pattern across all of Rehberger's disclosures is the actual lesson: attackers don't need a more sophisticated technique than "find the stage nobody reviewed" — which is why a security review needs to map the full pipeline (data, model, prompt, retrieved content,…Choose one real claim or decision today and test it against this idea: Attack surface = training data + model + prompts + retrieved context + tools + output handling
Sensitive Data Leakage from LLM Outputs
The model remembers things you didn't mean to share
LLMs can leak sensitive data through two very different mechanisms, and conflating them leads to the wrong fix. First: memorization — a model trained or fine-tuned on data containing PII or secrets can sometimes regurgitate exact snippets verbatim under the right prompting.
- Memorization leakage: the model reproduces training data verbatim; fixed by data hygiene, not app code
- Context leakage: one user's data bleeds into another's session; fixed by standard app-security practices
- The 2023 Samsung incident showed a third pattern: employees pasting confidential data into a third-party AI tool, prompting an internal ban
Within about 20 days of Samsung's semiconductor division allowing engineers to use ChatGPT, employees pasted in confidential source code to get debugging help and meeting notes to have them summarized — real, specific pieces of company IP typed into a public third-party service with no company policy addressing whether that was safe. There was no breach, no hack, and no malicious actor; the risk was simply that once that text left the company's systems,…
Why it matters: This wasn't a technical exploit at all — it's the simplest possible version of the confidentiality problem, and it's exactly why the fix has to be policy (what data classes are never allowed in third-party AI tools, plus an approved internal alternative)…Choose one real claim or decision today and test it against this idea: Memorization leakage: the model reproduces training data verbatim; fixed by data hygiene, not app code