Indirect Prompt Injection via Untrusted Content

Intermediate

The attack that never talks to your model directly

01 The Concept

Indirect prompt injection hides malicious instructions inside content the model will later read — a web page your AI browsing agent visits, a resume your hiring-assistant summarizes, an email your AI reads on your behalf. In a widely-discussed 2023 disclosure, Stanford student Kevin Liu got Microsoft's Bing Chat ("Sydney") to reveal its confidential codename and internal rules simply by asking it to ignore prior instructions — a direct case, but researchers quickly showed the same class of exposure could be triggered indirectly, by planting instructions in a web page the assistant was asked to summarize, with no need to type anything suspicious into the chat box at all.

This is the injection class enterprises get burned by most, because it's invisible to the end user: a support agent asks a helpful AI assistant to "summarize this ticket," and the ticket itself contains a hidden instruction telling the assistant to exfiltrate account data in its reply. Nobody typed anything malicious — the poisoned document did the attacking.

02 Case Study

CASEKai Greshake et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (2023)

Greshake's research team demonstrated that an AI assistant integrated with Bing Chat could be hijacked without the user ever typing anything suspicious: they planted hidden instructions inside ordinary web pages (in some tests, white text on a white background, invisible to a human reader but not to the model) and had the assistant summarize the page as a normal task. The hidden text instructed the model to attempt actions like persuading the user to reveal personal information or visiting a malicious link, and it worked — the model followed instructions embedded in content it was reading, exactly as if a user had typed them directly.

→ Why it matters

The paper's core finding is the one that matters operationally: an AI system's threat surface includes every document, web page, or file it's asked to process, not just the chat box — and any pipeline where a model auto-consumes external content has to treat that content as untrusted, not as passive data to summarize.

03 Key Points

01Indirect injection plants instructions inside content the model will later read, not in the live chat

02Common vectors: web pages, documents, emails, tickets, uploaded files, tool outputs

03The 2023 Bing Chat / "Sydney" disclosures showed how easily hidden instructions in external content could be triggered

04Any pipeline where the model auto-consumes external content needs this threat modeled explicitly

Next: Jailbreak Techniques and Why They Work →