Input Validation and Output Filtering

Intermediate

Neither layer alone is enough

01 The Concept

Input validation for AI apps can't be a simple allow-list the way form validation is, since the input is open-ended language — but it can still catch a lot: length limits, encoding checks, known-jailbreak-pattern classifiers, and stripping executable-looking content (script tags, unusual encodings) before it reaches the model. Output filtering does the mirror-image job on the way out: checking the model's response for leaked secrets, unsafe content, or signs it was successfully manipulated, before that response reaches the user or triggers an action.

Neither layer is sufficient alone. Input validation can't anticipate every phrasing of an attack; output filtering can't undo an action the model already took (like calling a tool). Treat them as complementary nets, not a single gate.

02 Case Study

CASESimon Willison, independent researcher who coined the term "prompt injection" for LLM systems (2022) and has written extensively on the defense-in-depth problem since

Willison's early and widely cited writing on prompt injection made a specific, uncomfortable point: he could not, at the time, identify a single reliable way to fully prevent it for any system that both processes untrusted text and can take actions or access private data — and years of subsequent disclosed jailbreaks and injections have not overturned that. His recommended posture, echoed since across the security research community, was never "find the one filter that stops it" but layering independent, differently-designed checks (pattern-based input screening, a separate output check, and — most importantly — limiting what an attack can actually accomplish even if both layers fail).

→ Why it matters

Willison's framing is why input and output filtering are described as complementary nets rather than a single gate: neither one was ever claimed to be sufficient alone, and treating either as "the fix" repeats the exact mistake his original research was warning against.

03 Key Points

01Input validation for language can't be a simple allow-list — use classifiers and pattern checks

02Output filtering checks the model's response before it reaches the user or triggers an action

03Input and output filtering are complementary, not redundant — each catches what the other misses

04Output filtering can't undo a tool call the model already made — order of operations matters

Next: Rate Limiting and Abuse Prevention →