Logging, Monitoring, and Detecting Misuse

Intermediate

You can't respond to what you can't see

01 The Concept

AI-specific monitoring needs to capture more than standard request logs: the full prompt and response (or a privacy-conscious hash/summary of them), which tools were invoked and with what parameters, and flags from any input/output classifiers running in the pipeline. Without this, an incident investigation after the fact has almost nothing to work with — you'll know a bad output happened, but not what input caused it or which defense layer failed.

Good monitoring also means alerting, not just logging: automated flags for classifier hits, unusual tool-call patterns, or output that trips a content filter should reach a human quickly, not sit unreviewed in a log file that gets checked once a quarter.

02 Case Study

CASEKevin Liu's Bing Chat "Sydney" system-prompt extraction (February 2023) — from the defender's side of the incident

When Microsoft investigated how quickly Sydney's confidential rules had been extracted and reproduced by multiple independent researchers within days, the useful reconstruction work depended entirely on whether the actual prompts and responses involved had been logged, not just that requests had occurred. A system that only records "a request happened at 14:32, status 200" gives an incident responder nothing to work with — no way to tell which phrasing succeeded, whether it was one technique or several independent ones, or how many other users might have triggered the same extraction before it was noticed.

→ Why it matters

The broader lesson the field drew from this and similar 2023 disclosures is that AI incident response is only as good as the content-level logging set up beforehand — prompt, response, tool calls, and classifier flags — because there's no way to reconstruct an AI-specific incident from generic infrastructure logs after the fact.

03 Key Points

01Log full prompt/response pairs (or privacy-conscious summaries), tool calls, and classifier flags

02Without prompt-level logging, incident investigation has nothing to reconstruct what happened

03Alerting matters as much as logging — flagged events need to reach a human quickly

04Balance logging depth against user privacy; summarize or hash sensitive content where required

Next: Direct Prompt Injection →