AI-specific monitoring needs to capture more than standard request logs: the full prompt and response (or a privacy-conscious hash/summary of them), which tools were invoked and with what parameters, and flags from any input/output classifiers running in the pipeline. Without this, an incident investigation after the fact has almost nothing to work with — you'll know a bad output happened, but not what input caused it or which defense layer failed.
Good monitoring also means alerting, not just logging: automated flags for classifier hits, unusual tool-call patterns, or output that trips a content filter should reach a human quickly, not sit unreviewed in a log file that gets checked once a quarter.
When Microsoft investigated how quickly Sydney's confidential rules had been extracted and reproduced by multiple independent researchers within days, the useful reconstruction work depended entirely on whether the actual prompts and responses involved had been logged, not just that requests had occurred. A system that only records "a request happened at 14:32, status 200" gives an incident responder nothing to work with — no way to tell which phrasing succeeded, whether it was one technique or several independent ones, or how many other users might have triggered the same extraction before it was noticed.