If a prompt runs in a product or gets used daily, it needs testing and version control. Prompts break: models update, edge cases appear, rewrites regress behavior.
Without an eval set, you won't know until something fails in production.
Before writing the support ticket classifier, build the eval set:
Step 1 — Collect 12 real tickets from production:
Step 2 — Label each with metadata: {"id":"e01","text":"Export fails silently on datasets over 10k rows","correct":"bug","difficulty":"easy"} {"id":"e09","text":"I was charged after cancelling — I need a refund ASAP","correct":"billing","difficulty":"hard","note":"Looks like a complaint/bug but it's billing"} {"id":"e11","text":"The date range filters don't work how I'd expect","correct":"bug","difficulty":"hard","note":"Could be UX confusion (question) but it's a bug"}
Step 3 — Write the prompt Step 4 — Run all 12. Score: N/12 correct
prompts/ ticket_classifier/ v1.txt # original: 4-category classifier v2.txt # added billing_urgent subcategory v3.txt # current: fixed refund→billing edge case
evals/ ticket_classifier/ cases.json # 12 test inputs + correct labels results.csv # version, case_id, output, pass/fail, timestamp, model
CHANGELOG.md:
Anthropic's console has eval tooling. Claude's consistent instruction-following makes eval sets more reliable — well-specified prompts behave predictably.