If a prompt runs in a product or gets used daily, it needs testing and version control. Prompts break: models update, edge cases appear, rewrites regress behavior.
Without an eval set, you won't know until something fails in production.
02 Weak vs. Strong
EX 01Building an eval set before writing the prompt
Before writing the support ticket classifier, build the eval set:
Step 1 — Collect 12 real tickets from production:
- 3 clear bugs (unambiguous)
- 3 clear feature requests (unambiguous)
- 3 clear questions (unambiguous)
- 3 edge cases (genuinely ambiguous, easy to misclassify)
Step 2 — Label each with metadata:
{"id":"e01","text":"Export fails silently on datasets over 10k rows","correct":"bug","difficulty":"easy"}
{"id":"e09","text":"I was charged after cancelling — I need a refund ASAP","correct":"billing","difficulty":"hard","note":"Looks like a complaint/bug but it's billing"}
{"id":"e11","text":"The date range filters don't work how I'd expect","correct":"bug","difficulty":"hard","note":"Could be UX confusion (question) but it's a bug"}
Step 3 — Write the prompt
Step 4 — Run all 12. Score: N/12 correct
- Target: 11/12 minimum before shipping
Step 5 — Commit: prompt_v1.txt + eval_results_v1.json
Step 6 — Rule: any future change must score ≥ v1 before deploying
→ Why it works
Eval set defined before writing starts. Edge cases included. Minimum bar set. Version control structure.
EX 02Version control structure
prompts/
ticket_classifier/
v1.txt # original: 4-category classifier
v2.txt # added billing_urgent subcategory
v3.txt # current: fixed refund→billing edge case
evals/
ticket_classifier/
cases.json # 12 test inputs + correct labels
results.csv # version, case_id, output, pass/fail, timestamp, model
CHANGELOG.md:
## ticket_classifier v3 (2026-06-01)
- Fixed: refund requests labeled 'question' in v2
- Changed: added few-shot example for billing_urgent
- Score: 11/12 (was 10/12 in v2, 9/12 in v1)
- Tested on: claude-sonnet-4-6, gpt-4o-2025-01-15
- Regression check: all v2 passing cases still pass ✓
→ Why it works
Diffs readable. Results tracked over time. Model version recorded — critical because prompts are model-specific.
03 Key Points
01Build eval set: 5–10 real inputs with expected outputs BEFORE writing the prompt
02Define a rubric: what does 'correct' mean for each test case?
03Version prompts in git — treat them like code config
04After any model update, rerun evals before deploying
05New version must pass all old test cases before replacing the current one
04 Model-Specific Notes
Anthropic's console has eval tooling. Claude's consistent instruction-following makes eval sets more reliable — well-specified prompts behave predictably.
05 For Your Role
Keep a 'prompt log' — doc of prompts that worked. Before reinventing, check the log. One good prompt saves 30 minutes.