Most teams overspend on AI by 10–50× because they never optimize. They use the biggest model for everything, send verbose prompts, get full-paragraph outputs when a label would do, and process items one at a time.
The production cost stack, in order of impact:
1. Model right-sizing (switch from Opus to Haiku: ~20× cheaper)
2. Prompt compression (trim system prompt from 3,000 to 500 tokens: ~6× on input)
3. Prompt caching (cache the static system prompt: ~90% off input costs)
4. Output truncation (label-only instead of explanations: ~10× on output)
5. Batching (20 items per call instead of 1: ~20× fewer API calls)
Applied together, these can reduce a $4,000/month pipeline to under $40 — with identical accuracy.
02 Weak vs. Strong
EX 01Auditing and optimizing a real classification pipeline
Audit our AI classification pipeline and produce an optimization plan.
Current setup:
- Model: Claude Sonnet (mid-tier)
- Volume: 10,000 tickets/day
- System prompt: 2,800 tokens (includes company history, full product docs, 15 few-shot examples)
- User prompt: ticket text, avg 200 tokens
- Output: category label + 3-sentence explanation + confidence score (avg 150 tokens)
- Processing: one ticket per API call
- Monthly cost: ~$4,200
Accuracy: 94% on our eval set (target: ≥92%)
For each optimization lever:
1. What to change
2. Estimated cost reduction (percentage and dollar amount)
3. Risk to accuracy (none / low / medium / high)
4. Implementation effort (hours)
Constraints:
- Accuracy must stay ≥92%
- We can't change our ticketing system (Zendesk)
- 2-person eng team, max 1 week of work
Format: ranked list by impact (highest savings first), then a summary table with cumulative savings.
→ Why it works
Full current-state provided: model, volume, token counts, accuracy, cost. Optimization requires knowing what exists. Constraints prevent unrealistic suggestions.
EX 02Model routing for a multi-stage pipeline
I have a 3-stage document processing pipeline. Help me pick the optimal model for each stage based on the task requirements.
Stage 1 — Document Summarization:
- Input: 10–40 page vendor contracts (PDF)
- Output: 2-page structured summary (sections: key terms, obligations, risks, renewal dates)
- Volume: 50 documents/week
- Quality: HIGH — legal team makes decisions based on these summaries
- Latency: can wait up to 2 minutes per document
Stage 2 — Data Extraction:
- Input: the Stage 1 summary
- Output: structured JSON (12 fields: vendor name, contract value, start/end dates, auto-renewal Y/N, etc.)
- Volume: same 50/week
- Quality: MEDIUM-HIGH — feeds a database, errors caught in weekly review
- Latency: under 10 seconds
Stage 3 — Notification Routing:
- Input: the Stage 2 JSON
- Output: one of 4 routing labels (legal_review, finance_approval, auto_approve, flag_for_renewal)
- Volume: same 50/week
- Quality: MEDIUM — humans review the routed items anyway
- Latency: under 2 seconds
For each stage, recommend:
- Model (specific: Claude Opus/Sonnet/Haiku, GPT-5/4o-mini, Gemini Ultra/Flash)
- Why this model fits this stage's requirements
- Estimated cost per document at this stage
- What you'd lose by using a cheaper model
Then: total cost per document across all 3 stages.
→ Why it works
Each stage has different quality/latency/complexity requirements. Specifying them separately enables task-matched model routing instead of one-model-fits-all.
03 Key Points
01Right-size the model first — most classification and extraction tasks don't need frontier models
02Prompt caching is free money: static system prompts cached across calls cost 90% less per token
03Output format directly affects cost — 'respond with only the label' vs. full explanations is 10× difference
04Batch processing: 'classify these 20 tickets' in one call vs. 20 separate calls saves overhead and enables caching
05Measure before and after: track cost-per-task, not just total spend — it's your unit economics
04 Model-Specific Notes
Claude's model tiers (Opus/Sonnet/Haiku) are designed for routing. Use Opus for reasoning-heavy tasks, Sonnet for balanced work, Haiku for classification/extraction. Prompt caching is native — enable cache_control on system prompts.
05 For Your Role
You don't need the most expensive model for everything. Simple tasks (classify, extract, label) work perfectly on cheap models. Save the expensive model for tasks that actually need intelligence.