Cost Optimization & Token Economics

Advanced

Same output. 99% less spend.

01 The Concept

Most teams overspend on AI by 10–50× because they never optimize. They use the biggest model for everything, send verbose prompts, get full-paragraph outputs when a label would do, and process items one at a time.

The production cost stack, in order of impact:

Model right-sizing (switch from Opus to Haiku: ~20× cheaper)
Prompt compression (trim system prompt from 3,000 to 500 tokens: ~6× on input)
Prompt caching (cache the static system prompt: ~90% off input costs)
Output truncation (label-only instead of explanations: ~10× on output)
Batching (20 items per call instead of 1: ~20× fewer API calls)

Applied together, these can reduce a $4,000/month pipeline to under $40 — with identical accuracy.

02 Weak vs. Strong

EX 01Auditing and optimizing a real classification pipeline

Audit our AI classification pipeline and produce an optimization plan.

Current setup:

Model: Claude Sonnet (mid-tier)
Volume: 10,000 tickets/day
System prompt: 2,800 tokens (includes company history, full product docs, 15 few-shot examples)
User prompt: ticket text, avg 200 tokens
Output: category label + 3-sentence explanation + confidence score (avg 150 tokens)
Processing: one ticket per API call
Monthly cost: ~$4,200

Accuracy: 94% on our eval set (target: ≥92%)

For each optimization lever:

What to change
Estimated cost reduction (percentage and dollar amount)
Risk to accuracy (none / low / medium / high)
Implementation effort (hours)

Constraints:

Accuracy must stay ≥92%
We can't change our ticketing system (Zendesk)
2-person eng team, max 1 week of work

Format: ranked list by impact (highest savings first), then a summary table with cumulative savings.

→ Why it works

Full current-state provided: model, volume, token counts, accuracy, cost. Optimization requires knowing what exists. Constraints prevent unrealistic suggestions.

EX 02Model routing for a multi-stage pipeline

I have a 3-stage document processing pipeline. Help me pick the optimal model for each stage based on the task requirements.

Stage 1 — Document Summarization:

Input: 10–40 page vendor contracts (PDF)
Output: 2-page structured summary (sections: key terms, obligations, risks, renewal dates)
Volume: 50 documents/week
Quality: HIGH — legal team makes decisions based on these summaries
Latency: can wait up to 2 minutes per document

Stage 2 — Data Extraction:

Input: the Stage 1 summary
Output: structured JSON (12 fields: vendor name, contract value, start/end dates, auto-renewal Y/N, etc.)
Volume: same 50/week
Quality: MEDIUM-HIGH — feeds a database, errors caught in weekly review
Latency: under 10 seconds

Stage 3 — Notification Routing:

Input: the Stage 2 JSON
Output: one of 4 routing labels (legal_review, finance_approval, auto_approve, flag_for_renewal)
Volume: same 50/week
Quality: MEDIUM — humans review the routed items anyway
Latency: under 2 seconds

For each stage, recommend:

Model (specific: Claude Opus/Sonnet/Haiku, GPT-5/4o-mini, Gemini Ultra/Flash)
Why this model fits this stage's requirements
Estimated cost per document at this stage
What you'd lose by using a cheaper model

Then: total cost per document across all 3 stages.

→ Why it works

Each stage has different quality/latency/complexity requirements. Specifying them separately enables task-matched model routing instead of one-model-fits-all.

03 Key Points

01Right-size the model first — most classification and extraction tasks don't need frontier models

02Prompt caching is free money: static system prompts cached across calls cost 90% less per token

03Output format directly affects cost — 'respond with only the label' vs. full explanations is 10× difference

04Batch processing: 'classify these 20 tickets' in one call vs. 20 separate calls saves overhead and enables caching

05Measure before and after: track cost-per-task, not just total spend — it's your unit economics

04 Model-Specific Notes

Claude's model tiers (Opus/Sonnet/Haiku) are designed for routing. Use Opus for reasoning-heavy tasks, Sonnet for balanced work, Haiku for classification/extraction. Prompt caching is native — enable cache_control on system prompts.

05 For Your Role

You don't need the most expensive model for everything. Simple tasks (classify, extract, label) work perfectly on cheap models. Save the expensive model for tasks that actually need intelligence.

Next: Context Engineering →