Context Windows & Caching Strategies

Expert

Optimize memory retrieval to minimize token cost and latency.

01 The Concept

Large context windows enable loading massive documents, but they scale cost and latency linearly.

Prompt Caching at the model level allows you to persist static system prompts, database schemas, and few-shot examples in memory.

By design, reusing cached prefixes reduces token costs by up to 90% and cuts time-to-first-token (TTFT) by 2-10x, enabling highly responsive, cost-efficient agentic architectures.

02 Weak vs. Strong

EX 01Prompt Layout for Cache Hit (OpenAI/Anthropic)

System Prompt: You are an SQL database assistant.

Context Database Schema (Static - Cached Prefix): (10,000 tokens of static table definitions)...

User Query (Dynamic): Find client X's refund status.

→ Why it works

Since the schema block is static and placed first, it matches the cached prefix across sequential calls. Only the short user query at the end changes, resulting in a 99% cache hit rate.

03 Key Points

01TTFT Reduction: Cache hits return tokens instantly, bypassing the need to re-process input files.

02Prefix Matching: Caching works from the beginning of the prompt. Keep static data first and dynamic text last.

03Cache Eviction: Cached blocks remain active for 5-10 minutes. Design system pipelines to keep reuse high.

04Context Layout: Order schemas, documentation, and few-shot examples at the start of your system prompt.

05Cost Cut: Cached tokens are priced at a fraction of standard input token rates, enabling cheap agentic pipelines.

04 Model-Specific Notes

Anthropic supports prompt caching (minimum 1,024 tokens for Sonnet/Haiku). Use the 'cache_control' parameter in messages/system parameters to declare cache breakpoints.

05 For Your Role

It's like keeping a heavy encyclopedia open on your desk instead of closing it and looking up the index from scratch every single time.

Next: RAG Prompt Optimization →