CoT Decoding Controls

Expert

Enforce exact formatting limits on reasoning steps to minimize cost.

01 The Concept

While Chain-of-Thought (CoT) improves reasoning, verbose thoughts increase token latency and cost.

CoT Decoding Controls calibrate reasoning density by instructing the model to limit thinking steps to a specific number of bullet points or bounding thoughts strictly within <thinking> tags under a strict word limit, keeping reasoning cost-effective.

02 Weak vs. Strong

EX 01CoT Decoding Control Setup

Classify the support ticket.

Rules:

First, output your logical reasoning inside tags.
The thinking trace must be a maximum of 3 bullet points, under 100 words.
Output final classification after the tags.

→ Why it works

Keeps the benefits of reasoning while preventing the model from outputting long, costly descriptions, saving latency and money.

03 Key Points

01Reasoning Density: Maximizing logical thinking per token in reasoning traces.

02Thinking Constraints: Setting strict word or bullet point budgets on reasoning traces.

03Delimiter Isolation: Bounding thoughts inside custom tags (e.g. ...).

04Latency Mitigation: Compressed thinking blocks reduce end-to-end API response time.

05Cost control: Prevents models from generating 1,000+ conversational thinking tokens for simple tasks.

04 Model-Specific Notes

Claude is highly compliant with thinking limits. Bounding reasoning in XML tags makes it easy to strip before presenting to users.

05 For Your Role

It's like telling someone: 'Do your math rough work, but keep it to one corner of the page and don't write a novel.'

Next: Context Retrieval Sorting →