Skeleton-of-Thought (SoT)

Expert

Generate outlines first, then fill details in parallel for low latency.

01 The Concept

Standard sequential generation is slow because LLMs decode tokens one by one. Skeleton-of-Thought (SoT) addresses this by decomposing generation into two phases:

1. Skeleton Stage: The model generates a structured table of contents or outline (the skeleton). 2. Expansion Stage: The application sends multiple parallel API requests to expand each outline point in parallel.

This approach reduces user-facing latency by up to 80% while often yielding deeper, more structured content since each section has its own full context window.

02 Weak vs. Strong

EX 01Document Generator (Python)

import asyncio from openai import AsyncOpenAI client = AsyncOpenAI() # Step 1: Generate the Skeleton skeleton_res = await client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Outline a 5-point technical feature spec for a new Stripe subscription module. Output only 5 bullet points with clear, short titles."}] ) points = parse_bullets(skeleton_res.choices[0].message.content) # Step 2: Expand in Parallel async def expand_point(p): res = await client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": f"Write a detailed technical specification for this section of a Stripe module: {p}"}] ) return res.choices[0].message.content full_spec = await asyncio.gather(*(expand_point(p) for p in points))

→ Why it works

Bypasses serial decoding. The initial outline generation takes 1 second, and the 5 parallel expansion calls take 3 seconds total. Total latency is 4 seconds instead of 20+ seconds for a single-pass run.

03 Key Points

01Parallel Speedup: Bypasses serial token limitations by distributing work across parallel API calls.

02Context Budgeting: Gives each section its own dedicated generation, preventing early truncation or rushed summaries.

03Skeleton Control: The first-turn prompt must output a highly parseable structure (e.g. XML tags or JSON keys).

04Dynamic Expansion: Ideal for reports, ebooks, feature specifications, and slide generators.

04 Model-Specific Notes

Claude handles highly structured outlines perfectly. Ensure the skeleton prompt requires clean XML tags for easy parsing.

05 For Your Role

It's like writing an essay by drafting the index first, then letting five friends write one chapter each at the same time.

Next: Graph-of-Thought (GoT) →