Run a Quality–Latency–Cost Bake-Off

Outcome. You can compare candidate models under identical conditions and calculate task success, p95 latency, total cost, and cost per successful outcome.

Context limits, prices, cached-input discounts, reasoning-token accounting, tool charges, batch discounts, and model names change quickly. A durable course should teach learners to fetch current documentation and run a dated test rather than memorize a scoreboard.

Freeze the workload, prompt, tools, retrieval, retry policy, region, concurrency, and grading before comparison. Record exact model IDs and dates. Measure task success and severe failures alongside time to first token, end-to-end p50/p95 latency, throughput, input/output/reasoning/tool usage, and total cost. Streaming can improve perceived latency without reducing completion time; caching can reduce repeated-prefix cost while the cached content still consumes usable context.

Calculate cost per successful task: total experiment cost divided by successful outcomes under the acceptance criteria. Include retries and fallback calls. A cheap model that causes repeated failures may be more expensive than a higher-priced model that finishes reliably.

Mental model. A model price is an input rate; product economics depend on how much work produces one acceptable result.

Evidence trail — reviewed 23 July 2026. FlashAttention: https://arxiv.org/abs/2205.14135 and PagedAttention/vLLM: https://arxiv.org/abs/2309.06180 explain serving efficiency. Fetch current candidate details from official model and pricing pages on the test date.