Outcome. You can design a model-routing policy with confidence gates, escalation, fallback, circuit breaking, and a failure budget without hiding severe errors behind an average.
Different requests need different capability, latency, and cost. A router may send common low-risk classifications to a small fast model, complex cases to a stronger model, and high-impact uncertain cases to a human. Routing can use known task type, input size, language, risk, policy, or a calibrated classifier. It should not rely only on a model saying “I am confident.”
Fallback is not simply “call another model after an error.” Models can share training data, prompts, retrieval, and failure modes, so a second answer may repeat the first. Define which failures trigger retry, alternate model, retrieval repair, deterministic path, or human escalation. Writes must remain idempotent across retries.
A failure budget states how much failure the product can tolerate over a period, with severe failures counted separately. Use circuit breakers when a provider, tool, or quality monitor degrades. Preserve traces across routes so teams can learn whether the router, model, tool, or grader caused the result.
Mental model. Routing allocates capability by task and risk; fallback changes the recovery path; neither removes the need for verification.
Evidence trail — reviewed 23 July 2026. Calibration concepts: https://arxiv.org/abs/2012.00955. NIST AI 600-1 supports measured lifecycle risk: https://doi.org/10.6028/NIST.AI.600-1. Apply ordinary distributed-systems controls to tools and providers.
Rules: deterministic filter handles spam; small model handles short EN/KM classification only if calibrated score ≥0.9 and evidence present; low score, policy exceptions, long attachments, or high-impact accounts route to stronger model; refund/identity cases always require human approval; provider outage triggers read-only fallback; all writes use one idempotency key.