The stronger design can explain why the loop continues, prove when it succeeded, and stop safely when it cannot.
Add state containing queries already tried, sources accepted/rejected, remaining evidence gaps, step count, and cost. Stop successfully only when three sources pass criteria; stop unsuccessfully after six steps or repeated no-progress; ask the user when the goal is ambiguous; verify that every cited URL was actually retrieved.
Specific thresholds make selection and later migration defensible.
Task: classify and summarize inbound tickets for human agents. Must: no autonomous customer action; EN/KM pass rate ≥92%; zero cross-customer data leaks in the test set; JSON schema ≥99.5%; p95 ≤4s; evidence quote for urgency; approved data retention. Target: ≤$0.03 per successful ticket. Preference: provider-hosted tools. Define 50 representative cases and severe failures.
The experiment makes multilingual cost visible while preventing a single sample from becoming a claim about every model or every Khmer text.
Choose one current model. Count tokens for the same 80–120 word message in English and Khmer using that model’s official counter or SDK. Record model/tokenizer, characters, words, tokens, and tokens per 100 characters. Repeat with names, numbers, and mixed English–Khmer text.
A matrix prevents one successful demo from becoming an unsupported claim about an entire modality.
For one current API, record exact model ID, accepted input types, produced output types, size/duration limits, preprocessing, pricing units, and five task tests: clean OCR, small chart labels, object counting, noisy speech, and temporal video question. Mark unsupported tasks explicitly.
The design separates conversational instruction following from enforceable authorization.
Keep the policy in the application as well as the prompt. The model may propose {order_id, amount, reason}; a policy service verifies ownership and amount; refunds above $100 require human approval; the tool executes only an authorized request; all attempts are logged.
The decision uses the full system economics and requirements rather than one appealing property.
For a document classifier, record data-residency requirement, monthly volume, latency target, accuracy threshold, model license, parameter/quantization format, measured throughput, hardware amortization, energy/cloud cost, staffing, and API alternative. Include failure recovery and upgrade work.
The improved contract makes both semantic validation and safe retry behavior explicit.
Tool: request_refund(order_id, amount, currency, reason, idempotency_key). Description states eligibility and when approval is required. Executor authenticates user, verifies order ownership/currency/refundable balance, caps amount, escalates high values, records the idempotency key, and returns {status, approved_amount, reason_code}.
The diagram reveals that location alone did not protect the credential or prevent untrusted content from steering a write.
Draw: user mailbox → attachment parser → model provider → tool proposal → company gateway → CRM. Use a read-only identity for lookup and a scoped write identity for approved fields; strip secrets; treat email content as untrusted; require user confirmation for ownership/status changes; log data sent and the final CRM response.
A shared protocol makes the trade-off auditable and repeatable next month.
Use the same 30-case frozen set, prompt contract, tool access, concurrency, and grader. Warm up each endpoint. Run repeated trials. Capture exact version, pass/severe-failure counts, p50/p95 end-to-end latency, time to first token, total token/tool cost, retries, and cost per pass. Note any unsupported parameter rather than silently changing conditions.
The policy makes resource allocation and recovery explicit instead of improvising after failure.
Rules: deterministic filter handles spam; small model handles short EN/KM classification only if calibrated score ≥0.9 and evidence present; low score, policy exceptions, long attachments, or high-impact accounts route to stronger model; refund/identity cases always require human approval; provider outage triggers read-only fallback; all writes use one idempotency key.
Budgeting and positional tests turn “supports 32K” from a marketing number into an application requirement.
Create a budget for a support request with a hypothetical 32K input allowance: 1K instructions, 3K recent history, 12K retrieved evidence, 2K tool definitions/results, 2K safety margin, and the remaining input allowance unused unless evidence requires it. Set output separately according to the API. Test the key fact near the start, middle, and end.
The stronger design treats streaming as an unreliable event channel around probabilistic output, not as a finished answer arriving character by character.
Buffer events by response ID; distinguish text deltas, tool calls, errors, and completion; show partial text as provisional; validate the final structured result; make downstream actions idempotent; and retry from a defined boundary without sending the email twice.
The comparison teaches that input representation is an engineering choice, not a neutral upload step.
Test four inputs: full PDF, rendered page, crop around the totals, and extracted text/table. Ask for total, currency, page, and exact evidence. Repeat on five invoices containing discounts, tax, and similar subtotal/total labels. Record accuracy, latency, and input cost.
The parity card reveals exactly where localization is complete and where the product only looks localized.
Create 20 parallel EN/KM cases across summarization, extraction, safety, instruction following, local names/addresses, KHR/USD amounts, dates, and mixed Khmer–English technical text. Use fluent review. Count tokens and score semantic accuracy, naturalness, terminology, format, and refusal parity for each language.
The stronger prompt has an audience, contract, epistemic rule, and a dataset that can reveal regressions.
For an on-call engineer, return JSON with: issue (≤20 words), customer_impact, evidence_quotes[], urgency (low|medium|high), next_action, and unknowns[]. Use only the ticket. If impact is not stated, put it in unknowns rather than guessing. Then test five tickets: normal, angry but low impact, outage, missing evidence, and prompt injection inside the ticket.
Repeated task-level measurement reveals both randomness and provider behavior without turning a dial into a personality label.
Choose one factual extraction task and one ideation task. Run each 20 times under every supported configuration. Record schema pass rate, task correctness, duplicate rate, and qualitative diversity. Save model ID, date, seed if supported, and all parameters. If a parameter is ignored, record that as an API finding.
The control checks whether the source supports the claim rather than whether a citation-shaped object exists.
Require claims as {claim, evidence_quote, source_id, support: entailed|partial|none}. Reject or abstain when the quote does not entail the claim. Test missing policy, conflicting versions, irrelevant retrieval, and a malicious document. Route policy exceptions to a human.
The stage-level report prevents prompt edits from masking a retrieval or governance failure.
For ten policy questions, label the gold passage. Compare keyword, dense, and hybrid retrieval at k=5; record whether gold appears, rank, stale-version errors, and authorization leaks. Then evaluate answer support only on cases where gold was retrieved. Inspect chunk boundaries around the missed cancellation clause.
The eval ties selection to the product’s cases and makes the trade-off inspectable.
Create 20 support cases: 10 common, 4 boundary, 2 missing-evidence, 2 adversarial, and 2 Khmer. Freeze prompts and tool access. Define pass criteria per case. Run each candidate at least three times where variability matters. Record severe failures, pass rate, p95 latency, and cost per pass. Blind human review to model identity.
The design earns complexity through measured value instead of architecture theatre.
Baseline one agent with retrieval and a rubric. Add a separate researcher only if source gathering is the bottleneck; add an independent reviewer only if it catches measured errors. Give workers scoped tools, pass structured handoffs, cap rounds, and compare quality, latency, and cost with the baseline.
Finding the first broken assumption prevents later verification from merely documenting an avoidable mistake.
Before clicking, identify product and button semantically; verify domain, account, quantity, and displayed price; prepare the action; pause for user confirmation with a summary; execute once; capture the confirmation number and final amount; stop if any element or price differs.
The experiment turns “thinking harder” into a quality–latency–cost curve.
Build 12 cases: four simple extractions, four moderate constraint problems, and four difficult planning or code-debugging cases. Run supported low/medium/high effort modes. Score correctness, constraint compliance, latency, and total tokens. Set a routing rule before looking at production traffic.