Prompt Drift occurs when a model provider updates an underlying model, causing previously working prompts to generate formatting errors, changed tones, or logical failures.
Calibration is the practice of systematically evaluating prompt responses against a golden test set on each model version.
By comparing outputs against a ground-truth dataset, you can adjust temperature, system instructions, or few-shot examples to maintain consistent output distributions.
GOLDEN_TESTS = [ {"input": "Ref #123: Card declined", "expected": {"type": "billing", "urgency": "high"}}, {"input": "Change email to [email protected]", "expected": {"type": "account", "urgency": "medium"}} ]
def calibrate_prompt(prompt_template, model_name): passed = 0 for test in GOLDEN_TESTS: out = call_llm(model_name, prompt_template.format(test['input'])) if parse_json(out) == test['expected']: passed += 1 accuracy = passed / len(GOLDEN_TESTS) print(f"Model {model_name} Accuracy: {accuracy*100}%") assert accuracy >= 0.95, "Regression detected!"
Anthropic locks model versions securely. Pin your API calls to specific date-named releases to prevent background updates from causing drift.