Sampling and Reproducibility

Outcome. You can explain temperature and nucleus sampling with a toy distribution, then design a reproducibility test that does not assume temperature zero means identical output.

After logits are converted into probabilities, a decoder chooses among candidate tokens. Temperature rescales the logits before softmax: lower values sharpen the distribution, while higher values flatten it. Top-p limits sampling to the smallest set of candidates whose cumulative probability reaches a threshold. Top-k keeps a fixed number of candidates. Providers may constrain, deprecate, ignore, or replace these controls for particular model families, especially reasoning-oriented models.

These controls affect variability, not factuality in a simple monotonic way. Lower randomness can stabilize formatting but can also repeat a consistently wrong answer. Higher randomness may diversify creative output while increasing variance. Use one primary control unless current provider guidance explicitly recommends a combination.

Reproducibility is an empirical property of the whole serving path. Hardware kernels, batching, model updates, hidden routing, tools, and nondeterministic operations may vary. If identical bytes matter, validate a structured result or use deterministic software for that step rather than depending on generative decoding.

Mental model. Sampling changes which plausible path is selected; an eval measures whether the path still satisfies the task.

Evidence trail — reviewed 23 July 2026. Nucleus sampling: https://arxiv.org/abs/1904.09751. Check the exact current generation-configuration documentation for the chosen model; sampling support is vendor- and version-specific.