Inference Configuration

What it is

The per-call generation parameters that shape how a model produces its answer: sampling temperature and nucleus cut-off, maximum output length, stop sequences, repetition penalties, random seed, requested output format, and — on models that expose it — the reasoning or thinking budget. It is the dial panel, as distinct from the prompt that goes into the model and the version of the model receiving it.

Why it matters

These parameters change output far more than their obscurity suggests, and they are usually set once by whoever wrote the first call and never revisited. A maximum output length copied from an example truncates a long structured answer in the middle of a JSON object. A temperature left at a creative default makes an extraction task non-reproducible. A stop sequence that also occurs inside normal content silently cuts answers short. None of these produce an error: they produce a wrong answer at full price.

Key decisions

• Whether configuration is per task profile or per call site. Profiles — extraction, drafting, classification — keep related call sites consistent and make a change reviewable in one place.
• Where the configuration lives: compiled into the application, in a configuration file released with it, or in a runtime store that can be changed without a deploy. The last is convenient and removes generation behaviour from code review.
• How much output headroom to allocate. The maximum output length must exceed the longest legitimate answer plus any reasoning tokens the model consumes before producing it.
• Whether to use both temperature and nucleus sampling at once, or fix one. Tuning both simultaneously makes the effect of either hard to attribute.
• How configuration is treated during evaluation: as part of the artefact under test, so a score is attached to a prompt, a model version, and a configuration together.

Failure modes

• The output limit is hit mid-token-stream; the response is syntactically invalid JSON and the application reports a parsing error rather than a truncation, so the real cause is never found.
• A seed is set and the result is assumed reproducible; it is not, because batching, hardware, kernel selection, and floating-point non-associativity move the sampled token on a near-tie even at temperature zero.
• Temperature zero is treated as deterministic and used to justify skipping validation of structured output.
• A configuration tuned for one model family is carried unchanged to another, where the same numeric value for nucleus sampling or a repetition penalty does not mean the same thing.
• A runtime configuration store is changed to fix one tenant's output and silently changes generation behaviour for every tenant sharing the profile.
• Reasoning budgets are left at a high default on high-volume simple traffic, and the invisible tokens dominate the bill without improving any measured outcome.

Related capabilities

• Model Versioning — the other half of what must be pinned for an output to be reproducible
• Token Budget Management — output ceilings set here are the input to budget arithmetic
• Model Benchmarking — the only way to choose these values on evidence rather than habit
• Multi-model Support — where parameter meanings stop being portable across providers
• Model Fallback — per-attempt timeouts and output limits are configuration decisions too

Applies: Production readiness · Design questions