Inference Configuration — Patterns

Named task profiles

Problem. Generation parameters are set inline at each call site, copied from whichever example the author read first. Nobody can state the current temperature for extraction tasks without grepping, and changing it means finding every copy.

Approach. Define a small, closed set of named profiles describing intent — deterministic extraction, factual answering, long-form drafting, creative generation — each with concrete values and a written rationale. Call sites name a profile. New call sites must either pick an existing profile or justify a new one, which keeps the set small.

Trade-off. Profiles are a shared resource: changing one to help a specific call site changes behaviour for every other site using it. Resisting the pressure to add per-call exceptions is continuous work, and losing that fight returns you to inline parameters with extra indirection.

Output headroom derived from the schema

Problem. The maximum output length is a magic number. Set too low it truncates valid answers; set to the model's maximum it removes a useful cost ceiling and makes runaway generation expensive.

Approach. Derive the limit from the expected answer rather than from a default: for structured output, from the schema's realistic worst case with all optional fields populated and arrays at their documented bounds, plus a margin; for prose, from an observed length distribution with a percentile chosen deliberately. Where a model consumes reasoning tokens before visible output, add that budget separately rather than hiding it in the margin.

Trade-off. The derivation goes stale when the schema or the prompt changes, so it needs to be re-derived as part of changing either. It also requires treating truncation as a distinct, alarmed outcome instead of an error, which is extra plumbing.

Configuration as part of the evaluated artefact

Problem. An evaluation score is recorded against a prompt and a model. The parameters are not captured, so a later run that disagrees cannot be explained, and a configuration change can regress quality without any evaluation noticing.

Approach. Hash the resolved configuration — every parameter actually sent, including defaults the library filled in — and store the hash with the prompt version and model version on every evaluation result and every production trace. Make a changed hash a first-class reason to re-run the suite.

Trade-off. Capturing library-filled defaults means reaching into what was actually transmitted rather than what the application specified, which is more intrusive than it sounds. The hash also makes evaluation history fragment whenever an unrelated parameter is touched, so expect to re-baseline more often than feels necessary.

Refusing to equate low temperature with determinism

Problem. Teams set temperature to zero, conclude that the output is now a pure function of the input, and build caching, testing, and audit arguments on top of that belief.

Approach. Treat the model as a sampler whose variance has been reduced, not removed. Assert on properties — schema validity, required fields, numeric tolerance, presence of a resolvable citation — rather than on exact strings. Where byte-exact reproduction is a real requirement, store the output and reproduce it from storage rather than from the model.

Trade-off. Property assertions are weaker than exact-match assertions and will pass some outputs a human would reject, so the properties must be chosen to cover what actually matters. The alternative is a test suite that fails randomly and is eventually disabled.