Inference Configuration — Stack

Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.

Selection criteria

The question is less "which library" than "where do these values live and who may change them".

• Reviewability. Is a change to a generation parameter visible in a diff, attributable to a person, and revertible to a known-good state?
• Scoping. Can a value be overridden per tenant, per environment, or per experiment without forking the whole profile?
• Propagation latency and safety. For runtime stores: how fast a change reaches running processes, and whether a bad value can be rolled back faster than it propagates.
• Validation at the edge. Whether out-of-range or mutually contradictory values — nucleus cut-off above one, an output limit exceeding the model's maximum, a stop sequence that is the empty string — are rejected on write rather than discovered on a call.
• Model-awareness. Whether the mechanism knows which parameters the target model actually accepts, since sending an unsupported parameter is an error on some APIs and silently ignored on others. Each family defines its own valid ranges, parameter names, and defaults, and only some expose a reasoning budget at all, so a profile is portable only across models whose accepted parameter set, maximum output length, seed handling, and unknown-parameter behaviour have all been checked.
• Evaluation coupling. Whether a configuration can be referenced by an immutable hash from an evaluation record, so a score means something later.

Options