Token Budget Management — References

Sources for counting tokens correctly and for the behaviour of long inputs.

Tokenization

• tiktoken — byte-pair encodings used by OpenAI models, and the clearest demonstration that a token count is a property of an encoding.
• Hugging Face Tokenizers — fast tokenizer implementations and the tokenizer.json artefact that pins an open-weight model's encoding.
• SentencePiece — subword tokenization used by several open model families, relevant when a tokenizer is not a byte-pair encoding at all.

Usage reporting and limits

• OpenAI API reference — the usage object, including cached input tokens and reasoning tokens reported separately from visible output.
• Anthropic Messages API — input and output token reporting, cache read and write token fields, and the token-counting endpoint.

Why long contexts are not free

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023), arXiv:2307.03172 — evidence that filling a context window degrades use of material placed in its middle, which is the quality argument for a budget rather than the cost argument.

Provider pricing pages are the authority on what each token class costs and are deliberately not linked here, because a stale price is worse than an absent one.