Module token 

Canonical token accounting — the rule that makes budget honesty checkable (SPEC.md §B3, ADR 0003).

Budget honesty is CGP’s flagship guarantee, but before this rule existed the conformance suite could only verify arithmetic: it summed the costs a provider declared and compared the total to the budget. A provider reporting token_cost: 1 on a ten-thousand-token frame satisfied that check perfectly while destroying the host’s actual budget. The one lie that mattered was the one lie the suite could not catch.

budget_tokens closes that hole by making cost a function of bytes both sides observe:

budget_tokens(content) = ceil(utf8_byte_length(content) / 4)

This is an accounting unit, not a tokenizer

A budget token is deliberately not a prediction of any model’s tokenizer. Its job is to make every provider’s cost claims comparable and verifiable, which no real tokenizer can do without being mandated in every language an implementation might be written in.

The approximation is honest about its direction. At roughly four bytes per token it tracks English prose closely, and it under-estimates dense source code (≈3–3.5 bytes/token) and CJK text (≈3 bytes/token). A host therefore MUST NOT treat one budget token as one model token: it maps its real model budget into budget tokens with a safety factor. See SUGGESTED_HOST_SAFETY_FACTOR.

Scope

The count covers ContextFrame::content only — not title, not citation_label, not provenance, and not the fences and labels a host wraps around a frame. content is the one field the provider fully controls and whose exact bytes both sides observe identically, so it is the only input on which a byte-exact check can be built. The host’s own rendering chrome is the host’s cost to budget.

Constants

BYTES_PER_BUDGET_TOKEN

Bytes per budget token. See the module docs for why this constant is an accounting convention rather than an empirical tokenizer ratio.

SUGGESTED_HOST_SAFETY_FACTOR

The factor a host is advised to apply when converting a real model context budget into budget tokens, compensating for the under-estimate on source code and CJK text.

Functions

budget_from_model_tokens

Convert a real model context budget into budget tokens, applying factor as headroom against the under-estimate documented on budget_tokens.

budget_tokens

The canonical budget-token cost of a piece of frame content.