Module token 

Token-cost estimation (v0.4.0 P0-3).

Used by the bench gate to assert code-mode delivers ≥95% token reduction vs JSON-tool mode. The estimator is a deterministic linear approximation of the OpenAI / Anthropic tokenizers — close enough to make the assertion meaningful without pulling in a proper BPE library. Hat-tip OpenAI’s “1 token ≈ 4 chars of English” rule.

Functions

estimate_code_mode_tokens

Estimate token cost of a code-mode exchange given the same query

estimate_json_mode_tokens

Estimate token cost of a JSON-mode tool exchange given a recall query + cited records (the standard MCP tools/call → tools/result envelope).

estimate_tokens

Estimate token cost of a UTF-8 string.