Module encoder 

Claude-compatible token counting.

WCGW counts tokens with the Xenova/claude-tokenizer (Hugging Face tokenizers). We embed that same tokenizer definition in the binary and load it lazily, so token budgets and truncation match the model that actually runs the agent. Small inputs use a byte-length upper bound and never initialize it; if loading fails for a larger input we fall back to a cheap character/word estimate.

Functions

budget_from_env

Read a token-budget override from env var var (e.g. WINX_CODING_TOKEN_BUDGET), falling back to default when unset, zero, or unparseable. Lets large-context clients tune how much of each file is pulled into context — and how much saved memory is kept — without a rebuild.

count_tokens

Count tokens the way Claude does. Falls back to estimate_tokens on failure.

decode_ids

Decode Claude token ids back into text. Returns None on failure.

definitely_fits_token_budget

Return true when byte length alone proves text fits the token budget.

encode_ids

Encode text into Claude token ids. Returns None if the tokenizer is unavailable so callers can pick a byte-based fallback.

estimate_tokens

Cheap fallback estimate used only when the tokenizer is unavailable.