Char-count approximation: 4 chars ≈ 1 token.
Used when no real tokeniser is available. More accurate than byte-count
for CJK text (3 bytes/char but ~0.5 tokens/char).
Cheaply cloneable token engine shared across the context subsystem.
All token counting and truncation goes through this single object —
pressure, compression, and render use the same backend.
spc_011-C-01: real-BPE-backed counter, the production default. CharApproxCounter’s
char/4 divisor only holds for English text — on CJK-heavy text it underestimates real BPE
counts by 40-70% (its own doc comment claims “~0.5 tokens/char” for CJK, but the
implementation applies the same 0.25 tokens/char divisor to every script). This wraps
deepstrike-tokenizer’s real cl100k BPE tokenizer (previously an orphaned crate — not a
workspace member, zero call sites anywhere) and adds a fixed margin on top. That only
guarantees a value above the selected cl100k_base estimate; it does not prove that the
result is conservative for every provider model. Native Anthropic/Gemini token counts take
precedence where callers explicitly request them.