Module token_engine 

Structs

CharApproxCounter

Char-count approximation: 4 chars ≈ 1 token. Used when no real tokeniser is available. More accurate than byte-count for CJK text (3 bytes/char but ~0.5 tokens/char).

ContextTokenEngine

Cheaply cloneable token engine shared across the context subsystem. All token counting and truncation goes through this single object — pressure, compression, and render use the same backend.

Traits

TokenCounter

Token counting and truncation interface. Implementations must be deterministic and must never panic on any valid UTF-8 input.