Struct ContextTokenEngine 

pub struct ContextTokenEngine(/* private fields */);

Cheaply cloneable token engine shared across the context subsystem. All token counting and truncation goes through this single object — pressure, compression, and render use the same backend.

Implementations

impl ContextTokenEngine

pub fn char_approx() -> Self

Deterministic char/4 approximation. Kept as an explicit, opt-in constructor for tests that pin exact token counts (many do, calibrated to this specific math) — not used by any production call site as of spc_011-C-01; see Self::fallback_estimator.

pub fn fallback_estimator() -> Self

spc_011-C-01: the production default token engine. Real-BPE-backed (see FallbackEstimator), replacing the previous char_approx() default that underestimated CJK-heavy text by 40-70%.

pub fn count(&self, text: &str) -> u32

pub fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str

pub fn token_budget_to_bytes(&self, tokens: u32) -> usize

pub fn count_message(&self, msg: &Message) -> u32

pub fn truncate_message(&self, msg: &Message, max_tokens: u32) -> Message

Truncate a text message to max_tokens. Returns the message unchanged if it fits. Parts messages are never truncated — mangling structured content produces worse outcomes than a minor token overrun.

Trait Implementations

impl Clone for ContextTokenEngine

fn clone(&self) -> ContextTokenEngine

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.