Struct FallbackEstimator 

pub struct FallbackEstimator { /* private fields */ }

spc_011-C-01: real-BPE-backed counter, the production default. CharApproxCounter’s char/4 divisor only holds for English text — on CJK-heavy text it underestimates real BPE counts by 40-70% (its own doc comment claims “~0.5 tokens/char” for CJK, but the implementation applies the same 0.25 tokens/char divisor to every script). This wraps deepstrike-tokenizer’s real cl100k BPE tokenizer (previously an orphaned crate — not a workspace member, zero call sites anywhere) and adds a fixed margin on top. That only guarantees a value above the selected cl100k_base estimate; it does not prove that the result is conservative for every provider model. Native Anthropic/Gemini token counts take precedence where callers explicitly request them.

Implementations

impl FallbackEstimator

pub fn new(backend: TokenizerBackend, safety_margin: f64) -> Self

Trait Implementations

impl Default for FallbackEstimator

fn default() -> Self

Returns the “default value” for a type.

impl TokenCounter for FallbackEstimator

fn count(&self, text: &str) -> u32

Count tokens in a UTF-8 string.

fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str

Return the longest prefix of text that fits within max_tokens. The returned slice is always a valid UTF-8 prefix of text.