Struct TokenizerProfile 

pub struct TokenizerProfile {
    pub chars_per_token: f32,
    pub bpe: Tokenizer,
    pub inline_json_cost: f32,
    pub toon_overhead: f32,
    pub format_factors: BTreeMap<String, f32>,
}

Cost model for one tokenizer family.

Captures the empirical observation that the same encoder produces wildly different token counts depending on the receiving model’s tokenizer (e.g., inline_json_cost is 2.2x on Anthropic-class but 1.0x on Ollama BPE). See Paper 2 §Encoder Bug Postmortem (2026-04-25).

bpe selects the actual byte-pair encoder used to count tokens. When set to Tokenizer::Heuristic (default for backward compat), chars_per_token drives the estimate. When set to a real BPE variant, that BPE is used and chars_per_token is informational only.

Fields

chars_per_token: f32

Average characters per token observed for this tokenizer. Only used when bpe == Heuristic.

bpe: Tokenizer

Real BPE tokenizer to use for accurate counts. Falls back to the chars_per_token heuristic when set to heuristic.

inline_json_cost: f32

Penalty multiplier for inline-JSON cells inside markdown tables. Use to decide between inline-JSON nested cells vs. recursive sections.

toon_overhead: f32

Multiplicative cost of TOON encoding vs json_compact for this tokenizer. (TOON’s “−40% tokens” claim is only valid for openai_o200k.)

format_factors: BTreeMap<String, f32>

Optional per-format cost factors (multiplied with raw-char-based estimate).

Implementations

impl TokenizerProfile

pub fn count_tokens(&self, text: &str) -> usize

Count tokens in text using this profile’s resolved tokenizer.

• If bpe == Heuristic, applies text.len() / chars_per_token (ceiled).
• Otherwise delegates to the real BPE encoder.

Trait Implementations

impl Clone for TokenizerProfile

fn clone(&self) -> TokenizerProfile

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerProfile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerProfile

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TokenizerProfile

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenizerProfile

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.