Struct HfTokenCounter 

pub struct HfTokenCounter { /* private fields */ }

Available on crate feature tokenizer-hf only.

TokenCounter backed by a HuggingFace Tokenizer.

Cloning is cheap — the tokenizer sits behind an Arc so every clone shares one parsed instance. Construct once at app boot, share across ChatModelConfig and ingestion pipelines.

Implementations

impl HfTokenCounter

pub fn from_bytes( bytes: &[u8], encoding_name: impl Into<String>, ) -> Result<HfTokenCounter, HfTokenizerError>

Construct a counter from a tokenizer.json byte payload. The supplied encoding_name surfaces on TokenCounter::encoding_name and the OTel gen_ai.tokenizer.name attribute — pick a stable identifier for the model whose tokenizer the bytes encode ("llama-3", "qwen-2.5", "mistral", …).

The encoding name is leaked once into a &'static str — see the crate-level docs for the rationale.

pub const fn encoding(&self) -> &'static str

Inspect the configured encoding name.

Trait Implementations

impl Clone for HfTokenCounter

fn clone(&self) -> HfTokenCounter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl TokenCounter for HfTokenCounter

fn count(&self, text: &str) -> u64

Count the tokens in text under this counter’s encoding.

fn encoding_name(&self) -> &'static str

Vendor-published encoding name ("cl100k_base", "o200k_base", "claude", "gemini-tokenizer", …) — surfaced on OTel gen_ai.tokenizer.name and operator diagnostics.

fn count_messages(&self, msgs: &[Message]) -> u64

Sum the token count across every text-bearing content part of a message slice. The default impl walks crate::ir::ContentPart::Text parts; non-text parts (image, tool-use, tool-result blocks) are vendor-specific in their token cost — counters that need an exact tally for those shapes override this method.