Skip to main content

Tokenizer

context_weaver::assembler

Trait Tokenizer 

Source 
pub trait Tokenizer: Send + Sync {
    // Required method
    fn estimate_tokens(&self, text: &str) -> usize;
}
Expand description

Trait for estimating token counts of text content.

The host application can provide a tokenizer that matches their target model (e.g. tiktoken for OpenAI, sentencepiece for others). The assembler uses this for budget enforcement.

Required Methods§

Source

fn estimate_tokens(&self, text: &str) -> usize

Estimate the number of tokens in the given text.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§