Trait Tokenizer 

pub trait Tokenizer: Send + Sync {
    // Required method
    fn count(&self, text: &str) -> usize;

    // Provided method
    fn encode(&self, _text: &str) -> Option<Vec<u32>> { ... }
}

Counts tokens for a &str.

Implementations must be cheap and Send + Sync because the builder calls count once per contribution per turn.

Required Methods

fn count(&self, text: &str) -> usize

Number of tokens the model would see for text.

Provided Methods

fn encode(&self, _text: &str) -> Option<Vec<u32>>

Token IDs for providers that need them. Default returns None; only tokenizers that have a real BPE backing (e.g. tiktoken) override.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl<T: Tokenizer + ?Sized> Tokenizer for Arc<T>

Convenience: every Arc<dyn Tokenizer> already satisfies Tokenizer via blanket deref-style forwarding. This impl lets the builder hold an Arc<dyn Tokenizer> and pass it around without unwrapping.

fn count(&self, text: &str) -> usize

fn encode(&self, text: &str) -> Option<Vec<u32>>

Implementors

impl Tokenizer for CharApproxTokenizer