Trait Tokenizer 

pub trait Tokenizer: Send + Sync {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Counts tokens for a piece of text. Rust equivalent of upstream TokenizerProtocol.

Required Methods

fn count_tokens(&self, text: &str) -> usize

Count the tokens represented by text.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Tokenizer for ApproxTokenizer