Trait Tokenizer 

pub trait Tokenizer:
    Send
    + Sync
    + 'static {
    // Required method
    fn count(&self, text: &str) -> usize;
}

Counts approximate tokens for a string.

Required Methods

fn count(&self, text: &str) -> usize

Returns the estimated token count.

Implementors

impl Tokenizer for ApproximateTokenizer