Trait text_splitter::TokenCount

pub trait TokenCount {
    // Required method
    fn token_count(&self, text: &str) -> usize;
}

Generic interface for tokenizers to calculate number of tokens.

Required Methods

fn token_count(&self, text: &str) -> usize

Returns the number of tokens in a given text after tokenization.

Implementations on Foreign Types

impl TokenCount for CoreBPE

Available on crate feature tiktoken only.

fn token_count(&self, text: &str) -> usize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

impl TokenCount for Tokenizer

Available on crate feature huggingface only.

fn token_count(&self, text: &str) -> usize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

Implementors