Trait text_splitter::ChunkSizer

pub trait ChunkSizer {
    // Required method
    fn chunk_size(&self, chunk: &str) -> usize;
}

Determines the size of a given chunk.

Required Methods

fn chunk_size(&self, chunk: &str) -> usize

Determine the size of a given chunk to use for validation

Implementations on Foreign Types

impl ChunkSizer for CoreBPE

Available on crate feature tiktoken-rs only.

fn chunk_size(&self, text: &str) -> usize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

impl ChunkSizer for Tokenizer

Available on crate feature tokenizers only.

fn chunk_size(&self, chunk: &str) -> usize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

Implementors

impl ChunkSizer for Characters