Trait text_splitter::ChunkSizer

pub trait ChunkSizer {
    // Required method
    fn chunk_size(
        &self,
        chunk: &str,
        capacity: &impl ChunkCapacity
    ) -> ChunkSize;
}

Determines the size of a given chunk.

Required Methods

fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize

Determine the size of a given chunk to use for validation

Object Safety

This trait is not object safe.

Implementations on Foreign Types

impl ChunkSizer for &CoreBPE

Available on crate feature tiktoken-rs only.

fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize

Returns the number of tokens in a given text after tokenization.

impl ChunkSizer for &Tokenizer

Available on crate feature tokenizers only.

fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

impl ChunkSizer for CoreBPE

Available on crate feature tiktoken-rs only.

fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize

Returns the number of tokens in a given text after tokenization.

impl ChunkSizer for Tokenizer

Available on crate feature tokenizers only.

fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize

Returns the number of tokens in a given text after tokenization.

Panics

Will panic if you don’t have a byte-level tokenizer and the splitter encounters text it can’t tokenize.

Implementors

impl ChunkSizer for Characters