Struct TokenTextSplitter 

pub struct TokenTextSplitter { /* private fields */ }

Splits text by estimated token count using a ~4 chars/token heuristic.

Splits at word boundaries to keep chunks readable. Consistent with the token estimation used in ConversationTokenBufferMemory.

Implementations

impl TokenTextSplitter

pub fn new(chunk_size: usize) -> Self

Create a new token text splitter.

chunk_size is in estimated tokens (not characters).

pub fn with_chunk_overlap(self, overlap: usize) -> Self

Trait Implementations

impl TextSplitter for TokenTextSplitter

fn split_text(&self, text: &str) -> Vec<String>

Split a string into chunks.

fn split_documents(&self, docs: Vec<Document>) -> Vec<Document>

Split documents by splitting each document’s content and producing new documents for each chunk. Metadata is preserved on each chunk.