Struct TokenAwareSplitter 

pub struct TokenAwareSplitter { /* private fields */ }

Splits text so each chunk’s token count (per the supplied Tokenizer) stays under max_tokens. Falls back to a recursive char splitter for the structural cuts; just adds a token-aware re-pack step on top.

Implementations

impl TokenAwareSplitter

pub fn new(tokenizer: Arc<dyn Tokenizer>, max_tokens: usize) -> Self

Build with a tokenizer + max-token cap.

pub fn with_overlap_tokens(self, n: usize) -> Self

Token-overlap between adjacent chunks. Clamped to max_tokens - 1 so the second-pass walker always makes progress; if the caller passes a larger value, the chunk would just feed itself back in as overlap and the splitter would loop or emit chunks > max.

Trait Implementations

impl TextSplitter for TokenAwareSplitter

fn split(&self, doc: &Document) -> Vec<Document>

Split doc into chunks.

fn split_all(&self, docs: &[Document]) -> Vec<Document>

Convenience: split many documents and concatenate the chunks.