Skip to main content

TextSplitter

cognis_rag::splitters

Trait TextSplitter 

Source 
pub trait TextSplitter: Send + Sync {
    // Required method
    fn split(&self, doc: &Document) -> Vec<Document>;

    // Provided method
    fn split_all(&self, docs: &[Document]) -> Vec<Document> { ... }
}
Expand description

A text splitter takes a document and emits chunk-sized documents.

Implementations preserve the source’s metadata on each chunk and add a chunk_index field so callers can re-order if needed.

Required Methods§

Source

fn split(&self, doc: &Document) -> Vec<Document>

Split doc into chunks.

Provided Methods§

Source

fn split_all(&self, docs: &[Document]) -> Vec<Document>

Convenience: split many documents and concatenate the chunks.

Implementors§