Skip to main content

split_into_chunks

Function split_into_chunks 

Source
pub fn split_into_chunks(
    text: &str,
    source: &DocumentSource,
    config: &ChunkConfig,
) -> Vec<Chunk>
Expand description

Split text into overlapping chunks, respecting paragraph boundaries.

Empty text produces no chunks. Paragraphs are split on double newlines. If a paragraph fits within chunk_size, it’s kept whole. Large paragraphs are split at chunk_size boundaries with chunk_overlap overlap.