Skip to main content

chunk_text

Function chunk_text 

Source
pub fn chunk_text(
    text: &str,
    chunk_chars: usize,
    overlap_chars: usize,
    terminators: &[char],
) -> Vec<String>
Expand description

Split text into chunks of at most chunk_chars characters. Paragraphs (separated by blank lines) are kept whole when they fit; longer ones are split on sentence boundaries, and sentences longer than a chunk on word boundaries. Each chunk after the first starts with up to overlap_chars characters from the end of the previous one, cut at a word boundary.