Skip to main content

split_into_chunks_hierarchical

Function split_into_chunks_hierarchical 

Source
pub fn split_into_chunks_hierarchical(
    body: &str,
    tokenizer: &Tokenizer,
) -> Vec<Chunk>
Expand description

Splits body into chunks using MarkdownSplitter with a real tokenizer. Respects Markdown semantic boundaries (H1-H6, paragraphs, blocks). For plain text without Markdown markers, falls back to paragraph and sentence breaks.