Skip to main content

chunk_source

Function chunk_source 

Source
pub fn chunk_source(
    source: &str,
    language: Option<&Language>,
    desired_length: usize,
) -> Vec<ChunkBoundary>
Expand description

Chunk source text via tree-sitter AST-merge or line-merge fallback.

language = Some(lang) parses with tree-sitter and runs the AST-merge algorithm. language = None runs the line-merge fallback.

Empty / whitespace-only input returns an empty Vec.