Skip to main content

chunk_text

Function chunk_text 

Source
pub fn chunk_text(
    path: &Path,
    source: &str,
    chunk_config: &ChunkConfig,
) -> Vec<CodeChunk>
Expand description

Split source text into overlapping sliding windows.

Each window is chunk_config.window_size bytes with chunk_config.window_overlap bytes of overlap. Window boundaries are adjusted to line breaks to avoid splitting mid-line.

This is used as the fallback for files without tree-sitter support (plain text, unknown extensions) and for large semantic chunks that exceed max_chunk_bytes.