Skip to main content

chunk_text

Function chunk_text 

Source
pub fn chunk_text(
    text: &str,
    config: &ChunkingConfig,
    token_counter: &dyn TokenCounter,
) -> Vec<TextChunk>
Expand description

Chunk text using the configured strategy and token counter.

Returns an empty vec for empty input. Short text (≤ max_size) is returned as a single chunk.