Module document_chunker 

Document Chunker

A production-grade document chunking engine that splits text into semantically coherent chunks for embedding and retrieval. Supports multiple chunking strategies including fixed-size windows, sentence boundaries, paragraph-based, and semantic grouping.

Example

use ipfrs_semantic::{DocumentChunker, DocumentChunkerConfig, ChunkStrategy};

let config = DocumentChunkerConfig {
    strategy: ChunkStrategy::SentenceBoundary {
        max_chunk_chars: 512,
        overlap_sentences: 1,
    },
    preserve_whitespace: false,
    min_chunk_chars: 10,
};

let mut chunker = DocumentChunker::new(config);
let chunks = chunker.chunk_text("doc-001", "Hello world. This is a test.");
assert!(!chunks.is_empty());

Structs

ChunkStats

Aggregate statistics over a collection of TextChunks.

DocumentChunker

Document chunking engine.

DocumentChunkerConfig

Configuration for DocumentChunker.

TextChunk

A single text chunk produced by DocumentChunker.

Enums

ChunkStrategy

Strategy that controls how a document is split into chunks.