Struct DocumentChunker 

pub struct DocumentChunker {
    pub config: DocumentChunkerConfig,
    pub chunks_produced: u64,
    pub documents_processed: u64,
}

Document chunking engine.

Maintains running counters for chunks produced and documents processed across the lifetime of the instance. All heavy work is done in the chunk_* family of methods.

Fields

config: DocumentChunkerConfig

Active configuration.

chunks_produced: u64

Total number of TextChunks emitted since construction.

documents_processed: u64

Total number of documents submitted to DocumentChunker::chunk_text.

Implementations

impl DocumentChunker

pub fn new(config: DocumentChunkerConfig) -> Self

Construct a new DocumentChunker with the supplied configuration.

pub fn chunk_text(&mut self, doc_id: &str, text: &str) -> Vec<TextChunk>

Chunk text using the strategy stored in self.config, filter chunks shorter than min_chunk_chars, and update internal counters.

pub fn chunk_fixed_size( &self, doc_id: &str, text: &str, size: usize, overlap: usize, ) -> Vec<TextChunk>

Split text into fixed-size windows of size characters stepping by size - overlap characters between each window.

If overlap >= size it is clamped to size.saturating_sub(1) to guarantee progress.

pub fn chunk_sentence_boundary( &self, doc_id: &str, text: &str, max_chars: usize, overlap_sentences: usize, ) -> Vec<TextChunk>

Split text at sentence boundaries and group sentences so that each chunk stays within max_chars characters. The last overlap_sentences sentences of each chunk are prepended to the next chunk.

pub fn chunk_paragraph( &self, doc_id: &str, text: &str, max_chars: usize, ) -> Vec<TextChunk>

Split text at \n\n paragraph boundaries. Paragraphs that exceed max_chars are further split at sentence boundaries.

pub fn chunk_semantic( &self, doc_id: &str, text: &str, max_chars: usize, _threshold: f64, ) -> Vec<TextChunk>

Group sentences into chunks of at most max_chars characters. _threshold is a placeholder for future embedding-based similarity.

pub fn merge_small_chunks( chunks: Vec<TextChunk>, min_chars: usize, ) -> Vec<TextChunk>

Merge chunks shorter than min_chars into the following chunk. The last chunk absorbs any trailing short chunks. Offsets and indices are updated in place.

pub fn stats(chunks: &[TextChunk]) -> ChunkStats

Compute aggregate statistics over a slice of chunks.

pub fn rechunk_with_strategy( &mut self, doc_id: &str, text: &str, strategy: ChunkStrategy, ) -> Vec<TextChunk>

Temporarily override the strategy on this chunker, produce chunks, then restore the original strategy.

pub fn set_metadata(chunks: &mut [TextChunk], key: &str, value: &str)

Add key = value metadata to every chunk in chunks.

pub fn chunker_stats(&self) -> (u64, u64)

Return (chunks_produced, documents_processed) counters.

Trait Implementations

impl Clone for DocumentChunker

fn clone(&self) -> DocumentChunker

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DocumentChunker

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.