Struct Chunker 

pub struct Chunker {
    pub size: usize,
    pub overlap: f32,
    pub unit: ChunkUnit,
}

A configured Markdown chunker.

Fields

size: usize

Target chunk size, measured in units (default 300).

overlap: f32

Fractional overlap between consecutive chunks (default 0.05 = 5%).

unit: ChunkUnit

The unit size/overlap are measured in.

Implementations

impl Chunker

pub fn from_config(cfg: &RagConfig) -> Self

Build a chunker from the resolved config.

pub fn chunk(&self, doc_id: &str, markdown: &str) -> Vec<Chunk>

Chunk a Markdown document into Chunks owned by doc_id.

Implemented over StreamingChunker so batch and streaming ingestion share one code path (and cannot diverge).

pub fn streaming(&self, doc_id: &str) -> StreamingChunker

Start an incremental chunking session for one document. Feed Markdown pieces with StreamingChunker::push as they arrive from the parser and collect completed chunks immediately; call StreamingChunker::finish after the last piece.

Trait Implementations

impl Clone for Chunker

fn clone(&self) -> Chunker

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Chunker

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Chunker

fn default() -> Self

Returns the “default value” for a type.