Struct ChunkerConfig 

pub struct ChunkerConfig {
    pub target_tokens: usize,
    pub min_tokens: usize,
    pub max_tokens: usize,
    pub overlap_messages: usize,
}

Configuration for the chunker.

Fields

target_tokens: usize

Target tokens per chunk (default: 1500)

min_tokens: usize

Minimum tokens — don’t create tiny chunks unless it’s the last window (default: 500)

max_tokens: usize

Maximum tokens — force split if exceeded (default: 2500)

overlap_messages: usize

Number of messages to overlap between consecutive windows (default: 2)

Trait Implementations

impl Clone for ChunkerConfig

fn clone(&self) -> ChunkerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkerConfig

fn default() -> Self

Returns the “default value” for a type.