Struct ChunkingStrategy 

pub struct ChunkingStrategy {
    pub max_window_tokens: usize,
    pub overlap_tokens: Vec<usize>,
    pub summary_fallback_weight: f32,
    pub summary_fallback_tokens: usize,
    pub chunk_weight_floor: f32,
    /* private fields */
}

Controls how long text sections are chunked and weighted.

Fields

max_window_tokens: usize

Max tokens per window when slicing a section into chunks.

overlap_tokens: Vec<usize>

Overlap sizes (in tokens) used when sliding windows across a section.

summary_fallback_weight: f32

Weight assigned to summary-fallback chunks (when generated).

summary_fallback_tokens: usize

Max tokens for summary-fallback chunks (0 disables fallback chunks).

chunk_weight_floor: f32

Floor applied to per-chunk weight after offset or summary fallback weighting.

Implementations

impl ChunkingStrategy

pub fn register_preprocessor( &mut self, p: impl TextPreprocessor + 'static, ) -> &mut Self

Register a crate::preprocessor::TextPreprocessor to run before chunking.

Preprocessors are applied in registration order; if any returns None the section is dropped and produces no chunks.

pub fn preprocessors(&self) -> &[Arc<dyn TextPreprocessor>]

Return the slice of registered preprocessors.

Trait Implementations

impl Clone for ChunkingStrategy

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkingStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkingStrategy

fn default() -> Self

Returns the “default value” for a type.