Struct ChunkOptions 

pub struct ChunkOptions {
    pub max_tokens: usize,
    pub overlap_tokens: usize,
}

Options controlling sentence-aware chunking.

Fields

max_tokens: usize

Maximum estimated tokens per chunk.

overlap_tokens: usize

Target token overlap between consecutive chunks.

Implementations

impl ChunkOptions

pub fn new(max_tokens: usize, overlap_tokens: usize) -> Self

Create chunking options with the given token budget and overlap.

max_tokens must be greater than zero; overlap_tokens may be zero to disable overlap. If overlap_tokens is greater than or equal to max_tokens it is clamped down to max_tokens - 1 so progress is still guaranteed.

Trait Implementations

impl Clone for ChunkOptions

fn clone(&self) -> ChunkOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ChunkOptions

impl Debug for ChunkOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkOptions

fn default() -> Self

Default options: 512 tokens per chunk with 64 tokens of overlap.

A sensible middle ground for short-context retrieval: chunks are small enough to feed most embedding models, while the overlap keeps adjacent chunks sharing context across their boundary.