pub fn chunk_text(text: &str, opts: &ChunkOptions) -> Vec<String>Expand description
Split text into token-budgeted chunks along sentence boundaries.
The algorithm packs sentence units greedily into a chunk until adding the
next unit would exceed ChunkOptions::max_tokens. When a new chunk
starts, trailing units from the just-pushed chunk are carried over as
overlap so consecutive chunks share boundary context. Overlap is
best-effort but never stalls: up to ChunkOptions::overlap_tokens of
trailing units are carried when they fit; if even the budget is exceeded
the last unit is still carried (so context is preserved); overlap is
skipped only when the emitted chunk was a single unit or overlap_tokens
is zero.
A single unit that on its own exceeds the budget is emitted as its own chunk — content is never dropped. Empty or whitespace-only input yields an empty vector.
Performance: the running window is re-estimated with
crate::tokens::estimate_tokens on each unit addition, so the cost is
O(k²) in the number of units per chunk. This is fine for typical retrieval
chunk sizes (hundreds of tokens); very large inputs with many tiny units
are not the intended use case.