Module chunk 

Document chunking by sentence boundary and token budget. Document chunking by sentence boundary and token budget.

Splits text into token-budgeted chunks along sentence boundaries, with optional overlap between consecutive chunks so boundary context is preserved. Sentence terminators are script-aware: ASCII ., !, ? and the CJK full-stop/exclamation/question marks (U+3002, U+FF01, U+FF1F). Newlines also act as unit boundaries.

Token counts reuse crate::tokens::estimate_tokens; this module adds no new dependencies.

Structs

ChunkOptions

Options controlling sentence-aware chunking.

Functions

chunk_text

Split text into token-budgeted chunks along sentence boundaries.