Module chunk 

Deterministic text chunking for the packing stage.

No chunker exists anywhere else in the workspace (every other “chunk” is vector batching), so this is the reference implementation. Guarantees:

• Deterministic: same text + same policy ⇒ same chunks, same ranges.
• UTF-8 safe: never cuts inside a multi-byte char.
• Fence-atomic: never cuts inside a triple-backtick-fenced code block — a fence larger than ChunkPolicy::max_chunk_bytes stays one oversized chunk rather than being broken (the packing layer decides its fate whole).
• Covering: without overlap, the chunk ranges partition the input — concatenating them reconstructs the text byte for byte.

Structs

ChunkPolicy

How oversized fragments are split before packing.

TextChunk

One chunk of a larger text.

Enums

ChunkBoundary

Which boundaries the chunker prefers to cut at.

Functions

chunk_text

Split text into chunks under policy. Empty text yields no chunks.