Module chunking 

Token-aware chunking utilities for bodies that exceed the embedding window. Semantic chunking for embedding inputs (Markdown-aware, 512-token limit).

Splits bodies using text_splitter::MarkdownSplitter with overlap so multi-chunk memories preserve context across chunk boundaries.

Structs

Chunk

Constants

CHUNK_OVERLAP_CHARS

CHUNK_SIZE_CHARS

Functions

aggregate_embeddings

chunk_text

needs_chunking

split_into_chunks

split_into_chunks_by_token_offsets

split_into_chunks_hierarchical

Splits body into chunks using MarkdownSplitter with a real tokenizer. Respects Markdown semantic boundaries (H1-H6, paragraphs, blocks). For plain text without Markdown markers, falls back to paragraph and sentence breaks.