Module chunk 

Markdown-aware chunking.

The chunker parses Markdown into sections (split at every heading, carrying a heading path), then slides a fixed-size window with fractional overlap over the words of each section. A chunk never crosses a heading boundary, and each chunk is prefixed with its heading path so the embedded text keeps its context.

The section parsing and windowing live in docling::chunker::WindowChunker (shared with the Python/Node bindings); this module maps its chunks onto retrievable Chunks and adds the incremental StreamingChunker buffer.

Structs

Chunker

A configured Markdown chunker.

Section

A contiguous run of body text under a heading path.

StreamingChunker

Incremental Markdown chunker: buffers streamed pieces and emits chunks for every section completed so far. A section only completes at the next heading (chunks never cross headings), so the buffer is cut at the start of the last heading line — everything before it is fully-formed blocks. Heading lines inside code fences are not cut points.

Functions

docling_chunks

Chunk a converted document with docling’s structure-aware chunkers (RAG_CHUNKER=hierarchical|hybrid), mapping each DocChunk onto a retrievable Chunk. The embedded text is docling’s contextualize() rendering (heading path + chunk body), the analogue of the window chunker’s heading-context prefix; the heading path and source item refs are kept as chunk metadata.

docling_chunks_with

Streaming docling_chunks: sink receives each retrievable Chunk as the docling chunkers produce it, so chunks can flow into embedding while the rest of the document is still being chunked. A false return from sink cancels the chunking.