Module chunker 

Document chunking for RAG pipelines — the Rust port of docling-core’s docling_core.transforms.chunker.

Two chunkers, matching docling’s semantics output-for-output:

• HierarchicalChunker walks the document tree and yields one chunk per top-level item (paragraph, whole list, table, picture caption, …), each carrying the heading path it sits under. Tables are serialized in docling’s triplet form (row, column = value), pictures contribute their captions.
• HybridChunker refines the hierarchical chunks with a tokenizer: oversized chunks are split (at item boundaries first, then within the text by docling’s semchunk algorithm), and undersized neighbours that share the same headings are merged back together.

contextualize renders a chunk to the string an embedding model should see: the heading path plus the chunk text.

Anything tokenizer-related is abstracted behind ChunkTokenizer; a HuggingFace tokenizers implementation ships behind the chunking cargo feature as [HuggingFaceTokenizer].

Structs

ChunkItem

One document item contributing to a chunk — the analogue of an entry in docling’s DocMeta.doc_items.

DocChunk

One chunk — the analogue of docling’s DocChunk (text + DocMeta).

HierarchicalChunker

Structure-driven chunker: one chunk per document item, lists and inline groups kept whole, heading path tracked as metadata — docling-core’s HierarchicalChunker with default parameters.

HybridChunker

Tokenization-aware chunker on top of HierarchicalChunker — docling’s HybridChunker with default parameters (merge_peers, repeat_table_header on; omit_header_on_overflow off).

Enums

ChunkItemKind

What kind of document item a ChunkItem points at — only what the hybrid splitting logic needs to know.

Traits

ChunkTokenizer

Token counting for HybridChunker — docling’s BaseTokenizer.

Functions

contextualize

Render a chunk for embedding: the heading path, then the text, joined with newlines — docling’s BaseChunker.contextualize().

semchunk

Split text into chunks of at most chunk_size tokens using the most semantically meaningful splitter available — the semchunk algorithm docling’s HybridChunker delegates plain-text splitting to.