Skip to main content

Module chunks

Module chunks 

Source
Expand description

Chunk-record JSON export shared by the CLI (--to chunks) and the HTTP server (to=chunks): the hierarchical chunker’s records always, plus the hybrid chunker’s when a tokenizer is available — DOCLING_CHUNK_TOKENIZER, or .models/chunk/tokenizer.json as populated by scripts/install/download_dependencies.sh (requires the chunking build feature; DOCLING_CHUNK_MAX_TOKENS overrides the default budget of 256).

Per-call ChunkOptions (#256, mirroring the fields of docling’s service-datamodel HybridChunkerOptions) select a single chunker and override the environment-derived tokenizer/budget; the env knobs stay the operator-side defaults.

Structs§

ChunkOptions
Per-call chunking configuration (#256). Every field falls back to the environment-derived behavior chunk_records always had, so ChunkOptions::default() is exactly the legacy export.

Enums§

ChunkerKind
Which chunker’s records to=chunks returns (docling’s service-datamodel ChunkerType, #256). None on ChunkOptions keeps the legacy “both” shape: hierarchical always, hybrid best-effort.

Functions§

chunk_records
The chunk records for document as a JSON object ({"hierarchical": [...], "hybrid": [...]?}). Tokenizer problems don’t fail the export — the hybrid records are skipped and the problem is reported through warn. Equivalent to chunk_records_with under ChunkOptions::default().
chunk_records_with
chunk_records with per-call ChunkOptions (#256). Errors only on an explicitly requested configuration that cannot be honored: chunker: hybrid without a usable tokenizer (or in a build without the chunking feature). The legacy both-chunkers mode still degrades through warn.