Expand description
Chunk-record JSON export shared by the CLI (--to chunks) and the HTTP
server (to=chunks): the hierarchical chunker’s records always, plus the
hybrid chunker’s when a tokenizer is available — DOCLING_CHUNK_TOKENIZER,
or .models/chunk/tokenizer.json as populated by
scripts/install/download_dependencies.sh (requires the chunking build
feature; DOCLING_CHUNK_MAX_TOKENS overrides the default budget of 256).
Per-call ChunkOptions (#256, mirroring the fields of docling’s
service-datamodel HybridChunkerOptions) select a single chunker and
override the environment-derived tokenizer/budget; the env knobs stay the
operator-side defaults.
Structs§
- Chunk
Options - Per-call chunking configuration (#256). Every field falls back to the
environment-derived behavior
chunk_recordsalways had, soChunkOptions::default()is exactly the legacy export.
Enums§
- Chunker
Kind - Which chunker’s records
to=chunksreturns (docling’s service-datamodelChunkerType, #256).NoneonChunkOptionskeeps the legacy “both” shape: hierarchical always, hybrid best-effort.
Functions§
- chunk_
records - The chunk records for
documentas a JSON object ({"hierarchical": [...], "hybrid": [...]?}). Tokenizer problems don’t fail the export — the hybrid records are skipped and the problem is reported throughwarn. Equivalent tochunk_records_withunderChunkOptions::default(). - chunk_
records_ with chunk_recordswith per-callChunkOptions(#256). Errors only on an explicitly requested configuration that cannot be honored:chunker: hybridwithout a usable tokenizer (or in a build without thechunkingfeature). The legacy both-chunkers mode still degrades throughwarn.