Module chunk_map 

Virtual-id maps of chunked text fields (seg_<id>.chunks).

A text field declared chunked indexes every value as its own scoring unit: term postings and positions are keyed by a dense, segment-local virtual id instead of the document id. This file maps each virtual id back to (doc_id, ordinal) and records the chunk’s token count for BM25 length normalisation. See docs/chunked-text-fields.md.

[magic "CHNK"][version u32, 1..=5][num_sections u32]
TOC × num_sections: [field_id u32][kind u32][count u32][total_tokens u64][data_offset u64]
kind 0 (chunk map):   doc_ids u32 × n | ordinals u16 × n | lengths u16 × n
kind 2 (addressed map): kind 0 columns | logical-order physical slots u32 × n
kind 1 (doc lengths): lengths u16 × num_docs        (norms of a plain text field)
kind 3 (document map, V4): addressed map with document scoring semantics
kind 4 (byte norms, V5): byte4 norm codes × num_docs, exact total_tokens in TOC

Version 1 files have 24-byte entries without kind and hold chunk maps only; they are still read.

Virtual ids are assigned in indexing order, and documents are indexed in doc-id order, so doc_ids starts out non-decreasing. A reorder pass on a field with the reorder attribute permutes the virtual ids (BP over the field’s postings, segment/text_reorder.rs). Readers verify doc-id order at open before enabling ordered query paths. Merges concatenate sections and add the document offset to doc_ids; ordinals and lengths are copied verbatim.

A doc-length section stores the token count of the field in every document of the segment (0 when the document has no value), so BM25 can normalise plain fields by their real length instead of tf.

Structs

ChunkMap

Read-only chunk map of one field, backed by the mapped .chunks file.

ChunkMapBuilder

In-memory map of one chunked field while a segment is being built.

ChunkMapFile

Everything a .chunks file holds.

ChunkMapSource

One source section of a merged chunk map.

DocLengths

Read-only per-document lengths of one plain text field, backed by the mapped .chunks file.

DocLengthsColumn

Per-document token counts of one plain text field, ready to be written.

DocLengthsSource

One source of a merged length column: the source segment’s column when it has one, and its document count (zeros are written for a missing column).

Constants

MAX_CHUNK_LENGTH

Token count stored per chunk; longer chunks saturate.

Functions

read_chunk_maps

Parse a .chunks file into per-field chunk maps and length columns.

write_chunk_maps

Write every chunked field’s map and every plain field’s length column as one .chunks file.

write_merged_chunk_maps

Write the merged .chunks file: per field, the sources’ sections are concatenated in order (virtual ids of a later source are offset by the chunk counts of the earlier ones, matching the posting merge; length columns follow the document order of the merge).