Module sections 

Section payload formats (v1).

Each non-binary section starts with a 12-byte header:

[0..4)   u32 version  (LE) - currently 1
[4..12)  u64 count    (LE) - number of entries

Then a payload that depends on the section. Embeddings use a different shape (no per-entry header - dim/count come from the file header).

All multi-byte integers are little-endian. Strings are raw UTF-8 bytes prefixed by a u32 length (no NUL terminators).

Re-exports

pub use blob::BLOB_DATA_PAYLOAD_VERSION;

pub use blob::BLOB_REF_NONE;

pub use blob::BLOB_REFS_PAYLOAD_VERSION;

pub use blob::BLOB_SPAN_OVERLAY_PAYLOAD_VERSION;

pub use blob::BlobDataTable;

pub use blob::BlobRefRecord;

pub use blob::BlobSpanEntry;

pub use blob::decode_blob_data_table;

pub use blob::decode_blob_refs;

pub use blob::decode_blob_span_overlay;

pub use blob::encode_blob_data;

pub use blob::encode_blob_refs;

pub use blob::encode_blob_span_overlay;

Modules

blob

blob section codecs (media blob references and the blob-relative span overlay). both are OPTIONAL and EXCLUDED from content_hash, additive within frozen format v1, and never touch the embedding hot path. refs is the 0x14 table of content-hash references to source media (self-contained or catalog); span_overlay is the 0x16 per-chunk blob-relative span replacement for chunks_original_spans (0x03).

graph

graph section codecs (chunk-to-chunk csr adjacency and, later, the entity/property-graph sections). all are OPTIONAL and EXCLUDED from content_hash, additive within frozen format v1, and never touch the embedding hot path. G1 ships only adjacency (SECTION_GRAPH_ADJACENCY, 0x0C); the entity sections (0x11..=0x13) land in G2.

Structs

OriginalSpan

SearchContract

SpaceEntry

one row of the 0x15 table. space_index addresses the band: the vectors live at section 0x20 + space_index (and the optional fp source at 0x30 + space_index). model_hash fingerprints the model that produced THIS space’s vectors.

Constants

REPACK_KIND_CHUNK_IDS

intpack (encoding id 4) repack kinds. the kind byte leads the packed payload so the wire codec can rebuild the exact canonical bytes of the section it was packed from, keeping content_hash stable. canonical sections are never version-bumped; this is a wire encoding, not a payload-format change.

REPACK_KIND_SPANS

SPACE_DTYPE_F16

SPACE_DTYPE_F32

dtype codes for the band slab (mirror the manifest dtype strings).

SPACE_DTYPE_I4

SPACE_DTYPE_I8

SPACE_TABLE_PAYLOAD_VERSION

Functions

decode_chunk_ids

decode_chunk_ids_intpack

reconstruct the canonical (raw-encoding) chunk-ids payload from the body of an intpack repack (the bytes after the kind byte). the output is byte-identical to encode_chunk_ids so content_hash is preserved.

decode_chunks_canonical

decode_chunks_original_spans

decode_chunks_original_spans_intpack

reconstruct the canonical (raw-encoding) spans payload from the body of an intpack repack (the bytes after the kind byte). byte-identical to encode_chunks_original_spans so content_hash is preserved.

decode_intpack_repack

decode an intpack repack payload (the full wire bytes, including the leading kind byte) back to the canonical section payload it was packed from. dispatched by encoding::decode_payload for encoding id 4.

decode_provenance

decode_search_contract

decode_space_table

decode the payload back to entries. typed errors on truncation, a version mismatch, hostile counts, or an invalid entry; never panics.

decode_txt_streams

decode a txt_streams payload (the full wire bytes, including the leading kind/version byte) back to the canonical chunks_canonical section payload. byte-identical to encode_chunks_canonical, so content_hash is unchanged. dispatched by encoding::decode_payload for encoding id 10 (parallel to decode_intpack_repack).

encode_chunk_ids

encode_chunk_ids_intpack

encode the chunk-ids section as an intpack repack payload: a kind byte, the count, then the 32 raw digest bytes per id. returns None when any id is not a canonical sha256:<64 lowercase hex> string, so the writer falls back to the raw encoding and reconstruction stays guaranteed byte-exact.

encode_chunks_canonical

encode_chunks_original_spans

encode_chunks_original_spans_intpack

encode the spans section as an intpack repack: a kind byte, the count, a deduped uri pool, then three bitpacked columns (uri index, byte_start, and byte_end - byte_start). lengths are stored as a wrapping difference so reconstruction is byte-exact for any input.

encode_provenance

encode_search_contract

encode_space_table

encode entries into the 0x15 payload. deterministic: same entries in the same order, same bytes. rejects duplicate indices/names.