Module quantization 

Vector quantization for IVF indexes

Residual Product Quantization with OPQ is the trained float codec; TurboQuant (tq) is the training-free per-segment codec.

Structs

DistanceTable

Precomputed distance table for fast asymmetric distance computation

PQCodebook

Learned codebook for Product Quantization with OPQ rotation

PQConfig

Configuration for residual Product Quantization with OPQ.

TqCodec

Complete TQ codec for one field dimension. Cheap to build (sub-millisecond) and immutable; share via Arc per open segment.

TqEncodeScratch

Reusable per-thread encode buffers (hot-path allocation hygiene).

TqFlatBuilder

Streaming builder for one segment’s TQ payload: doc/ordinal columns plus a block-packed codes column ready for ann_disk serialization.

TqQueryPlan

Per-query LUTs: padded_dim × 16 i8 tables (globally-scaled quantizations) for the block kernels. The intermediate f32 tables are dropped after quantization — they are not read on the search path.

Constants

TQ_BLOCK_LANES

Vectors per scoring block; one lane per vector.

TQ_CODEC_VERSION

Bumping this refuses to mix payloads across incompatible codec revisions.

Functions

tq_block_bytes

Bytes of one scoring block: 16 f32 gammas + 16 packed nibble rows.

tq_codes_column_len

Total codes-column bytes for count vectors (final block zero-padded).

tq_codes_column_len_checked

Overflow-checked tq_codes_column_len for untrusted header values.

tq_expected_fingerprint

Fingerprint every payload built for dim must carry (no codebook build).

tq_ivf_block_bytes

Bytes of one IVF-TQ scoring block: 16 f32 residual scales + 16 f32 gammas

tq_ivf_codes_column_len_checked

Overflow-checked IVF-TQ codes-column length for untrusted header values.

tq_pack_block

Pack up to 16 nibble rows (+ gammas) into one block. Missing lanes are zero-filled. rows are padded_dim-length 0..=15 values.

tq_pack_ivf_block

Pack an IVF-TQ block: per-lane residual scales, gammas, then nibbles.

tq_padded_dim

Padded (power-of-two, ≥ [TQ_MIN_PADDED_DIM]) dimension for an input dimension. Cheap; usable for header validation without building a codec.

tq_score_block

Score one block (16 lanes) into scores. block is [16 × f32 gamma][padded_dim × 8 packed nibbles]; lanes past the run’s vector count hold zero padding and must be ignored by the caller.

tq_score_ivf_block

Score one IVF-TQ block: score[lane] = cluster_dot + scale · (base + gamma · qjl), where cluster_dot = ⟨normalized query, centroid⟩ is the probed cluster’s shared contribution and scale = ‖residual‖.