Expand description
§Vector Quantizer
Production-grade product quantization (PQ) for compressing high-dimensional embedding vectors into compact codes for efficient approximate nearest-neighbor (ANN) search.
§Overview
Product Quantization divides a high-dimensional vector into M subspaces, each of
dimension D/M, and quantizes each subspace independently using a small codebook of
K centroids learned via k-means. Each sub-vector is then represented by the index
of its nearest centroid — a single u8 value — so the full vector is stored as M
bytes regardless of the original dimensionality.
§Algorithm
-
Training: For each subspace, run k-means on the projected sub-vectors to learn
codes_per_subspacecentroids. Initialization picks everyN/K-th training vector to seed the codebook (deterministic, no external RNG dependency). -
Encoding: Map each sub-vector to the index (
u8) of its nearest centroid. -
Decoding: Reconstruct the approximate full vector by concatenating the centroid vectors retrieved from each codebook.
-
Distance: Asymmetric distance computes exact sub-vector distances from query to reconstructed codes; symmetric distance decodes both codes first.
Structs§
- Codebook
- A learned codebook for one PQ subspace.
- Quantization
Config - Configuration for
VectorQuantizer. - Quantization
Stats - Runtime statistics for a
VectorQuantizer. - Quantizer
Code - Compact quantized representation of a vector.
- Vector
Quantizer - Product-quantization based vector compressor.
Enums§
- VqError
- Errors produced by
VectorQuantizeroperations.