Expand description
Product Quantization (PQ): 8-16x compression for large datasets.
Splits D-dimensional vectors into M subvectors, clusters each subspace with K=256 centroids via k-means. Each vector is encoded as M bytes (one centroid index per subvector).
Distance is computed via precomputed lookup tables: for each query,
build a [M][K] table of distances from the query’s subvectors to
all centroids. Then the distance to any encoded vector is just M
table lookups + additions — O(M) per candidate vs O(D) for FP32.
Trade-off: 2-5% recall loss vs SQ8’s <1%, but 2-4x more compression (8-16x total vs 4x for SQ8). Best for cost-sensitive large datasets.
Structs§
- PqCodec
- PQ codec with trained codebooks.