Expand description
SQ8 scalar quantization codecs for approximate distance computation in ANN indexes.
Two codecs with different encoding strategies:
§Sq8Codec — per-dimension affine, for dot product / cosine
Each dimension is mapped to [0, 255] using its own observed min/max.
Dot product and cosine require per-dim scale accuracy; the residual-corrected
path (approx_dot, approx_cosine_dist) preserves ordinal ranking.
§GsSq8Codec — global-scale affine, for L2 (Vamana acquisition)
A single shared scale gs = max_range_across_dims / 255 is used for all dims;
per-dim min_i offsets are still subtracted before quantizing.
L2² in code space: gs² × Σ (a_i - b_i)² — exact after the lossy f32→u8 encode
(offsets cancel, one scalar factorizes out). No anisotropy gate or residual pass for
in-distribution vectors; OOD queries (components outside the trained range) fall back
to exact f32 in the caller (see VamanaIndex::search). Small-range dims contribute
proportionally fewer codes and proportionally less L2 signal — an honest trade-off
documented in ADR-052.
§Hot-loop NEON helpers (u8_dot_u32, u8_l2sq_u32)
Both codecs share these inner functions:
u8_dot_u32: NEONvmull_u8(16-wide u8→u16→u32) or chunked portable fallback.u8_l2sq_u32: NEONvabdq_u8+vmull_u8squaring or chunked portable fallback.
Structs§
- Encoded
Vector - A corpus vector encoded by
Sq8Codec. - GsEncoded
Vector - A corpus vector encoded by
GsSq8Codec. - GsSq8
Codec - Global-scale SQ8 codec for L2 distance — the Vamana acquisition path.
- Sq8Codec
- Per-dimension affine SQ8 codec for dot product and cosine distance.
Enums§
- Quant
Error - Errors returned by the
try_*codec constructors and encoders.
Functions§
- u8_
l2sq_ u32 - Compute
Σ (a_i - b_i)²overu8slices as au32accumulator using NEONvabdq_u8(absolute difference) +vmull_u8squaring on aarch64, or a chunked portable fallback elsewhere.