Expand description
SQ8 scalar quantization codecs for approximate distance computation in ANN indexes.
Two codecs with different encoding strategies:
§Sq8Codec — per-dimension affine, for dot product / cosine
Each dimension is mapped to [0, 255] using its own observed min/max. Dot product and cosine reconstruct each dimension in f64 before summing, preserving small dimensions beside a much wider one.
§GsSq8Codec — global-scale affine, for L2 (Vamana acquisition)
A single shared scale gs = max_range_across_dims / 255 is used for all dims;
per-dim min_i offsets are still subtracted before quantizing.
L2² in code space: gs² × Σ (a_i - b_i)² — exact after the lossy f32→u8 encode
(offsets cancel, one scalar factorizes out). No anisotropy gate or residual pass for
in-distribution vectors; OOD queries (components outside the trained range) fall back
to exact f32 in the caller (see VamanaIndex::search). Small-range dims contribute
proportionally fewer codes and proportionally less L2 signal — an honest trade-off
documented in ADR-052.
§Hot-loop NEON helper (u8_l2sq_u32)
GsSq8Codec uses u8_l2sq_u32: NEON vabdq_u8 + vmull_u8
squaring or a chunked portable fallback. The integer dot helper remains
test-only; its shared-mean scaling loses small per-dimension weights.
See docs/api/codecs.md for the full function-by-function reference and
docs/design.md for the anisotropy-gating rationale behind GsSq8Codec.
Structs§
- Encoded
Vector - A corpus vector encoded by
Sq8Codec. - GsEncoded
Vector - A corpus vector encoded by
GsSq8Codec. - GsSq8
Codec - Global-scale SQ8 codec for L2 distance — the Vamana acquisition path.
- Sq8Codec
- Per-dimension affine SQ8 codec for dot product and cosine distance.
Enums§
- Quant
Error - Errors returned by the
try_*codec constructors and encoders.
Functions§
- u8_
l2sq_ u32 Σ (a_i - b_i)²over equal-lengthu8slices as au32accumulator (NEON on aarch64, chunked portable fallback elsewhere). Seedocs/api/codecs.mdfor the kernel breakdown.