Module vecquant 

2g: data-oblivious vector fingerprints (TurboQuant-style, arXiv 2504.19874).

Beginner’s map of what happens and why it needs NO training:

1. ROTATE the vector by a fixed random rotation. A rotation preserves all distances and dot products, but after it every coordinate of a unit vector looks like a small Gaussian: sigma = 1/sqrt(dim), regardless of what the data means. That statistical guarantee is the whole trick – it replaces the per-dataset codebook training PQ needs.
2. QUANTIZE each rotated coordinate independently against a FIXED ladder of 2^BITS levels sized for that Gaussian. The recipe depends only on (dimension, seed); both live in the catalog, so any process can encode or decode any vector at any time, forever.
3. Store the original LENGTH (one f32): the code approximates the vector’s direction; the norm restores its scale.

The rotation is a fast Walsh-Hadamard transform (FWHT) with seeded random sign flips, ROUNDS times: O(d log d) instead of a d*d matrix multiply, and zero bytes of stored matrix. FWHT needs a power-of-two width, so vectors are zero-padded up to one (1536 -> 2048).

Search math: dot(x, q) = |x| * dot(unit_code(x), rotate(q)) because rotations preserve dots. One estimated dot yields L2, cosine and dot scores alike. The estimate is APPROXIMATE – callers oversample and rescore against the exact f32 rows (D23), so approximation can only ever cost a MISS, never a wrong distance in the final ranking.

Structs

AffineQuery

The affine estimate (2g.2): because the quantizer ladder is UNIFORM, level value v(l) = l * step - SPAN (in coordinate units), so dot(x, q) ~ norm * ( step * SUM(l_j * q_j) - SPAN * SUM(q_j) ) / sd The second sum is one constant per query; the first needs no table at all – unpack each nibble to an integer, convert, multiply-add. Every step of that loop is straight-line arithmetic the compiler can vectorize, unlike a table lookup which never vectorizes. Four accumulators break the dependency chain as before.

Encoder

The prepared recipe: sign vectors and scales are FIXED per (seed, width), so they are computed once here, never per vector. Before this hoist the encode path allocated fresh sign vectors and re-ran the PRNG for EVERY vector – measured 3.3x on the 250K ingest ladder; the hoist plus ROUNDS 3 -> 1 (recall unchanged at 1.000 on both the gaussian and the hostile sparse dataset; the identity-rotation mutation fails the sparse case at 0.555) brought bulk 15.9s -> 8.6s.

Constants

DEFAULT_BITS

Default code width. 4-bit = 16 levels, dim/2 bytes; 2-bit = 4 levels, dim/4 bytes – half the scan I/O and half the unpack work, the lever that matters when the code keyspace outgrows the pool. Recorded per store in the catalog; the recall gate decides the default.

MAX_LEVELS

Functions

code_len

Bytes of code per vector for a padded width (norm f32 NOT included).

dot_est_affine

dot estimate via the affine form, 4-bit codes (nibble-packed).

dot_est_affine2

level_table

The 16 reconstruction values, in coordinate units (already divided by sqrt(padded)): dequant(level) * sd = the ladder midpoint.

pad_dim