iqdb-quantize 1.0.0

Vector quantization (scalar, product, binary) for memory-efficient vector search - part of the iQDB family.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

# iqdb-quantize v0.4.0 — Binary Quantization + Feature Freeze

**The last scheme lands, and the surface freezes.** v0.4.0 adds binary quantization (BQ, 32× compression), completing the three-scheme set behind the `Quantizer` trait. With SQ8, PQ, and BQ all present and asymmetric distance implemented for each, the public API is declared **feature frozen** — no new public items before 1.0.

## What is iqdb-quantize?

The memory-efficiency layer of the iQDB vector database — scalar (SQ8, 4×), product (PQ, up to 192×), and binary (BQ, 32×) quantization behind one `Quantizer` trait, with distance computed directly on the compressed form.

## What's new in 0.4.0

### `BinaryQuantizer` (BQ)

One bit per dimension, thresholded against a trained per-dimension mean: the bit is `1` when the component is at or above the mean, `0` otherwise. Bits pack into `Vec<u64>` words, and the unused high bits of the trailing word are always zero, so they cannot contribute to Hamming distance. BQ is the highest-compression scheme — 32× — and is scored directly on the packed words via XOR + popcount.

```rust
use iqdb_quantize::{BinaryQuantizer, Quantizer};
use iqdb_types::DistanceMetric;

let mut bq = BinaryQuantizer::new();
bq.train(&[&[0.0_f32, 1.0, 2.0][..], &[2.0_f32, 1.0, 0.0][..]]).expect("ok");

let code = bq.quantize(&[0.5_f32, 1.5, 2.5]).expect("dim matches");
assert_eq!(code.dim(), 3);
assert_eq!(code.as_words().len(), 1); // 3 bits fit in one u64 word

let d = bq
    .distance(&[0.5_f32, 1.5, 2.5], &code, DistanceMetric::Hamming)
    .expect("Hamming is supported");
assert!(d.is_finite());
```

BQ supports `DistanceMetric::Hamming` **only**; every other metric returns `IqdbError::InvalidMetric` — a one-bit code carries no magnitude for a Euclidean or dot-product comparison.

### `BqCode` — owned and immutable

The packed `u64` words plus the original dimension, produced only by `BinaryQuantizer::quantize`, with `dim`, `is_empty`, and `as_words` accessors and no public mutators.

## Feature freeze

The public surface is now **complete and frozen for 1.x** — additive, non-breaking changes remain allowed, anything else waits for 2.0:

- Trait: `Quantizer` (`Quantized`, `train`, `quantize`, `dequantize`, `distance`).
- Quantizers: `ScalarQuantizer`, `BinaryQuantizer`, `ProductQuantizer`.
- Codes: `Sq8Code`, `BqCode`, `PqCode`.
- Batch ADC: `PqAdcTables`, `ProductQuantizer::build_query_tables`.
- Constant: `VERSION`.

There is no `todo!` or `unimplemented!` anywhere in shipping code. PQ `Cosine` stays intentionally unsupported (no per-subvector global norm; L2-normalize and use `DotProduct`) — a settled decision, not a deferral.

## Breaking changes

**Pre-1.0 API churn.** Everything here is additive over 0.3.0. No existing item changed shape.

## Verification

BQ adds property tests asserting the packed-word Hamming distance matches a naive per-dimension popcount reference, and that distance is finite and non-negative for every valid pair. The full gate runs on the CI matrix on stable and the 1.87 MSRV:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
```

MSRV: Rust 1.87.

## What's next

- **v0.5.0 — recall validation + API freeze.** End-to-end recall measured against full-`f32` baselines, training instrumented through `tracing`, a criterion bench harness, and the public API locked for the 1.x series.

## Installation

```toml
[dependencies]
iqdb-quantize = "0.4"
```

## Documentation

- [README](https://github.com/jamesgober/iqdb-quantize/blob/main/README.md)
- [API reference](https://github.com/jamesgober/iqdb-quantize/blob/main/docs/API.md)
- [ROADMAP](https://github.com/jamesgober/iqdb-quantize/blob/main/dev/ROADMAP.md)
- [Standards (REPS)](https://github.com/jamesgober/iqdb-quantize/blob/main/REPS.md)
- [CHANGELOG](https://github.com/jamesgober/iqdb-quantize/blob/main/CHANGELOG.md)

---

**Full diff:** [`v0.3.0...v0.4.0`](https://github.com/jamesgober/iqdb-quantize/compare/v0.3.0...v0.4.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/iqdb-quantize/blob/main/CHANGELOG.md).