turboquant 0.1.1

Implementation of Google's TurboQuant algorithm for vector quantization
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

# Contributing

## Scope

TurboQuant is a Rust library for KV-cache quantization research. Contributions should prioritize correctness, typed errors, reproducible benchmarks, and honest documentation over feature count.

## Prerequisites

- Rust `1.87.0`
- `cargo fmt`, `clippy`, `llvm-tools-preview`
- Optional for trace export: Python 3 with `torch`, `transformers`, `numpy`, and `safetensors`
- Optional for real-model ONNX export: Python 3 with the pinned packages in `scripts/requirements-real-model.txt`

## Local Setup

```bash
cargo fmt -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo check --examples --all-features
```

For coverage and dependency review:

```bash
cargo llvm-cov --workspace --all-features --summary-only
cargo audit
```

## Benchmark Smoke Tests

Synthetic:

```bash
cargo run --release --example benchmark -- --workload synthetic --quick
```

Trace:

```bash
cargo run --release --example benchmark -- \
  --workload trace \
  --trace traces/example.safetensors
```

Real-model compare run:

```bash
cargo run --release --example benchmark -- \
  --workload real-model \
  --real-model-dir artifacts/smollm2-135m-onnx \
  --prompt "Explain KV-cache quantization in one sentence." \
  --real-eval-mode compare \
  --bits 4 \
  --real-key-strategy prod \
  --max-new-tokens 16
```

Experimental WGPU batch path:

```bash
cargo run --release --example benchmark --features gpu -- --workload synthetic --quick --backend wgpu
```

## Real-Model Export Setup

```bash
python3 -m venv .venv
. .venv/bin/activate
pip install -r scripts/requirements-real-model.txt
python3 scripts/export_hf_decoder_onnx.py \
  --preset smollm2-135m-instruct \
  --output-dir artifacts/smollm2-135m-onnx
```

## CI-Safe and Manual Tests

CI-safe:

- `cargo test --all-features`
  Includes the tiny checked-in ONNX fixture.

Manual heavier smoke test:

```bash
TURBOQUANT_REAL_MODEL_DIR=artifacts/smollm2-135m-onnx \
  cargo test --all-features manual_exported_real_model_smoke_test -- --ignored --nocapture
```

## Contribution Rules

- Keep public APIs explicit about preconditions and failure modes.
- Do not silently clamp, truncate, or reinterpret malformed external input.
- Add or update tests for every bug fix and every new documented behavior.
- If you change measured behavior, update the benchmark path or documentation in the same pass.
- Keep experimental surfaces labeled as experimental until their guarantees materially improve.
- Do not relabel synthetic or trace workloads as real-model runs.

## Code Review Expectations

- Formatting and clippy must be clean.
- All tests must pass with `--all-features`.
- New user-visible behavior must be documented in `README.md` and `CHANGELOG.md`.
- Non-obvious design choices should be explained either in code comments or in `ARCHITECTURE.md`.

## Release Hygiene

Before tagging a release candidate:

- update `CHANGELOG.md`
- update the status/limitations sections in `README.md`
- rerun lint, tests, coverage, audit, and benchmark smoke commands
- rerun at least one manual real-model smoke test on an exported bundle