# quant-eval
`quant-eval` is a Rust crate for evidence-first compression and retrieval evaluation. It provides deterministic benchmark scaffolds, typed result shapes, benchmark receipts, RAG fixture metrics, and HyperQuant primitive evaluation before any codec is promoted into a governor or runtime path.
Current status: prototype-to-evidence benchmark substrate. It contains real metric code and deterministic fixtures, but some harnesses still use synthetic data or simulated compression paths. It should not be described as proving workload-level codec quality, production readiness, or model performance until real codec adapters and corpus receipts exist.
## What this gives you
`quant-eval` gives compression and retrieval crates a place to produce evidence before integration:
- **Compression benchmark scaffolding** — deterministic synthetic vector corpus, exact nearest-neighbor baseline, recall@K, MRR, and overlap-derived similarity summaries.
- **Semantic-memory search scaffolding** — synthetic index/query generation, precision@K, recall@K, NDCG@K, MAP, and degradation-ratio calculations.
- **Admissibility harness** — profile-oriented checks over deterministic standard vectors.
- **Benchmark receipts** — timestamped receipt structures with machine fingerprint, result list, JSON serialization, hashes, and diffs.
- **RAG fixture metrics** — local recall@K, NDCG@K, and exact-rerank recovery over caller-supplied query/retrieval fixtures.
- **HyperQuant primitive evaluation** — deterministic Z1/A2 evaluation through the published `hyperquant` crate, with mean/max MSE, estimated bytes, rejected-vector counts, receipt counts, and explicit claim boundaries.
- **Conservative public surface** — measurement APIs first; no silent production claims.
## Evidence pipeline
```text
fixtures / synthetic corpora
↓
codec or retrieval harness
↓
metrics: MSE, recall@K, MRR, NDCG, MAP, recovery
↓
benchmark receipts + diffs
↓
policy/admission decisions in downstream crates
```
The crate is intentionally upstream of runtime policy. It measures and records; it does not decide that a codec is admissible for a truth-bearing system.
## Installation
```toml
[dependencies]
quant-eval = "0.1.1"
```
From the RecursiveIntell Libraries workspace:
```toml
[dependencies]
quant-eval = { path = "../quant-eval" }
```
## Quick start: compression benchmark scaffold
```rust
use quant_eval::{CompressionBenchmark, CompressionBenchmarkConfig};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let benchmark = CompressionBenchmark::with_config(CompressionBenchmarkConfig {
dim: 64,
db_size: 100,
queries: 10,
seed: 42,
top_k: 5,
iterations: 10,
});
let report = benchmark.run()?;
println!("recall@{} = {}", report.top_k, report.recall_at_k);
println!("mrr = {}", report.mrr);
Ok(())
}
```
## Quick start: HyperQuant primitive evaluation
```rust
use quant_eval::{run_hyperquant_eval, HyperQuantEvalConfig};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let result = run_hyperquant_eval(&HyperQuantEvalConfig {
dim: 16,
vectors: 64,
seed: 42,
scale: 8.0,
})?;
for profile in &result.profiles {
println!(
"{:?}: mean_mse={} max_mse={} receipts={}",
profile.kind,
profile.mean_mse,
profile.max_mse,
profile.receipt_count
);
}
println!("claim boundary: {}", result.claim_boundary);
Ok(())
}
```
## Public API
The crate re-exports:
- `AdmissibilityTest`
- `CodecProfile`
- `CompressionBenchmark`
- `CompressionBenchmarkConfig`
- `SemanticMemoryBenchmark`
- `SemanticMemoryConfig`
- `QuantEvalError`
- `MachineFingerprint`
- `BenchmarkReceipt`
- `BenchmarkResult`
- `ReceiptDiff`
- `evaluate_rag_fixture`
- `RagEvalResult`
- `RagQueryFixture`
- `RagRetrievedDoc`
- `run_hyperquant_eval`
- `HyperQuantEvalConfig`
- `HyperQuantEvalResult`
- `HyperQuantProfileEval`
## Implemented modules
### Admissibility harness
File: `src/benchmarks/admissibility.rs`
Implemented:
- `CodecProfile` presets: `fast`, `balanced`, and `high_compression`.
- `AdmissibilityTest` over caller-provided `TestSetEntry` values.
- Standard synthetic test vectors for zero, unit, and deterministic pseudo-random vectors.
- Summary counts per profile.
Important limitation:
- This harness still simulates codec behavior from `should_succeed` and profile quality targets. It does not yet call a shared `quant-codec-core` trait.
### Compression benchmark scaffold
File: `src/benchmarks/compression.rs`
Implemented:
- Deterministic synthetic corpus and query generation.
- Raw nearest-neighbor computation with cosine similarity.
- Recall@K and MRR calculations over exact-vs-estimated result sets.
- Similarity-style summary statistics over top-K overlap.
Important limitations:
- Compression is currently simulated by returning exact result sets.
- It does not yet measure real encoded byte size, compression ratio, per-block ratios, wire formats, or codec theoretical ratios.
- The reported cosine-similarity statistics are derived from top-K overlap, not raw-vs-decoded vector cosine.
### Semantic-memory benchmark scaffold
File: `src/benchmarks/semantic.rs`
Implemented:
- Deterministic synthetic index and query generation.
- Raw search baseline using cosine similarity.
- Synthetic relevance judgments from raw top-K results.
- Precision@K, Recall@K, NDCG@K, MAP, and degradation-ratio calculations.
Important limitation:
- Compressed search currently delegates to raw search, so degradation is simulated/minimal by construction. It is not evidence of real codec preservation quality.
### RAG fixture harness
File: `src/rag.rs`
Implemented:
- Query fixtures with explicit relevant document IDs.
- Retrieved document list with scores.
- Recall@K.
- NDCG@K.
- Exact-rerank recovery for top-ranked relevant result.
- Duplicate retrieved-doc suppression.
### HyperQuant primitive harness
File: `src/hyperquant_eval.rs`
Implemented:
- `HyperQuantEvalConfig`
- `HyperQuantProfileEval`
- `HyperQuantEvalResult`
- `run_hyperquant_eval`
- deterministic synthetic fixture generation;
- triangular A2 fixture where A2 should match or beat Z1;
- Z1/A2 metrics through the published `hyperquant` crate;
- conservative claim-boundary string on every result.
Important limitation:
- This is primitive-level evidence only. It is not HyperQuant paper parity, model-quality evidence, or production admissibility.
### Benchmark receipts
Files: `src/receipt.rs`, `src/fingerprint.rs`
Implemented:
- `BenchmarkReceipt` with timestamp, commit hash, machine fingerprint string, result list, and optional note.
- `BenchmarkResult` timing fields.
- Receipt JSON serialization/deserialization.
- Receipt hash and receipt diff helpers.
- `MachineFingerprint` derived from available host/user/arch/OS/CPU-count/machine-id inputs.
## Claim boundary
Safe to claim today:
- `quant-eval` provides deterministic Rust benchmark scaffolds and fixture metrics.
- `quant-eval` can evaluate current HyperQuant Z1/A2 primitive behavior.
- `quant-eval` emits typed metrics and benchmark receipt structures.
- `quant-eval` has local tests, clippy, and publish dry-run receipts for this release.
Not safe to claim today:
- real codec admissibility across production workloads;
- actual compression-ratio measurements for all codecs;
- model-quality preservation;
- superiority of any codec;
- production readiness;
- integrated policy enforcement for `poly-kv`, `fib-quant`, `turbo-quant`, `semantic-memory`, or `quant-governor`;
- `quant_codec_core::EvalReport` emission.
Those are reasonable next targets, but they need implementation evidence before becoming public claims.
## Verification
Release gate for v0.1.1:
```bash
cargo fmt -p quant-eval
cargo test -p quant-eval -- --nocapture
cargo test -p hyperquant -- --nocapture
cargo check -p quant-eval --all-targets
cargo clippy -p quant-eval --all-targets -- -D warnings
cargo publish -p quant-eval --dry-run --allow-dirty
```
Expected current test surface:
- 21 unit tests in `quant-eval` library modules.
- 4 HyperQuant integration tests.
- 5 general integration tests.
- 5 RAG fixture tests.
- 35 `quant-eval` tests total.
- 18 `hyperquant` tests for the dependency surface.
## Development
Run focused HyperQuant evaluation tests:
```bash
cargo test -p quant-eval hyperquant_eval -- --nocapture
```
Run all quant-eval tests:
```bash
cargo test -p quant-eval -- --nocapture
```
Run lint gate:
```bash
cargo clippy -p quant-eval --all-targets -- -D warnings
```
## Integration path
Recommended adoption order:
```text
quant-eval fixture metrics
-> quant-codec-core adapter reports
-> quant-governor policy/admissibility
-> turbo-quant / fib-quant comparative benchmarks
-> poly-kv or semantic-memory only with exact fallback and disclosure
```
`quant-eval` should remain evidence infrastructure. Policy decisions belong in governor/runtime crates.
## Dependencies
Runtime dependencies:
- `serde`
- `serde_json`
- `thiserror`
- `chrono`
- `sha2`
- `blake3`
- `hyperquant`
Dev dependency:
- `tempfile`
The crate currently contains no platform-specific code, FFI, async runtime dependency, CUDA, or HuggingFace integration.
## Roadmap
Near-term:
1. Add a codec evaluation trait or adapter layer so harnesses can call real encode/decode implementations.
2. Replace simulated compression paths with actual compressed/decompressed vector comparisons.
3. Add encoded-byte accounting and compression-ratio reports.
4. Emit or convert into `quant-codec-core` report shapes when that boundary is ready.
5. Add cross-crate integration tests for `hyperquant`, `fib-quant`, and `turbo-quant` adapters.
Medium-term:
1. Add real corpus fixtures for semantic-memory embeddings.
2. Add before/after receipt diffs for codec promotion reviews.
3. Add admissibility gates that can be consumed by `quant-governor`.
4. Add visual report export for benchmark receipts.
## License
MIT. See `LICENSE-MIT` for details.