quant-eval

quant-eval is a Rust crate for evidence-first compression and retrieval evaluation. It provides deterministic benchmark scaffolds, typed result shapes, benchmark receipts, RAG fixture metrics, and HyperQuant primitive evaluation before any codec is promoted into a governor or runtime path.

Current status: prototype-to-evidence benchmark substrate. It contains real metric code and deterministic fixtures, but some harnesses still use synthetic data or simulated compression paths. It should not be described as proving workload-level codec quality, production readiness, or model performance until real codec adapters and corpus receipts exist.

What this gives you

quant-eval gives compression and retrieval crates a place to produce evidence before integration:

• Compression benchmark scaffolding — deterministic synthetic vector corpus, exact nearest-neighbor baseline, recall@K, MRR, and overlap-derived similarity summaries.
• Semantic-memory search scaffolding — synthetic index/query generation, precision@K, recall@K, NDCG@K, MAP, and degradation-ratio calculations.
• Admissibility harness — profile-oriented checks over deterministic standard vectors.
• Benchmark receipts — timestamped receipt structures with machine fingerprint, result list, JSON serialization, hashes, and diffs.
• RAG fixture metrics — local recall@K, NDCG@K, and exact-rerank recovery over caller-supplied query/retrieval fixtures.
• HyperQuant primitive evaluation — deterministic Z1/A2 evaluation through the published hyperquant crate, with mean/max MSE, estimated bytes, rejected-vector counts, receipt counts, and explicit claim boundaries.
• Conservative public surface — measurement APIs first; no silent production claims.

Evidence pipeline

fixtures / synthetic corpora
        ↓
codec or retrieval harness
        ↓
metrics: MSE, recall@K, MRR, NDCG, MAP, recovery
        ↓
benchmark receipts + diffs
        ↓
policy/admission decisions in downstream crates

The crate is intentionally upstream of runtime policy. It measures and records; it does not decide that a codec is admissible for a truth-bearing system.

Installation

[dependencies]
quant-eval = "0.1.1"

From the RecursiveIntell Libraries workspace:

[dependencies]
quant-eval = { path = "../quant-eval" }

Quick start: compression benchmark scaffold

use quant_eval::{CompressionBenchmark, CompressionBenchmarkConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let benchmark = CompressionBenchmark::with_config(CompressionBenchmarkConfig {
        dim: 64,
        db_size: 100,
        queries: 10,
        seed: 42,
        top_k: 5,
        iterations: 10,
    });

    let report = benchmark.run()?;
    println!("recall@{} = {}", report.top_k, report.recall_at_k);
    println!("mrr = {}", report.mrr);
    Ok(())
}

Quick start: HyperQuant primitive evaluation

use quant_eval::{run_hyperquant_eval, HyperQuantEvalConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let result = run_hyperquant_eval(&HyperQuantEvalConfig {
        dim: 16,
        vectors: 64,
        seed: 42,
        scale: 8.0,
    })?;

    for profile in &result.profiles {
        println!(
            "{:?}: mean_mse={} max_mse={} receipts={}",
            profile.kind,
            profile.mean_mse,
            profile.max_mse,
            profile.receipt_count
        );
    }

    println!("claim boundary: {}", result.claim_boundary);
    Ok(())
}

Public API

The crate re-exports:

• AdmissibilityTest
• CodecProfile
• CompressionBenchmark
• CompressionBenchmarkConfig
• SemanticMemoryBenchmark
• SemanticMemoryConfig
• QuantEvalError
• MachineFingerprint
• BenchmarkReceipt
• BenchmarkResult
• ReceiptDiff
• evaluate_rag_fixture
• RagEvalResult
• RagQueryFixture
• RagRetrievedDoc
• run_hyperquant_eval
• HyperQuantEvalConfig
• HyperQuantEvalResult
• HyperQuantProfileEval

Implemented modules

Admissibility harness

File: src/benchmarks/admissibility.rs

Implemented:

• CodecProfile presets: fast, balanced, and high_compression.
• AdmissibilityTest over caller-provided TestSetEntry values.
• Standard synthetic test vectors for zero, unit, and deterministic pseudo-random vectors.
• Summary counts per profile.

Important limitation:

• This harness still simulates codec behavior from should_succeed and profile quality targets. It does not yet call a shared quant-codec-core trait.

Compression benchmark scaffold

File: src/benchmarks/compression.rs

Implemented:

• Deterministic synthetic corpus and query generation.
• Raw nearest-neighbor computation with cosine similarity.
• Recall@K and MRR calculations over exact-vs-estimated result sets.
• Similarity-style summary statistics over top-K overlap.

Important limitations:

• Compression is currently simulated by returning exact result sets.
• It does not yet measure real encoded byte size, compression ratio, per-block ratios, wire formats, or codec theoretical ratios.
• The reported cosine-similarity statistics are derived from top-K overlap, not raw-vs-decoded vector cosine.

Semantic-memory benchmark scaffold

File: src/benchmarks/semantic.rs

Implemented:

• Deterministic synthetic index and query generation.
• Raw search baseline using cosine similarity.
• Synthetic relevance judgments from raw top-K results.
• Precision@K, Recall@K, NDCG@K, MAP, and degradation-ratio calculations.

Important limitation:

• Compressed search currently delegates to raw search, so degradation is simulated/minimal by construction. It is not evidence of real codec preservation quality.

RAG fixture harness

File: src/rag.rs

Implemented:

• Query fixtures with explicit relevant document IDs.
• Retrieved document list with scores.
• Recall@K.
• NDCG@K.
• Exact-rerank recovery for top-ranked relevant result.
• Duplicate retrieved-doc suppression.

HyperQuant primitive harness

File: src/hyperquant_eval.rs

Implemented:

• HyperQuantEvalConfig
• HyperQuantProfileEval
• HyperQuantEvalResult
• run_hyperquant_eval
• deterministic synthetic fixture generation;
• triangular A2 fixture where A2 should match or beat Z1;
• Z1/A2 metrics through the published hyperquant crate;
• conservative claim-boundary string on every result.

Important limitation:

• This is primitive-level evidence only. It is not HyperQuant paper parity, model-quality evidence, or production admissibility.

Benchmark receipts

Files: src/receipt.rs, src/fingerprint.rs

Implemented:

• BenchmarkReceipt with timestamp, commit hash, machine fingerprint string, result list, and optional note.
• BenchmarkResult timing fields.
• Receipt JSON serialization/deserialization.
• Receipt hash and receipt diff helpers.
• MachineFingerprint derived from available host/user/arch/OS/CPU-count/machine-id inputs.

Claim boundary

Safe to claim today:

• quant-eval provides deterministic Rust benchmark scaffolds and fixture metrics.
• quant-eval can evaluate current HyperQuant Z1/A2 primitive behavior.
• quant-eval emits typed metrics and benchmark receipt structures.
• quant-eval has local tests, clippy, and publish dry-run receipts for this release.

Not safe to claim today:

• real codec admissibility across production workloads;
• actual compression-ratio measurements for all codecs;
• model-quality preservation;
• superiority of any codec;
• production readiness;
• integrated policy enforcement for poly-kv, fib-quant, turbo-quant, semantic-memory, or quant-governor;
• quant_codec_core::EvalReport emission.

Those are reasonable next targets, but they need implementation evidence before becoming public claims.

Verification

Release gate for v0.1.1:

cargo fmt -p quant-eval
cargo test -p quant-eval -- --nocapture
cargo test -p hyperquant -- --nocapture
cargo check -p quant-eval --all-targets
cargo clippy -p quant-eval --all-targets -- -D warnings
cargo publish -p quant-eval --dry-run --allow-dirty

Expected current test surface:

• 21 unit tests in quant-eval library modules.
• 4 HyperQuant integration tests.
• 5 general integration tests.
• 5 RAG fixture tests.
• 35 quant-eval tests total.
• 18 hyperquant tests for the dependency surface.

Development

Run focused HyperQuant evaluation tests:

cargo test -p quant-eval hyperquant_eval -- --nocapture

Run all quant-eval tests:

cargo test -p quant-eval -- --nocapture

Run lint gate:

cargo clippy -p quant-eval --all-targets -- -D warnings

Integration path

Recommended adoption order:

quant-eval fixture metrics
  -> quant-codec-core adapter reports
  -> quant-governor policy/admissibility
  -> turbo-quant / fib-quant comparative benchmarks
  -> poly-kv or semantic-memory only with exact fallback and disclosure

quant-eval should remain evidence infrastructure. Policy decisions belong in governor/runtime crates.

Dependencies

Runtime dependencies:

• serde
• serde_json
• thiserror
• chrono
• sha2
• blake3
• hyperquant

Dev dependency:

• tempfile

The crate currently contains no platform-specific code, FFI, async runtime dependency, CUDA, or HuggingFace integration.

Roadmap

Near-term:

1. Add a codec evaluation trait or adapter layer so harnesses can call real encode/decode implementations.
2. Replace simulated compression paths with actual compressed/decompressed vector comparisons.
3. Add encoded-byte accounting and compression-ratio reports.
4. Emit or convert into quant-codec-core report shapes when that boundary is ready.
5. Add cross-crate integration tests for hyperquant, fib-quant, and turbo-quant adapters.

Medium-term:

1. Add real corpus fixtures for semantic-memory embeddings.
2. Add before/after receipt diffs for codec promotion reviews.
3. Add admissibility gates that can be consumed by quant-governor.
4. Add visual report export for benchmark receipts.

License

MIT. See LICENSE-MIT for details.