Wickra Benchmark

A reproducible, golden-verified benchmark suite for quant backtests. Take a curated (strategy, dataset, expected report) case, recompute it, and confirm it reproduces byte-for-byte — the same result in ten languages, or the build goes red.

Part of the Wickra ecosystem. Built on the same deterministic backtest engine and ten-language binding surface as wickra-backtest, wickra-proof, wickra-verify and the rest.

wickra-benchmark is the "ImageNet for trading-strategy reproducibility": not a new backtest engine, but the curated, hash-pinned suite you check an engine against. Each case pins a wickra-backtest StrategySpec, a deterministic candle dataset, the expected BacktestReport, and the blake3 hash of that report's canonical form. Running a case recomputes the report with the pinned engine and returns two independent booleans: passed (the recomputed report is byte-exact equal to the frozen expectation) and hash_match (its canonical hash equals the frozen expected_hash).

It is a free reproducibility harness, not a hosted service: a CLI plus ten language bindings over one small deterministic core. Nothing you run ever leaves your machine.

# Recompute every curated case and confirm it still reproduces, byte for byte.
wickra-benchmark run-suite --suite cases/suite.json --data-root datasets

id                      passed  hash_match  hash
atr-breakout-filter-01  true    true        9d79e79f8323
bollinger-breakout-01   true    true        5d25f870f8a9
breakout-channel-01     true    true        2b1ef11f989c
buy-and-hold-01         true    true        c1f6820a3de2
donchian-long-range-01  true    true        66c89cd7e025
ema-trend-follow-01     true    true        97a97c31a400
ema-whipsaw-01          true    true        67f1c7ffed09
macd-trend-01           true    true        21ed980a9c53
roc-momentum-01         true    true        ccc3cb4f26ca
rsi-mean-reversion-01   true    true        664558550a58
rsi-short-downtrend-01  true    true        863447c37345
sma-crossover-01        true    true        8f5e84ff8862
sma-long-trend-01       true    true        ff7647521e22
sma-timed-exit-01       true    true        f3c209e5debc
wma-crossover-01        true    true        365525935325
15/15 passed

Exit code 0 means every case reproduced, 1 that at least one did not — so a drifting engine turns a build red rather than going unnoticed.

Determinism is the product

• Recompute, never trust — a case passes only when a fresh run reproduces the frozen report; a stale engine, a changed default, a numerical drift all turn the case red.
• Two independent checks — passed (byte-exact report equality) and hash_match (canonical-hash equality) are reported separately, so a case whose expected and expected_hash disagree is caught, not masked.
• Canonical hashes — every report is hashed under the same canonicalization wickra-proof uses (keys sorted, no whitespace, floats quantized to 1e-8, no NaN/±inf), so the hash is identical in every language.
• Byte-identical across languages and runners — a SuiteReport is re-sorted by case id and is byte-for-byte the same in all ten bindings and between the parallel (rayon) and sequential (WASM) runners; the cross-language golden tests assert it.

Status

Pre-release — functionally complete, CI-verified, not yet published. The core, the CLI, all ten language bindings, the curated case registry, the golden corpus, the property + fuzz suites, the benchmarks and one runnable example per language are built and green across Linux, macOS and Windows. Packages are not yet on the registries. Track progress in ROADMAP.md.

Documentation

• benchmark.wickra.org — the site: what the suite is, the case registry, and the per-language quickstarts.
• ARCHITECTURE.md — how the pieces fit together.
• docs/CASES.md — the BenchmarkCase/Suite schema and how to contribute a case.
• docs/DATASETS.md — how the datasets are curated, generated and hash-pinned.
• docs/HASHING.md — the canonicalization + blake3 contract shared with wickra-proof.
• docs/REPRODUCING.md — recompute the suite in every language.
• docs/Cookbook.md — recipes, including "gate engine reproducibility in CI".

Quickstart

# Recompute a whole suite against its datasets and confirm every case reproduces.
cargo run -p wickra-benchmark -- run-suite \
  --suite cases/suite.json --data-root datasets

# Or a single case, as JSON.
cargo run -p wickra-benchmark -- run-case \
  --case cases/sma-crossover-01.json --data-root datasets --format json

# Exit 0 = every case reproduced, 1 = at least one failed (CI-friendly).

The bundled suite self-passes — run-suite reports passed 5, failed 0 — so a red build means the engine, not the suite, changed.

Case and suite format

A case is one curated reproducibility unit:

• id — a stable, unique, kebab-case key (the sort and tie key).
• strategy — the embedded wickra-backtest StrategySpec (indicators, entry/exit rules, sizing, costs).
• dataset_ref — the dataset CSV the case runs on, under the data root.
• expected + expected_hash — the frozen BacktestReport and its canonical blake3 hash. Generated, never hand-written (see cases/README.md for the bless flow); running the case recomputes and checks against both.

cases/suite.json bundles the cases into a named, id-unique suite. Full schema in docs/CASES.md.

Reproduce the suite in any language

The core is a JSON-over-C-ABI data API (Benchmark::command_json) exposed natively in Rust, Python, Node.js and WASM, and over the C ABI hub in C, C++, C#, Go, Java and R. Every binding drives the same run_case / run_suite / list_cases / version commands and returns the core's canonical response verbatim; the golden/ fixtures pin one blessed response per command and the cross-language golden tests assert byte-for-byte equality — the same passed, the same hash_match, the same blake3 hashes, everywhere. One runnable example per language lives under examples/; per-binding quickstarts are in each bindings/<lang>/README.md.

Contributing a case

A good case is small, deterministic and non-degenerate (it actually trades). Add or reuse a dataset under datasets/, write the draft with a fresh id, description, strategy and dataset_ref, then bless it — let the engine fill in expected and expected_hash — and add it to cases/suite.json. The full flow, including the never-edit-by-hand rule, is in cases/README.md and docs/CASES.md.

Project layout

crates/benchmark-core       the library: case + suite + runner + hash + command
crates/benchmark-cli        reference CLI, binary `wickra-benchmark`
crates/benchmark-bench      Criterion benchmarks
bindings/{c,python,node,wasm,go,csharp,java,r}   ten-language surface
datasets/                   deterministic candle CSVs + blake3 MANIFEST.json
cases/                      curated BenchmarkCases + suite.json
golden/                     command envelopes -> byte-exact canonical responses
examples/                   runnable per-language demos
fuzz/                       cargo-fuzz targets (case/suite parse, run_case, command_json)

Building everything from source

# Rust core + tests + lints
cargo test --workspace --all-features
cargo test --workspace --no-default-features   # the sequential runner
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo bench -p benchmark-bench

# Python binding (requires a Rust toolchain + maturin)
cd bindings/python && maturin develop --release && pytest

# Node binding (requires @napi-rs/cli)
cd bindings/node && npm install && npm run build && npm test

# WASM binding (requires wasm-pack)
cd bindings/wasm && wasm-pack build --target nodejs --out-dir pkg && node --test tests/*.test.js

# C ABI (cdylib + staticlib + generated header)
cargo build -p wickra-benchmark-c --release

# C# binding (requires the .NET 8 SDK; links the C ABI above)
dotnet test bindings/csharp/WickraBenchmark.Tests/WickraBenchmark.Tests.csproj

# Go binding (requires a C compiler for cgo; links the C ABI above)
cd bindings/go && go test ./...

# Java binding (requires JDK 22+ and Maven; links the C ABI above)
mvn -f bindings/java test

# R binding (requires a C toolchain / Rtools; links the C ABI above)
R CMD INSTALL bindings/r

The Go, Java and R bindings load the C ABI shared library at run time; put target/release (or target/debug) on the library path. Fuzzing requires a nightly toolchain — see fuzz/; the same never-panic invariants are covered on stable by the property tests.

Re-blessing after an engine bump is one command, and it writes every copy of the corpus — cases/, cases/suite.json, golden/ and examples/data/ — from the same value:

WICKRA_BLESS=1 cargo test -p wickra-benchmark-core --test golden
python scripts/check_corpus_sync.py

Testing

The commands are in Building everything from source.

• wickra-benchmark-core — 14 unit tests over case and suite validation, the CSV candle loader, canonicalization and hashing. Plus five integration suites: 12 conformance tests (determinism, ordering, the pass/fail tally), 11 indicator-conformance tests (every frozen expected_hash is downstream of the indicator arithmetic, so a moved hash has two possible causes; these separate them — one per family the cases name, plus the guard that fails when a case names a family nothing pins), 4 property tests, the path-vs-inline equivalence test (run_suite and run_suite_inline must agree on the same data), and the golden runner.
• benchmark-cli — 4 tests over argument parsing.
• bindings/c — 6 Rust tests driving the ABI itself, including its error paths, so a null or malformed command is proven to be reported rather than dereferenced.
• bindings/python — 10 pytest cases: smoke, golden parity, surface completeness, batch equivalence. bindings/node — 12 node --test cases, same shape. bindings/wasm — 7 against the built package.
• bindings/csharp — 7 xUnit cases. bindings/java — 7 JUnit cases. bindings/go — 7 go test cases. bindings/r — 3 script suites.
• fuzz/ — four targets over the untrusted-input surface: the case parser, the suite parser, run_case, and the command_json envelope.

On top of those, all ten languages replay the shared, language-neutral golden corpus in golden/ — eighteen command envelopes — and assert their response is byte-identical to the committed one.

What "parity" means here, precisely. The responses are compared byte for byte, not to a tolerance. That is possible because every binding returns the core's canonical string verbatim — the arithmetic is not reimplemented anywhere — and because a report is canonicalized to sorted keys and round-trippable floats before it is hashed. It is not free, though: a case may name any indicator the engine offers, and some of those call a transcendental from the platform's math library (ln, exp and friends). No mainstream libm rounds those correctly, and implementations differ in the last bit. A case built on one would have to compare to a relative tolerance instead. None currently does, and that is a property of the corpus worth keeping deliberately rather than by accident.

Requirements

Rust 1.86 (workspace) / 1.88 (Node binding). Per-binding toolchains: Python 3.9+, Node.js 22+, .NET 8, JDK 22+, Go 1.23+, R 4.1+, and a C11/C++14 compiler with CMake for the C example.

Benchmarks

Criterion benchmarks for run_suite at 10/100/1000 cases (parallel vs sequential) live in crates/benchmark-bench; numbers and methodology are in BENCHMARKS.md.

Ecosystem

Part of the Wickra family — each one a data-driven core with a CLI and the same ten-language binding surface:

• wickra — main library (Rust core + Python / Node.js / WASM bindings + a C ABI for C / C++ / C# / Go / Java / R)
• wickra-playground — a polyglot strategy playground: one StrategySpec live side by side in Python, Rust, JS and Go, entirely in the browser
• wickra-backtest — the deterministic engine every case here is recomputed with
• wickra-exchange — unified market-data + execution across ten crypto exchanges
• wickra-terminal — the trading terminal: a TUI and a browser renderer over the stack
• wickra-screener — parallel multi-symbol screening over 514 streaming indicators
• wickra-xray — market-microstructure explorer: footprint, order-book heatmap, liquidation map, funding/OI divergence
• wickra-radar — perp-universe alert radar: OI delta, funding flip, book imbalance, liquidation clusters, OI/price divergence
• wickra-copilot — local market copilot grounded in real order-book, liquidation and funding microstructure
• wickra-shazam — match an asset's current microstructure fingerprint against its entire history
• wickra-benchmark — this repository: the curated, hash-pinned suite you check an engine against
• wickra-strategy-ci — Jest for trading strategies: golden-pin the report, catch regressions in CI, property-test against fuzzed data
• wickra-verify — confirm or refute a claimed backtest report against its strategy and data, in ten languages
• wickra-proof — Proof-of-Backtest: deterministic (spec, data) → report + blake3 hash, recomputable byte-for-byte in ten languages
• wickra-zk — prove a backtest zero-knowledge — on-chain-verifiable performance without revealing the data or the strategy
• wickra-impact — the backtester that knows you would have moved the market: agent-based fills on the real historical L2 order book
• wickra-darwin — evolutionary strategy search at millions of backtests per second, mutating and crossing JSON specs across the 514-indicator space
• wickra-gym — a Gymnasium-compatible, microstructure-aware backtest environment with O(1) steps for deterministic RL rollouts
• wickra-feature-store — OHLCV and microstructure streams into ML-ready feature matrices over 514 O(1) streaming indicators
• wickra-genome — a vector database of the whole market: every asset a 514-dim live vector, for similarity search, clustering and anomaly detection
• wickra-timemachine — scrub the whole market like a video — every symbol, full order book, rewound to any moment via deterministic re-fold
• wickra-synth — deterministic synthetic market microstructure: OHLCV, order book, trades and funding from a single seed
• wickra-compile — compile a strategy spec into a standalone deployable: a WASM module, a self-contained binary, or a no_std artifact
• wickra-embed — allocation-free, no_std streaming indicators for bare-metal and HFT, byte-for-byte identical to the core
• wickra-pico — the O(1) indicator core running bare-metal on a $5 Raspberry Pi Pico — the LED blinks on the EMA cross

The indicator core underneath documents itself at docs.wickra.org.

Contributing

See CONTRIBUTING.md and the Code of Conduct. Every change runs the full CI matrix (all ten languages × three OSes) plus CodeQL, Scorecard, zizmor and a dataset-manifest integrity check.

Security

Report vulnerabilities per SECURITY.md. The threat model is in THREAT_MODEL.md.

License

Dual-licensed under either MIT or Apache-2.0, at your option.

Disclaimer

wickra-benchmark is research and engineering tooling, not financial advice. A passing case attests only that a report is the deterministic result of a given strategy over given data — it makes no claim about the quality, profitability or future performance of any strategy, nor about whether the data is representative of any market. Trading carries risk; you are responsible for your own decisions. wickra-benchmark is free software you run yourself: no hosted service, no data collection, no warranty.