# CacheKit Benchmarks
This directory contains the benchmark suite for CacheKit cache policies.
## Benchmark Organization
### Criterion Benchmarks (Performance Measurement)
**`workloads.rs`** - Policy comparison across standard workloads
- Hit rate benchmarks for all policies × all workloads
- Scan resistance tests
- Adaptation speed tests
- Comprehensive benchmarks with latency/throughput
- Run: `cargo bench --bench workloads`
**`ops.rs`** - Micro-operation benchmarks
- Individual get/insert operation latency
- Measures raw policy overhead
- Run: `cargo bench --bench ops`
**`comparison.rs`** - External crate comparison
- Compares CacheKit policies against `lru` and `quick_cache` crates
- Run: `cargo bench --bench comparison`
**`policy/*.rs`** - Policy-specific benchmarks
- Unique operations for each policy
- Run: `cargo bench --bench policy_lru`, etc.
### Report Generators
**`reports.rs`** - Human-readable console reports
- Prints comparison tables for policies
- No Criterion overhead, just raw measurements
- Run: `cargo bench --bench reports -- <report_type>`
- Available reports:
- `hit_rate` - Hit rate comparison table
- `extended` - Extended workload comparison
- `scan` - Scan resistance metrics
- `adaptation` - Adaptation speed metrics
- `detailed` - Single benchmark deep dive
- `memory` - Memory overhead comparison
- `comprehensive` - Full policy tables
- `all` - Run all reports
**`runner.rs`** - JSON artifact generator
- Produces structured JSON results for automation
- Captures full environment metadata (git, rustc, CPU)
- Output: `target/benchmarks/<timestamp>/results.json`
- Run: `cargo bench --bench runner`
## Common Infrastructure
**`common/`** - Shared benchmark utilities
- `registry.rs` - Central registry of policies and workloads
- `metrics.rs` - Benchmark measurement utilities
- `workload.rs` - Workload generators
- `json_results.rs` - JSON output schema
## Quick Start
### Run hit rate comparison
```bash
cargo bench --bench reports -- hit_rate
```
### Generate JSON artifact
```bash
cargo bench --bench runner
# Output: target/benchmarks/<timestamp>/results.json
```
### Run full Criterion benchmarks
```bash
cargo bench --bench workloads
```
### View Criterion HTML reports
```bash
# After running criterion benchmarks
open target/criterion/report/index.html
```
## Benchmark Configuration
Default parameters (defined in each benchmark file):
- **Capacity**: 4,096 entries
- **Universe**: 16,384 possible keys
- **Operations**: 200,000 per benchmark
- **Seed**: 42 (for reproducibility)
## Policy Registry
All benchmarks use the central policy registry (`common/registry.rs`).
Current policies:
- **LRU** - Least Recently Used
- **LRU-K** - LRU with K-distance tracking
- **LFU** - Least Frequently Used
- **Heap-LFU** - LFU with heap-based eviction
- **Clock** - Clock/Second-Chance algorithm
- **S3-FIFO** - Simple Scalable FIFO variant
- **2Q** - Two-Queue algorithm
To add a new policy:
1. Add entry to `for_each_policy!` macro in `common/registry.rs`
2. All benchmarks automatically include it
## Workload Registry
Current standard workloads:
- `uniform` - Uniform random access
- `hotset_90_10` - 90% of accesses to 10% of keys
- `scan` - Sequential access
- `zipfian_1.0` - Zipfian distribution (α=1.0)
- `scrambled_zipf` - Zipfian with hashed keys
- `latest` - Recent keys are more popular
- `scan_resistance` - Mixed point queries and sequential scans
- `flash_crowd` - Sudden traffic spikes
Extended workloads include: `zipfian_0.8`, `shifting_hotspot`, `exponential`, `pareto`, `correlated`, `loop_small`, `working_set_churn`, `bursty`, `mixture`.
To add a new workload:
1. Add entry to `STANDARD_WORKLOADS` or `EXTENDED_WORKLOADS` in `common/registry.rs`
2. All benchmarks automatically test it
## Output Formats
### Criterion
- HTML reports in `target/criterion/`
- Statistical analysis with confidence intervals
- Automatic regression detection
### Console Reports
- Plain text tables
- Hit rate percentages
- Throughput (ops/sec)
- Latency distribution (p50, p95, p99)
### JSON Artifacts
- Structured data in `target/benchmarks/<timestamp>/results.json`
- Includes metadata (git commit, rustc version, CPU model)
- Schema version 1.0.0
- ~60KB per run, 91 results
## Best Practices
### For Performance Testing
1. Use release builds: `cargo bench`
2. Close other applications to reduce noise
3. Run multiple times and check consistency
4. Use Criterion for statistical rigor
### For Development
1. Use `reports.rs` for quick comparisons
2. Check `hit_rate` first to catch regressions
3. Use `runner.rs` for CI/CD integration
### For Release
1. Run full benchmark suite: `cargo bench`
2. Generate JSON artifact: `cargo bench --bench runner`
3. Compare with previous releases
4. Document significant changes in CHANGELOG
## Troubleshooting
### Benchmarks take too long
- Reduce `OPS` constant in benchmark file
- Run specific workloads: `cargo bench --bench workloads -- uniform`
- Use `reports.rs` instead of Criterion
### Results vary between runs
- Ensure system is idle during benchmarking
- Disable CPU frequency scaling if possible
- Run on dedicated hardware for consistent results
- Use longer warmup periods
### Adding new policies fails
- Check `for_each_policy!` macro syntax
- Verify policy implements `Cache<K, V>` trait
- Ensure type parameters match (some policies use `Arc<V>`, others use `V`)
## See Also
- [Benchmark Quick Start](../docs/benchmarks/QUICKSTART.md)
- [Benchmark Docs](../docs/benchmarks/README.md)
- [Workload Documentation](../docs/benchmarks/workloads.md)