# Benchmark Guide for linch
This guide explains how to use the `bench_runner.sh` script to manage benchmarks, baselines, and comparisons.
## Prerequisites
1. **Install critcmp** (for baseline comparisons):
```bash
cargo install critcmp
```
2. **Ensure you're in a git repository** (the script requires git for tagging)
## Quick Start
### Running Benchmarks
```bash
# Basic benchmark run with automatic commit and tagging
./bench_runner.sh run "my-optimization"
# Run with custom commit message
./bench_runner.sh run "async-improvements" --message "Improved async channel performance"
# Run without committing changes
./bench_runner.sh run "test-run" --no-commit
# Run without git tagging
./bench_runner.sh run "experimental" --no-tag
```
### Listing Baselines
```bash
# List all available baselines
./bench_runner.sh list
```
### Comparing Baselines
```bash
# Compare two baselines
./bench_runner.sh compare baseline1 baseline2
# Compare three baselines
./bench_runner.sh compare old-version current-version new-optimization
# Compare with latest baseline
./bench_runner.sh compare bench_20240831_105800_optimization bench_20240831_110200_improvement
```
## How It Works
### 1. Benchmark Run Process
When you run `./bench_runner.sh run "name"`, the script:
1. **Commits Changes**: Automatically commits any uncommitted changes with a descriptive message
2. **Creates Git Tag**: Creates a git tag with format `bench_YYYYMMDD_HHMMSS_name`
3. **Runs Benchmarks**: Executes `cargo bench --bench bench_channel`
4. **Saves Baseline**: Saves results with the same name as the git tag
### 2. Naming Convention
- **Git tags**: `bench_20240831_105800_optimization`
- **Baselines**: Same as git tag name
- **Format**: `bench_[timestamp]_[user-provided-name]`
### 3. Baseline Storage
Baselines are stored in `target/criterion/` and can be managed with critcmp.
## Available Benchmarks
The script runs all benchmarks in `benches/bench_channel.rs`, including:
- **sync_send_async_recv**: Mixed synchronous/asynchronous patterns
- **async_send_sync_recv**: Mixed asynchronous/synchronous patterns
- **congestion_***: Performance under different congestion levels
- **multi_producer_consumer_***: Multiple producers and consumers
- **realistic_workload_***: Real-world scenarios with CPU work
- **receive_workload_***: Lightweight senders with heavy receivers
Each benchmark tests both `linch` and `schannel` implementations.
## Example Workflow
```bash
# 1. Make some optimizations to your code
# ... edit source files ...
# 2. Run benchmark and save baseline
./bench_runner.sh run "lock-free-optimization" -m "Implemented lock-free queue"
# 3. Make more changes
# ... edit more files ...
# 4. Run another benchmark
./bench_runner.sh run "memory-optimization" -m "Reduced memory allocations"
# 5. List all baselines
./bench_runner.sh list
# 6. Compare the two optimizations
./bench_runner.sh compare bench_20240831_105800_lock-free-optimization bench_20240831_110200_memory-optimization
# 7. Compare with an older baseline
./bench_runner.sh compare old-baseline bench_20240831_105800_lock-free-optimization bench_20240831_110200_memory-optimization
```
## Tips
1. **Use descriptive names**: `./bench_runner.sh run "async-send-optimization"` is better than `./bench_runner.sh run "test"`
2. **Regular benchmarking**: Run benchmarks before and after significant changes to track performance impact
3. **Baseline cleanup**: Old baselines can be removed from `target/criterion/` if needed
4. **Git integration**: The automatic tagging helps correlate performance results with code changes
5. **CI/CD integration**: The script can be used in automated pipelines with `--no-commit` flag
## Troubleshooting
### "critcmp not found"
```bash
cargo install critcmp
```
### "No baselines found"
Run at least one benchmark first:
```bash
./bench_runner.sh run "initial-baseline"
```
### "Not a git repository"
Initialize git or run from within a git repository:
```bash
git init
```
### Comparison fails
Check baseline names with:
```bash
./bench_runner.sh list
```