---
description: Measure solver performance and compare output quality between branches.
icon: gauge-max
layout:
width: default
title:
visible: true
description:
visible: true
tableOfContents:
visible: true
outline:
visible: true
pagination:
visible: true
metadata:
visible: true
tags:
visible: true
---
# Benchmarking
Fynd ships with a benchmark tool (`fynd-benchmark`) for load-testing a solver and comparing output quality between two solver instances. Both features live in `tools/benchmark/` and run against live solver instances.
{% hint style="info" %}
**Prerequisite:** You need a running solver before using any benchmark command. See [quickstart](../get-started/quickstart/ "mention") for setup instructions.
{% endhint %}
## Load Testing
Measures latency (round-trip, solve time, overhead) and throughput for a single solver instance.
```bash
cargo run -p fynd-benchmark --release -- load [OPTIONS]
```
{% hint style="warning" %}
Always build with `--release`. Debug builds produce misleading latency numbers.
{% endhint %}
### Options
| `--solver-url` | `http://localhost:3000` | Solver URL to benchmark against |
| `-n` | `1` | Number of requests to send |
| `-m` | `sequential` | Parallelization mode |
| `--requests-file` | _(none)_ | Path to JSON file with request templates |
| `--output-file` | _(none)_ | Output file for JSON results |
### Parallelization Modes
Control how requests are dispatched with the `-m` flag:
* **`sequential`** — Send one request at a time, wait for each response before sending the next. Good for measuring single-request latency.
* **`fixed:N`** — Maintain exactly N concurrent in-flight requests (e.g., `fixed:5`). Good for simulating sustained load.
* **`rate:Nms`** — Fire a new request every N milliseconds regardless of pending responses (e.g., `rate:100`). Good for testing behavior under a fixed request rate.
### Examples
```bash
# Measure single-request latency (10 sequential requests)
cargo run -p fynd-benchmark --release -- load -n 10
# Simulate 10 concurrent users sending 100 total requests
cargo run -p fynd-benchmark --release -- load -m fixed:10 -n 100
# Fire a request every 50ms using custom request templates
cargo run -p fynd-benchmark --release -- load \
-m rate:50 -n 100 \
--requests-file tools/benchmark/requests_set.json
# Export results to JSON for further analysis
cargo run -p fynd-benchmark --release -- load \
-m fixed:10 -n 1000 \
--output-file results.json
```
### Output
The tool prints real-time progress, summary statistics (min, max, mean, median, p95, p99, stddev), and ASCII histograms of timing distributions. Pass `--output-file` to export the full results as JSON.
***
## Comparing Two Solvers
Sends identical quote requests to two running solver instances and compares output quality: amount out (in bps), gas estimates, and route selection.
```bash
cargo run -p fynd-benchmark --release -- compare [OPTIONS]
```
### Setup
You need two Fynd instances running simultaneously — typically from different git branches. Since both share the same binary target directory and metrics port, use **git worktrees** to avoid conflicts.
#### 1. Create a worktree for the baseline
```bash
# From the main repo
git worktree add ../fynd-baseline main
```
#### 2. Start solver A (baseline) in the worktree
```bash
cd ../fynd-baseline
RUST_LOG=info cargo run --release -- serve \
--protocols uniswap_v2,uniswap_v3 \
--http-port 3000 \
--tycho-url <TYCHO_URL> \
--tycho-api-key <API_KEY>
```
#### 3. Start solver B (your branch) in the original repo
```bash
cd /path/to/fynd
RUST_LOG=info cargo run --release -- serve \
--protocols uniswap_v2,uniswap_v3 \
--http-port 3001 \
--tycho-url <TYCHO_URL> \
--tycho-api-key <API_KEY>
```
#### 4. Wait for both solvers to be healthy
```bash
curl http://localhost:3000/v1/health
curl http://localhost:3001/v1/health
```
Both should return `{"healthy": true, ...}` before running the comparison.
#### 5. Run the comparison
```bash
cargo run -p fynd-benchmark --release -- compare \
--url-a http://localhost:3000 \
--url-b http://localhost:3001 \
--label-a main \
--label-b my-branch \
-n 100
```
### Options
| `--url-a` | `http://localhost:3000` | Solver A (baseline) URL |
| `--url-b` | `http://localhost:3001` | Solver B (candidate) URL |
| `--label-a` | `main` | Label for solver A in output |
| `--label-b` | `branch` | Label for solver B in output |
| `-n` | `500` | Number of requests to send |
| `--requests-file` | _(none)_ | Path to JSON file with custom requests |
| `--output` | `comparison_results.json` | Path for full results JSON |
| `--timeout-ms` | `15000` | Per-request timeout in milliseconds |
| `--seed` | `42` | Random seed for reproducibility |
### Net-of-Gas Comparison
The compare tool uses the server-computed `amount_out_net_gas` field for net-of-gas output comparison. This value represents the output amount minus gas cost denominated in the output token, calculated by the solver. It works for all token pairs.
### Output
Prints a summary table to stdout showing win/loss counts and bps differences (both gross and net-of-gas). Writes detailed per-request results to the output JSON file. Positive bps diffs mean solver B returned more output than solver A.
***
## CPU Scaling
Measures how solver throughput (req/s) scales with worker thread count. The tool builds a solver in-process for each worker count, runs a load test, shuts down, and repeats.
```bash
cargo run -p fynd-benchmark --release -- scale [OPTIONS]
```
{% hint style="warning" %}
Requires a `worker_pools.toml` with exactly **one** pool defined.
{% endhint %}
### Options
| `--base-config` | `worker_pools.toml` | Single-pool TOML config |
| `--worker-counts` | _(required)_ | Comma-separated worker counts (e.g. `1,2,4,8,16`) |
| `--protocols` | _(required)_ | Comma-separated protocols for solver |
| `--tycho-url` | `localhost:4242` | Tycho WebSocket URL |
| `--tycho-api-key` | _(none)_ | Tycho API key |
| `--disable-tls` | `false` | Disable TLS for Tycho connection |
| `--rpc-url` | _(none)_ | Node RPC URL |
| `--chain` | `ethereum` | Chain name |
| `--http-port` | `3000` | Solver HTTP port |
| `-n` | `100` | Requests per iteration |
| `-m` | `fixed:8` | Parallelization mode |
| `--requests-file` | _(none)_ | Custom request templates |
| `--warmup-secs` | `30` | Seconds to wait after health before benchmarking |
| `--health-timeout-secs` | `300` | Max seconds to wait for solver health |
| `--output-file` | _(none)_ | JSON output file |
### Example
```bash
cargo run -p fynd-benchmark --release -- scale \
--base-config single_pool.toml \
--worker-counts 1,2,4,8,16 \
--protocols uniswap_v2,uniswap_v3 \
--tycho-url wss://tycho.example.com \
--tycho-api-key $TYCHO_API_KEY \
-n 200 \
-m fixed:8 \
--output-file scale_results.json
```
### Output
The tool prints a summary table showing throughput, latency, and per-worker efficiency at each worker count:
```
=== CPU Scaling Results ===
Pool: most_liquid_2_hops_fast (algorithm: most_liquid)
Requests per run: 200, Mode: fixed:8
1 | 8.50 | 95 | 142 | 8.50
2 | 16.20 | 50 | 98 | 8.10
4 | 30.10 | 28 | 65 | 7.53
8 | 48.90 | 18 | 52 | 6.11
16 | 62.30 | 15 | 48 | 3.89
```
Pass `--output-file` to export the full results as JSON for further analysis.
***
## Request Data
By default, the load test uses a single WETH→USDC swap and the compare tool samples from a built-in set of 50 real aggregator trades. Both commands accept `--requests-file` to supply custom requests.
### Downloading More Trades
For broader coverage, download a larger set of real aggregator trades (10k):
```bash
cargo run -p fynd-benchmark --release -- download-trades
```
Then use it with either command via `--requests-file aggregator_trades_10k.json`.
### Custom Request Format
The file should be a JSON array of quote request bodies:
```json
[
{
"orders": [{
"token_in": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
"token_out": "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
"amount": "1000000000000000000000",
"side": "sell",
"sender": "0x0000000000000000000000000000000000000001"
}]
}
]
```
See `tools/benchmark/requests_set.json` in the repository for a complete example.