<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://private-user-images.githubusercontent.com/1433964/551822927-b10bec0e-d196-4ebe-97ce-2fb9c720b8d3.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzE0NTc0NzksIm5iZiI6MTc3MTQ1NzE3OSwicGF0aCI6Ii8xNDMzOTY0LzU1MTgyMjkyNy1iMTBiZWMwZS1kMTk2LTRlYmUtOTdjZS0yZmI5YzcyMGI4ZDMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDIxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjAyMThUMjMyNjE5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YWQxOWVlY2NlYjI5Mjg4NDFmMzIyYzBmMTYyNzBlZWZhYzVmODk2ZDRlZmM0ODE1M2MzMTBmNWIyYjlmMDgyYyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.arY04-hBiWJRfUcweZTgpYdox9mR4y-6qxxu1bKy6SQ">
<source media="(prefers-color-scheme: light)" srcset="https://private-user-images.githubusercontent.com/1433964/551822940-e123170a-ef87-4330-964d-ba4d63e1e032.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzE0NTc0NzksIm5iZiI6MTc3MTQ1NzE3OSwicGF0aCI6Ii8xNDMzOTY0LzU1MTgyMjk0MC1lMTIzMTcwYS1lZjg3LTQzMzAtOTY0ZC1iYTRkNjNlMWUwMzIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDIxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjAyMThUMjMyNjE5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZmY3MDE4MjI2NGQ2MjI5YzJlOTQwNmRhNDRhZGFkNmJiM2U0Y2RkMzBmZTE1ZWRjY2FkZWUzYjE5MmVjZTA1MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.E8fiF79Rgi8B2rdpM6fMFMCABCclMH5E7jgeNsSeUWQ">
<img alt="Logo">
</picture>
This is a companion CLI tool for the [SCARF Benchmark](https://github.com/scarfbench/benchmark). It provides a commandline interface to list and test benchmarks, run agents, submit solutions, view and explore leaderboard among other useful tasks.
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Prerequisites](#prerequisites)
- [Build from Source](#build-from-source)
- [Usage](#usage)
- [Development](#development)
- [Development Dependencies](#development-dependencies)
- [Testing](#testing)
- [Building and Testing](#building-and-testing)
## Features
- List available benchmarks
- Test and validate benchmarks
- Run agents on benchmark problems
- Submit solutions (to be added)
- View and explore leaderboards (to be added)
## Installation
### Prerequisites
Before installing the SCARF CLI, ensure you have the following tools installed:
- **Docker** ([Installation Guide](https://docs.docker.com/get-docker/)) - Runs benchmarks in isolated environments
- **Make** - Builds and runs projects as specified in makefiles
- **Python** - If you want to install `scarf` with pip (optional)
### Clone the repository
```bash
git clone https://github.com/scarfbench/scarf.git
cd scarf
```
### Install with `pip`
You can install the SCARF CLI using `pip` for easier management:
```bash
pip install -U .
```
Note: Depending on your os, pip may be called `pip3`.
### Install with `cargo`
We have provided a handy wrapper for cargo called `cargow` (`cargow.bat` if you are on windows), you can just install with
```bash
./cargow install --path $PWD --root $HOME/.local [--force]
```
> Note: when you provide a --root folder, cargo will go ahead and create a bin folder within that directory and put the binary there. So make sure you have it in your path. For example, in the above, that path will be `$HOME/.local/bin`.
### Build from Source
1. **Clone the repository:**
```bash
git clone https://github.com/scarfbench/scarf.git
cd scarf
```
2. **Build the project:**
```bash
./cargow build --release
```
The compiled binary will be located in `target/release/scarf`.
3. **Run the CLI:**
```bash
./target/release/scarf --help
ScarfBench CLI: The command line helper tool for scarf bench
Usage: scarf [OPTIONS] <COMMAND>
Commands:
bench A series of subcommands to run on the benchmark applications.
eval Subcommands to run evaluation over the benchmark
help Print this message or the help of the given subcommand(s)
Options:
-v, --verbose... Increase verbosity (-v, -vv, -vvv).
-h, --help Print help
-V, --version Print version
```
Optionally, add the binary to your system's PATH for easier access.
## Usage
After installation, you can use the SCARF CLI to interact with the SCARF Benchmark. Here are some common commands:
### 1. List Benchmarks
```bash
❯ ./target/release/scarf bench list --help
List the application(s) in the benchmark.
Usage: scarf bench list [OPTIONS] --benchmark-dir <ROOT>
Options:
--benchmark-dir <ROOT> Path to the root of the scarf benchmark.
-v, --verbose... Increase verbosity (-v, -vv, -vvv). If RUST_LOG is set, it takes precedence.
--layer <LAYER> Application layer to list.
-h, --help Print help
```
This should give you something like below
```bash
❯ ./target/release/scarf bench list --benchmark-dir /home/rkrsn/workspace/scarfbench/benchmark --layer business_domain
┌─────────────────┬──────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────┐
│ Layer ┆ Application ┆ Framework ┆ Path │
╞═════════════════╪══════════════╪═══════════╪═════════════════════════════════════════════════════════════════════════════════╡
│ business_domain ┆ cart ┆ jakarta ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/jakarta │
│ business_domain ┆ cart ┆ quarkus ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/quarkus │
│ business_domain ┆ cart ┆ spring ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/spring │
│ business_domain ┆ converter ┆ jakarta ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/jakarta │
│ business_domain ┆ converter ┆ quarkus ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/quarkus │
│ business_domain ┆ converter ┆ spring ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/spring │
│ business_domain ┆ counter ┆ jakarta ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/jakarta │
│ business_domain ┆ counter ┆ quarkus ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/quarkus │
│ business_domain ┆ counter ┆ spring ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/spring │
│ business_domain ┆ helloservice ┆ jakarta ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/jakarta │
│ business_domain ┆ helloservice ┆ quarkus ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/quarkus │
│ business_domain ┆ helloservice ┆ spring ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/spring │
│ business_domain ┆ standalone ┆ jakarta ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/jakarta │
│ business_domain ┆ standalone ┆ quarkus ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/quarkus │
│ business_domain ┆ standalone ┆ spring ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/spring │
└─────────────────┴──────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────┘
```
### 2. Test Benchmark Layer(s)
You can use the `scarf bench test` command to test specific benchmark layers or the whole benchmark. Here are some examples:
```bash
❯ ./target/release/scarf bench test --help
Run regression tests (with `make test`) on the benchmark application(s).
Usage: scarf bench test [OPTIONS] --benchmark-dir <ROOT>
Options:
--benchmark-dir <ROOT> Path to the root of the scarf benchmark.
-v, --verbose... Increase verbosity (-v, -vv, -vvv). If RUST_LOG is set, it takes precedence.
--layer <LAYER> Application layer to test.
--dry-run Use dry run instead of full run.
-h, --help Print help
```
For example, to test the `persistence` layer:
```bash
❯ ./target/release/scarf bench test --benchmark-dir /home/rkrsn/workspace/scarfbench/benchmark --layer persistence
```
This will run `make tests` in all the apps in `persistence` layer and provide a summary of the results.
```bash
┌─────────────────────────────────────────────────────────────────────────────┬─────────┐
│ Application Path ┆ Result │
╞═════════════════════════════════════════════════════════════════════════════╪═════════╡
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/jakarta ┆ Failure │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/spring ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/quarkus ┆ Failure │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/quarkus ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/jakarta ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/spring ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/quarkus ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/jakarta ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/spring ┆ Success │
└─────────────────────────────────────────────────────────────────────────────┴─────────┘
```
## Development
### Development Dependencies
Use `make setup` to install and verify tooling (`rustup`, `rustfmt`, `clippy`, `cargo-nextest`, `cargo-llvm-cov`). You can also install them manually:
1. **Clippy** - Linting and code quality checks
```bash
./rustupw component add clippy
```
2. **Rustfmt** - Code formatting
```bash
./rustupw component add rustfmt
```
3. **LLVM Coverage Tools** - Coverage analysis
```bash
./rustupw component add llvm-tools-preview
./cargow install cargo-llvm-cov
```
4. **Nextest** - Advanced test runner
```bash
./cargow install cargo-nextest --locked
```
### Testing
The project follows idiomatic Rust testing practices:
- **Unit tests**: Located within each module under the `#[cfg(test)]` attribute
- **Integration tests**: Place in `tests/` (not currently present) for CLI-level coverage.
- For intergation tests, use descriptive names for test files, e.g., `cli_tests.rs`.
### Building and Testing
A [Makefile](Makefile) is provided to streamline development tasks. Run `make help` to see available commands. You can run `make help` to see all available targets:
| Target | Description |
|------------|------------------------------------------------------------------|
| `all` | Run full pipeline (setup → fmt → clippy → build → test → coverage) |
| `setup` | Check/install rustup, cargo, components, nextest, llvm-cov |
| `fmt` | Run `cargo fmt --all` |
| `clippy` | Run `cargo clippy` with warnings denied |
| `build` | Run `cargo build` |
| `test` | Run tests using `cargo nextest` |
| `coverage` | Run coverage using `cargo llvm-cov` + nextest |
| `clean` | Run `cargo clean` |
| `help` | Show help message |
Run the full pipeline with:
```bash
make
```
To build a release binary:
```bash
./cargow build --release
./target/release/scarf --help
```