scarfbench-cli 0.1.0-rc.1

scarfbench-cli-0.1.0-rc.1 is not a library.

This is a companion CLI tool for the SCARF Benchmark. It provides a commandline interface to list and test benchmarks, run agents, submit solutions, view and explore leaderboard among other useful tasks.

Features
Installation
- Prerequisites
- Build from Source
Usage
Development

Features

List available benchmarks
Test and validate benchmarks
Run agents on benchmark problems
Submit solutions (to be added)
View and explore leaderboards (to be added)

Installation

Prerequisites

Before installing the SCARF CLI, ensure you have the following tools installed:

Docker (Installation Guide) - Runs benchmarks in isolated environments
Make - Builds and runs projects as specified in makefiles
Python - If you want to install scarf with pip (optional)

Clone the repository

git clone https://github.com/scarfbench/scarf.git
cd scarf

Install with `pip`

You can install the SCARF CLI using pip for easier management:

pip install -U .

Note: Depending on your os, pip may be called pip3.

Install with `cargo`

We have provided a handy wrapper for cargo called cargow (cargow.bat if you are on windows), you can just install with

./cargow install --path $PWD --root $HOME/.local [--force]

Note: when you provide a --root folder, cargo will go ahead and create a bin folder within that directory and put the binary there. So make sure you have it in your path. For example, in the above, that path will be $HOME/.local/bin.

Build from Source

Clone the repository:

git clone https://github.com/scarfbench/scarf.git
cd scarf

Build the project:
```
./cargow build --release
```
The compiled binary will be located in target/release/scarf.

Run the CLI:

./target/release/scarf --help

 ScarfBench CLI: The command line helper tool for scarf bench
 
 Usage: scarf [OPTIONS] <COMMAND>
 
 Commands:
 bench  A series of subcommands to run on the benchmark applications.
 eval   Subcommands to run evaluation over the benchmark
 help   Print this message or the help of the given subcommand(s)
 
 Options:
 -v, --verbose...  Increase verbosity (-v, -vv, -vvv).
 -h, --help        Print help
 -V, --version     Print version

Optionally, add the binary to your system's PATH for easier access.

Usage

After installation, you can use the SCARF CLI to interact with the SCARF Benchmark. Here are some common commands:

1. List Benchmarks

❯ ./target/release/scarf bench list --help
List the application(s) in the benchmark.

Usage: scarf bench list [OPTIONS] --benchmark-dir <ROOT>

Options:
      --benchmark-dir <ROOT>    Path to the root of the scarf benchmark.
  -v, --verbose...     Increase verbosity (-v, -vv, -vvv). If RUST_LOG is set, it takes precedence.
      --layer <LAYER>  Application layer to list.
  -h, --help           Print help

This should give you something like below

❯ ./target/release/scarf bench list --benchmark-dir /home/rkrsn/workspace/scarfbench/benchmark --layer business_domain
┌─────────────────┬──────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────┐
│ Layer           ┆ Application  ┆ Framework ┆ Path                                                                            │
╞═════════════════╪══════════════╪═══════════╪═════════════════════════════════════════════════════════════════════════════════╡
│ business_domain ┆ cart         ┆ jakarta   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/jakarta         │
│ business_domain ┆ cart         ┆ quarkus   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/quarkus         │
│ business_domain ┆ cart         ┆ spring    ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/cart/spring          │
│ business_domain ┆ converter    ┆ jakarta   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/jakarta    │
│ business_domain ┆ converter    ┆ quarkus   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/quarkus    │
│ business_domain ┆ converter    ┆ spring    ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/converter/spring     │
│ business_domain ┆ counter      ┆ jakarta   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/jakarta      │
│ business_domain ┆ counter      ┆ quarkus   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/quarkus      │
│ business_domain ┆ counter      ┆ spring    ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/counter/spring       │
│ business_domain ┆ helloservice ┆ jakarta   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/jakarta │
│ business_domain ┆ helloservice ┆ quarkus   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/quarkus │
│ business_domain ┆ helloservice ┆ spring    ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/helloservice/spring  │
│ business_domain ┆ standalone   ┆ jakarta   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/jakarta   │
│ business_domain ┆ standalone   ┆ quarkus   ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/quarkus   │
│ business_domain ┆ standalone   ┆ spring    ┆ /home/rkrsn/workspace/scarfbench/benchmark/business_domain/standalone/spring    │
└─────────────────┴──────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────┘

2. Test Benchmark Layer(s)

You can use the scarf bench test command to test specific benchmark layers or the whole benchmark. Here are some examples:

❯ ./target/release/scarf bench test --help
Run regression tests (with `make test`) on the benchmark application(s).

Usage: scarf bench test [OPTIONS] --benchmark-dir <ROOT>

Options:
      --benchmark-dir <ROOT>    Path to the root of the scarf benchmark.
  -v, --verbose...     Increase verbosity (-v, -vv, -vvv). If RUST_LOG is set, it takes precedence.
      --layer <LAYER>  Application layer to test.
      --dry-run        Use dry run instead of full run.
  -h, --help           Print help

For example, to test the persistence layer:

❯ ./target/release/scarf bench test --benchmark-dir /home/rkrsn/workspace/scarfbench/benchmark --layer persistence

This will run make tests in all the apps in persistence layer and provide a summary of the results.

┌─────────────────────────────────────────────────────────────────────────────┬─────────┐
│ Application Path                                                            ┆ Result  │
╞═════════════════════════════════════════════════════════════════════════════╪═════════╡
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/jakarta        ┆ Failure │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/spring        ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/quarkus        ┆ Failure │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/quarkus       ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/roster/jakarta       ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/spring  ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/quarkus ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/address-book/jakarta ┆ Success │
│ /home/rkrsn/workspace/scarfbench/benchmark/persistence/order/spring         ┆ Success │
└─────────────────────────────────────────────────────────────────────────────┴─────────┘

Development

Development Dependencies

Use make setup to install and verify tooling (rustup, rustfmt, clippy, cargo-nextest, cargo-llvm-cov). You can also install them manually:

Clippy - Linting and code quality checks
```
./rustupw component add clippy
```
Rustfmt - Code formatting
```
./rustupw component add rustfmt
```

LLVM Coverage Tools - Coverage analysis

./rustupw component add llvm-tools-preview
./cargow install cargo-llvm-cov

Nextest - Advanced test runner

./cargow install cargo-nextest --locked

Testing

The project follows idiomatic Rust testing practices:

Unit tests: Located within each module under the #[cfg(test)] attribute
Integration tests: Place in tests/ (not currently present) for CLI-level coverage.
- For intergation tests, use descriptive names for test files, e.g., cli_tests.rs.

Building and Testing

A Makefile is provided to streamline development tasks. Run make help to see available commands. You can run make help to see all available targets:

Target	Description
`all`	Run full pipeline (setup → fmt → clippy → build → test → coverage)
`setup`	Check/install rustup, cargo, components, nextest, llvm-cov
`fmt`	Run `cargo fmt --all`
`clippy`	Run `cargo clippy` with warnings denied
`build`	Run `cargo build`
`test`	Run tests using `cargo nextest`
`coverage`	Run coverage using `cargo llvm-cov` + nextest
`clean`	Run `cargo clean`
`help`	Show help message

Run the full pipeline with:

make

To build a release binary:

./cargow build --release
./target/release/scarf --help