Awesome Sails Benchmarks

Note: Built for the Sails framework. It is highly recommended to study the Sails Documentation before using this crate.

A benchmarking suite for Sails programs. This crate provides tools to measure gas consumption and a CLI utility to analyze performance regressions.

Features

• gtest (default): Enables built-in support for the Gear gtest framework and provides the MeasureGas implementation for System.
• ascii-table: Enables ASCII table output for terminal reports.
• cli: Enables the bench-analyzer CLI utility.

Installation

Add to your Cargo.toml:

[dev-dependencies]
# Standard usage (with gtest support)
awesome-sails-benchmarks = { version = "x.y.z", features = ["ascii-table"] }

# If you have version conflicts with gtest, disable default features:
awesome-sails-benchmarks = { version = "x.y.z", default-features = false }

Usage

1. Gas Measurement

Use the MeasureGas trait to capture gas during tests. Benchmarks MUST be run in --release mode.

The MeasureGas trait is always available, but the implementation for the Gear gtest::System requires the gtest feature (enabled by default).

use awesome_sails_benchmarks::{MeasureGas, BenchStorage, ToBenchmarkMap};
use std::collections::BTreeMap;

struct MyServiceResults {
    op_gas: BTreeMap<u32, u64>,
}

impl ToBenchmarkMap for MyServiceResults {
    fn to_benchmark_map(self) -> BTreeMap<String, u64> {
        let mut map = BTreeMap::new();
        for (count, gas) in self.op_gas {
            map.insert(format!("my_op.{}", count), gas);
        }
        map
    }
}

#[tokio::test]
#[ignore]
async fn bench_my_service() {
    let env = create_env();
    // System from gtest implements MeasureGas if the 'gtest' feature is enabled.
    let system = env.system();
    let mut service = deploy_program(&env).await;

    let mut op_gas = BTreeMap::new();
    for &count in &[0, 100, 1000] {
        let gas = system.measure_gas(|| {
            service.operation(count).send_one_way().unwrap()
        });
        op_gas.insert(count, gas);
    }

    BenchStorage::from_default_path()
        .update("my_service", MyServiceResults { op_gas })
        .expect("Failed to save benchmarks");
}

2. Gas Trace Tree (Qualitative Profiling)

While MeasureGas answers "how much gas?", GasTrace answers "where did the gas go?" — it reconstructs a reply-linked message tree from one or more BlockRunResults, decorates each node with its gas cost, and resolves (interface_id, entry_id) pairs back to Service::method names via a MethodRegistry. Output is a human-readable ASCII tree.

use awesome_sails_benchmarks::{GasTrace, MethodRegistry};

let block = system.run_next_block();

let registry = MethodRegistry::new()
    .register_service::<MyServiceMeta>("MyService");

let tree = GasTrace::new(&block)
    .with_registry(&registry)
    .with_actor_name(user_id, "alice")
    .build();

println!("{}", tree);

Example output:

[0x1001..abcd] alice -> MyProgram::MyService::increment  12,400 gas
  `-- [0x1002..9876] [reply] Ok  - gas
[event] MyProgram
Total: 12,400 gas | 3 messages | depth 1

Limitations (gtest runtime constraints, not tool design):

• The tree is built from reply_to linkage. Cross-program sub-calls (A → B → C) appear as separate roots because gtest::CoreLog doesn't expose parent-message causality.
• block.log() surfaces replies and events but not the originating request. GasTrace synthesizes a root for any gas_burned entry whose id isn't represented elsewhere in the tree and nests its reply under it — this recovers gas attribution but the synthesized root has no payload, so its method is shown as [raw] unless a future gtest revision exposes the full request.

3. Analysis

The bench-analyzer utility compares generated results with a baseline.

Installation

Install the tool globally via cargo:

cargo install awesome-sails-benchmarks --features cli

Comparison Modes

• Informational (Default): Generates a report but always exits with code 0.
• Regression Check (--fail-on-regression): Exits with code 1 if a significant regression (based on --threshold) is detected.
• Strict Mode (--strict): Exits with code 1 if ANY metric deviates from baseline by more than the threshold (including improvements).

Usage

bench-analyzer \
  --current benchmarks/bench_data.json \
  --other benchmarks/baseline.json \
  --threshold 5.0 \
  --fail-on-regression \
  --output report.md

CLI Reference

A CLI tool to compare two benchmark JSON files and detect performance regressions.

Usage: bench-analyzer [OPTIONS] --current <CURRENT> --other <OTHER>

Options:
      --current <CURRENT>      Path to the current benchmark results (the ones you just generated)
      --other <OTHER>          Path to the baseline benchmark results (the reference file to compare against)
      --output <OUTPUT>        Path to save the comparison report in Markdown format
      --threshold <THRESHOLD>  Custom regression threshold percentage (e.g. 5.0)
      --fail-on-regression     Exit with code 1 if a significant regression is detected
      --strict                 Enable strict mode: the tool will fail if ANY metric deviates from baseline by more than the threshold (including improvements). Useful for CI self-checks to ensure benchmark stability
  -h, --help                   Print help