substrate-benchmark-machine 0.0.3

CLI for benchmarking substrate validator node
Documentation

substrate-benchmark-machine

Different Substrate chains can have different hardware requirements.
It is therefore important to be able to quickly gauge if a piece of hardware fits a chains' requirements.
The substrate-benchmark-machine command archives this by measuring key metrics and making them comparable.

Installing the command looks like this:

cargo install --git https://github.com/btwiuse/substrate-benchmark-machine

Output

Perform quick hardware check:

$ substrate-benchmark-machine
[INFO] Performing quick hardware check...
[INFO] 🏁 CPU score: 1.42 GiBs (✅ Blake2256: expected minimum 1.00 GiBs)
[INFO] 🏁 Memory score: 17.93 GiBs (✅ MemCopy: expected minimum 14.32 GiBs)
[INFO] 🏁 Disk score (seq. writes): 397.35 MiBs (❌ DiskSeqWrite: expected minimum 450.00 MiBs)
[INFO] 🏁 Disk score (rand. writes): 193.44 MiBs (❌ DiskRndWrite: expected minimum 200.00 MiBs)
[INFO] ⚠  The hardware does not meet the minimal requirements for role 'Authority'.

Run full benchmarks:

$ substrate-benchmark-machine --full
[INFO] Running full machine benchmarks...
[INFO]
+----------+----------------+-------------+-------------+-------------------+
| Category | Function       | Score       | Minimum     | Result            |
+===========================================================================+
| CPU      | BLAKE2-256     | 1.42 GiBs   | 1.00 GiBs   | ✅ Pass (141.3 %) |
|----------+----------------+-------------+-------------+-------------------|
| CPU      | SR25519-Verify | 726.21 KiBs | 666.00 KiBs | ✅ Pass (109.0 %) |
|----------+----------------+-------------+-------------+-------------------|
| Memory   | Copy           | 18.10 GiBs  | 14.32 GiBs  | ✅ Pass (126.4 %) |
|----------+----------------+-------------+-------------+-------------------|
| Disk     | Seq Write      | 398.56 MiBs | 450.00 MiBs | ❌ Fail ( 88.6 %) |
|----------+----------------+-------------+-------------+-------------------|
| Disk     | Rnd Write      | 193.37 MiBs | 200.00 MiBs | ✅ Pass ( 96.7 %) |
+----------+----------------+-------------+-------------+-------------------+
From 5 benchmarks in total, 4 passed and 1 failed (10% fault tolerance).
[INFO] The hardware fails to meet the requirements
Error: One of the benchmarks had a score that was lower than its requirement

The score is the average result of each benchmark. It always adheres to "higher is better".

The category indicate which part of the hardware was benchmarked:

  • CPU Processor intensive task
  • Memory RAM intensive task
  • Disk Hard drive intensive task

The function is the concrete benchmark that was run:

  • BLAKE2-256 The throughput of the Blake2-256 cryptographic hashing function with 32 KiB input. The blake2_256 function is used in many places in Substrate. The throughput of a hash function strongly depends on the input size, therefore we settled to use a fixed input size for comparable results.
  • SR25519 Verify Sr25519 is an optimized version of the Curve25519 signature scheme. Signature verification is used by Substrate when verifying extrinsics and blocks.
  • Copy The throughput of copying memory from one place in the RAM to another.
  • Seq Write The throughput of writing data to the storage location sequentially. It is important that the same disk is used that will later-on be used to store the chain data.
  • Rnd Write The throughput of writing data to the storage location in a random order. This is normally much slower than the sequential write.

The score needs to reach the minimum in order to pass the benchmark. This can be reduced with the --tolerance flag.

The result indicated if a specific benchmark was passed by the machine or not. The percent number is the relative score reached to the minimum that is needed. The --tolerance flag is taken into account for this decision. For example a benchmark that passes even with 95% since the tolerance was set to 10% would look like this: ✅ Pass ( 95.0 %).

Interpretation

Ideally all results show a Pass and the program exits with code 0. Currently some of the benchmarks can fail even on reference hardware; they are still being improved to make them more deterministic.
Make sure to run nothing else on the machine when benchmarking it.
You can re-run them multiple times to get more reliable results.

Arguments

  • --tolerance A percent number to reduce the minimum requirement. This should be used to ignore outliers of the benchmarks. The default value is 10%.
  • --verify-duration How long the verification benchmark should run.
  • --disk-duration How long the read and write benchmarks should run each.
  • --allow-fail Always exit the program with code 0.
  • --chain / --dev Specify the chain config to use. This will be used to compare the results with the requirements of the chain (WIP).
  • --base-path

License: Apache-2.0

FAQs

How to solve ⚠ The hardware does not meet the minimal requirements for role 'Authority'.

This error message may appear when a substrate validator node does quick hardware check while starting up for different reasons:

a) You node have low specs.

b) Your node meets the minimum requirement, but

  • b.1) you have other CPU/IO intensive process running while the node starts.

  • b.2) the quick check doesn't have any tolerance of slightly lower scores.

In order to decide if you should upgrade your node spec, you should stop the gear node and other resource-heavy processes, and run the full machine benchmark:

substrate-benchmark-machine --full

If some of the benchmarks still fail, you should upgrade your node spec accordingly until you get all 5 ✅; otherwise you don't have to.