Crate try_runtime_cli

source ·
Expand description

Try-runtime

Substrate’s ultimate testing framework for the power users.

As the name suggests, try-runtime is a detailed testing framework that gives you a lot of control over what is being executed in which environment. It is recommended that user’s first familiarize themselves with substrate in depth, particularly the execution model. It is critical to deeply understand how the wasm/client/runtime interactions, and the runtime apis work in the substrate runtime, before commencing to working with try-runtime.

Resources

Some resources about the above:

  1. https://docs.substrate.io/reference/command-line-tools/try-runtime/
  2. https://www.crowdcast.io/e/substrate-seminar/41
  3. https://docs.substrate.io/fundamentals/runtime-development/

Background Knowledge

The basis of all try-runtime commands is the same: connect to a live node, scrape its state and put it inside a TestExternalities, then call into a specific runtime-api using the given state and some runtime.

Alternatively, the state could come from a snapshot file.

All of the variables in the above statement are made italic. Let’s look at each of them:

  1. State is the key-value pairs of data that comprise the canonical information that any blockchain is keeping. A state can be full (all key-value pairs), or be partial (only pairs related to some pallets/prefixes). Moreover, some keys are especial and are not related to specific pallets, known as [well_known_keys] in substrate. The most important of these is the :CODE: key, which contains the code used for execution, when wasm execution is chosen.

  2. A runtime-api call is a call into a function defined in the runtime, on top of a given state. Each subcommand of try-runtime utilizes a specific runtime-api.

  3. Finally, the runtime is the actual code that is used to execute the aforementioned runtime-api. Everything in this crate assumes wasm execution, which means the runtime that you use is the one stored onchain, namely under the :CODE: key.

To recap, a typical try-runtime command does the following:

  1. Download the state of a live chain, and write to an externalities.
  2. Overwrite the :CODE: with a given wasm blob
  3. Test some functionality via calling a runtime-api.

Usage

To use any of the provided commands, [SharedParams] must be provided. The most important of which being [SharedParams::runtime], which specifies which runtime to use. Furthermore, [SharedParams::overwrite_state_version] can be used to alter the state-version (see https://forum.polkadot.network/t/state-trie-migration/852 for more info).

Then, the specific command has to be specified. See [Command] for more information about each command’s specific customization flags, and assumptions regarding the runtime being used.

Said briefly, this CLI is capable of executing:

  • [Command::OnRuntimeUpgrade]: execute all the on_runtime_upgrade hooks.
  • [Command::ExecuteBlock]: re-execute the given block.
  • [Command::OffchainWorker]: re-execute the given block’s offchain worker code path.
  • [Command::FollowChain]: continuously execute the blocks of a remote chain on top of a given runtime.
  • [Command::CreateSnapshot]: Create a snapshot file from a remote node.

Finally, To make sure there are no errors regarding this, always run any try-runtime command with executor=trace logging targets, which will specify which runtime is being used per api call. Moreover, remote-ext, try-runtime and runtime logs targets will also be useful.

Spec name check

A common pitfall is that you might be running some test on top of the state of chain x, with the runtime of chain y. To avoid this all commands do a spec-name check before executing anything by default. This will check the, if any alterations are being made to the :CODE:, then the spec names match. The spec versions are warned, but are not mandated to match.

If anything, in most cases, we expect spec-versions to NOT match, because try-runtime is all about testing unreleased runtimes.

Note on signature and state-root checks

All of the commands calling into TryRuntime_execute_block ([Command::ExecuteBlock] and [Command::FollowChain]) disable both state root and signature checks. This is because in 99% of the cases, the runtime that is being tested is different from the one that is stored in the canonical chain state. This implies:

  1. the state root will NEVER match, because :CODE: is different between the two.
  2. replaying all transactions will fail, because the spec-version is part of the transaction signature.

Best Practices

Try-runtime is all about battle-testing unreleased runtime. The following list of suggestions help developers maximize the testing coverage and make base use of try-runtime.

Adding pre/post hooks

One of the gems that come only in the try-runtime feature flag is the pre_upgrade and post_upgrade hooks for OnRuntimeUpgrade. This trait is implemented either inside the pallet, or manually in a runtime, to define a migration. In both cases, these functions can be added, given the right flag:

ⓘ
#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, TryRuntimeError> {}

#[cfg(feature = "try-runtime")]
fn post_upgrade(state: Vec<u8>) -> Result<(), TryRuntimeError> {}

(The pallet macro syntax will support this simply as a part of #[pallet::hooks]).

These hooks allow you to execute some code, only within the on-runtime-upgrade command, before and after the migration. Moreover, pre_upgrade can return a Vec<u8> that contains arbitrary encoded data (usually some pre-upgrade state) which will be passed to post_upgrade after upgrading and used for post checking.

State Consistency

Similarly, each pallet can expose a function in #[pallet::hooks] section as follows:

ⓘ
#[cfg(feature = "try-runtime")]
fn try_state(_: BlockNumber) -> Result<(), TryRuntimeError> {}

which is called on numerous code paths in the try-runtime tool. These checks should ensure that the state of the pallet is consistent and correct. See frame_support::try_runtime::TryState for more info.

Logging

It is super helpful to make sure your migration code uses logging (always with a runtime log target prefix, e.g. runtime::balance) and state exactly at which stage it is, and what it is doing.

Guarding migrations

Always make sure that any migration code is guarded either by StorageVersion, or by some custom storage item, so that it is NEVER executed twice, even if the code lives in two consecutive runtimes.

Examples

For the following examples, we assume the existence of the following:

  1. a substrate node compiled without --feature try-runtime, called substrate. This will be the running node that you connect to. then, after some changes to this node, you compile it with --features try-runtime. This gives you:
  2. a substrate binary that has the try-runtime sub-command enabled.
  3. a wasm blob that has try-runtime functionality.
# this is like your running deployed node.
cargo build --release && cp target/release/substrate .

# this is like your WIP branch.
cargo build --release --features try-runtime
cp target/release/substrate substrate-try-runtime
cp ./target/release/wbuild/kitchensink-runtime/kitchensink_runtime.wasm runtime-try-runtime.wasm

The above example is with substrate’s kitchensink-runtime, but is applicable to any substrate-based chain that has implemented try-runtime-cli.

  • If you run try-runtime subcommand against substrate binary listed above, you get the following error.
[substrate] ./substrate try-runtime
Error: Input("TryRuntime wasn't enabled when building the node. You can enable it with `--features try-runtime`.")
  • If you run the same against substrate-try-runtime, it will work.
[substrate] ./substrate-try-runtime try-runtime
Try some command against runtime state

Usage: substrate-try-runtime try-runtime [OPTIONS] --runtime <RUNTIME> <COMMAND>

Commands:
  on-runtime-upgrade  Execute the migrations of the "local runtime"
  execute-block       Executes the given block against some state
  offchain-worker     Executes *the offchain worker hooks* of a given block against some state
  follow-chain        Follow the given chain's finalized blocks and apply all of its extrinsics
  create-snapshot     Create a new snapshot file
  help                Print this message or the help of the given subcommand(s)

Options:
      --chain <CHAIN_SPEC>
          Specify the chain specification
      --dev
          Specify the development chain
  -d, --base-path <PATH>
          Specify custom base path
  -l, --log <LOG_PATTERN>...
          Sets a custom logging filter. Syntax is `<target>=<level>`, e.g. -lsync=debug
      --detailed-log-output
          Enable detailed log output
      --disable-log-color
          Disable log color output
      --enable-log-reloading
          Enable feature to dynamically update and reload the log filter
      --tracing-targets <TARGETS>
          Sets a custom profiling filter. Syntax is the same as for logging: `<target>=<level>`
      --tracing-receiver <RECEIVER>
          Receiver to process tracing messages [default: log] [possible values: log]
      --runtime <RUNTIME>
          The runtime to use
      --wasm-execution <METHOD>
          Type of wasm execution used [default: compiled] [possible values: interpreted-i-know-what-i-do, compiled]
      --wasm-instantiation-strategy <STRATEGY>
          The WASM instantiation method to use [default: pooling-copy-on-write] [possible values: pooling-copy-on-write, recreate-instance-copy-on-write, pooling, recreate-instance, legacy-instance-reuse]
      --heap-pages <HEAP_PAGES>
          The number of 64KB pages to allocate for Wasm execution. Defaults to [`sc_service::Configuration.default_heap_pages`]
      --overwrite-state-version <OVERWRITE_STATE_VERSION>
          Overwrite the `state_version`
  -h, --help
          Print help information (use `--help` for more detail)
  -V, --version
          Print version information
  • Run the migrations of a given runtime on top of a live state.
# assuming there's `./substrate --dev --tmp --ws-port 9999` or similar running.
./substrate-try-runtime \
    try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    on-runtime-upgrade \
    live --uri ws://localhost:9999
  • Same as the previous one, but run it at specific block number’s state. This means that this block hash’s state shall not yet have been pruned in rpc.polkadot.io.
./substrate-try-runtime \
    try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    on-runtime-upgrade \
    live --uri ws://localhost:9999 \
    # replace with your desired block hash!
    --at 0xa1b16c1efd889a9f17375ec4dd5c1b4351a2be17fa069564fced10d23b9b3836
  • Executing the same command with the [Runtime::Existing] will fail because the existing runtime, stored onchain in substrate binary that we compiled earlier does not have try-runtime feature!
./substrate-try-runtime try-runtime --runtime existing -lruntime=debug on-runtime-upgrade live --uri ws://localhost:9999
...
Error: Input("given runtime is NOT compiled with try-runtime feature!")
  • Now, let’s use a snapshot file. First, we create the snapshot:
./substrate-try-runtime try-runtime --runtime existing -lruntime=debug create-snapshot --uri ws://localhost:9999
2022-12-13 10:28:17.516  INFO main try-runtime::cli: snapshot path not provided (-s), using 'node-268@latest.snap'
2022-12-13 10:28:17.516  INFO                 main remote-ext: since no at is provided, setting it to latest finalized head, 0xe7d0b614dfe89af65b33577aae46a6f958c974bf52f8a5e865a0f4faeb578d22
2022-12-13 10:28:17.516  INFO                 main remote-ext: since no prefix is filtered, the data for all pallets will be downloaded
2022-12-13 10:28:17.550  INFO                 main remote-ext: writing snapshot of 1611464 bytes to "node-268@latest.snap"
2022-12-13 10:28:17.551  INFO                 main remote-ext: initialized state externalities with storage root 0x925e4e95de4c08474fb7f976c4472fa9b8a1091619cd7820a793bf796ee6d932 and state_version V1

Note that the snapshot contains the existing runtime, which does not have the correct try-runtime feature. In the following commands, we still need to overwrite the runtime.

Then, we can use it to have the same command as before, on-runtime-upgrade

./substrate-try-runtime try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    on-runtime-upgrade \
    snap -s node-268@latest.snap
  • Execute the latest finalized block with the given runtime.
./substrate-try-runtime try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    execute-block live \
    --uri ws://localhost:9999

This can still be customized at a given block with --at. If you want to use a snapshot, you can still use --block-ws-uri to provide a node form which the block data can be fetched.

Moreover, this runs the frame_support::try_runtime::TryState hooks as well. The hooks to run can be customized with the --try-state. For example:

./substrate-try-runtime try-runtime \
   --runtime runtime-try-runtime.wasm \
   -lruntime=debug \
   execute-block \
   --try-state System,Staking \
   live \
   --uri ws://localhost:9999 \
   --pallet System Staking

Will only run the try-state of the two given pallets. When running try-state against some real chain data it can take a long time for the command to execute since it has to query all the key-value pairs. In scenarios like above where we only want to run the try-state for some specific pallets, we can use the --pallet option to specify from which pallets we want to query the state. This will greatly decrease the execution time.

See [frame_try_runtime::TryStateSelect] for more information.

  • Follow our live chain’s blocks using follow-chain, whilst running the try-state of 3 pallets in a round robin fashion
./substrate-try-runtime \
    try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    follow-chain \
    --uri ws://localhost:9999 \
    --try-state rr-3