Crate xshell

Expand description

xshell is a swiss-army knife for writing cross-platform “bash” scripts in Rust.

It doesn’t use the shell directly, but rather re-implements parts of scripting environment in Rust. The intended use-case is various bits of glue code, which could be written in bash or python. The original motivation is xtask development.

Here’s a quick example:

use xshell::{Shell, cmd};

let sh = Shell::new()?;
let branch = "main";
let commit_hash = cmd!(sh, "git rev-parse {branch}").read()?;

Goals:

Ergonomics and DWIM (“do what I mean”): cmd! macro supports interpolation, writing to a file automatically creates parent directories, etc.
Reliability: no shell injection by construction, good error messages with file paths, non-zero exit status is an error, independence of the host environment, etc.
Frugality: fast compile times, few dependencies, low-tech API.

Guide

For a short API overview, let’s implement a script to clone a github repository and publish it as a crates.io crate. The script will do the following:

Clone the repository.
cd into the repository’s directory.
Run the tests.
Create a git tag using a version from Cargo.toml.
Publish the crate with an optional --dry-run.

Start with the following skeleton:

use xshell::{cmd, Shell};

fn main() -> anyhow::Result<()> {
    let sh = Shell::new()?;

    Ok(())
}

Only two imports are needed – the Shell struct the and cmd! macro. By convention, an instance of a Shell is stored in a variable named sh. All the API is available as methods, so a short name helps here. For “scripts”, the anyhow crate is a great choice for an error-handling library.

Next, clone the repository:

cmd!(sh, "git clone https://github.com/matklad/xshell.git").run()?;

The cmd! macro provides a convenient syntax for creating a command – the Cmd struct. The Cmd::run method runs the command as if you typed it into the shell. The whole program outputs:

$ git clone https://github.com/matklad/xshell.git
Cloning into 'xshell'...
remote: Enumerating objects: 676, done.
remote: Counting objects: 100% (220/220), done.
remote: Compressing objects: 100% (123/123), done.
remote: Total 676 (delta 106), reused 162 (delta 76), pack-reused 456
Receiving objects: 100% (676/676), 136.80 KiB | 222.00 KiB/s, done.
Resolving deltas: 100% (327/327), done.

Note that the command itself is echoed to stderr (the $ git ... bit in the output). You can use Cmd::quiet to override this behavior:

cmd!(sh, "git clone https://github.com/matklad/xshell.git")
    .quiet()
    .run()?;

To make the code more general, let’s use command interpolation to extract the username and the repository:

let user = "matklad";
let repo = "xshell";
cmd!(sh, "git clone https://github.com/{user}/{repo}.git").run()?;

Note that the cmd! macro parses the command string at compile time, so you don’t have to worry about escaping the arguments. For example, the following command “touches” a single file whose name is contains a space:

let file = "contains a space";
cmd!(sh, "touch {file}").run()?;

Next, cd into the folder you have just cloned:

sh.change_dir(repo);

Each instance of Shell has a current directory, which is independent of the process-wide std::env::current_dir. The same applies to the environment.

Next, run the tests:

let test_args = ["-Zunstable-options", "--report-time"];
cmd!(sh, "cargo test -- {test_args...}").run()?;

Note how the so-called splat syntax (...) is used to interpolate an iterable of arguments.

Next, read the Cargo.toml so that we can fetch crate’ declared version:

let manifest = sh.read_file("Cargo.toml")?;

Shell::read_file works like std::fs::read_to_string, but paths are relative to the current directory of the Shell. Unlike std::fs, error messages are much more useful. For example, if there isn’t a Cargo.toml in the repository, the error message is:

Error: failed to read file `xshell/Cargo.toml`: no such file or directory (os error 2)

xshell doesn’t implement string processing utils like grep, sed or awk – there’s no need to, built-in language features work fine, and it’s always possible to pull extra functionality from crates.io.

To extract the version field from Cargo.toml, str::split_once is enough:

let manifest = sh.read_file("Cargo.toml")?;
let version = manifest
    .split_once("version = \"")
    .and_then(|it| it.1.split_once('\"'))
    .map(|it| it.0)
    .ok_or_else(|| anyhow::format_err!("can't find version field in the manifest"))?;

cmd!(sh, "git tag {version}").run()?;

The splat (...) syntax works with any iterable, and in Rust options are iterable. This means that ... can be used to implement optional arguments. For example, here’s how to pass --dry-run when not running in CI:

let dry_run = if sh.var("CI").is_ok() { None } else { Some("--dry-run") };
cmd!(sh, "cargo publish {dry_run...}").run()?;

Putting everything altogether, here’s the whole script:

use xshell::{cmd, Shell};

fn main() -> anyhow::Result<()> {
    let sh = Shell::new()?;

    let user = "matklad";
    let repo = "xshell";
    cmd!(sh, "git clone https://github.com/{user}/{repo}.git").run()?;
    sh.change_dir(repo);

    let test_args = ["-Zunstable-options", "--report-time"];
    cmd!(sh, "cargo test -- {test_args...}").run()?;

    let manifest = sh.read_file("Cargo.toml")?;
    let version = manifest
        .split_once("version = \"")
        .and_then(|it| it.1.split_once('\"'))
        .map(|it| it.0)
        .ok_or_else(|| anyhow::format_err!("can't find version field in the manifest"))?;

    cmd!(sh, "git tag {version}").run()?;

    let dry_run = if sh.var("CI").is_ok() { None } else { Some("--dry-run") };
    cmd!(sh, "cargo publish {dry_run...}").run()?;

    Ok(())
}

xshell itself uses a similar script to automatically publish oneself to crates.io when the version in Cargo.toml changes:

https://github.com/matklad/xshell/blob/master/examples/ci.rs

Maintenance

Minimum Supported Rust Version: 1.59.0. MSRV bump is not considered semver breaking. MSRV is updated conservatively.

The crate isn’t comprehensive yet, but this is a goal. You are hereby encouraged to submit PRs with missing functionality!

duct is a crate for heavy-duty process herding, with support for pipelines.

Most of what this crate provides can be open-coded using std::process::Command and std::fs. If you only need to spawn a single process, using std is probably better (but don’t forget to check the exit status!).

Implementation Notes

The design is heavily inspired by the Julia language:

Smaller influences are the duct crate and Ruby’s FileUtils module.

The cmd! macro uses a simple proc-macro internally. It doesn’t depend on helper libraries, so the fixed-cost impact on compile times is moderate. Compiling a trivial program with cmd!("date +%Y-%m-%d") takes one second. Equivalent program using only std::process::Command compiles in 0.25 seconds.

To make IDEs infer correct types without expanding proc-macro, it is wrapped into a declarative macro which supplies type hints.