Crate goldenscript

This crate provides the Goldenscript testing framework, loosely based on Cockroach Labs’ datadriven framework for Go. It combines several testing techniques that make it easy and efficient to write and update test cases:

• Golden master testing (aka characterization testing or historical oracle)
• Data-driven testing (aka table-driven testing or parameterized testing)
• Keyword-driven testing

A goldenscript is a plain text file that contains a set of arbitrary input commands and their expected text output, separated by ---:

command
---
output

command argument
command key=value
---
output

The commands are executed by a provided Runner. The expected output is usually not written by hand, but instead generated by running tests with the environment variable UPDATE_GOLDENFILES=1:

$ UPDATE_GOLDENFILES=1 cargo test

The files are then verified by inspection and checked in to version control. Tests will fail with a diff if they don’t match the expected output.

This approach is particularly useful when testing complex stateful systems, such as database operations, network protocols, or language parsing. It can be tedious and labor-intensive to write and assert such cases by hand, so scripting and recording these interactions often yields much better test coverage at a fraction of the cost.

Internally, the goldenfile crate is used to manage golden files.

Examples

For real-world examples, see e.g.:

• toyDB Raft: distributed consensus cluster.
• toyDB MVCC: ACID transactions.
• goldenscript parser: Goldenscript uses itself to test its parser and runner.

Below is a basic example, testing the Rust standard library’s BTreeMap.

# Tests the Rust standard library BTreeMap.

# Get and range returns nothing for an empty map.
get foo
range
---
get → None

# Inserting keys out of order will return them in order. Silence the insert
# output with ().
(insert b=2 a=1 c=3)
range
---
a=1
b=2
c=3

# Getting a key returns its value.
get b
---
get → Some("2")

# Bounded scans, where the end is exclusive.
range b
---
b=2
c=3

range a c
---
a=1
b=2

# An end bound less than the start bound panics. Expect the failure with !.
!range b a
---
Panic: range start is greater than range end in BTreeMap

# Replacing a key updates the value and returns the old one.
insert b=foo
get b
---
insert → Some("2")
get → Some("foo")

The corresponding runner for this script:

#[derive(Default)]
struct BTreeMapRunner {
    map: std::collections::BTreeMap<String, String>,
}

impl goldenscript::Runner for BTreeMapRunner {
    fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
        let mut output = String::new();
        match command.name.as_str() {
            // get KEY: fetches the value of the given key, or None if it does not exist.
            "get" => {
                let mut args = command.consume_args();
                let key = &args.next_pos().ok_or("key not given")?.value;
                args.reject_rest()?;
                let value = self.map.get(key);
                writeln!(output, "get → {value:?}")?;
            }

            // insert KEY=VALUE...: inserts the given key/value pairs, returning the old value.
            "insert" => {
                let mut args = command.consume_args();
                for arg in args.rest_key() {
                    let old = self.map.insert(arg.key.clone().unwrap(), arg.value.clone());
                    writeln!(output, "insert → {old:?}")?;
                }
                args.reject_rest()?;
            }

            // range [FROM] [TO]: iterates over the key/value pairs in the range from..to.
            "range" => {
                use std::ops::Bound::*;
                let mut args = command.consume_args();
                let from = args.next_pos().map(|a| Included(a.value.clone())).unwrap_or(Unbounded);
                let to = args.next_pos().map(|a| Excluded(a.value.clone())).unwrap_or(Unbounded);
                args.reject_rest()?;
                for (key, value) in self.map.range((from, to)) {
                    writeln!(output, "{key}={value}")?;
                }
            }

            name => return Err(format!("invalid command {name}").into()),
        };
        Ok(output)
    }
}

#[test]
fn btreemap() {
    goldenscript::run(&mut BTreeMapRunner::default(), "btreemap").expect("goldenscript failed")
}

Syntax

Blocks

A goldenscript consists of one or more input/output blocks. Each block has a set of one or more input commands on individual lines (empty or comment lines are ignored), a --- separator, and arbitrary output terminated by an empty line. A minimal goldenscript with two blocks might be:

command
---
output

command 1
command 2
---
output 1
output 2

Commands

A Command must have a command name, which can be any arbitrary string, e.g.:

command
"command with space and 🚀"
---

It may additionally have:

• Arguments: any number of space-separated arguments. These have a string value, and optionally also a string key as key=value. Keys and values can be empty, and duplicate keys are allowed by the parser (the runner can handle this as desired).

  command argument key=value
  command "argument with space" "key with space"="value with space"
  command "" key=  # Empty argument values.
  ---

• Prefix: an optional :-terminated string prefix before the command. The command’s output will be given the same prefix. The prefix can be used by the test runner, e.g. to signify two different clients.

  client1: put key=value
  client2: get key
  ---
  client1: put ok
  client2: get key=value

• Silencing: a command wrapped in () will have its output suppressed. This can be useful e.g. for setup commands whose output are not of interest in the current test case and would only add noise.

  echo foo
  (echo bar)
  ---
  foo

• Failure: if ! precedes the command, it is expected to fail with an error or panic, and the failure message is used as output. If the command unexpectedly succeeds, the test fails. If the line contains other symbols before the command name (e.g. a prefix or silencing), the ! must be used immediately before the command name.

  ! command error=foo
  prefix: ! command panic=bar
  (!command error=foo)
  ---
  Error: foo
  prefix: Panic: bar

• Tags: an optional comma- or space-separated list of tags (strings) enclosed in [] before or after the command and arguments. This can be used by the runner e.g. to modify the execution of a command.

  command [tag]
  command arg key=value [a,b c]
  [tag] command
  prefix:[tag]!> command arg
  ---

• Literal: if > precedes the command, the entire rest of the line is taken to be the command name (except leading whitespace). Arguments, tags, comments, and any other special characters are ignored and used as-is. As a special case (currently only with >), lines can span multiple lines by ending the line with .

  > a long command name including key=value, [tags], # a comment and exclamation!
  prefix: [tag] ! > a long, failing command with tags and a prefix
  ---
  
  > a very \
  long line \
  with line \
  continuation
  ---

Output

The command output following a --- separator can contain any arbitrary Unicode string until an empty line (or end of file). If the command output contains empty lines, the entire output will automatically be prefixed with > . If no commands in a block yield any output, it defaults to “ok”.

echo "output 1"
echo "output 2"
---
output 1
output 2

echo "Paragraph 1.\n\nParagraph 2."
---
> Paragraph 1.
>
> Paragraph 2.

echo "输出\n# Comment\n🚀"
---
输出
# Comment
🚀

Comments

Comments begin with # or // and run to the end of the line.

# This is a comment.
// As is this.
command argument # Comments can follow commands too.
---

Strings

Unquoted strings can only contain alphanumeric ASCII characters [a-zA-Z0-9] and a handful of special characters: _ - . / @ (only _ at the start of a string).

Strings can be quoted using " or ', in which case they can contain arbitrary Unicode characters. \ is used as an escape character, both to escape quotes \" and \' as well as itself \\, and also \0 (null), \n (newline), \r (carriage return), and \t (tab). \x can be used to represent arbitrary hexadecimal bytes (e.g. \x7a) and \u{} can be used to represent arbitrary Unicode characters (e.g. \u{1f44b})

string
"string with spaces and \"quotes\""
'字符串'
---

Writing Tests

In the simplest case, a goldenscript test might be:

struct Runner;

impl goldenscript::Runner for Runner {
    fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
        match command.name.as_str() {
            "echo" => {
                let lines: Vec<&str> = command.args.iter().map(|a| a.value.as_str()).collect();
                Ok(lines.join("\n"))
            }
            name => return Err(format!("invalid command {name}").into())
        }
    }
}

#[test]
fn test() -> std::io::Result<()> {
    goldenscript::run(&mut Runner, "tests/scripts/test")
}

Argument Processing

Arguments can be processed manually via Command::args, or using the Command::consume_args() helper which simplifies common argument handling. For example:

impl goldenscript::Runner for Runner {
    /// Implement a send command, which sends a string message to a list
    /// of nodes, optionally retrying.
    ///
    /// send [retry=BOOL] MESSAGE ID...
    ///
    /// Example: send foo 1 2 3
    fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
        if command.name != "send" {
            return Err(format!("invalid command {}", command.name).into())
        }

        let mut args = command.consume_args();

        // The first positional argument is a required string message.
        let message = &args.next_pos().ok_or("message not given")?.value;

        // The remaining positional arguments are numeric node IDs.
        let ids: Vec<u32> = args.rest_pos().iter().map(|a| a.parse()).collect::<Result<_, _>>()?;
        if ids.is_empty() {
            return Err("no node IDs given".into())
        }

        // An optional retry=bool key/value argument can also be given.
        let retry: bool = args.lookup_parse("retry")?.unwrap_or(false);

        // Any other arguments that haven't been processed above should error.
        args.reject_rest()?;

        // Execute the send.
        self.send(&ids, message, retry)
    }
}

Managing State

The runner is free to manage internal state as desired. If it is stateful, it is recommended to persist state within a single goldenscript (across commands and blocks), but not across goldenscripts since this can be hard to reason about and depend on the execution order of scripts. This is most easily done by instantiating a new runner for each script.

Initial state setup should generally be done via explicit setup commands, to make it more discoverable.

Running All Scripts in a Directory

External crates can be used to automatically generate and run individual tests for each goldenscript in a directory. For example, the test_each_file crate:

use test_each_file::test_each_path;

test_each_path! { in "tests/scripts" as scripts => test_goldenscript }

fn test_goldenscript(path: &std::path::Path) {
    goldenscript::run(&mut Runner, path).unwrap()
}

Hooks

Runners have various hooks that will be called during script execution: Runner::start_script, Runner::end_script, Runner::start_block, Runner::end_block, Runner::start_command, and Runner::end_command. These can be used e.g. for initial setup, invariant assertions, or to output the current state.

Structs

Argument

A command argument.

ArgumentConsumer

Helper for argument processing, by returning and removing arguments on demand.

Command

A command.

Traits

Runner

Runs goldenscript commands, returning their output.

Functions

generate

Generates output for a goldenscript input, without comparing them.

run

Runs a goldenscript at the given path.