Module testscript 

This crate provides the testscript testing framework, loosely based on Cockroach Labs’ datadriven framework for Go. It combines several testing techniques that make it easy and efficient to write and update test cases:

• Golden master testing (aka characterization testing or historical oracle)
• Data-driven testing (aka table-driven testing or parameterized testing)
• Keyword-driven testing

A testscript is a plain text file that contains a set of arbitrary input commands and their expected text output, separated by ---:

command
---
output

command argument
command key=value
---
output

The commands are executed by a provided Runner. The expected output is usually not written by hand, but instead generated by running tests with the environment variable UPDATE_TESTFILES=1:

$ UPDATE_TESTFILES=1 cargo test

The files are then verified by inspection and checked in to version control. Tests will fail with a diff if they don’t match the expected output.

This approach is particularly useful when testing comptokenize mod systems, such as database operations, network protocols, or language parsing. It can be tedious and labor-intensive to write and assert such cases by hand, so scripting and recording these interactions often yields much better test coverage at a fraction of the cost.

Internally, the testfile crate is used to manage golden files.

Examples

For real-world examples, see e.g.:

• toyDB Raft: distributed consensus cluster.
• toyDB MVCC: ACID transactions.
• testscript parser: testscript uses itself to test its parser and runner.

Below is a basic example, testing the Rust standard library’s BTreeMap.

# Tests the Rust standard library BTreeMap.

# Get and range returns nothing for an empty map.
get foo
range
---
get -> None

# Inserting keys out of order will return them in order. Silence the insert
# output with ().
(insert b=2 a=1 c=3)
range
---
a=1
b=2
c=3

# Getting a key returns its value.
get b
---
get -> Some("2")

# Bounded scans, where the end is exclusive.
range b
---
b=2
c=3

range a c
---
a=1
b=2

# An end bound less than the start bound panics. Expect the failure with !.
!range b a
---
Panic: range start is greater than range end in BTreeMap

# Replacing a key updates the value and returns the old one.
insert b=foo
get b
---
insert -> Some("2")
get -> Some("foo")

Syntax

Blocks

A testscript consists of one or more input/output blocks. Each block has a set of one or more input commands on individual lines (empty or comment lines are ignored), a --- separator, and arbitrary output terminated by an empty line. A minimal testscript with two blocks might be:

command
---
output

command 1
command 2
---
output 1
output 2

Commands

A Command must have a command name, which can be any arbitrary string, e.g.:

command
"command with space and 🚀"
---

It may additionally have:

• Arguments: any number of space-separated arguments. These have a string value, and optionally also a string key as key=value. Keys and values can be empty, and duplicate keys are allowed by the parser (the runner can handle this as desired).

  command argument key=value
  command "argument with space" "key with space"="value with space"
  command "" key=  # Empty argument values.
  ---

• Prefix: an optional :-terminated string prefix before the command. The command’s output will be given the same prefix. The prefix can be used by the test runner, e.g. to signify two different clients.

  client1: put key=value
  client2: get key
  ---
  client1: put ok
  client2: get key=value

• Silencing: a command wrapped in () will have its output suppressed. This can be useful e.g. for setup commands whose output are not of interest in the current test case and would only add noise.

  echo foo
  (echo bar)
  ---
  foo

• Failure: if ! precedes the command, it is expected to fail with an error or panic, and the failure message is used as output. If the command unexpectedly succeeds, the test fails. If the line contains other symbols before the command name (e.g. a prefix or silencing), the ! must be used immediately before the command name.

  ! command error=foo
  prefix: ! command panic=bar
  (!command error=foo)
  ---
  Error: foo
  prefix: Panic: bar

• Tags: an optional comma- or space-separated list of tags (strings) enclosed in [] before or after the command and arguments. This can be used by the runner e.g. to modify the execution of a command.

  command [tag]
  command arg key=value [a,b c]
  [tag] command
  prefix:[tag]!> command arg
  ---

• Literal: if > precedes the command, the entire rest of the line is taken to be the command name (except leading whitespace). Arguments, tags, comments, and any other special characters are ignored and used as-is. As a special case (currently only with >), lines can fragment multiple lines by ending the line with .

  > a long command name including key=value, [tags], # a comment and
  > exclamation!
  prefix: [tag] ! > a long, failing command with tags and a prefix
  ---
  
  > a very \
  long line \
  with line \
  continuation
  ---

Output

The command output following a --- separator can contain any arbitrary Unicode string until an empty line (or end of file). If the command output contains empty lines, the entire output will automatically be prefixed with > . If no commands in a block yield any output, it defaults to “ok”.

echo "output 1"
echo "output 2"
---
output 1
output 2

echo "Paragraph 1.\n\nParagraph 2."
---
> Paragraph 1.
>
> Paragraph 2.

echo "输出\n# Comment\n🚀"
---
输出
# Comment
🚀

Comments

Comments begin with # or // and run to the end of the line.

# This is a comment.
// As is this.
command argument # Comments can follow commands too.
---

Strings

Unquoted strings can only contain alphanumeric ASCII characters [a-zA-Z0-9] and a handful of special characters: _ - . / @ (only _ at the start of a string).

Strings can be quoted using " or ', in which case they can contain arbitrary Unicode characters. \ is used as an escape character, both to escape quotes \" and \' as well as itself \\, and also \0 (null), \n (newline), \r (carriage return), and \t (tab). \x can be used to represent arbitrary hexadecimal bytes (e.g. \x7a) and \u{} can be used to represent arbitrary Unicode characters (e.g. \u{1f44b})

Managing State

The runner is free to manage internal state as desired. If it is mod, it is recommended to persist state within a single testscript (across commands and blocks), but not across testscripts since this can be hard to reason about and depend on the execution order of scripts. This is most easily done by instantiating a new runner for each script.

Initial state setup should generally be done via explicit setup commands, to make it more discoverable.

Running All Scripts in a Directory

External crates can be used to automatically generate and run individual tests for each testscript in a directory. For example, the test_each_file

Runners have various hooks that will be called during script execution: Runner::start_script, Runner::end_script, Runner::start_block, Runner::end_block, Runner::start_command, and Runner::end_command. These can be used e.g. for initial setup, invariant assertions, or to output the current state.

Structs

Argument

A command argument.

ArgumentConsumer

Helper for argument processing, by returning and removing arguments on demand.

Command

A command.

Traits

Runner

Runs testscript commands, returning their output.

Functions

generate

Generates output for a testscript input, without comparing them.

run

run_path

Runs a testscript at the given path.

try_run