reifydb_testing/testscript/

mod.rs

// SPDX-License-Identifier: Apache-2.0
// Copyright (c) 2025 ReifyDB

// This file includes portions of code from https://github.com/erikgrinaker/goldenscript (Apache 2 License).
// Original Apache 2 License Copyright (c) erikgrinaker 2024.

//! This crate provides the testscript testing framework, loosely based on
//! Cockroach Labs' [`datadriven`](https://github.com/cockroachdb/datadriven)
//! framework for Go. It combines several testing techniques that make it easy
//! and efficient to write and update test cases:
//!
//! * [Golden master testing](https://en.wikipedia.org/wiki/Characterization_test) (aka characterization testing or
//!   historical oracle)
//! * [Data-driven testing](https://en.wikipedia.org/wiki/Data-driven_testing) (aka table-driven testing or
//!   parameterized testing)
//! * [Keyword-driven testing](https://en.wikipedia.org/wiki/Keyword-driven_testing)
//!
//! A testscript is a plain text file that contains a set of arbitrary input
//! commands and their expected text output, separated by `---`:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command argument
//! command key=value
//! ---
//! output
//! ```
//!
//! The commands are executed by a provided [`Runner`]. The expected output is
//! usually not written by hand, but instead generated by running tests with the
//! environment variable `UPDATE_TESTFILES=1`:
//!
//! ```sh
//! $ UPDATE_TESTFILES=1 cargo test
//! ```
//!
//! The files are then verified by inspection and checked in to version control.
//! Tests will fail with a diff if they don't match the expected output.
//!
//! This approach is particularly useful when testing complex
//! systems, such as database operations, network protocols, or language
//! parsing. It can be tedious and labor-intensive to write and assert such
//! cases by hand, so scripting and recording these interactions often yields
//! much better test coverage at a fraction of the cost.
//!
//! Internally, the
//! [`testfile`](https://docs.rs/testfile/latest/testfile/) crate is used
//! to manage golden files.
//!
//! # Examples
//!
//! For real-world examples, see e.g.:
//!
//! * [toyDB Raft](https://github.com/erikgrinaker/toydb/tree/master/src/raft/testscripts/node): distributed consensus
//!   cluster.
//! * [toyDB MVCC](https://github.com/erikgrinaker/toydb/tree/master/src/store/testscripts/mvcc): ACID transactions.
//! * [testscript parser](https://github.com/erikgrinaker/testscript/tree/main/tests/scripts): testscript uses itself to
//!   test its parser and runner.
//!
//! Below is a basic example, testing the Rust transaction library's
//! [`BTreeMap`](https://doc.rust-lang.org/std/collections/struct.BTreeMap.html).
//!
//! ```yaml
//! # Tests the Rust transaction library BTreeMap.
//!
//! # Get and range returns nothing for an empty map.
//! get foo
//! range
//! ---
//! get -> None
//!
//! # Inserting keys out of order will return them in order. Silence the insert
//! # output with ().
//! (insert b=2 a=1 c=3)
//! range
//! ---
//! a=1
//! b=2
//! c=3
//!
//! # Getting a key returns its value.
//! get b
//! ---
//! get -> Some("2")
//!
//! # Bounded scans, where the end is exclusive.
//! range b
//! ---
//! b=2
//! c=3
//!
//! range a c
//! ---
//! a=1
//! b=2
//!
//! # An end bound less than the start bound panics. Expect the failure with !.
//! !range b a
//! ---
//! Panic: range start is greater than range end in BTreeMap
//!
//! # Replacing a key updates the value and returns the old one.
//! insert b=foo
//! get b
//! ---
//! insert -> Some("2")
//! get -> Some("foo")
//! ```
//!
//! # Syntax
//!
//! ## Blocks
//!
//! A testscript consists of one or more input/output blocks. Each block has a
//! set of one or more input commands on individual lines (empty or comment
//! lines are ignored), a `---` separator, and arbitrary output terminated by an
//! empty line. A minimal testscript with two blocks might be:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command 1
//! command 2
//! ---
//! output 1
//! output 2
//! ```
//!
//! ## Commands
//!
//! A [`Command`] must have a command name, which can be any arbitrary
//! [string](#strings), e.g.:
//!
//! ```text
//! command
//! "command with space and 🚀"
//! ---
//! ```
//!
//! It may additionally have:
//!
//! * [**Arguments:**](Argument) any number of space-separated arguments. These have a string [value](Argument::value),
//!   and optionally also a string [key](Argument::key) as `key=value`. Keys and values can be empty, and duplicate keys
//!   are allowed by the parser (the runner can handle this as desired).
//!
//!     ```text
//!     command argument key=value
//!     command "argument with space" "key with space"="value with space"
//!     command "" key=  # Empty argument values.
//!     ---
//!     ```
//!
//! * [**Prefix:**](Command::prefix) an optional :-terminated string prefix before the command. The command's output
//!   will be given the same prefix. The prefix can be used by the test runner, e.g. to signify two different clients.
//!
//!     ```text
//!     client1: put key=value
//!     client2: get key
//!     ---
//!     client1: put ok
//!     client2: get key=value
//!     ```
//!
//! * [**Silencing:**](Command::silent) a command wrapped in `()` will have its output suppressed. This can be useful
//!   e.g. for setup commands whose output are not of interest in the current test case and would only add noise.
//!
//!     ```text
//!     echo foo
//!     (echo bar)
//!     ---
//!     foo
//!     ```
//!
//! * [**Failure:**](Command::fail) if `!` precedes the command, it is expected to fail with an error or panic, and the
//!   failure message is used as output. If the command unexpectedly succeeds, the test fails. If the line contains
//!   other symbols before the command name (e.g. a prefix or silencing), the `!` must be used immediately before the
//!   command name.
//!
//!     ```text
//!     ! command error=foo
//!     prefix: ! command panic=bar
//!     (!command error=foo)
//!     ---
//!     Error: foo
//!     prefix: Panic: bar
//!     ```
//!
//! * [**Tags:**](Command::tags) an optional comma- or space-separated list of tags (strings) enclosed in [] before or
//!   after the command and arguments. This can be used by the runner e.g. to modify the execution of a command.
//!
//!     ```text
//!     command [tag]
//!     command arg key=value [a,b c]
//!     [tag] command
//!     prefix:[tag]!> command arg
//!     ---
//!     ```
//!
//!  * **Literal:** if `>` precedes the command, the entire rest of the line is taken to be the command name (except
//!    leading whitespace). Arguments, tags, comments, and any other special characters are ignored and used as-is. As a
//!    special case (currently only with `>`), lines can fragment multiple lines by ending the line with \.
//!
//!    ```text
//!    > a long command name including key=value, [tags], # a comment and
//!    > exclamation!
//!    prefix: [tag] ! > a long, failing command with tags and a prefix
//!    ---
//!
//!    > a very \
//!    long line \
//!    with line \
//!    continuation
//!    ---
//!    ```
//!
//! ## Output
//!
//! The command output following a `---` separator can contain any arbitrary
//! Unicode string until an empty line (or end of file). If the command output
//! contains empty lines, the entire output will automatically be prefixed with
//! `> `. If no commands in a block yield any output, it defaults to "ok".
//!
//! ```text
//! echo "output 1"
//! echo "output 2"
//! ---
//! output 1
//! output 2
//!
//! echo "Paragraph 1.\n\nParagraph 2."
//! ---
//! > Paragraph 1.
//! >
//! > Paragraph 2.
//!
//! echo "输出\n# Comment\n🚀"
//! ---
//! 输出
//! # Comment
//! 🚀
//! ```
//!
//! ## Comments
//!
//! Comments begin with `#` or `//` and run to the end of the line.
//!
//! ```text
//! # This is a comment.
//! // As is this.
//! command argument # Comments can follow commands too.
//! ---
//! ```
//!
//! ## Strings
//!
//! Unquoted strings can only contain alphanumeric ASCII characters
//! `[a-zA-Z0-9]` and a handful of special characters: `_ - . / @`
//! (only `_` at the start of a string).
//!
//! Strings can be quoted using `"` or `'`, in which case they can contain
//! arbitrary Unicode characters. `\` is used as an escape character, both to
//! escape quotes `\"` and `\'` as well as itself `\\`, and also `\0` (null),
//! `\n` (newline), `\r` (carriage return), and `\t` (tab). `\x` can be used to
//! represent arbitrary hexadecimal bytes (e.g. `\x7a`) and `\u{}` can be used
//! to represent arbitrary Unicode characters (e.g. `\u{1f44b}`)
//! ## Managing State
//!
//! The runner is free to manage internal state as desired. If it is stateful,
//! it is recommended to persist state within a single testscript (across
//! commands and blocks), but not across testscripts since this can be hard to
//! reason about and depend on the execution order of scripts. This is most
//! easily done by instantiating a new runner for each script.
//!
//! Initial state setup should generally be done via explicit setup commands, to
//! make it more discoverable.
//!
//! ## Running All Scripts in a Directory
//!
//! External crates can be used to automatically generate and run individual
//! tests for each testscript in a directory. For example, the
//! [`test_each_file`](https://docs.rs/test_each_file/latest/test_each_file/)
//!
//! Runners have various hooks that will be called during script execution:
//! [`Runner::start_script`], [`Runner::end_script`], [`Runner::start_block`],
//! [`Runner::end_block`], [`Runner::start_command`], and
//! [`Runner::end_command`]. These can be used e.g. for initial setup, invariant
//! assertions, or to output the current state.

pub mod command;
pub mod parser;
pub mod runner;