goldenscript 0.7.0

//! This crate provides the Goldenscript testing framework, loosely based on
//! Cockroach Labs' [`datadriven`](https://github.com/cockroachdb/datadriven)
//! framework for Go. It combines several testing techniques that make it easy
//! and efficient to write and update test cases:
//!
//! * [Golden master testing](https://en.wikipedia.org/wiki/Characterization_test)
//!   (aka characterization testing or historical oracle)
//! * [Data-driven testing](https://en.wikipedia.org/wiki/Data-driven_testing)
//!   (aka table-driven testing or parameterized testing)
//! * [Keyword-driven testing](https://en.wikipedia.org/wiki/Keyword-driven_testing)
//!
//! A goldenscript is a plain text file that contains a set of arbitrary input
//! commands and their expected text output, separated by `---`:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command argument
//! command key=value
//! ---
//! output
//! ```
//!
//! The commands are executed by a provided [`Runner`]. The expected output is
//! usually not written by hand, but instead generated by running tests with the
//! environment variable `UPDATE_GOLDENFILES=1`:
//!
//! ```sh
//! $ UPDATE_GOLDENFILES=1 cargo test
//! ```
//!
//! The files are then verified by inspection and checked in to version control.
//! Tests will fail with a diff if they don't match the expected output.
//!
//! This approach is particularly useful when testing complex stateful systems,
//! such as database operations, network protocols, or language parsing. It can
//! be tedious and labor-intensive to write and assert such cases by hand, so
//! scripting and recording these interactions often yields much better test
//! coverage at a fraction of the cost.
//!
//! Internally, the
//! [`goldenfile`](https://docs.rs/goldenfile/latest/goldenfile/) crate is used
//! to manage golden files.
//!
//! # Examples
//!
//! For real-world examples, see e.g.:
//!
//! * [toyDB Raft](https://github.com/erikgrinaker/toydb/tree/master/src/raft/testscripts/node):
//!   distributed consensus cluster.
//! * [toyDB MVCC](https://github.com/erikgrinaker/toydb/tree/master/src/storage/testscripts/mvcc):
//!   ACID transactions.
//! * [goldenscript parser](https://github.com/erikgrinaker/goldenscript/tree/main/tests/scripts):
//!   Goldenscript uses itself to test its parser and runner.
//!
//! Below is a basic example, testing the Rust standard library's
//! [`BTreeMap`](https://doc.rust-lang.org/std/collections/struct.BTreeMap.html).
//!
//! ```yaml
//! # Tests the Rust standard library BTreeMap.
//!
//! # Get and range returns nothing for an empty map.
//! get foo
//! range
//! ---
//! get → None
//!
//! # Inserting keys out of order will return them in order. Silence the insert
//! # output with ().
//! (insert b=2 a=1 c=3)
//! range
//! ---
//! a=1
//! b=2
//! c=3
//!
//! # Getting a key returns its value.
//! get b
//! ---
//! get → Some("2")
//!
//! # Bounded scans, where the end is exclusive.
//! range b
//! ---
//! b=2
//! c=3
//!
//! range a c
//! ---
//! a=1
//! b=2
//!
//! # An end bound less than the start bound panics. Expect the failure with !.
//! !range b a
//! ---
//! Panic: range start is greater than range end in BTreeMap
//!
//! # Replacing a key updates the value and returns the old one.
//! insert b=foo
//! get b
//! ---
//! insert → Some("2")
//! get → Some("foo")
//! ```
//!
//! The corresponding runner for this script:
//!
//! ```
//! # use std::error::Error;
//! # use std::fmt::Write as _;
//! #[derive(Default)]
//! struct BTreeMapRunner {
//!     map: std::collections::BTreeMap<String, String>,
//! }
//!
//! impl goldenscript::Runner for BTreeMapRunner {
//!     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
//!         let mut output = String::new();
//!         match command.name.as_str() {
//!             // get KEY: fetches the value of the given key, or None if it does not exist.
//!             "get" => {
//!                 let mut args = command.consume_args();
//!                 let key = &args.next_pos().ok_or("key not given")?.value;
//!                 args.reject_rest()?;
//!                 let value = self.map.get(key);
//!                 writeln!(output, "get → {value:?}")?;
//!             }
//!
//!             // insert KEY=VALUE...: inserts the given key/value pairs, returning the old value.
//!             "insert" => {
//!                 let mut args = command.consume_args();
//!                 for arg in args.rest_key() {
//!                     let old = self.map.insert(arg.key.clone().unwrap(), arg.value.clone());
//!                     writeln!(output, "insert → {old:?}")?;
//!                 }
//!                 args.reject_rest()?;
//!             }
//!
//!             // range [FROM] [TO]: iterates over the key/value pairs in the range from..to.
//!             "range" => {
//!                 use std::ops::Bound::*;
//!                 let mut args = command.consume_args();
//!                 let from = args.next_pos().map(|a| Included(a.value.clone())).unwrap_or(Unbounded);
//!                 let to = args.next_pos().map(|a| Excluded(a.value.clone())).unwrap_or(Unbounded);
//!                 args.reject_rest()?;
//!                 for (key, value) in self.map.range((from, to)) {
//!                     writeln!(output, "{key}={value}")?;
//!                 }
//!             }
//!
//!             name => return Err(format!("invalid command {name}").into()),
//!         };
//!         Ok(output)
//!     }
//! }
//!
//! #[test]
//! fn btreemap() {
//!     goldenscript::run(&mut BTreeMapRunner::default(), "btreemap").expect("goldenscript failed")
//! }
//! ```
//!
//! # Syntax
//!
//! ## Blocks
//!
//! A goldenscript consists of one or more input/output blocks. Each block has a
//! set of one or more input commands on individual lines (empty or comment
//! lines are ignored), a `---` separator, and arbitrary output terminated by an
//! empty line. A minimal goldenscript with two blocks might be:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command 1
//! command 2
//! ---
//! output 1
//! output 2
//! ```
//!
//! ## Commands
//!
//! A [`Command`] must have a command name, which can be any arbitrary
//! [string](#strings), e.g.:
//!
//! ```text
//! command
//! "command with space and 🚀"
//! ---
//! ```
//!
//! It may additionally have:
//!
//! * [**Arguments:**](Argument) any number of space-separated arguments.
//!   These have a string [value](Argument::value), and optionally also a string
//!   [key](Argument::key) as `key=value`. Keys and values can be empty, and
//!   duplicate keys are allowed by the parser (the runner can handle this as
//!   desired).
//!
//!     ```text
//!     command argument key=value
//!     command "argument with space" "key with space"="value with space"
//!     command "" key=  # Empty argument values.
//!     ---
//!     ```
//!
//! * [**Prefix:**](Command::prefix) an optional :-terminated string prefix
//!   before the command. The command's output will be given the same prefix.
//!   The prefix can be used by the test runner, e.g. to signify two different
//!   clients.
//!
//!     ```text
//!     client1: put key=value
//!     client2: get key
//!     ---
//!     client1: put ok
//!     client2: get key=value
//!     ```
//!
//! * [**Silencing:**](Command::silent) a command wrapped in `()` will have its
//!   output suppressed. This can be useful e.g. for setup commands whose output
//!   are not of interest in the current test case and would only add noise.
//!
//!     ```text
//!     echo foo
//!     (echo bar)
//!     ---
//!     foo
//!     ```
//!
//! * [**Failure:**](Command::fail) if `!` precedes the command, it is expected
//!   to fail with an error or panic, and the failure message is used as output.
//!   If the command unexpectedly succeeds, the test fails. If the line contains
//!   other symbols before the command name (e.g. a prefix or silencing), the
//!   `!` must be used immediately before the command name.
//!
//!     ```text
//!     ! command error=foo
//!     prefix: ! command panic=bar
//!     (!command error=foo)
//!     ---
//!     Error: foo
//!     prefix: Panic: bar
//!     ```
//!
//! * [**Tags:**](Command::tags) an optional comma- or space-separated list of
//!   tags (strings) enclosed in [] before or after the command and arguments.
//!   This can be used by the runner e.g. to modify the execution of a command.
//!
//!     ```text
//!     command [tag]
//!     command arg key=value [a,b c]
//!     [tag] command
//!     prefix:[tag]!> command arg
//!     ---
//!     ```
//!
//!  * **Literal:** if `>` precedes the command, the entire rest of the line is
//!    taken to be the command name (except leading whitespace). Arguments,
//!    tags, comments, and any other special characters are ignored and used
//!    as-is. As a special case (currently only with `>`), lines can span
//!    multiple lines by ending the line with \.
//!
//!    ```text
//!    > a long command name including key=value, [tags], # a comment and exclamation!
//!    prefix: [tag] ! > a long, failing command with tags and a prefix
//!    ---
//!
//!    > a very \
//!    long line \
//!    with line \
//!    continuation
//!    ---
//!    ```
//!
//! ## Output
//!
//! The command output following a `---` separator can contain any arbitrary
//! Unicode string until an empty line (or end of file). If the command output
//! contains empty lines, the entire output will automatically be prefixed with
//! `> `. If no commands in a block yield any output, it defaults to "ok".
//!
//! ```text
//! echo "output 1"
//! echo "output 2"
//! ---
//! output 1
//! output 2
//!
//! echo "Paragraph 1.\n\nParagraph 2."
//! ---
//! > Paragraph 1.
//! >
//! > Paragraph 2.
//!
//! echo "输出\n# Comment\n🚀"
//! ---
//! 输出
//! # Comment
//! 🚀
//! ```
//!
//! ## Comments
//!
//! Comments begin with `#` or `//` and run to the end of the line.
//!
//! ```text
//! # This is a comment.
//! // As is this.
//! command argument # Comments can follow commands too.
//! ---
//! ```
//!
//! ## Strings
//!
//! Unquoted strings can only contain alphanumeric ASCII characters
//! `[a-zA-Z0-9]` and a handful of special characters: `_ - . / @`
//! (only `_` at the start of a string).
//!
//! Strings can be quoted using `"` or `'`, in which case they can contain
//! arbitrary Unicode characters. `\` is used as an escape character, both to
//! escape quotes `\"` and `\'` as well as itself `\\`, and also `\0` (null),
//! `\n` (newline), `\r` (carriage return), and `\t` (tab). `\x` can be used to
//! represent arbitrary hexadecimal bytes (e.g. `\x7a`) and `\u{}` can be used
//! to represent arbitrary Unicode characters (e.g. `\u{1f44b}`)
//!
//! ```text
//! string
//! "string with spaces and \"quotes\""
//! '字符串'
//! ---
//! ```
//!
//! # Writing Tests
//!
//! In the simplest case, a goldenscript test might be:
//!
//! ```no_run
//! # use std::error::Error;
//! struct Runner;
//!
//! impl goldenscript::Runner for Runner {
//!     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
//!         match command.name.as_str() {
//!             "echo" => {
//!                 let lines: Vec<&str> = command.args.iter().map(|a| a.value.as_str()).collect();
//!                 Ok(lines.join("\n"))
//!             }
//!             name => return Err(format!("invalid command {name}").into())
//!         }
//!     }
//! }
//!
//! #[test]
//! fn test() -> std::io::Result<()> {
//!     goldenscript::run(&mut Runner, "tests/scripts/test")
//! }
//! ```
//!
//! ## Argument Processing
//!
//! Arguments can be processed manually via [`Command::args`], or using the
//! [`Command::consume_args()`] helper which simplifies common argument
//! handling. For example:
//!
//! ```
//! # use std::error::Error;
//! # struct Runner;
//! # impl Runner {
//! #   fn send(&self, ids: &[u32], message: &str, retry: bool) -> Result<String, Box<dyn Error>> {
//! #     Ok(String::new())
//! #   }
//! # }
//! #
//! impl goldenscript::Runner for Runner {
//!     /// Implement a send command, which sends a string message to a list
//!     /// of nodes, optionally retrying.
//!     ///
//!     /// send [retry=BOOL] MESSAGE ID...
//!     ///
//!     /// Example: send foo 1 2 3
//!     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
//!         if command.name != "send" {
//!             return Err(format!("invalid command {}", command.name).into())
//!         }
//!
//!         let mut args = command.consume_args();
//!
//!         // The first positional argument is a required string message.
//!         let message = &args.next_pos().ok_or("message not given")?.value;
//!
//!         // The remaining positional arguments are numeric node IDs.
//!         let ids: Vec<u32> = args.rest_pos().iter().map(|a| a.parse()).collect::<Result<_, _>>()?;
//!         if ids.is_empty() {
//!             return Err("no node IDs given".into())
//!         }
//!
//!         // An optional retry=bool key/value argument can also be given.
//!         let retry: bool = args.lookup_parse("retry")?.unwrap_or(false);
//!
//!         // Any other arguments that haven't been processed above should error.
//!         args.reject_rest()?;
//!
//!         // Execute the send.
//!         self.send(&ids, message, retry)
//!     }
//! }
//! ```
//!
//! ## Managing State
//!
//! The runner is free to manage internal state as desired. If it is stateful,
//! it is recommended to persist state within a single goldenscript (across
//! commands and blocks), but not across goldenscripts since this can be hard to
//! reason about and depend on the execution order of scripts. This is most
//! easily done by instantiating a new runner for each script.
//!
//! Initial state setup should generally be done via explicit setup commands, to
//! make it more discoverable.
//!
//! ## Running All Scripts in a Directory
//!
//! External crates can be used to automatically generate and run individual
//! tests for each goldenscript in a directory. For example, the
//! [`test_each_file`](https://docs.rs/test_each_file/latest/test_each_file/)
//! crate:
//!
//! ```no_run
//! # use std::error::Error;
//! # struct Runner;
//! #
//! # impl goldenscript::Runner for Runner {
//! #     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> { todo!() }
//! # }
//! use test_each_file::test_each_path;
//!
//! test_each_path! { in "tests/scripts" as scripts => test_goldenscript }
//!
//! fn test_goldenscript(path: &std::path::Path) {
//!     goldenscript::run(&mut Runner, path).unwrap()
//! }
//! ```
//!
//! ## Hooks
//!
//! Runners have various hooks that will be called during script execution:
//! [`Runner::start_script`], [`Runner::end_script`], [`Runner::start_block`],
//! [`Runner::end_block`], [`Runner::start_command`], and
//! [`Runner::end_command`]. These can be used e.g. for initial setup, invariant
//! assertions, or to output the current state.

#![warn(clippy::all)]

mod command;
mod parser;
mod runner;

pub use command::{Argument, ArgumentConsumer, Command};
pub use runner::{generate, run, Runner};