1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335
//! This crate provides the Goldenscript testing framework, loosely based on
//! Cockroach Labs' [`datadriven`](https://github.com/cockroachdb/datadriven)
//! framework for Go. It combines several testing techniques that make it easy
//! and efficient to write and update test cases:
//!
//! * [Golden master testing](https://en.wikipedia.org/wiki/Characterization_test)
//! (aka characterization testing or historical oracle)
//! * [Data-driven testing](https://en.wikipedia.org/wiki/Data-driven_testing)
//! (aka table-driven testing or parameterized testing)
//! * [Keyword-driven testing](https://en.wikipedia.org/wiki/Keyword-driven_testing)
//!
//! A goldenscript is a plain text file that contains a set of arbitrary input
//! commands and their expected text output, separated by `---`:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command argument key=value
//! ---
//! output
//! ```
//!
//! The commands are executed by a provided [`Runner`]. The expected output is
//! usually not written by hand, but instead generated by running tests with the
//! environment variable `UPDATE_GOLDENFILES=1` and then verified by inspection
//! before it is checked in to version control. Tests will fail with a diff if
//! they don't match the expected output.
//!
//! This approach is particularly useful when testing complex stateful systems,
//! such as computer language parsing, operations on a key/value store,
//! concurrent transactions in a SQL database, or communication between a
//! cluster of Raft nodes. It can be very tedious and labor-intensive to write
//! and assert such cases by hand, so scripting and recording these interactions
//! often yields much better test coverage at a fraction of the cost.
//!
//! Internally, the
//! [`goldenfile`](https://docs.rs/goldenfile/latest/goldenfile/) crate is used
//! to manage golden files.
//!
//! # Example
//!
//! We'll test the [`dateparser`](https://docs.rs/dateparser/latest/dateparser/)
//! crate which parses timestamp strings in various formats.
//!
//! We write an initial goldenscript `tests/scripts/dateparser` containing test
//! cases for a `parse` command. A goldenscript may contain multiple commands,
//! either as individual input/output blocks or grouped together (which will
//! append their output). The input ends with a `---` separator, and the output
//! ends with a blank line. Note that we don't yet specify any expected output
//! after the `---` separator, this will be autogenerated later.
//!
//! ```text
//! parse 2024-04-30
//! ---
//!
//! # Test various date formats.
//! parse 2024-Apr-30
//! parse 2024.04.30
//! parse 04/30/2024
//! ---
//!
//! # Test some error cases.
//! parse 30.04.2024
//! parse 30/04/2024
//! parse 30/04/24
//! ---
//!
//! # Strings containing special characters must be quoted using " or '.
//! parse "2024-04-30 11:55:32"
//! parse '2024年04月30日11时55分32秒'
//! ---
//! ```
//!
//! We write a runner that recognizes this `parse` command and its timestamp
//! argument, and outputs the parsed date/time in RFC 3339 format. We also add a
//! test using it to run the goldenscript `tests/scripts/dateparser`.
//!
//! ```no_run
//! struct DateParserRunner;
//!
//! impl goldenscript::Runner for DateParserRunner {
//! fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> {
//! // Only accept a parse command with a single argument.
//! if command.name != "parse" {
//! return Err(format!("invalid command {}", command.name).into())
//! }
//! if command.args.len() != 1 {
//! return Err("parse takes 1 argument".into())
//! }
//!
//! // Parse the timestamp, and output the RFC 3339 timestamp or error string.
//! let input = &command.args[0].value;
//! match dateparser::parse_with(input, &chrono::offset::Utc, chrono::NaiveTime::MIN) {
//! Ok(datetime) => Ok(datetime.to_rfc3339()),
//! Err(error) => Ok(format!("Error: {error}")),
//! }
//! }
//! }
//!
//! #[test]
//! fn dateparser() -> std::io::Result<()> {
//! goldenscript::run(&mut DateParserRunner, "tests/scripts/dateparser")
//! }
//! ```
//!
//! Running `UPDATE_GOLDENFILES=1 cargo test` will populate the file with the
//! runner's output. We verify by inspection that it is correct. Later runs of
//! `cargo test` will assert that the output matches the file.
//!
//! ```text
//! parse 2024-04-30
//! ---
//! 2024-04-30T00:00:00+00:00
//!
//! # Test various other date formats.
//! parse 2024-Apr-30
//! parse 2024.04.30
//! parse 04/30/2024
//! ---
//! 2024-04-30T00:00:00+00:00
//! 2024-04-30T00:00:00+00:00
//! 2024-04-30T00:00:00+00:00
//!
//! # Test some error cases.
//! parse 30.04.2024
//! parse 30/04/2024
//! parse 30/04/24
//! ---
//! Error: 30.04.2024 did not match any formats.
//! Error: 30/04/2024 did not match any formats.
//! Error: 30/04/24 did not match any formats.
//!
//! # Strings containing special characters must be quoted using " or '.
//! parse "2024-04-30 11:55:32"
//! parse '2024年04月30日11时55分32秒'
//! ---
//! 2024-04-30T11:55:32+00:00
//! 2024-04-30T11:55:32+00:00
//! ```
//!
//! # Syntax
//!
//! ## Blocks
//!
//! A goldenscript consists of one or more input/output blocks. Each block has a
//! set of one or more input commands on individual lines (empty or comment
//! lines are ignored), a `---` separator, and arbitrary output terminated by an
//! empty line. A minimal goldenscript with two blocks might be:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command 1
//! command 2
//! ---
//! output 1
//! output 2
//! ```
//!
//! ## Commands
//!
//! A [`Command`] must have a command name, which can be any arbitrary non-empty
//! [string](#strings), e.g.:
//!
//! ```text
//! command
//! "command with space and 🚀"
//! ---
//! ```
//!
//! It may additionally have:
//!
//! * [**Arguments:**](Argument) any number of space-separated arguments.
//! These have a string [value](Argument::value), and optionally also a string
//! [key](Argument::key) as `key=value`. Values can be empty, and duplicate
//! keys are allowed by the parser (the runner can handle this as desired).
//!
//! ```text
//! command argument key=value
//! command "argument with space" "key with space"="value with space"
//! command "" key= # Empty argument values.
//! ---
//! ```
//!
//! * [**Prefix:**](Command::prefix) an optional :-terminated string prefix
//! before the command name. The command's output will be given the same
//! prefix. The prefix can be used by the test runner, e.g. to signify two
//! different clients.
//!
//! ```text
//! client1: put key=value
//! client2: get key
//! ---
//! client1: put ok
//! client2: get key=value
//! ```
//!
//! * [**Silencing:**](Command::silent) a command wrapped in `()` will have its
//! output suppressed. This can be useful e.g. for setup commands whose output
//! are not of interest in the current test case and would only add noise.
//!
//! ```text
//! echo foo
//! (echo bar)
//! ---
//! foo
//! ```
//!
//! ## Output
//!
//! The command output following a `---` separator can contain any arbitrary
//! Unicode string until an empty line (or end of file). If the command output
//! contains empty lines, the entire output will automatically be prefixed with
//! `> `. If no commands in a block yield any output, it defaults to "ok".
//!
//! ```text
//! echo "output 1"
//! echo "output 2"
//! ---
//! output 1
//! output 2
//!
//! echo "Paragraph 1.\n\nParagraph 2."
//! ---
//! > Paragraph 1.
//! >
//! > Paragraph 2.
//!
//! echo "输出\n# Comment\n🚀"
//! ---
//! 输出
//! # Comment
//! 🚀
//! ```
//!
//! ## Comments
//!
//! Comments begin with `#` or `//` and run to the end of the line.
//!
//! ```text
//! # This is a comment.
//! // As is this.
//! command argument # Comments can follow commands too.
//! ---
//! ```
//!
//! ## Strings
//!
//! Unquoted strings can only contain alphanumeric ASCII characters
//! `[a-zA-Z0-9]` and a handful of special characters: `_ - . /`
//! (only `_` at the start of a string).
//!
//! Strings can be quoted using `"` or `'`, in which case they can contain
//! arbitrary Unicode characters. `\` is used as an escape character, both to
//! escape quotes `\"` and `\'` as well as itself `\\`, and also `\0` (null),
//! `\n` (newline), `\r` (carriage return), and `\t` (tab).
//!
//! ```text
//! string
//! "string with spaces and \"quotes\""
//! '字符串'
//! ---
//! ```
//!
//! # Writing Tests
//!
//! In the simplest case, a goldenscript test is:
//!
//! ```no_run
//! struct Runner;
//!
//! impl goldenscript::Runner for Runner {
//! fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> {
//! todo!();
//! }
//! }
//!
//! #[test]
//! fn test() -> std::io::Result<()> {
//! goldenscript::run(&mut Runner, "tests/scripts/test")
//! }
//! ```
//!
//! ## Managing State
//!
//! The runner is free to manage internal state as desired. If it is stateful,
//! it is recommended to persist state within a single goldenscript (across
//! commands and blocks), but not across goldenscripts since this can be hard to
//! reason about and depend on the execution order of scripts. This is most
//! easily done by instantiating a new runner for each script.
//!
//! Initial state setup should generally be done via explicit setup commands, to
//! make it more discoverable.
//!
//! ## Running All Scripts in a Directory
//!
//! External crates can be used to automatically generate and run individual
//! tests for each goldenscript in a directory. For example, the
//! [`test_each_file`](https://docs.rs/test_each_file/latest/test_each_file/)
//! crate:
//!
//! ```no_run
//! # struct Runner;
//! #
//! # impl goldenscript::Runner for Runner {
//! # fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> { todo!() }
//! # }
//! use test_each_file::test_each_path;
//!
//! test_each_path! { in "tests/scripts" as scripts => test_goldenscript }
//!
//! fn test_goldenscript(path: &std::path::Path) {
//! goldenscript::run(&mut Runner, path).unwrap()
//! }
//! ```
//!
//! ## Hooks
//!
//! Runners have various hooks that will be called during script execution:
//! [`Runner::start_script`], [`Runner::end_script`], [`Runner::start_block`],
//! and [`Runner::end_block`]. These can be used e.g. for initial setup,
//! invariant assertions, or to output the current state.
#![warn(clippy::all)]
mod command;
mod parser;
mod runner;
pub use command::{Argument, Command};
pub use runner::{generate, run, Runner};