1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
//! This crate provides the Goldenscript testing framework, loosely based on
//! Cockroach Labs' [`datadriven`](https://github.com/cockroachdb/datadriven)
//! framework for Go. It combines several testing techniques that make it easy
//! and efficient to write and update test cases:
//!
//! * [Golden master testing](https://en.wikipedia.org/wiki/Characterization_test)
//!   (aka characterization testing or historical oracle)
//! * [Data-driven testing](https://en.wikipedia.org/wiki/Data-driven_testing)
//!   (aka table-driven testing or parameterized testing)
//! * [Keyword-driven testing](https://en.wikipedia.org/wiki/Keyword-driven_testing)
//!
//! A goldenscript is a plain text file that contains a set of arbitrary input
//! commands and their expected text output, separated by `---`:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command argument key=value
//! ---
//! output
//! ```
//!
//! The commands are executed by a provided [`Runner`]. The expected output is
//! usually not written by hand, but instead generated by running tests with the
//! environment variable `UPDATE_GOLDENFILES=1` and then verified by inspection
//! before it is checked in to version control. Tests will fail with a diff if
//! they don't match the expected output.
//!
//! This approach is particularly useful when testing complex stateful systems,
//! such as computer language parsing, operations on a key/value store,
//! concurrent transactions in a SQL database, or communication between a
//! cluster of Raft nodes. It can be very tedious and labor-intensive to write
//! and assert such cases by hand, so scripting and recording these interactions
//! often yields much better test coverage at a fraction of the cost.
//!
//! Internally, the
//! [`goldenfile`](https://docs.rs/goldenfile/latest/goldenfile/) crate is used
//! to manage golden files.
//!
//! # Example
//!
//! We'll test the [`dateparser`](https://docs.rs/dateparser/latest/dateparser/)
//! crate which parses timestamp strings in various formats.
//!
//! We write an initial goldenscript `tests/scripts/dateparser` containing test
//! cases for a `parse` command. A goldenscript may contain multiple commands,
//! either as individual input/output blocks or grouped together (which will
//! append their output). The input ends with a `---` separator, and the output
//! ends with a blank line. Note that we don't yet specify any expected output
//! after the `---` separator, this will be autogenerated later.
//!
//! ```text
//! parse 2024-04-30
//! ---
//!
//! # Test various date formats.
//! parse 2024-Apr-30
//! parse 2024.04.30
//! parse 04/30/2024
//! ---
//!
//! # Test some error cases.
//! parse 30.04.2024
//! parse 30/04/2024
//! parse 30/04/24
//! ---
//!
//! # Strings containing special characters must be quoted using " or '.
//! parse "2024-04-30 11:55:32"
//! parse '2024年04月30日11时55分32秒'
//! ---
//! ```
//!
//! We write a runner that recognizes this `parse` command and its timestamp
//! argument, and outputs the parsed date/time in RFC 3339 format. We also add a
//! test using it to run the goldenscript `tests/scripts/dateparser`.
//!
//! ```no_run
//! struct DateParserRunner;
//!
//! impl goldenscript::Runner for DateParserRunner {
//!     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> {
//!         // Only accept a parse command with a single argument.
//!         if command.name != "parse" {
//!             return Err(format!("invalid command {}", command.name).into())
//!         }
//!         if command.args.len() != 1 {
//!             return Err("parse takes 1 argument".into())
//!         }
//!
//!         // Parse the timestamp, and output the RFC 3339 timestamp or error string.
//!         let input = &command.args[0].value;
//!         match dateparser::parse_with(input, &chrono::offset::Utc, chrono::NaiveTime::MIN) {
//!             Ok(datetime) => Ok(datetime.to_rfc3339()),
//!             Err(error) => Ok(format!("Error: {error}")),
//!         }
//!     }
//! }
//!
//! #[test]
//! fn dateparser() -> std::io::Result<()> {
//!     goldenscript::run(&mut DateParserRunner, "tests/scripts/dateparser")
//! }
//! ```
//!
//! Running `UPDATE_GOLDENFILES=1 cargo test` will populate the file with the
//! runner's output. We verify by inspection that it is correct. Later runs of
//! `cargo test` will assert that the output matches the file.
//!
//! ```text
//! parse 2024-04-30
//! ---
//! 2024-04-30T00:00:00+00:00
//!
//! # Test various other date formats.
//! parse 2024-Apr-30
//! parse 2024.04.30
//! parse 04/30/2024
//! ---
//! 2024-04-30T00:00:00+00:00
//! 2024-04-30T00:00:00+00:00
//! 2024-04-30T00:00:00+00:00
//!
//! # Test some error cases.
//! parse 30.04.2024
//! parse 30/04/2024
//! parse 30/04/24
//! ---
//! Error: 30.04.2024 did not match any formats.
//! Error: 30/04/2024 did not match any formats.
//! Error: 30/04/24 did not match any formats.
//!
//! # Strings containing special characters must be quoted using " or '.
//! parse "2024-04-30 11:55:32"
//! parse '2024年04月30日11时55分32秒'
//! ---
//! 2024-04-30T11:55:32+00:00
//! 2024-04-30T11:55:32+00:00
//! ```
//!
//! # Syntax
//!
//! ## Blocks
//!
//! A goldenscript consists of one or more input/output blocks. Each block has a
//! set of one or more input commands on individual lines (empty or comment
//! lines are ignored), a `---` separator, and arbitrary output terminated by an
//! empty line. A minimal goldenscript with two blocks might be:
//!
//! ```text
//! command
//! ---
//! output
//!
//! command 1
//! command 2
//! ---
//! output 1
//! output 2
//! ```
//!
//! ## Commands
//!
//! A [`Command`] must have a command name, which can be any arbitrary non-empty
//! [string](#strings), e.g.:
//!
//! ```text
//! command
//! "command with space and 🚀"
//! ---
//! ```
//!
//! It may additionally have:
//!
//! * [**Arguments:**](Argument) any number of space-separated arguments.
//!   These have a string [value](Argument::value), and optionally also a string
//!   [key](Argument::key) as `key=value`. Values can be empty, and duplicate
//!   keys are allowed by the parser (the runner can handle this as desired).
//!
//!     ```text
//!     command argument key=value
//!     command "argument with space" "key with space"="value with space"
//!     command "" key=  # Empty argument values.
//!     ---
//!     ```
//!
//! * [**Prefix:**](Command::prefix) an optional :-terminated string prefix
//!   before the command name. The command's output will be given the same
//!   prefix. The prefix can be used by the test runner, e.g. to signify two
//!   different clients.
//!
//!     ```text
//!     client1: put key=value
//!     client2: get key
//!     ---
//!     client1: put ok
//!     client2: get key=value
//!     ```
//!
//! * [**Silencing:**](Command::silent) a command wrapped in `()` will have its
//!   output suppressed. This can be useful e.g. for setup commands whose output
//!   are not of interest in the current test case and would only add noise.
//!
//!     ```text
//!     echo foo
//!     (echo bar)
//!     ---
//!     foo
//!     ```
//!
//! ## Output
//!
//! The command output following a `---` separator can contain any arbitrary
//! Unicode string until an empty line (or end of file). If the command output
//! contains empty lines, the entire output will automatically be prefixed with
//! `> `. If no commands in a block yield any output, it defaults to "ok".
//!
//! ```text
//! echo "output 1"
//! echo "output 2"
//! ---
//! output 1
//! output 2
//!
//! echo "Paragraph 1.\n\nParagraph 2."
//! ---
//! > Paragraph 1.
//! >
//! > Paragraph 2.
//!
//! echo "输出\n# Comment\n🚀"
//! ---
//! 输出
//! # Comment
//! 🚀
//! ```
//!
//! ## Comments
//!
//! Comments begin with `#` or `//` and run to the end of the line.
//!
//! ```text
//! # This is a comment.
//! // As is this.
//! command argument # Comments can follow commands too.
//! ---
//! ```
//!
//! ## Strings
//!
//! Unquoted strings can only contain alphanumeric ASCII characters
//! `[a-zA-Z0-9]` and a handful of special characters: `_ - . /`
//! (only `_` at the start of a string).
//!
//! Strings can be quoted using `"` or `'`, in which case they can contain
//! arbitrary Unicode characters. `\` is used as an escape character, both to
//! escape quotes `\"` and `\'` as well as itself `\\`, and also `\0` (null),
//! `\n` (newline), `\r` (carriage return), and `\t` (tab).
//!
//! ```text
//! string
//! "string with spaces and \"quotes\""
//! '字符串'
//! ---
//! ```
//!
//! # Writing Tests
//!
//! In the simplest case, a goldenscript test is:
//!
//! ```no_run
//! struct Runner;
//!
//! impl goldenscript::Runner for Runner {
//!     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> {
//!         todo!();
//!     }
//! }
//!
//! #[test]
//! fn test() -> std::io::Result<()> {
//!     goldenscript::run(&mut Runner, "tests/scripts/test")
//! }
//! ```
//!
//! ## Managing State
//!
//! The runner is free to manage internal state as desired. If it is stateful,
//! it is recommended to persist state within a single goldenscript (across
//! commands and blocks), but not across goldenscripts since this can be hard to
//! reason about and depend on the execution order of scripts. This is most
//! easily done by instantiating a new runner for each script.
//!
//! Initial state setup should generally be done via explicit setup commands, to
//! make it more discoverable.
//!
//! ## Running All Scripts in a Directory
//!
//! External crates can be used to automatically generate and run individual
//! tests for each goldenscript in a directory. For example, the
//! [`test_each_file`](https://docs.rs/test_each_file/latest/test_each_file/)
//! crate:
//!
//! ```no_run
//! # struct Runner;
//! #
//! # impl goldenscript::Runner for Runner {
//! #     fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn std::error::Error>> { todo!() }
//! # }
//! use test_each_file::test_each_path;
//!
//! test_each_path! { in "tests/scripts" as scripts => test_goldenscript }
//!
//! fn test_goldenscript(path: &std::path::Path) {
//!     goldenscript::run(&mut Runner, path).unwrap()
//! }
//! ```
//!
//! ## Hooks
//!
//! Runners have various hooks that will be called during script execution:
//! [`Runner::start_script`], [`Runner::end_script`], [`Runner::start_block`],
//! and [`Runner::end_block`]. These can be used e.g. for initial setup,
//! invariant assertions, or to output the current state.

#![warn(clippy::all)]

mod command;
mod parser;
mod runner;

pub use command::{Argument, Command};
pub use runner::{generate, run, Runner};