trycmd_indygreg_fork/lib.rs
1//! # Snapshot testing for a herd of CLI tests
2//!
3//! > Treat your tests like cattle, instead of [pets](https://docs.rs/snapbox)
4//!
5//! `trycmd` is a test harness that will enumerate test case files and run them to verify the
6//! results, taking inspiration from
7//! [trybuild](https://crates.io/crates/trybuild) and [cram](https://bitheap.org/cram/).
8//!
9//! ## Which tool is right
10//!
11//! - [cram](https://bitheap.org/cram/): End-to-end CLI snapshotting agnostic of any programming language
12//! - `trycmd`: For running a lot of blunt tests (limited test predicates)
13//! - Particular attention is given to allow the test data to be pulled into documentation, like
14//! with [mdbook](https://rust-lang.github.io/mdBook/)
15//! - [snapbox](https://crates.io/crates/snapbox): When you want something like `trycmd` in one off
16//! cases or you need to customize `trycmd`s behavior.
17//! - [assert_cmd](https://crates.io/crates/assert_cmd) +
18//! [assert_fs](https://crates.io/crates/assert_fs): Test cases follow a certain pattern but
19//! special attention is needed in how to verify the results.
20//! - Hand-written test cases: for peculiar circumstances
21//!
22//! ## Getting Started
23//!
24//! To create a minimal setup, create a `tests/cli_tests.rs` with
25//! ```rust,no_run
26//! #[test]
27//! fn cli_tests() {
28//! trycmd_indygreg_fork::TestCases::new()
29//! .case("tests/cmd/*.toml")
30//! .case("README.md");
31//! }
32//! ```
33//! and write out your test cases in your `.toml` files along with examples in your `README.md`.
34//!
35//! Run this with `cargo test` like normal. [`TestCases`] will enumerate all test case files and
36//! run the contained commands, verifying they run as expected.
37//!
38//! To temporarily override the results, you can do:
39//! ```rust,no_run
40//! #[test]
41//! fn cli_tests() {
42//! trycmd_indygreg_fork::TestCases::new()
43//! .case("tests/cmd/*.toml")
44//! .case("README.md")
45//! // See Issue #314
46//! .fail("tests/cmd/buggy-case.toml");
47//! }
48//! ```
49//!
50//! ## Workflow
51//!
52//! To generate snapshots, run
53//! ```console
54//! $ TRYCMD=dump cargo test --test cli_tests
55//! ```
56//! This will write all of the `.stdout` and `.stderr` files in a `dump/` directory.
57//!
58//! You can then copy over to `tests/cmd` the cases you want to test
59//!
60//! To update snapshots, run
61//! ```console
62//! $ TRYCMD=overwrite cargo test --test cli_tests
63//! ```
64//! This will overwrite any existing `.stdout` and `.stderr` file in `tests/cmd`
65//!
66//! To filter the tests to those with `name1`, `name2`, etc in their file names, you can run:
67//! ```console
68//! cargo test --test cli_tests -- cli_tests trycmd=name1 trycmd=name2...
69//! ```
70//!
71//! To debug what `trycmd` is doing, run `cargo test -F trycmd/debug`.
72//!
73//! ## File Formats
74//!
75//! For `tests/cmd/help.trycmd`, `trycmd` will look for:
76//! - `tests/cmd/help.in/`
77//! - `tests/cmd/help.out/`
78//!
79//! Say you have `tests/cmd/help.toml`, `trycmd` will look for:
80//! - `tests/cmd/help.stdin`
81//! - `tests/cmd/help.stdout`
82//! - `tests/cmd/help.stderr`
83//! - `tests/cmd/help.in/`
84//! - `tests/cmd/help.out/`
85//!
86//! ### `*.trycmd`
87//!
88//! `*.trycmd` / `*.md` files are literate test cases good for:
89//! - Markdown-compatible syntax for directly rendering them
90//! - Terminal-like appearance for extracting subsections into documentation
91//! - Reducing the proliferation of files
92//! - Running multiple commands within the same temp dir (if a `*.out/` directory is present)
93//!
94//! The syntax is:
95//! - Test cases live inside of ` ``` ` fenced code blocks
96//! - Everything out of them is ignored
97//! - Blocks with info strings with an unsupported language (not `trycmd`, `console`) or the
98//! `ignore` attribute are ignored
99//! - "`$ `" line prefix starts a new command
100//! - "`> `" line prefix appends to the prior command
101//! - "`? <status>`" line indicates the exit code (like `echo "? $?"`) and `<status>` can be
102//! - An exit code
103//! - `success` *(default)*, `failed`, `interrupted`, `skipped`
104//! - All following lines are treated as stdout + stderr
105//!
106//! The command is then split with [shlex](https://crates.io/crates/shlex), allowing quoted content
107//! to allow spaces. The first argument is the program to run which maps to `bin.name` in the
108//! `.toml` file.
109//!
110//! Example:
111//!
112//! With a `[[bin]]` like:
113//! ```rust,ignore
114//! fn main() {
115//! println!("Hello world");
116//! }
117//! ```
118//!
119//! You can verify a code block like:
120//! ~~~md
121//! ```console
122//! $ my-cmd
123//! Hello world
124//!
125//! ```
126//! ~~~
127//!
128//! For a more complete example, see:
129//! <https://github.com/assert-rs/trycmd/tree/main/examples/demo_trycmd>.
130//!
131//! ### `*.toml`
132//!
133//! As an alternative to `.trycmd`, the `toml` are good for:
134//! - Precise control over current dir, stdin/stdout/stderr (including binary support)
135//! - 1-to-1 with dumped results
136//! - `TRYCMD=overwrite` support
137//!
138//! [See full schema](https://github.com/assert-rs/trycmd/blob/main/schema.json):
139//! Basic parameters:
140//! - `bin.name`: The name of the binary target from `Cargo.toml` to be used to find the file path
141//! - `args`: the arguments (including flags and option) passed to the binary
142//!
143//! #### `*.stdin`
144//!
145//! Data to pass to `stdin`.
146//! - If not present, nothing will be written to `stdin`
147//! - If `binary = false` in `*.toml` (the default), newlines and path separators will be normalized.
148//!
149//! #### `*.stdout` and `*.stderr`
150//!
151//! Expected results for `stdout` or `stderr`.
152//! - If not present, we'll not verify the output
153//! - If `binary = false` in `*.toml` (the default), newlines and path separators will be normalized before comparing
154//!
155//! **Eliding Content**
156//!
157//! Sometimes the output either includes:
158//! - Content that changes from run-to-run (like time)
159//! - Content out of scope of your tests and you want to exclude it to reduce brittleness
160//!
161//! To elide a section of content:
162//! - `...` as its own line: match all lines until the next one. This is equivalent of
163//! `\n(([^\n]*\n)*)?`.
164//! - `[..]` as part of a line: match any characters. This is equivalent of `[^\n]*?`.
165//! - `[EXE]` as part of the line: On Windows, matches `.exe`, ignored otherwise
166//! - `[ROOT]` as part of the line: The root directory for where the test is running
167//! - `[CWD]` as part of the line: The current working directory within the root
168//! - `[YOUR_NAME_HERE]` as part of the line: See [`TestCases::insert_var`]
169//!
170//! We will preserve these with `TRYCMD=dump` and will make a best-effort at preserving them with
171//! `TRYCMD=overwrite`.
172//!
173//! ### `*.in/`
174//!
175//! When present, this will automatically be picked as the CWD for the command.
176//!
177//! `.keep` files will be ignored but their parent directories will be created.
178//!
179//! Tests are assumed to not modify files in `*.in/` unless an `*.out/` is provided or
180//! `fs.sandbox = true` is set in the `.toml` file.
181//!
182//! ### `*.out/`
183//!
184//! When present, each file in this directory will be compared to generated or modified files.
185//!
186//! See also "Eliding Content" for `.stdout`
187//!
188//! `.keep` files will be ignored.
189//!
190//! Note: This implies `fs.sandbox = true`.
191//!
192//! ## Examples
193//!
194//! - Simple cargo binary: [trycmd's integration tests](https://github.com/assert-rs/trycmd/blob/main/tests/cli_tests.rs)
195//! - Simple example: [trycmd's integration tests](https://github.com/assert-rs/trycmd/blob/main/tests/example_tests.rs)
196//! - [typos](https://github.com/crate-ci/typos) (source code spell checker)
197//! - [clap](https://github.com/clap-rs/clap/) (CLI parser) to test examples
198//!
199//! ## Related crates
200//!
201//! For testing command line programs.
202//! - [escargot][escargot] for more control over configuring the crate's binary.
203//! - [duct][duct] for orchestrating multiple processes.
204//! - or [commandspec] for easier writing of commands
205//! - [`assert_cmd`][assert_cmd] for test cases that are individual pets, rather than herd of cattle
206//! - [`assert_fs`][assert_fs] for filesystem fixtures and assertions.
207//! - or [tempfile][tempfile] for scratchpad directories.
208//! - [rexpect][rexpect] for testing interactive programs.
209//! - [dir-diff][dir-diff] for testing file side-effects.
210//!
211//! For snapshot testing:
212//! - [insta](https://crates.io/crates/insta)
213//! - [fn-fixture](https://crates.io/crates/fn-fixture)
214//! - [runt](https://crates.io/crates/runt)
215//! - [turnt](https://github.com/cucapra/turnt)
216//! - [cram](https://bitheap.org/cram/)
217//! - [term-transcript](https://crates.io/crates/term-transcript): CLI snapshot testing, including colors
218//!
219//! [escargot]: http://docs.rs/escargot
220//! [rexpect]: https://crates.io/crates/rexpect
221//! [dir-diff]: https://crates.io/crates/dir-diff
222//! [tempfile]: https://crates.io/crates/tempfile
223//! [duct]: https://crates.io/crates/duct
224//! [assert_fs]: https://crates.io/crates/assert_fs
225//! [assert_cmd]: https://crates.io/crates/assert_cmd
226//! [commandspec]: https://crates.io/crates/commandspec
227
228#![cfg_attr(docsrs, feature(doc_auto_cfg))]
229// Doesn't distinguish between incidental sharing vs essential sharing
230#![allow(clippy::branches_sharing_code)]
231// Forces indentation that may not represent the logic
232#![allow(clippy::collapsible_else_if)]
233
234pub mod cargo;
235pub mod schema;
236
237mod cases;
238mod registry;
239mod runner;
240mod spec;
241
242pub use cases::TestCases;
243pub use snapbox::Error;
244
245pub(crate) use registry::BinRegistry;
246pub(crate) use runner::{Case, Mode, Runner};
247pub(crate) use spec::RunnerSpec;
248
249pub(crate) use snapbox::Data;