Crate trycmd_indygreg_fork
source ·Expand description
Snapshot testing for a herd of CLI tests
Treat your tests like cattle, instead of pets
trycmd
is a test harness that will enumerate test case files and run them to verify the
results, taking inspiration from
trybuild and cram.
Which tool is right
- cram: End-to-end CLI snapshotting agnostic of any programming language
trycmd
: For running a lot of blunt tests (limited test predicates)- Particular attention is given to allow the test data to be pulled into documentation, like with mdbook
- snapbox: When you want something like
trycmd
in one off cases or you need to customizetrycmd
s behavior. - assert_cmd + assert_fs: Test cases follow a certain pattern but special attention is needed in how to verify the results.
- Hand-written test cases: for peculiar circumstances
Getting Started
To create a minimal setup, create a tests/cli_tests.rs
with
#[test]
fn cli_tests() {
trycmd_indygreg_fork::TestCases::new()
.case("tests/cmd/*.toml")
.case("README.md");
}
and write out your test cases in your .toml
files along with examples in your README.md
.
Run this with cargo test
like normal. TestCases
will enumerate all test case files and
run the contained commands, verifying they run as expected.
To temporarily override the results, you can do:
#[test]
fn cli_tests() {
trycmd_indygreg_fork::TestCases::new()
.case("tests/cmd/*.toml")
.case("README.md")
// See Issue #314
.fail("tests/cmd/buggy-case.toml");
}
Workflow
To generate snapshots, run
$ TRYCMD=dump cargo test --test cli_tests
This will write all of the .stdout
and .stderr
files in a dump/
directory.
You can then copy over to tests/cmd
the cases you want to test
To update snapshots, run
$ TRYCMD=overwrite cargo test --test cli_tests
This will overwrite any existing .stdout
and .stderr
file in tests/cmd
To filter the tests to those with name1
, name2
, etc in their file names, you can run:
cargo test --test cli_tests -- cli_tests trycmd=name1 trycmd=name2...
To debug what trycmd
is doing, run cargo test -F trycmd/debug
.
File Formats
For tests/cmd/help.trycmd
, trycmd
will look for:
tests/cmd/help.in/
tests/cmd/help.out/
Say you have tests/cmd/help.toml
, trycmd
will look for:
tests/cmd/help.stdin
tests/cmd/help.stdout
tests/cmd/help.stderr
tests/cmd/help.in/
tests/cmd/help.out/
*.trycmd
*.trycmd
/ *.md
files are literate test cases good for:
- Markdown-compatible syntax for directly rendering them
- Terminal-like appearance for extracting subsections into documentation
- Reducing the proliferation of files
- Running multiple commands within the same temp dir (if a
*.out/
directory is present)
The syntax is:
- Test cases live inside of
```
fenced code blocks- Everything out of them is ignored
- Blocks with info strings with an unsupported language (not
trycmd
,console
) or theignore
attribute are ignored
- “
$
” line prefix starts a new command - “
>
” line prefix appends to the prior command - “
? <status>
” line indicates the exit code (likeecho "? $?"
) and<status>
can be- An exit code
success
(default),failed
,interrupted
,skipped
- All following lines are treated as stdout + stderr
The command is then split with shlex, allowing quoted content
to allow spaces. The first argument is the program to run which maps to bin.name
in the
.toml
file.
Example:
With a [[bin]]
like:
fn main() {
println!("Hello world");
}
You can verify a code block like:
```console
$ my-cmd
Hello world
```
For a more complete example, see: https://github.com/assert-rs/trycmd/tree/main/examples/demo_trycmd.
*.toml
As an alternative to .trycmd
, the toml
are good for:
- Precise control over current dir, stdin/stdout/stderr (including binary support)
- 1-to-1 with dumped results
TRYCMD=overwrite
support
See full schema: Basic parameters:
bin.name
: The name of the binary target fromCargo.toml
to be used to find the file pathargs
: the arguments (including flags and option) passed to the binary
*.stdin
Data to pass to stdin
.
- If not present, nothing will be written to
stdin
- If
binary = false
in*.toml
(the default), newlines and path separators will be normalized.
*.stdout
and *.stderr
Expected results for stdout
or stderr
.
- If not present, we’ll not verify the output
- If
binary = false
in*.toml
(the default), newlines and path separators will be normalized before comparing
Eliding Content
Sometimes the output either includes:
- Content that changes from run-to-run (like time)
- Content out of scope of your tests and you want to exclude it to reduce brittleness
To elide a section of content:
...
as its own line: match all lines until the next one. This is equivalent of\n(([^\n]*\n)*)?
.[..]
as part of a line: match any characters. This is equivalent of[^\n]*?
.[EXE]
as part of the line: On Windows, matches.exe
, ignored otherwise[ROOT]
as part of the line: The root directory for where the test is running[CWD]
as part of the line: The current working directory within the root[YOUR_NAME_HERE]
as part of the line: SeeTestCases::insert_var
We will preserve these with TRYCMD=dump
and will make a best-effort at preserving them with
TRYCMD=overwrite
.
*.in/
When present, this will automatically be picked as the CWD for the command.
.keep
files will be ignored but their parent directories will be created.
Tests are assumed to not modify files in *.in/
unless an *.out/
is provided or
fs.sandbox = true
is set in the .toml
file.
*.out/
When present, each file in this directory will be compared to generated or modified files.
See also “Eliding Content” for .stdout
.keep
files will be ignored.
Note: This implies fs.sandbox = true
.
Examples
- Simple cargo binary: trycmd’s integration tests
- Simple example: trycmd’s integration tests
- typos (source code spell checker)
- clap (CLI parser) to test examples
Related crates
For testing command line programs.
- escargot for more control over configuring the crate’s binary.
- duct for orchestrating multiple processes.
- or commandspec for easier writing of commands
assert_cmd
for test cases that are individual pets, rather than herd of cattleassert_fs
for filesystem fixtures and assertions.- or tempfile for scratchpad directories.
- rexpect for testing interactive programs.
- dir-diff for testing file side-effects.
For snapshot testing:
- insta
- fn-fixture
- runt
- term-transcript: CLI snapshot testing, including colors
Modules
- Interact with
cargo
cmd.toml
Schema
Structs
- Entry point for running tests