Cargo Regression

Collect test task from input files, execute them and compare results with golden.

Usage

cargo install cargo-regression

Build your test files like ./demo, then

cargo regression ./demo --debug

The tests will be exectued in ./tmp in default, change the dir by --work-dir path.

Set Extension(s)

cargo-regression will collect all files that match extensions as test tasks, you can set extensions in two ways:

• By commmand arguments

cargo regression ./demo --extensions py sh

• By xx/__all__.toml

# override for all sub-dir
extensions = ["py", "sh"]

Other Configurations

There are three types of configs:

• xx/__all__.toml: Affect all task for current and all sub directory.
• xxx.toml: Only affect for input task file xxx.xx
• Argument: Is equivalent to set it in the most top /__all__.toml.

The configs can inherit from upper, which means

Except extensions can only be define in xx/__all__.toml, the other configs can be define in both xx/__all__.toml and xxx.toml.

Variable Table

Advanced Features

Test Filter

Only test specified tasks.

# No filter
cargo regression ./demo
# Only include demo/trybuild/*
cargo regression ./demo --include demo/trybuild/*
# Exclude demo/trybuild/*
cargo regression ./demo --exclude demo/trybuild/*
# Combined filter
cargo regression ./demo --include demo/trybuild/* --exclude demo/trybuild/compile-ok.rs

Schedule Parallelism

permits and permit are virtual resource costs, you can define permits in arguments (default=1), and define permit in task toml config file (default=0).

cargo regression ./demo --include demo/test-premit/* --permits 1
cargo regression ./demo --include demo/test-premit/* --permits 2

assertion

exit-code

Assert the exit code, default is 0. See test-exit.toml

[assert]
exit-code = 1

equal

The output file should equal to the golden. See compile-fail.toml

[[assert.golden]]
file = "{{name}}.stderr"
# The task's stder' should equal to __golden__/{{name}}.stderr
equal = true

match

Match pattern and assert the number (count) of it. See test-match.toml

[[assert.golden]]
# match pattern from task stdout
file = "{{name}}.stdout"
match = [
  # regular expression match
  { pattern = 'f.*o', count = 4 },
  # should contain word "fo" at least once
  { pattern = '\bfo\b', count-at-least = 1 },
  # should contain word "fo" at most once
  { pattern = '\bfo0\b', count-at-most = 1 },
]

value

Capture float number and assert the value (count) of it. The epsilon is assert tolerance, if the epsilon is not defined, default epsilon is 1e-10 See test-value.toml

[[assert.golden]]
# match float value from task stdout
file = "{{name}}.stdout"
value = [
  # match 'f.*o' and then match a float, assert it should within 4±0.01
  { pattern-before = 'f.*o', value = 4.0, epsilon = 0.01 },
  # match a float and then match 'after\b', assert it should > 1-0.0000000001
  { pattern-after = 'after\b', value-at-least = 1 },
  # math the float between 'f.*o' and 'after\b', assert it should within 4±0.0000000001
  { pattern-before = 'f.*o', pattern-after = 'after\b', value = 4.0 }
]

Use it as API

see ./examples

use cargo_regression::{test, Args, TestExitCode};

#[tokio::main]
async fn main() -> TestExitCode {
  // Get arguments from CLI
  let args = Args::parse_from(std::env::args_os());
  // Or set fixed arguemnts
  let args = Args::new().debug(true).work_dir("tmp").root_dir("demo");
  args.test().await
}

Remind: For fixed argument, the include and exclude variables should be all files matched by yourself, use ["path/A", "path/B"] rather than ["path/*"].

TODO

• regolden
• full config document
• document