Expand description
The retest
crate is used to automate black box regression testing.
There is also a retest executable
(available precompiled for 64-bit Windows; in source form for other
platforms). This uses retest plan files
(.rt
).
There are two phases for regression testing.
-
First there is the generate phase. Here, the application to test is run and the output (whether a file or captured
stdout
) is saved in an expected folder. This is typically done initially, and subsequently whenever new tests are added. -
Second there’s the testing phase. Here, the application to test is run and the output is saved to an actual folder. Then the corresponding outputs in the expected and actual folder are compared, and any discrepencies reported. This is typically done many times.
Note also that it is also possible to just check the application to test’s exit code/error level, if that’s all that is required.
Retest’s two APIs
See also, Notes for Upgraders.
Using Retest Plan Files
This API uses the same retest plan files
(.rt
) used by the
retest executable. It offers these
benefits: you can write the plan files independently of your code, you can
generate the initial test expecteds and any subsequent ones using the
retest executable, and you can test using the retest executable, making
the whole process easier to debug.
This API is provided by the
Plan::new_from_rt()
and
Plan::new_from_rt_filtered()
constructors.
Using Builders
This API uses builder methods which means you don’t have to learn how to
create retest plan files
(.rt
), and may be easier
for integrating retest regression testing with other tools.
This API is provided by two separate builders that are used together. An
overall test Plan
is created using
Plan::new()
and the Plan
builder
methods. Then the individual tests are created using
Test::new()
and the Test
builder
methods. Each test is added to the Plan
using the
Plan::push()
method.
Generating and Retesting
Once a Plan
has been created (and populated with
tests) using either API, you can generate expecteds using the
Plan::generate()
method, or test and
compare using the Plan::retest()
method. There is also a Plan::apply()
method which will generate or retest depending on the RETEST_GENERATE
environment variable.
During generating or retesting, logging output is produced (or not)
depending on the logger you are using and the logging level you have set.
And when finished, both these methods return a
Counts
struct that summarizes the results.
Dependencies
To use retest, add this line your Cargo.toml
file’s [dependencies]
section:
retest = { package = "qtrac-retest", version = "4" }
Then, in your crate root, for Rust 2015 add extern crate retest
, and for
Rust 2018 add use retest
.
Note that the retest crate writes all its output to the current logger, so a logger must be available. (See the log crate and, for example, the simplelog crate.)
Examples
Retest Plan File Example
// === This is just a skeleton, so won't work as-is ===
use retest::{Counts, Plan, XResult};
// = use the log module of your choice =
// = initialize the logger =
fn manage_tests(rt_filename: &str, generate: bool) -> XResult<Counts> {
let plan = Plan::new_from_rt(&rt_filename)?;
if generate {
plan.generate()
} else {
plan.retest()
}
}
This code will generate or retest every test it finds in the .rt
file
that’s specified. If you want to limit generating or retesting to one or
more specific tests you can do so by creating a BTreeSet<u32>
of the
test numbers you want to generate or retest, and using the
Plan::new_from_rt_filtered()
method.
The XResult
type is an alias for Result<T, XError>
. The XError
type
is just a wrapper around the various errors retest can encounter.
For a complete example of this use case see the source code’s
src/bin/retest/main.rs
file.
Builders Example
// === This is just a skeleton, so won't work as-is ===
use retest::{Counts, DiffKind, Plan, Test, XResult};
// = use the log module of your choice =
// = initialize the logger =
fn run_tests() -> XResult<Counts> {
let app = r"V:\bin\myapp.exe";
let plan = Plan::new()
.expected_path(r"U:\myapp\expected")
.actual_path(r"U:\myapp\actual")
.push(Test::new(&app)
.name("Text test")
.args(&["-v", "--format=light", r"$OUT_PATH\01.txt"])
.build())
.push(Test::new(&app)
.name("Bad error level test")
.exit_code(1)
.build())
.push(Test::new(&app)
.name("Compare PDFs test")
.args(&["/out", r"$OUT_PATH\03.pdf")
.diff(DiffKind::custom(
r"C:\bin\comparepdfcmd\comparepdfcmd.exe")
.diff_args(&[r"$EXPECTED_PATH\03.pdf",
r"$ACTUAL_PATH\03.pdf"])
.build())
.build();
plan.retest() // Could generate using plan.generate().
// Or could use plan.apply() which will generate or retest
// depending on the RETEST_GENERATE environment variable.
}
For an example of this use case see the source code’s
src/plan/builder_tests.rs
file’s “plan13” test.
Note that the Plan
and Test
builder methods can be used as setters,
which is useful in conditionals, e.g.,
let mut planner = Plan::new();
planner.expected_path("/home/user/myapp/expected");
if !custom_actual_path.is_empty() {
planner.actual_path(&custom_actual_path);
}
let mut test = Test::new("myapp");
test.name("Help Test");
test.args(&["--help"]);
if expect_invalid {
test.exit_code(2);
}
planner.push(test.build());
// Create and push more tests...
let plan = planner.build();
plan.retest();
For a small example of this use case see the source code’s
src/plan/builder_tests.rs
file’s “plan14” test.
Notes for Upgraders
Upgrading from 3.x to 4.x
Remove any calls to Plan::app_path_env_var()
. Instead, when creating a
new Test
give the app
with a full path, e.g.,
Path::new(env!("CARGO_TARGET_DIR")).join("myapp.exe")
, unless it is in
thePATH
already.
Upgrading from 2.x to 3.x
- Rename
PlanBuilder
toPlan
; - Rename
Plan
’sappend()
method topush()
. - Replace
DiffKind::Custom
withDiffKind::custom
, e.g.:DiffKind::Custom("
diff".to_string()) //
old-styleDiffKind::custom("
diff") //
new-style
Naturally, if you recompile, rust will find all the places these changes are needed.
New methods:
Upgrading from 1.x to 2.x
- Rename
TestBuilder
toTest
; - Rename
Test
’sdiff_by()
todiff()
; - Ensure that
diff()
is passed aDiffKind
rather than a&str
.
License
Retest is free open source software (FOSS) licensed under the GNU General Public License version 3 (GPLv3).
Macros
Structs
Holds the number of generated or retested tests, passes, fails, and errors.
Holds one or more Test
s which can be used to
generate expected files or to retest by creating actual files and
comparing them with the expecteds.
Represents a single test which can be generated or retested.
Enums
The kind of comparison to do when retesting.