Cargo Darwin
cargo-darwin
is a plugin over cargo
tool.
Darwin mutates your code, if your code still passes check tests, then your code isn't enough tested.
Installation
cargo install cargo-darwin
Usage
Will display something like
[Missing] : Tests pass, the mutation hasn't been caught, suspicion of missing test
[OK] : Tests failed, the mutation has been caught
[Timeout] : Mutation introduces infinite loop, inconclusive
[Killed] : Mutation introduces non buildable modification
---
[OK] : Mutation #0 replace - by + in function "sub" of file src\a\toto.rs at line 11:6
[OK] : Mutation #1 replace - by * in function "sub" of file src\a\toto.rs at line 11:6
[Killed] : Mutation #2 replace - by && in function "sub" of file src\a\toto.rs at line 11:6
[Missing] : Mutation #3 replace + by - in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #4 replace + by * in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #5 replace + by - in function "add" of file src\lib.rs at line 5:10
[Missing] : Mutation #6 replace + by * in function "add" of file src\lib.rs at line 5:10
There is a --dry-run
mode to just list mutation without actually apply tests.
Details
Darwin walks the provided path (if none provided get the current dir).
For each file ending by .rs extension, Darwin analyze the file and try to found mutable
function.
A function is mutable if there is no #[test]
or #[tokio::test]
attribute over it.
fn mutable() {}
#[test]
fn non_mutable() {}
#[tokio::test]
async fn non_mutable_async() {}
The project is in its really early stage, so the mutation are quite limited, actually just binary expressions
like a + b
or a - b
.
For example this mutable function
fn add(x: u8, y:u8) -> u8 {
x + y
}
will become
fn add(x: u8, y:u8) -> u8 {
x - y
}
Then Darwin create a copy of the actual project and apply the modification on the copied file
Once the project mutated, Darwin runs a cargo build
, if the project compile, then the mutation is sustainable
If so, Darwin runs the cargo test
, There are 3 possibilities:
- project tests pass : the project is inefficiently tested as the mutation isn't catch
- tests fail : the project has at least one test which catches the mutation
- timeout : the mutation even if compiles, introduce a loop or something that makes the test run forever
Reports
All reports can be found in the mutation path in a reports folder which. For example if you have run darwin with
You will get this tree
Mutated projects
If the --keep
flag is defined, after tests, you can walk to generated projects
Each one has a mutation ID and the associated mutation ID can be found in summary file
Summary
Summarize the mutation applied and the result of each.
[OK] : Mutation #0 replace - by + in function "sub" of file src\a\toto.rs at line 11:6
[OK] : Mutation #1 replace - by * in function "sub" of file src\a\toto.rs at line 11:6
[Killed] : Mutation #2 replace - by && in function "sub" of file src\a\toto.rs at line 11:6
[Missing] : Mutation #3 replace + by - in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #4 replace + by * in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #5 replace + by - in function "add" of file src\lib.rs at line 5:10
[Missing] : Mutation #6 replace + by * in function "add" of file src\lib.rs at line 5:10
For more information about the mutation, check the associated mutation_ID.log file
Mutation report
reports/mutation_X.log
files are the detailed view of the mutation.
There are build with the following nomenclature
- Mutated file
- Mutation
- Mutation status
- Diff of the mutation
- Test or build output Below an example of output
Mutation of file F:\Projets\Lab\Rust\darwin\playground\src\a\toto.rs
Mutation reason: replace - by *
Status : OK => Mutation Caught
Mutation diff:
@@ -8,7 +8,7 @@
//
fn sub(x: i8, y: i8) -> i8 {
let u = 8;
- x - y
+ x * y
}
Output:
#[test]
stderr:
Compiling playground v0.1.0 (F:\Projets\Lab\Rust\darwin\tmp\1)
Finished test [unoptimized + debuginfo] target(s) in 0.18s
Running unittests src\lib.rs (target\debug\deps\playground-29148ab9d23d3c5d.exe)
error: test failed, to rerun pass `--lib`
stdout:
running 2 tests
test a::toto::test_sub ... FAILED
test a::toto::async_test_sub ... FAILED
failures:
---- a::toto::test_sub stdout ----
thread 'a::toto::test_sub' panicked at src\a\toto.rs:16:5:
assertion `left == right` failed
left: 10
right: 3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- a::toto::async_test_sub stdout ----
thread 'a::toto::async_test_sub' panicked at src\a\toto.rs:21:5:
assertion `left == right` failed
left: 10
right: 3
failures:
a::toto::async_test_sub
a::toto::test_sub
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
As a test has failed, the mutation has been caught, so the code is enough tested for this particular mutation
Limits
This project has done in the sole goal to understand the mutation testing and how it can be implemented.
That's why the set of mutation is hardcoded so far.
And very limited:
a + b
gives:
a - b
a * b
a - b
gives:
a + b
a * b
a && b
Trivia
Darwin stands for the "Natural selection law", the life mutates to adapt to environment so is doing cargo-darwin but this time the mutation are unwanted.
I would like to call it "Malcom" in honor of "Ian Malcom" in Jurassic Park, but the reference is too capillotracted ^^'