Crate cargo_darwin

cargo-darwin is a plugin over cargo tool.

Darwin mutates your code, if your code still passes check tests, then your code isn’t enough tested.

Usage

cargo darwin /path/to/project/to/test

Will display something like

[Missing] : Tests pass, the mutation hasn't been caught, suspicion of missing test
[OK]      : Tests failed, the mutation has been caught
[Timeout] : Mutation introduces infinite loop, inconclusive
[Killed]  : Mutation introduces non buildable modification
  ---
[OK] : Mutation #0 replace - by + in function "sub" of file src\a\toto.rs at line 11:6
[OK] : Mutation #1 replace - by * in function "sub" of file src\a\toto.rs at line 11:6
[Killed] : Mutation #2 replace - by && in function "sub" of file src\a\toto.rs at line 11:6
[Missing] : Mutation #3 replace + by - in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #4 replace + by * in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #5 replace + by - in function "add" of file src\lib.rs at line 5:10
[Missing] : Mutation #6 replace + by * in function "add" of file src\lib.rs at line 5:10

Details

Darwin walks the provided path (if none provided get the current dir).

For each file ending by .rs extension, Darwin analyze the file and try to found mutable function.

A function is mutable if there is no #[test] or #[tokio::test] attribute over it.

fn mutable() {}

#[test]
fn non_mutable() {}

#[tokio::test]
async fn non_mutable_async() {}

The project is in its really early stage, so the mutation are quite limited, actually just binary expressions like a + b or a - b.

For example this mutable function

fn add(x: u8, y:u8) -> u8 {
    x + y
}

will become

fn add(x: u8, y:u8) -> u8 {
    x - y
}

Then Darwin create a copy of the actual project and apply the modification on the copied file

Once the project mutated, Darwin runs a cargo build, if the project compile, then the mutation is sustainable

If so, Darwin runs the cargo test, There are 3 possibilities:

• project tests pass : the project is inefficiently tested as the mutation isn’t catch
• tests fail : the project has at least one test which catches the mutation
• timeout : the mutation even if compiles, introduce a loop or something that makes the test run forever

Reports

All reports can be found in the mutation path in a reports folder which.

For example if you have run darwin with

cargo darwin --mutation-path /tmp/darwin /path/to/project/to/test

You will get this tree

tmp/
├─ darwin/
│  ├─ reports/
│  │  ├─ mutation_0.log
│  │  ├─ mutation_1.log
│  │  ├─ summary
│  ├─ 0/
│  ├─ 1/

Mutated projects

If the --keep flag is defined, after tests, you can walk to generated projects

Each one has a mutation ID and the associated mutation ID can be found in summary file

Summary

Summarize the mutation applied and the result of each.

[OK] : Mutation #0 replace - by + in function "sub" of file src\a\toto.rs at line 11:6
[OK] : Mutation #1 replace - by * in function "sub" of file src\a\toto.rs at line 11:6
[Killed] : Mutation #2 replace - by && in function "sub" of file src\a\toto.rs at line 11:6
[Missing] : Mutation #3 replace + by - in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #4 replace + by * in function "add" of file src\lib.rs at line 5:6
[Missing] : Mutation #5 replace + by - in function "add" of file src\lib.rs at line 5:10
[Missing] : Mutation #6 replace + by * in function "add" of file src\lib.rs at line 5:10

For more information about the mutation, check the associated mutation_ID.log file

Mutation report

reports/mutation_X.log files are the detailed view of the mutation.

There are build with the following nomenclature

• Mutated file
• Mutation
• Mutation status
• Diff of the mutation
• Test or build output

Below an example of output

Mutation of file F:\Projets\Lab\Rust\darwin\playground\src\a\toto.rs
Mutation reason: replace - by *
Status : OK => Mutation Caught
Mutation diff:
@@ -8,7 +8,7 @@
 //
 fn sub(x: i8, y: i8) -> i8 {
     let u = 8;
-    x - y
+    x * y
 }

Output:

 #[test]
stderr:
   Compiling playground v0.1.0 (F:\Projets\Lab\Rust\darwin\tmp\1)
    Finished test [unoptimized + debuginfo] target(s) in 0.18s
     Running unittests src\lib.rs (target\debug\deps\playground-29148ab9d23d3c5d.exe)
error: test failed, to rerun pass `--lib`

stdout:

running 2 tests
test a::toto::test_sub ... FAILED
test a::toto::async_test_sub ... FAILED

failures:

---- a::toto::test_sub stdout ----
thread 'a::toto::test_sub' panicked at src\a\toto.rs:16:5:
assertion `left == right` failed
  left: 10
 right: 3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- a::toto::async_test_sub stdout ----
thread 'a::toto::async_test_sub' panicked at src\a\toto.rs:21:5:
assertion `left == right` failed
  left: 10
 right: 3


failures:
    a::toto::async_test_sub
    a::toto::test_sub

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

As a test has failed, the mutation has been caught, so the code is enough tested for this particular mutation

Functions

• run
  Main darwin function