Crate faine 

faine stands for FAultpoint INjection, Exhaustible/Exploring and is an implementation of testing technique known as fail points, fault injection, or chaos engineering, which allows testing otherwise hard or impossible to reproduce conditions such as I/O errors.

How this works

• You instrument the source code, adding (fail)points where normal code flow can be overridden externally, to, for instance, return a specific error instead of calling an I/O function (note that for surrounding code, triggering such failpoint would be the same as named I/O function returning an error).
• You trigger these failpoints in the tests, effectively simulating otherwise hard to reproduce failures, and check that your code behaves correctly under these conditions.

On top of supporting that, faine implements automated execution path exploration, running a tested code multiple times with different combinations of failpoints enabled and disabled (NB: in much more effective way than trying all N² possible combinations). This allows simpler tests (which do not know inner workings of the code, that is to know which failpoints to trigger and which effects to expect), with much higher coverage (as all possible code paths are tested).

Example

Let’s test a code which is supposed to atomically replace a file with given content.

Instrument the code by adding failpoint macros before (or around) each operation you want to simulate failures of:

use faine::inject_return_io_error;

fn atomic_replace_file(path: &Path, content: &str) -> io::Result<()> {
    inject_return_io_error!("create file"); // <- added failpoint
    let mut file = File::create(path)?;
    inject_return_io_error!("write file");  // <- added failpoint
    file.write_all(content.as_bytes())?;
    Ok(())
}

Now write a test, utilizing faine::Runner:

use faine::Runner;

#[test]
fn test_replace_file_is_atomic() {
    Runner::default().run(|_| {
        // prepare filesystem state for testing
        let tempdir = tempfile::tempdir().unwrap();
        let path = tempdir.path().join("myfile");
        File::create(&path).unwrap().write_all(b"old").unwrap();
        // run the tested code
        let res = atomic_replace_file(&path, "new");
        // check resulting filesystem state
        let contents = read_to_string(path).unwrap();
        assert!(
           res.is_ok() && contents == "new" ||
           res.is_err() && contents == "old"
        ); // fires!
    }).unwrap();
}

See examples/atomic_replace_file.rs for complete code for this example.

Quick reference

Specifying failpoints which inject early return

The complete macro signature allows to specify failpoint name and returned value:

inject_return!("failpoint name", Err(io::Error::other("injected error")));

You can omit failpoint name (in which case it’s generated from source file path, line and position), and, as testing I/O related code is quite common case, there are shortcuts which return Err(io::Error::other())) right away:

inject_return!(Err(io::Error::other("injected error")));  // name autogenerated
inject_return_io_error!("failpoint name");                // return io::Error
inject_return_io_error!();

Specifying failpoints which wrap expressions

There is a set of macros with the same variations which, instead of returning early, wrap an expression and replace it with something else when failpoint is activated:

let f = inject_override!(File::open("foo"), "failpoint name", Err(io::Error::other("injected error")));
let f = inject_override!(File::open("foo"), Err(io::Error::other("injected error")));
let f = inject_override_io_error!(File::open("foo"), "failpoint name");
let f = inject_override_io_error!(File::open("foo"));

These are particularly useful if you branch based on I/O operation result:

fn open_with_fallback() -> io::Result<File> {
    if let Ok(file) = inject_override_io_error!(File::open("main.dat")) {
        Ok(file)
    } else {
        inject_override_io_error!(File::open("backup.dat"))
    }
}

There’s also a similar set of macros inject_override_with_side_effect* which do the same, but still call an expression if a failpoint is activated, allowing its possible side effect to happen.

Executing the instrumented code

In the test, construct a default Runner and call its run() method with a code to test (with optional preparation code an asserts, just like a normal test):

#[test]
fn test_foobar() {
    Runner::default().run(|_| {
        // ...preparation...
        let res = tested_code();
        assert!(res);
    }).unwrap();
}

Controlling execution

Runner has methods to tune its behavior:

• with_branch_preference()

Enabling failpoints

You can toggle failpoints processing with enable_failpoints macro. This is particularly useful to test how subsequent runs of the code recover from any previous errors:

use faine::enable_failpoints;
tested_code(); // this fails in all possible ways
tested_code(); // this also fails, and sees the previous errors
enable_failpoints!(false);
tested_code(); // this recovers

Introspection

You may inspect which failpoints the code execution has passed through, and which of these were activated. It is possible from both the executed code (to examine current execution trace) and after Runner::run completion (to examine traces for all performed executions with different failpoint paths).

#[test]
fn test_foobar() {
    let report = Runner::default().run(|handle| {
        let res = tested_code();
        eprintln!("current trace: {:?}", handle.trace());
        // assert logic may take failpoint status in the current run into account
        if handle.trace().failpoint_status_first("commit transaction") == Some(Branch::Skip) {
            assert!(res);
        }
    }).unwrap();
    // prints the same set of traces
    eprintln!("all traces: {:#?}", report.traces);
}

By default, traces are printed in a fancy way to make them most readable. This may be disabled through fancy-traces feature. An example Report::traces dump for a linear code with three failpoints:

[
    (💥create temp file),
    (create temp file)→(💥write temp file),
    (create temp file)→(write temp file)→(💥replace file),
    (create temp file)→(write temp file)→(replace file),
]

Run cargo test --example atomic_replace_file -- --no-capture to reproduce.

Other implementations of the same concept

Neither supports path exploration as far as I know.

• chaos-rs
• fail
• fail-parallel
• failpoints
• fault-injection

Macros

enable_failpoints

Enable or disable failpoints

inject_override

Define failpoint which overrides an expression

inject_override_io_error

Define failpoint which overrides an expression with std::io::Error

inject_override_with_side_effect

Define failpoint which overrides an expression (which is still executed)

inject_override_with_side_effect_io_error

Define failpoint which overrides an expression (which is still executed) with std::io::Error

inject_return

Define failpoint which returns from an enclosing function

inject_return_io_error

Define failpoint which returns std::io::Error from an enclosing function

Structs

Report

Execution report

Runner

Runner for code instrumented with failpoints

Step

Record of a single failpoint passage

Trace

Execution trace

Enums

Branch

Path chosen when execution passes through a failpoint

Error

Error when executing tested code