rtest - Resource based test framework

There are many unit-test frameworks in rust. This framework focuses on integration-testing, that means external software, not necessarily written in rust.

rtest works by using stateful resources. It uses macros to build a executable binary that can handle all your filters and returns a nice output.

Imagine you are hosting a webshop and want to verify it works with integration-tests.

#[derive(rtest_derive::Resource)]
struct Orderinfo {
    item: String,
}
#[derive(rtest_derive::Resource)]
struct Order(String);

#[derive(Debug, thiserror::Error)]
pub enum ShopError {
    #[error("{0}")]
    Network(#[from] reqwest::Error),
}
impl rtest::TestError for ShopError {}

const SHOP: &str = "http://shop.example.com";

#[rtest_derive::rtest]
async fn place_order(info: Orderinfo) -> Result<Order, ShopError> {
    let client = reqwest::Client::new();
    let url = format!("{}/v1/order/{}", SHOP, info.item);
    let id = client.post(url).send().await?.text().await?;
    Ok(Order(id))
}

#[rtest_derive::rtest]
async fn check_order(order: Order) -> Result<Order, ShopError> {
    let res = reqwest::get(format!("{}/v1/order/{}", SHOP, order.0)).await?;
    assert_ne!(res.status(), 404);
    Ok(order)
}

#[rtest_derive::rtest]
async fn cancel_order(order: Order) -> Result<(), ShopError> {
    let client = reqwest::Client::new();
    let res = client.delete(format!("{}/v1/order/{}", SHOP, order.0)).send().await?;
    assert_eq!(res.status(), 200);
    Ok(())
}

pub fn main() -> std::process::ExitCode {
    let water = Orderinfo {
        item: "water".to_string(),
    };
    let pizza = Orderinfo {
        item: "pizza".to_string(),
    };
    let runconfig = rtest::RunConfig {
        context: rtest::Context::default().with_resource(water).with_resource(pizza),
        ..Default::default()
    };
    rtest_derive::run!(runconfig)
}

The test framework will know that in order to test the check_order function it first needs to have a Order. But the only way to generate such an order is through the place_order test. cancel_order will consume the order and no longer make it useable.

Yes, you can trick it by removing all tests that generate an Order - the framework will notice that on runtime and fail. It might be possible that multiple routes are valid to test all functions, in case of an error the route it took will be dumped. Let's assume checking an order with water will fail. The framework might decide to create another Order with pizza because it cannot verify deletion otherwise, tests might be executed multiple times.

rtest Results:
[✓]
  [✓] delete_file
[✓] create
  [✓] create_file
  [✓] setup_fileinfo
[x] read
  [✓] optional_test
  [✓] read_metadata
  [x] test_that_should_fail
      --- Run: 1/1 ---
      Error: test failure: No such file or directory (os error 2)
      Logs:
        2024-05-14T10:11:59.824647Z  INFO filesystem: Wubba Lubba dub-dub
  [x] test_that_should_panic - 183ms
      --- Run: 1/1 ---
      Error: Panic: 'Yes Rico, Kaboom'
      at rtest/examples/filesystem/main.rs:88
      Stacktrace:
         0: rust_begin_unwind
                   at /rustc/098d4fd74c078b12bfc2e9438a2a04bc18b393bc/library/std/src/panicking.rs:647:5
         1: core::panicking::panic_fmt
                   at /rustc/098d4fd74c078b12bfc2e9438a2a04bc18b393bc/library/core/src/panicking.rs:72:14
         2: filesystem::test_that_should_panic
                   at rtest/examples/filesystem/main.rs:88:5
      Logs:
        2024-05-14T10:11:59.641008Z  INFO filesystem: Kaboom?
Total Tests: 7. Total Runs: 6 Errors: 2
Failed

Examples

Execute the webshop with one test marked optional, rerun it and display the results:

cargo run --example=webshop -- run --optional-tests=run_test_on_2_servers -o rtest_result.json
cargo run --example=webshop -- re-run rtest_result.json
cargo run --example=webshop -- display rtest_result.json

Features:

• Allow any Input/Output Resources
• Custom Errors
• Custom Context (though rarely needed)
• Execution Model that takes costs (of processes/resources) into account
• Multithread support
• Async Support
• Capture logs
• Capture Panics (needed for asserts)
• Capture println
• External Log Capturing, e.g. capture logs of a kubernetes pod during test exection via an adapter.
• Json Input/Output to persistent runs, retry runs, compare with previous runs
• Markdown Output