Crate monkey_test

source ·
Expand description

§Monkey Test

Monkey Test is a property based testing (PBT) tool like QuickCheck (Wikipedia) (github) and similar libraries.

§Contents

§Property based testing core concepts

PBT is a complement to normal unit testing.

A normal unit test uses a singel specific example input and verifies it against a specific outcome.

// Unit tests with hard coded examples with expected results
assert_eq!(1_f64.sqrt(), 1_f64);
assert_eq!(4_f64.sqrt(), 2_f64);
assert_eq!(9_f64.sqrt(), 3_f64);
assert_eq!(16_f64.sqrt(), 4_f64);

With PBT a property of your code is validated against an arbitrary number of generated examples. A propery can loose some specificity, but can usually say something more general about the code under test compared to a specific test example and outcome. Further, using random examples in test can find aspects you missed when manually choosing examples to test.

// Testing more general properties
use monkey_test::*;

monkey_test()
    .with_generator(gen::f64::ranged(2.0..))
    .title("Lower bound")
    .assert_true(|n| n.sqrt() > 1.0)
    .title("Upper bound")
    .assert_true(|n| n.sqrt() < n);

So, what is the point of having the loose boundaries in the sqrt-properties tested above? Is there any value in testing these properties?

The answer is that usually when code goes wrong or has a bug, the return value is not just a little bit off, but many times it is way off and fail spectacularly, like returning a negative value or panicing.

In short, combining general property based tests with some specific unit tests is a powerful testing technique to both specify the precise behaviour and finding bugs you did not forsee yourself.

§Nomenclature

  • Generator - A source of random examples
  • Property - Your parameterized test
  • Shrinker - Generate smaller examples based on failing example, in order to simplify the failure

§Common classes of properties

How do you write a useful property that is valid for all generated examples? One baby step is to try parameterize an already existing example based test. As an inspiration, here follow some common classes of properties to test.

§No explosion

Just shoot examples at code under test and make sure there are no errors and no panics.

ⓘ
// Testing that there are no panics, but will panic on division by zero for
// example range 700..800.
use monkey_test::*;

monkey_test()
   .with_example_count(1_000)
   .with_generator(gen::i32::ranged(-10_000..10_000))
   .assert_no_panic(|n| { let _ = 1/(n / 100 - 7); });
ⓘ
// Testing that there are no Err, but will fail on -1 and lower values.
use monkey_test::*;

monkey_test()
   .with_generator(gen::i8::any())
   .assert_true(|n| u8::try_from(n).is_ok() );

§Simplification

Since the examples used are random, i.e. unknown, it can be hard to be specific about what result to expect. However, using the “loose boundaries” principle, you can usually at least specify some relaxed model - a simplified property to verify.

// We can at least specify that all results from `abs()` should be positive, if
// not being i8 minimum value which does not have a positive counterpart.
use monkey_test::*;

monkey_test()
   .with_generator(gen::i8::any())
   .assert_true(|n| n == i8::MIN || n.abs() >= 0 );

§Symmetry

Apply a function and its inverse and make sure you get back the same initial value. Some examples of this is write and read back something, like save to file system and load it again. Another example is to generate some ground truth data, transform it to an input format and make sure your business logic calculate an answer that corresponds to the ground truth data.

// Testing the serialize and deserialize round trip for a parser.
use monkey_test::*;

monkey_test()
   .with_generator(gen::u32::any())
   .assert_eq(|n| n, |n| n.to_string().parse::<u32>().unwrap());

§Idempotens

Applying the same function many times generate the same result. This can be useful for showing that there is no hidden state affecting the result of the code under test.

// Absolute value stays the same through several applications of "abs".
use monkey_test::*;

monkey_test()
   .with_generator(gen::i8::ranged(-127..))
   .assert_eq(|n| n.abs(), |n| n.abs().abs().abs().abs());

§Invariance

A property or value that is not changed by some specific operation or transformation.

// The negation operation does not affect the absolute value.
use monkey_test::*;

monkey_test()
   .with_generator(gen::i8::ranged(-127..))
   .assert_eq(|n| n.abs(), |n| (-n).abs());

§Oracle

Compare the function result against other trusted means to get the same result. Perhaps compare output to a model of one aspect, analogous function, other existing implementation, unoptimized code or legacy code. Perhaps compare the output of old and new code to enable reckless refactoring.

// Operation "n * 2" can be double checked with the oracle function "n + n",
// an analogous function to get to the same result.
use monkey_test::*;

monkey_test()
   .with_generator(gen::u8::ranged(..128))
   .assert_eq(|n| n + n, |n| n * 2 );

§Induction

Show that some property holds for P(0) and that P(n) + C = P(n+1).

// Via induction show that Vec::len() works as expected
use monkey_test::*;

// Induction base case with empty vector
assert_eq!{Vec::<i64>::new().len(), 0};

// Induction general case
monkey_test()
   .with_generator(gen::vec::any(gen::u8::any()))
   .assert_eq(
      |vec| vec.len() + 1,
      |mut vec| {
         vec.push(42 /* Add a single arbitrary item to vec */);
         vec.len()
      });

§Stateful testing

As one example, execute a series of commands against a stateful system, to then verify some property of the system.

ⓘ
// Make sure counter value is the same as the sum of all increments applied.
// In this case, it will fail with a series of increments like the shrunken
// failure example [-44, -128, -111, -128, -90] and failure
// reason "Actual value should equal expected -501, but got -500".

use monkey_test::*;

// The stateful system under test
struct Counter{acc: i64}
impl Counter {
   fn add(&mut self, value: i8) {
      self.acc += value as i64;
      // Buggy if statement...
      if self.acc < -500 {
         self.acc = -500;
      }
   }
}

monkey_test()
   .with_generator(gen::vec::any(gen::i8::any()))
   .assert_eq(
      // Expected sum of increments
      |vec| vec.iter().cloned().map(|x| x as i64).sum(),
      // Actual counter accumulation
      |vec| {
         let mut counter = Counter{ acc:0 };
         vec.iter().cloned().for_each(|x| counter.add(x));
         counter.acc
      });

Another example on stateful testing can be to poke at a stateful API with a random sequence of legal commands and verify that the API does not panic.

§Features

For a complete guide of all features in Monkey Test, refer to the generated API documentation (docs.rs). Additional usage examples can be found in code repository test folder (github.com). A summary is given below.

§Generators and shrinkers for basic types

Generators for bool, f32, f64 and for all integer types i8, i16, i32, i64, i128, isize, u8, u16 u32, u64 , u128 and usize.

use monkey_test::*;
let bytes = gen::u8::any();
let some_longs = gen::i64::ranged(10..=20);
let mostly_true = gen::bool::with_ratio(1,20);

There are some more specialized generators. In module gen::sized there are generators that return progressively larger and lager values, suitable for controlling the size of generated collections. In module gen::fixed there are generators that do not use randomness, which can be useful some times.

use monkey_test::*;
let progressively_larger_sizes: BoxGen<usize> = gen::sized::default();
let always_the_same_value: BoxGen<i32> = gen::fixed::constant(42);

§Generators and shrinkers for collections

There is a generator and a shrinker for vectors.

use monkey_test::*;
let int_vectors: BoxGen<Vec<i16>> = gen::vec::any(gen::i16::any());

monkey_test()
   .with_generator(int_vectors)
   .test_true(|vec| vec.iter().all(|&n| n <= 1337) )
   .assert_minimum_failure(vec![1338]);

§Pick values and mix generators

Create generators that pick among values and mix values from different generators

use monkey_test::*;
let fruits = gen::pick_evenly(&["banana", "apple", "orange"]);
let nuts = gen::pick_evenly(&["peanut", "almond", "pecan"]);
let snacks = gen::mix_with_ratio(&[(3, nuts), (1, fruits)]);

§Compose generators and shrinkers for more complex types

Generators and shrinkers for more complex types can be constructed from more basic ones, using one of zip, zip_3, …, zip_6 together with map. When constructing generators this way, you automatically also get a shrinker for the complex type.

use monkey_test::*;

#[derive(Clone)]
struct Point {x: u16, y: u16}

let points: BoxGen<Point> = gen::u16::any()
   .zip(gen::u16::any())
   .map(|(x, y)| Point{x, y}, |p| (p.x, p.y));

#[derive(Clone)]
struct Color {r: u8, g: u8, b: u8, a: u8}

let colors: BoxGen<Color> = gen::u8::any()
   .zip_4(gen::u8::any(), gen::u8::any(), gen::u8::any())
   .map(|(r, g, b, a)| Color{r, g, b, a}, |c| (c.r, c.g, c.b, c.a));

§Create generators and shrinkers from scratch

For implemementing a generator on your own, you only need to implement the Gen trait.

use monkey_test::*;
// Use the randomization source of your choosing
use rand::Rng;
use rand::SeedableRng;

#[derive(Clone)]
struct DiceGen {
   /// Die side count
   side_count: u32,
}

impl Gen<u32> for DiceGen {
    fn examples(&self, seed: u64) -> BoxIter<u32> {
        let distr =
            rand::distributions::Uniform::new_inclusive(1, self.side_count);
        let iter =
            rand_chacha::ChaCha8Rng::seed_from_u64(seed).sample_iter(distr);
        Box::new(iter)
    }

    fn shrinker(&self) -> BoxShrink<u32> {
        shrink::int()
        // Use shrink::none() for not providing any shrinking.
    }
}

fn dice_throw_generator_from_struct(side_count: u32) -> BoxGen<u32> {
    Box::new(DiceGen { side_count })
}

Some boilerplate code can be eliminated and the same functionality can be achieved by using gen::from_fn instead of implementing the Gen trait.

use monkey_test::*;
use rand::Rng;
use rand::SeedableRng;

fn dice_throw_generator_from_fn(side_count: u32) -> BoxGen<u32> {
    gen::from_fn(move |seed| {
        let distr = rand::distributions::Uniform::new_inclusive(1, side_count);
        rand_chacha::ChaCha8Rng::seed_from_u64(seed).sample_iter(distr)
    })
    .with_shrinker(shrink::int())
}

Similarly, a shrinker can be implemented by either implementing the Shrink trait directly, or just make use of shrink::from_fn.

§Key design principles of Monkey Test

  • configurability and flexibility - Leave a high degree of configurability and flexibility to the user by letting most details to be specified programatically. The aim is to have an declarative builder-style API like the Java library QuickTheories (github).

  • powerful shinking - Good shrinkers is a really important aspect of a property based testing tool. Let say that the failing example is a vector of 1000 elements and only 3 of the elements in combination is the actual failure cause. You are then unlikely to find the 3-element combination, if the shrinking is not powerful enough.

  • composability for complex test examples - Basic type generators and shrinkers are provided out of the box. User should also be able to genereate and shrink more complex types, by composing together more primitive generators and shrinkers into more complex ones. The main inspiration here is the Scala library ScalaCheck (homepage), which is fenomenal in this aspect, having the power to for example easily generate and shrink recursive data structures, by using composition.

  • minimize macro magic - In order to keep the tool simple, just avoid macros if same developer experience can be provided using normal Rust code. Macros-use is an complex escape hatch only to be used when normal syntax is insufficient.

Modules§

  • gen
    The gen module contains built in generators.
  • shrink
    The shrink module contains built in shrinkers.

Structs§

  • Conf
    Configuration for executing monkey tests.
  • ConfAndGen
    Configuration for executing monkey tests, including the generator.

Enums§

  • MonkeyResult
    Result summary from evaluation of a property tested.

Traits§

  • Gen
    The generator trait, for producing example values to test in a property.
  • MapWithGen
    Non-object-safe trait for providing generator mapping.
  • MapWithShrink
    Non-object-safe trait for providing shrinker mapping.
  • Shrink
    The shrinker trait, for shrinking a failed example values into smaller ones. What is determined as a smaller value can be subjective and is up to author or tester to determine, but as a rule of thumb a smaller value should be easier to interpret, when a property is proven wrong.
  • ZipWithGen
    Non-object-safe trait for providing generator zipping.
  • ZipWithShrink
    Non-object-safe trait for providing shrinker zipping.

Functions§

  • monkey_test
    Main entry point for writing property based tests using the monkey-test tool.
  • seed_to_use
    The standard source to get randimization seed from.

Type Aliases§

  • BoxGen
    A boxed generator of example type E
  • BoxIter
    A boxed iterator of example type E
  • BoxShrink
    A boxed shrinker of example type E
  • Property
    A property is something that should hold, for all given examples.