Crate heckcheck

A heckin small test case generator.

What is test case generation?

A test case generator is a program which writes programs to test other programs. The idea is that we can find more bugs if we test more paths in a program, and the best way to do that is to provide a wide range of inputs to our programs. And the best way to do that is to write a program designed to generate inputs. You may have heard this being referred to as “fuzzing” or “property testing” as well.

Examples

This is a basic roundtrip test for an RGB serializer and parser, which ensures that the output matches the original input.

use heckcheck::prelude::*;

/// A color value encoded as Red-Green-Blue
#[derive(Clone, Debug, Arbitrary, PartialEq)]
pub struct Rgb {
    pub r: u8,
    pub g: u8,
    pub b: u8,
}

impl Rgb {
    /// Convert from RGB to Hexadecimal.
    pub fn to_hex(&self) -> String {
        format!("#{:02X}{:02X}{:02X}", self.r, self.g, self.b)
    }

    /// Convert from Hexadecimal to RGB.
    pub fn from_hex(s: String) -> Self {
        let s = s.strip_prefix('#').unwrap();
        Rgb {
            r: u8::from_str_radix(&s[0..2], 16).unwrap(),
            g: u8::from_str_radix(&s[2..4], 16).unwrap(),
            b: u8::from_str_radix(&s[4..6], 16).unwrap(),
        }
    }
}

// Validate values can be converted from RGB to Hex and back.
heckcheck::check(|rgb: Rgb| {
    let hex = rgb.to_hex();
    let res = Rgb::from_hex(hex);
    assert_eq!(rgb, res);
    Ok(())
});

Philosophy

We believe that test case generation is an essential tool in modern programmer’s toolboxes, and both fuzzing and property testing play a role in this. Just like cargo-fuzz, heckcheck relies on the stable Arbitrary trait to modify structured data. This means that all code in a crate can just implement Arbitrary in order to be hooked up to a variety of different test case generation tools.

Unlike some other tools, heckcheck is also extensible. The built-in shrinker and permutation algorithms aren’t necessarily the best in class. But they can be modified in order to be improved. The end vision would be to see something like Arbitrary, cargo-fuzz, and heckcheck provided out of the box by the ::test module.

Replaying tests

When a test generated by heckcheck fails, we print a base64 string of the input data as part of the panic message. You can pass this string together with the failing code to heckcheck::replay to create a reproduction. This makes it quick to go from finding a failing test to having a reproduction available.

Conditional derives

If you only want to only derive Default when the code is being tested, you can use cfg_attr(test) to only compile it when code is being tested.

#[cfg_attr(test, derive(Arbitrary))]
#[derive(Clone, Debug, PartialEq)]
pub struct Rgb {
    pub r: u8,
    pub g: u8,
    pub b: u8,
}

Acknowledgements

This crate was built thanks to the work of the cargo-fuzz team on the arbitrary crate. It has been heavily inspired by burntsushi’s work on quickcheck. And we would like to thank Boxy who solved a critical bug that almost made us give up.

Re-exports

• pub use arbitrary;

Modules

• prelude
  The heckcheck prelude

Structs

• HeckCheck
  The main test checker.
• Shrinker
  The default test case shrinker.

Enums

• ShrinkReport
  The result of a shrinking pass.

Traits

• Shrink
  Test case shrinking.

Functions

• check
  Check a target.
• replay
  Replay a failing test from a base64 string.