Crate cliff

Find the load at which a benchmark falls over.

Most good benchmarks allow you to vary the offered load to the system, and then give you output that indicate whether the system-under-test is keeping up. This could be dropped packets, latency spikes, or whatever else is appropriate for the problem domain. Now, you want to find out how far you can push your system until it falls over. How do you do that?

This crate provides one answer: binary search. The idea is simple: first, you double offered load until the system falls over. As long as the system keeps up, you raise the lower bound of your estimate for the maximum tolerated load. When the system no longer keeps up, that gives you an upper limit on the throughput your system can support. At that point, you perform a binary search between the upper and lower bounds, tightening the range until you reach the fidelity you want.

So that you can easily support manual override, LoadIterator also supports a linear mode. In this mode, cliff just walks a pre-defined list of loads, and stops when the test runner indicates that the system is no longer keeping up.

Examples

use cliff::LoadIterator;

// First, we set the starting load. This is the initial lower bound.
let mut load = LoadIterator::search_from(500);
// The initial lower bound is the first load we try.
assert_eq!(load.next(), Some(500));
// Since we did not say that the system was overloaded,
// the iterator next produces twice the load of the previous step.
assert_eq!(load.next(), Some(1000));
// Same thing again.
assert_eq!(load.next(), Some(2000));
// Now, let's say the system did not keep up with the last load level:
load.failed();
// At this point, cliff will begin a binary search between
// 1000 (the highest supported load)
// and
// 2000 (the lowest unsupported load).
// The middle of that range is 1500, so that's what it'll try.
assert_eq!(load.next(), Some(1500));
// Let's say that succeeded.
// That means the cliff must lie between 1500 and 2000, so we try 1750:
assert_eq!(load.next(), Some(1750));
// And if that failed ...
load.failed();
// ... then the cliff must lie between 1500 and 1750, and so on.
// Ultimately, we reach the desired fidelity,
// which defaults to half the initial lower bound (here 250).
// At that point, no more benchmark runs are performed.
assert_eq!(load.next(), None);

Structs

LoadIterator

An iterator that helps determine maximum supported load for a system.