pub struct Attrition {
pub max_dead: usize,
pub prob_graceful: f64,
pub prob_crash: f64,
pub prob_wipe: f64,
pub recovery_delay_ms: Option<Range<usize>>,
pub grace_period_ms: Option<Range<usize>>,
pub scope: AttritionScope,
}Expand description
Built-in attrition configuration for automatic process reboots.
Provides a default chaos mechanism that randomly kills and restarts server
processes during the chaos phase. For custom fault injection strategies,
implement FaultInjector instead.
§Probabilities
The prob_* fields are weights that get normalized internally. They don’t
need to sum to 1.0, but all must be non-negative.
§Example
Attrition {
max_dead: 1,
prob_graceful: 0.3,
prob_crash: 0.5,
prob_wipe: 0.2,
recovery_delay_ms: None,
grace_period_ms: None,
scope: AttritionScope::PerProcess,
}Fields§
§max_dead: usizeMaximum number of simultaneously dead processes.
The attrition injector will not kill a process if the number of currently dead (not yet restarted) processes is already at this limit.
prob_graceful: f64Weight for RebootKind::Graceful reboots.
prob_crash: f64Weight for RebootKind::Crash reboots.
prob_wipe: f64Weight for RebootKind::CrashAndWipe reboots.
recovery_delay_ms: Option<Range<usize>>Recovery delay range in milliseconds.
After a process is killed (crash or force-kill after grace), it restarts after a seeded random delay drawn from this range.
Defaults to 1000..10000 (1-10 seconds) if not set.
grace_period_ms: Option<Range<usize>>Grace period range in milliseconds (for graceful reboots).
After the per-process shutdown token is cancelled, the process has this long to clean up before being force-killed. The actual duration is a seeded random value from this range.
Defaults to 2000..5000 (2-5 seconds) if not set.
scope: AttritionScopeThe failure domain each reboot targets.
AttritionScope::PerProcess (the default) kills one random process at
a time. AttritionScope::PerMachine / AttritionScope::PerZone kill
all collocated processes together, modeling correlated failure; they
require a .cluster()
topology and respect max_dead as a whole-group budget.
Implementations§
Source§impl Attrition
impl Attrition
Sourcepub fn swarm_for_seed(&self) -> Attrition
pub fn swarm_for_seed(&self) -> Attrition
Derive a per-seed swarm reboot regime from this base configuration.
Implements swarm testing (Groce et al., ISSTA 2012) for attrition: each seed exercises a random reboot regime rather than the fixed configured one. This surfaces bug classes that a single fixed regime hides — most importantly the never-reboot case (needed to find slow leaks / timer overflows) and single-mode cases (“always crash”, “graceful-only”).
Draws exactly four config_random_bool values from the independent
CONFIG_RNG stream (fixed sequence ⇒ reproducible per seed, never
perturbs in-run randomness). The first draw, with ~50% probability, sets
max_dead = 0 — the never-reboot regime, where the injector’s
dead_count() >= max_dead gate is always true so it never reboots. The
remaining three each mask one reboot-kind weight to 0.0 with ~50%
probability.
When all three kind weights are masked off, choose_kind
falls back to RebootKind::Crash — the “always crash” single-mode regime.