pub struct ChaosConfiguration {Show 22 fields
pub clog_probability: f64,
pub clog_duration: Range<Duration>,
pub partition_probability: f64,
pub partition_duration: Range<Duration>,
pub bit_flip_probability: f64,
pub bit_flip_min_bits: u32,
pub bit_flip_max_bits: u32,
pub bit_flip_cooldown: Duration,
pub partial_write_max_bytes: usize,
pub partial_read_max_bytes: usize,
pub random_close_probability: f64,
pub random_close_cooldown: Duration,
pub random_close_explicit_ratio: f64,
pub clock_drift_enabled: bool,
pub clock_drift_max: Duration,
pub buggified_delay_enabled: bool,
pub buggified_delay_max: Duration,
pub buggified_delay_probability: f64,
pub connect_failure_mode: ConnectFailureMode,
pub connect_failure_probability: f64,
pub max_pair_latency: Range<Duration>,
pub partition_strategy: PartitionStrategy,
}Expand description
Configuration for chaos injection in simulations.
This struct contains all settings related to fault injection and chaos testing,
following FoundationDB’s BUGGIFY patterns for deterministic testing.
Fields§
§clog_probability: f64Clogging probability for individual writes (0.0 - 1.0)
clog_duration: Range<Duration>Duration range for clog delays
partition_probability: f64Network partition probability (0.0 - 1.0)
partition_duration: Range<Duration>Duration range for network partitions
bit_flip_probability: f64Bit flip probability for packet corruption (0.0 - 1.0)
bit_flip_min_bits: u32Minimum number of bits to flip (power-law distribution lower bound)
bit_flip_max_bits: u32Maximum number of bits to flip (power-law distribution upper bound)
bit_flip_cooldown: DurationCooldown duration after bit flip to prevent excessive corruption
partial_write_max_bytes: usizeMaximum bytes for partial write simulation (BUGGIFY truncates writes to 0-max_bytes) Following FDB’s approach of truncating writes to test TCP backpressure handling
partial_read_max_bytes: usizeMaximum bytes for partial read simulation (BUGGIFY truncates reads to 1-max_bytes)
Mirrors FDB’s Sim2Conn partial delivery on the receiver to test short reads
and message reassembly. Always delivers at least one byte so a partial read is
never mistaken for EOF.
random_close_probability: f64Random connection close probability per I/O operation (0.0 - 1.0) FDB default: 0.00001 (0.001%) - see sim2.actor.cpp:584
random_close_cooldown: DurationCooldown duration after a random close event (prevents cascading failures) FDB uses connectionFailuresDisableDuration - see sim2.actor.cpp:583
random_close_explicit_ratio: f64Ratio of explicit exceptions vs silent failures (0.0 - 1.0) FDB default: 0.3 (30% explicit) - see sim2.actor.cpp:602
clock_drift_enabled: boolEnable clock drift simulation
When enabled, timer() can return a time up to clock_drift_max ahead of now()
FDB ref: sim2.actor.cpp:1058-1064
clock_drift_max: DurationMaximum clock drift (default 100ms per FDB)
timer() can be up to this much ahead of now()
buggified_delay_enabled: boolEnable buggified delays on sleep/timer operations When enabled, 25% of sleep operations get extra delay FDB ref: sim2.actor.cpp:1100-1105
buggified_delay_max: DurationMaximum additional delay for buggified sleep (default 100ms)
Uses power-law distribution: max_delay * pow(random01(), 1000.0)
FDB ref: sim2.actor.cpp:1104
buggified_delay_probability: f64Probability of adding buggified delay (default 25% per FDB)
connect_failure_mode: ConnectFailureModeConnection establishment failure mode (per FDB)
FDB ref: sim2.actor.cpp:1243-1250 (SIM_CONNECT_ERROR_MODE)
connect_failure_probability: f64Probability of connect failure when Probabilistic mode is enabled (default 50%)
max_pair_latency: Range<Duration>Permanent per-IP-pair latency range (FDB SimClogging::MAX_CLOGGING_LATENCY).
Each ordered IP pair samples a fixed latency from this range at first contact
(via sample_duration), then adds it to every delivery on that pair for the
whole run — modelling a stably-slow link. An all-zero range (end is zero)
disables it, leaving behavior unchanged. FDB’s MAX * random01() is the
ZERO..MAX case. FDB ref: sim2.actor.cpp ~294-299, 352-354.
partition_strategy: PartitionStrategyNetwork partition strategy.
Controls how nodes are selected for partitioning.
TigerBeetle ref: packet_simulator.zig partition modes
§Real-World Scenario
Different strategies test different failure scenarios:
- Random: unpredictable chaos
UniformSize: various quorum sizesIsolateSingle: single node isolation (common in production)
Implementations§
Source§impl ChaosConfiguration
impl ChaosConfiguration
Sourcepub fn disabled() -> Self
pub fn disabled() -> Self
Create a configuration with all chaos disabled (for fast local testing)
Sourcepub fn random_for_seed() -> Self
pub fn random_for_seed() -> Self
Create a randomized chaos configuration for seed-based testing
Sourcepub fn swarm_for_seed() -> Self
pub fn swarm_for_seed() -> Self
Create a swarm-testing chaos configuration for seed-based testing.
Starts from random_for_seed, then disables each
fault family with ~50% probability (drawn from the independent CONFIG_RNG
stream). This implements swarm testing (Groce et al., ISSTA 2012): each
seed exercises a random subset of fault families — including the all-off
subset — instead of every family being slightly on at once (which lets
families crowd each other out, the passive-suppression anti-pattern).
Sourcepub fn apply_buggify_knobs(&mut self)
pub fn apply_buggify_knobs(&mut self)
Spike selected fault magnitudes under buggify (FDB’s
if (randomize && BUGGIFY) KNOB = random(lo, hi)).
Composes on top of random_for_seed /
swarm_for_seed: each knob keeps its sampled value
unless its own buggify_knob! call site fires for the
seed, in which case it jumps to an aggressive value within bounds. A
representative subset — extend by adding more buggify_knob! lines.
Trait Implementations§
Source§impl Clone for ChaosConfiguration
impl Clone for ChaosConfiguration
Source§fn clone(&self) -> ChaosConfiguration
fn clone(&self) -> ChaosConfiguration
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more