Crate tower_resilience_chaos 

Chaos engineering layer for Tower services.

This crate provides a chaos engineering layer that can inject failures and latency into Tower services for testing resilience patterns. It’s designed to help you verify that your circuit breakers, retries, timeouts, and other resilience mechanisms work correctly under adverse conditions.

Features

• Error Injection: Inject errors at a configurable rate
• Latency Injection: Add random delays to requests
• Deterministic Testing: Use seeds for reproducible chaos
• Event System: Monitor chaos injection via event listeners
• Composable: Works with all other tower-resilience patterns

Safety

WARNING: This layer is intended for testing and development only. Never use it in production environments. Consider using feature flags or environment checks to ensure chaos layers are only enabled in non-production environments.

Basic Example

Latency-Only Chaos (no type parameters needed!)

use tower::ServiceBuilder;
use tower_resilience_chaos::ChaosLayer;
use std::time::Duration;

// No type parameters required for latency-only chaos!
let chaos = ChaosLayer::builder()
    .name("api-chaos")
    .latency_rate(0.2)  // 20% of requests delayed
    .min_latency(Duration::from_millis(50))
    .max_latency(Duration::from_millis(200))
    .build();

let service = ServiceBuilder::new()
    .layer(chaos)
    .service_fn(|req: String| async move {
        Ok::<String, std::io::Error>(format!("Response to: {}", req))
    });

Error Injection (types inferred from closure)

use tower::ServiceBuilder;
use tower_resilience_chaos::ChaosLayer;
use std::time::Duration;

// Types inferred from the error_fn closure signature
let chaos = ChaosLayer::builder()
    .name("api-chaos")
    .error_rate(0.1)  // 10% of requests fail
    .error_fn(|_req: &String| {
        std::io::Error::new(std::io::ErrorKind::Other, "chaos error!")
    })
    .latency_rate(0.2)  // 20% of remaining requests delayed
    .min_latency(Duration::from_millis(50))
    .max_latency(Duration::from_millis(200))
    .build();

let service = ServiceBuilder::new()
    .layer(chaos)
    .service_fn(|req: String| async move {
        Ok::<String, std::io::Error>(format!("Response to: {}", req))
    });

Testing Circuit Breakers

use tower::ServiceBuilder;
use tower_resilience_chaos::ChaosLayer;
use tower_resilience_circuitbreaker::CircuitBreakerLayer;
use std::time::Duration;

// Create a chaos layer that fails 60% of requests
// Types inferred from closure signature
let chaos = ChaosLayer::builder()
    .name("circuit-breaker-test")
    .error_rate(0.6)
    .error_fn(|_req: &String| {
        std::io::Error::new(std::io::ErrorKind::Other, "simulated failure")
    })
    .build();

// Wrap with a circuit breaker
let circuit_breaker = CircuitBreakerLayer::builder()
    .name("test-breaker")
    .failure_rate_threshold(0.5)
    .sliding_window_size(10)
    .build();

let service = ServiceBuilder::new()
    .layer(circuit_breaker)
    .layer(chaos)
    .service_fn(|req: String| async move {
        Ok::<String, std::io::Error>(req)
    });

// Make requests - circuit breaker should open after ~5 failures

Deterministic Testing

Use a seed for reproducible chaos injection:

use tower_resilience_chaos::ChaosLayer;

let chaos = ChaosLayer::builder()
    .error_rate(0.5)
    .error_fn(|_req: &()| {
        std::io::Error::new(std::io::ErrorKind::Other, "chaos")
    })
    .seed(42)  // Same seed = same sequence of failures
    .build();

// Running the same test multiple times will produce the same results

Event Monitoring

Track chaos injection with event listeners:

use tower_resilience_chaos::ChaosLayer;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;

let errors = Arc::new(AtomicUsize::new(0));
let latencies = Arc::new(AtomicUsize::new(0));

let e = errors.clone();
let l = latencies.clone();

let chaos = ChaosLayer::builder()
    .error_rate(0.1)
    .error_fn(|_req: &()| {
        std::io::Error::new(std::io::ErrorKind::Other, "chaos")
    })
    .latency_rate(0.2)
    .on_error_injected(move || {
        e.fetch_add(1, Ordering::SeqCst);
    })
    .on_latency_injected(move |delay: Duration| {
        l.fetch_add(1, Ordering::SeqCst);
    })
    .build();

// After running tests, check counters
println!("Errors injected: {}", errors.load(Ordering::SeqCst));
println!("Latencies injected: {}", latencies.load(Ordering::SeqCst));

Latency Injection Only

Test timeout handling without errors (no type parameters needed!):

use tower_resilience_chaos::ChaosLayer;
use std::time::Duration;

// No type parameters required for latency-only chaos!
let chaos = ChaosLayer::builder()
    .latency_rate(0.5)  // 50% of requests delayed
    .min_latency(Duration::from_millis(100))
    .max_latency(Duration::from_millis(500))
    .build();

// Use with TimeLimiter to test timeout behavior

Re-exports

pub use config::ChaosConfig;

pub use config::ChaosConfigBuilder;

pub use config::ChaosConfigBuilderWithRate;

pub use config::CustomErrorFn;

pub use config::ErrorInjector;

pub use config::NoErrorInjection;

pub use events::ChaosEvent;

pub use layer::ChaosLayer;

pub use service::Chaos;

Modules

config

Configuration types for chaos injection. Configuration for chaos engineering layer.

events

Event types emitted by chaos injection. Event types for chaos engineering layer.

layer

Tower Layer implementation for chaos injection. Tower layer for chaos engineering.

service

Tower Service implementation for chaos injection. Chaos service implementation.