Nine Lives 🐱

Resilience patterns for Rust with algebraic composition.

Nine Lives provides battle-tested resilience patterns (retry, circuit breaker, bulkhead, timeout) as composable tower layers with a unique algebraic composition system.

Features

• 🔁 Retry policies with exponential/linear/constant backoff and jitter
• ⚡ Circuit breakers with half-open state recovery
• 🚧 Bulkheads for concurrency limiting and resource isolation
• ⏱️ Timeout policies integrated with tokio
• 🧮 Algebraic composition via intuitive operators (+, |, &)
• 🏎️ Fork-join for concurrent racing (Happy Eyeballs pattern)
• 🔒 Lock-free implementations using atomics
• 🏗️ Tower-native - works with any tower Service

Quick Start

Add to your Cargo.toml:

[dependencies]
ninelives = "0.1"
tower = "0.5"
tokio = { version = "1", features = ["full"] }

Basic Usage

use ninelives::prelude::*;
use std::time::Duration;
use tower::{Service, ServiceBuilder, ServiceExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Apply a timeout to any service
    let mut svc = ServiceBuilder::new()
        .layer(TimeoutLayer::new(Duration::from_secs(1))?)
        .service_fn(|req: &str| async move {
            Ok::<_, std::io::Error>(format!("Response: {}", req))
        });

    let response = svc.ready().await?.call("hello").await?;
    println!("{}", response);
    Ok(())
}

Algebraic Composition - The Nine Lives Advantage

Compose resilience strategies using intuitive operators:

• Policy(A) + Policy(B) - Sequential composition: A wraps B
• Policy(A) | Policy(B) - Fallback: try A, fall back to B on error
• Policy(A) & Policy(B) - Fork-join: try both concurrently, return first success

Precedence: & > + > | (like * > + > bitwise-or in math)

Example: Fallback Strategy

Try an aggressive timeout first, fall back to a longer timeout on failure:

use ninelives::prelude::*;
use std::time::Duration;
use tower::{ServiceBuilder, Layer};

let fast = Policy(TimeoutLayer::new(Duration::from_millis(100))?);
let slow = Policy(TimeoutLayer::new(Duration::from_secs(5))?);
let policy = fast | slow;

let svc = ServiceBuilder::new()
    .layer(policy)
    .service_fn(|req| async { Ok::<_, std::io::Error>(req) });

Example: Fork-Join (Happy Eyeballs)

Race two strategies concurrently and return the first success:

use ninelives::prelude::*;
use std::time::Duration;

// Create two timeout policies with different durations
let ipv4 = Policy(TimeoutLayer::new(Duration::from_millis(100))?);
let ipv6 = Policy(TimeoutLayer::new(Duration::from_millis(150))?);

// Race them concurrently - first success wins
let policy = ipv4 & ipv6;

let svc = ServiceBuilder::new()
    .layer(policy)
    .service_fn(|req| async { Ok::<_, std::io::Error>(req) });

Example: Multi-Tier Resilience

Combine multiple strategies with automatic precedence:

use ninelives::prelude::*;
use std::time::Duration;

// Aggressive: just a fast timeout
let aggressive = Policy(TimeoutLayer::new(Duration::from_millis(50))?);

// Defensive: nested timeouts for retries
let defensive = Policy(TimeoutLayer::new(Duration::from_secs(10))?)
              + Policy(TimeoutLayer::new(Duration::from_secs(5))?);

// Try aggressive first, fall back to defensive
let policy = aggressive | defensive;
// Parsed as: Policy(Timeout50ms) | (Policy(Timeout10s) + Policy(Timeout5s))

Example: Circuit Breaker with Retry

use ninelives::prelude::*;
use std::time::Duration;

// Build a retry policy with exponential backoff
let retry = RetryPolicy::builder()
    .max_attempts(3)
    .backoff(Backoff::exponential(Duration::from_millis(100)))
    .with_jitter(Jitter::full())
    .build()?;

// Configure circuit breaker
let circuit_breaker = CircuitBreakerLayer::new(
    CircuitBreakerConfig::default()
        .failure_threshold(5)
        .timeout_duration(Duration::from_secs(10))
)?;

// Compose: circuit breaker wraps retry
let policy = Policy(circuit_breaker) + Policy(retry.into_layer());

Tower Integration

Nine Lives layers work seamlessly with tower's ServiceBuilder:

use ninelives::prelude::*;
use tower::ServiceBuilder;
use std::time::Duration;

let service = ServiceBuilder::new()
    .layer(TimeoutLayer::new(Duration::from_secs(30))?)
    .layer(CircuitBreakerLayer::new(CircuitBreakerConfig::default())?)
    .layer(BulkheadLayer::new(10)?)
    .service(my_inner_service);

Or use the algebraic syntax:

let policy = Policy(TimeoutLayer::new(Duration::from_secs(30))?)
           + Policy(CircuitBreakerLayer::new(CircuitBreakerConfig::default())?)
           + Policy(BulkheadLayer::new(10)?);

let service = ServiceBuilder::new()
    .layer(policy)
    .service(my_inner_service);

Available Layers

TimeoutLayer

Enforces time limits on operations:

use ninelives::prelude::*;
use std::time::Duration;

let timeout = TimeoutLayer::new(Duration::from_secs(5))?;

RetryLayer

Retries failed operations with configurable backoff and jitter:

use ninelives::prelude::*;
use std::time::Duration;

let retry = RetryPolicy::builder()
    .max_attempts(3)
    .backoff(Backoff::exponential(Duration::from_millis(100)))
    .with_jitter(Jitter::full())
    .build()?
    .into_layer();

Backoff strategies:

• Backoff::constant(duration) - Fixed delay
• Backoff::linear(base) - Linear increase: base * attempt
• Backoff::exponential(base) - Exponential: base * 2^attempt

Jitter strategies:

• Jitter::none() - No jitter
• Jitter::full() - Random [0, delay]
• Jitter::equal() - delay/2 + random [0, delay/2]
• Jitter::decorrelated() - AWS-style stateful jitter

CircuitBreakerLayer

Prevents cascading failures with three-state management (Closed/Open/HalfOpen):

use ninelives::prelude::*;
use std::time::Duration;

let circuit_breaker = CircuitBreakerLayer::new(
    CircuitBreakerConfig::default()
        .failure_threshold(5)        // Open after 5 failures
        .timeout_duration(Duration::from_secs(10))  // Stay open for 10s
        .half_open_max_calls(3)      // Allow 3 test calls in half-open
)?;

BulkheadLayer

Limits concurrent requests for resource isolation:

use ninelives::prelude::*;

let bulkhead = BulkheadLayer::new(10)?;  // Max 10 concurrent requests

Error Handling

All resilience errors are unified under ResilienceError<E>:

use ninelives::ResilienceError;

match service.call(request).await {
    Ok(response) => { /* success */ },
    Err(ResilienceError::Timeout { .. }) => { /* timeout */ },
    Err(ResilienceError::CircuitOpen { .. }) => { /* circuit breaker open */ },
    Err(ResilienceError::RetryExhausted { failures, .. }) => {
        // All retry attempts failed
        eprintln!("Failed after {} attempts", failures.len());
    },
    Err(ResilienceError::Bulkhead { .. }) => { /* capacity exhausted */ },
    Err(ResilienceError::Inner(e)) => { /* inner service error */ },
}

Operator Precedence

When combining operators, understand the precedence rules:

// & binds tighter than +, and + binds tighter than |
A | B + C & D   // Parsed as: A | (B + (C & D))

// Use parentheses for explicit control
(A | B) + C     // C wraps the fallback between A and B

Examples:

// Try fast, fallback to slow with retry
let policy = fast | retry + slow;
// Equivalent to: fast | (retry + slow)

// Retry wraps a fallback
let policy = retry + (fast | slow);

// Happy Eyeballs: race IPv4 and IPv6
let policy = ipv4 & ipv6;
// Both called concurrently, first success wins

// Complex composition
let policy = aggressive | defensive + (ipv4 & ipv6);
// Try aggressive, fallback to defensive wrapping parallel attempts

Testability

Nine Lives is designed for testing with dependency injection:

use ninelives::prelude::*;
use std::time::Duration;

// Use InstantSleeper for tests (no actual delays)
let retry = RetryPolicy::builder()
    .max_attempts(3)
    .backoff(Backoff::exponential(Duration::from_millis(100)))
    .with_sleeper(InstantSleeper)
    .build()?;

// TrackingSleeper records sleep durations for assertions
let tracker = TrackingSleeper::new();
let retry = RetryPolicy::builder()
    .max_attempts(3)
    .with_sleeper(tracker.clone())
    .build()?;

// ... exercise retry ...

let sleeps = tracker.get_sleeps();
assert_eq!(sleeps.len(), 2); // Slept twice before success

Roadmap

Nine Lives is evolving toward a fractal resilience framework with autonomous operation:

• v1.0 (Current Phase): Tower-native layers with algebraic composition ✅
• v1.5: Telemetry events, control plane for runtime tuning 🚧
• v2.0: Autonomous Sentinel with meta-policies, shadow evaluation 🔮
• v3.0: Rich adapter ecosystem (Redis, OTLP, Prometheus) 🌐

See ROADMAP.md for the full vision.

Performance

Nine Lives is built for production:

• Lock-free circuit breaker state transitions using atomics
• Zero-allocation backoff/jitter calculations with overflow protection
• Minimal overhead - resilience layers add < 1% latency in common cases

Benchmarks coming soon.

Comparison to Other Libraries

Nine Lives' unique advantage: Algebraic composition with fork-join support lets you express complex resilience strategies declaratively, including concurrent racing patterns like Happy Eyeballs, without nested builders or imperative code.

Examples

See the examples/ directory for runnable examples:

• timeout_fallback.rs - Timeout with fallback policy
• decorrelated_jitter.rs - AWS-style decorrelated jitter
• algebra_composition.rs - Complex algebraic composition patterns

Run with:

cargo run --example timeout_fallback

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be dual licensed as above, without any additional terms or conditions.

License

Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0)

@ 2025 • James Ross • 📧 • 🔗 FLYING•ROBOTS