Crate ninelives 

Nine Lives 🐱

Resilience patterns for Rust: retry policies, circuit breakers, bulkheads, and timeouts.

Features

• Retry policies with backoff strategies (constant, linear, exponential)
• Circuit breakers with half-open state recovery
• Bulkheads for concurrency limiting and resource isolation
• Timeout policies integrated with tokio
• Policy composition via tower-native layers and algebraic operators
• Lock-free implementations using atomics

Quick Start

Basic Usage - Single Layer

use ninelives::prelude::*;
use std::time::Duration;
use tower::{Service, ServiceBuilder};

// Create a simple service
let mut svc = ServiceBuilder::new()
    .layer(TimeoutLayer::new(Duration::from_secs(1))?)
    .service_fn(|req: &'static str| async move {
        Ok::<_, std::io::Error>(req)
    });

let response = svc.call("hello").await?;
assert_eq!(response, "hello");

Algebraic Composition - The Power of Policy

Compose layers using intuitive operators:

use ninelives::prelude::*;
use std::time::Duration;
use tower::{Service, ServiceBuilder};

// Define resilience strategy using algebra
// Try fast timeout, fallback to longer timeout
let fast = Policy(TimeoutLayer::new(Duration::from_millis(100))?);
let slow = Policy(TimeoutLayer::new(Duration::from_secs(5))?);
let policy = fast | slow;  // Fallback operator

// Apply to any service
let mut svc = ServiceBuilder::new()
    .layer(policy)
    .service_fn(|req: &'static str| async move {
        Ok::<_, std::io::Error>(req)
    });

let response = svc.call("hello").await?;

Complex Composition

Combine multiple strategies with precedence:

use ninelives::prelude::*;
use std::time::Duration;

// Aggressive: fast timeout only
let aggressive = Policy(TimeoutLayer::new(Duration::from_millis(50))?);

// Defensive: nested timeouts for resilience
let defensive = Policy(TimeoutLayer::new(Duration::from_secs(10))?)
              + Policy(TimeoutLayer::new(Duration::from_secs(5))?);

// Try aggressive first, fall back to defensive
// Operator precedence: + binds tighter than |
let _policy = aggressive | defensive;
// This creates: Policy(Timeout50ms) | (Policy(Timeout10s) + Policy(Timeout5s))

Algebraic Operators

• Policy(A) + Policy(B) - Sequential composition: A wraps B
• Policy(A) | Policy(B) - Fallback: try A, fall back to B on error
• Policy(A) & Policy(B) - Fork-join: try both concurrently, return first success

Precedence: & > + > | (like * > + > bitwise-or in math)

Example: A | B + C & D is parsed as A | (B + (C & D))

Available Layers

• TimeoutLayer - Enforce time limits on operations
• RetryLayer - Retry failed operations with backoff
• CircuitBreakerLayer - Prevent cascading failures
• BulkheadLayer - Limit concurrent requests

For more examples, see the algebra module documentation.

Modules

prelude

Convenient re-exports for common Nine Lives types.

telemetry

Telemetry and observability for Nine Lives policies.

Structs

Backoff

Backoff strategy wrapper preserving the existing API while delegating to concrete strategies.

BulkheadLayer

Tower-native bulkhead layer with optional telemetry.

BulkheadPolicy

Concurrency-limiting bulkhead. Clones share the same underlying semaphore via Arc, so they observe and affect the same in-flight count. max_concurrent is the configured permit ceiling.

CircuitBreakerConfig

Validated configuration for the circuit breaker.

CircuitBreakerLayer

Tower-native circuit breaker layer with optional telemetry.

CombinedLayer

Sequential composition layer that applies inner first, then outer.

ConstantBackoff

Constant backoff strategy.

ExponentialBackoff

Exponential backoff strategy.

FallbackLayer

Fallback composition layer that tries primary, falling back to secondary on error.

FallbackService

Tower service that executes primary, falling back to secondary on error.

ForkJoinLayer

Fork-join composition layer that tries both left and right concurrently.

ForkJoinService

Tower service that races two services concurrently, returning the first success.

InstantSleeper

Test sleeper that doesn’t actually sleep.

LinearBackoff

Linear backoff strategy.

MonotonicClock

Monotonic clock backed by Instant::now().

Policy

Opt-in wrapper enabling algebraic composition of tower layers.

RetryLayer

Tower-native retry layer with optional telemetry.

RetryPolicy

Retry policy combining backoff, jitter, predicate, and sleeper.

RetryPolicyBuilder

Builder for RetryPolicy.

RetryService

Retry service produced by RetryLayer.

TimeoutLayer

Tower-native timeout layer with optional telemetry.

TimeoutPolicy

Policy that enforces a maximum duration on async operations.

TokioSleeper

Production sleeper using tokio runtime.

TrackingSleeper

Test sleeper that tracks all sleep calls. Clones share the same recorded history via Arc, so any clone observing or pushing sleeps sees the combined timeline.

Enums

BackoffError

Errors returned by backoff configuration.

BuildError

Errors produced while building a retry policy.

BulkheadError

Errors produced while configuring a bulkhead (e.g., invalid permit counts).

CircuitBreakerError

Errors produced when validating breaker configuration.

CircuitState

Circuit breaker state machine.

Jitter

Jitter strategy for randomizing retry delays

ResilienceError

Unified error type for all resilience policies

TimeoutError

Errors returned when configuring timeouts.

Constants

MAX_BACKOFF

Maximum delay used when calculations overflow (1 day).

MAX_TIMEOUT

Maximum allowed timeout duration (30 days) to avoid runaway timers while permitting long jobs. Intended to guard accidental timeouts of u64::MAX; override via TimeoutPolicy::new_with_max when longer horizons are required.

Traits

BackoffStrategy

Trait implemented by all backoff strategies.

Clock

Thread-safe time source abstraction.

Sleeper

Contract for sleeping/waiting.