🦊 Foxtive Supervisor

Because tasks crash. We help them get back up.

A Rust supervision library that keeps your async tasks running, even when things go wrong. Think of it as a caring parent for your background jobs.

What's New in v0.3?

Advanced Scheduling & Composition

• Cron Scheduling: Schedule tasks with cron expressions (e.g., "*/1 * * * * * *" for every second)
• Task Groups: Atomic operations on related tasks with group-level health monitoring
• Supervisor Hierarchies: Create nested supervisor trees for complex architectures
• Task Pools: Load-balanced worker pools with RoundRobin, Random, and LeastLoaded strategies
• Conditional Dependencies: Environment-based dependency activation for flexible deployments
• Rate Limiting: Enforce minimum restart intervals to prevent resource exhaustion
• Time Windows: Restrict task execution to specific time periods
• Cascading Shutdown: Bottom-up shutdown propagation through supervisor hierarchies
• Complex Dependency Graphs: Support for diamond dependencies, fan-out/fan-in patterns, and more

Why?

In production, things fail:

• Network hiccups
• Temporary resource exhaustion
• Race conditions you didn't catch in testing
• That one edge case from 3am

Foxtive Supervisor automatically restarts your tasks with configurable policies, handles panics gracefully, and gives you hooks to observe what's happening.

Features

Core Supervision

• Automatic Restarts: Keep tasks running with customizable retry policies and backoff strategies.
• Panic Recovery: Gracefully handle task panics without crashing your application.
• Dependency Management: Ensure tasks start only after their dependencies are ready.
• Prerequisites Gates: Run async checks before any task starts (e.g., database connectivity).

Dynamic Task Management

• Runtime Task Control: Add, remove, pause, resume, or restart tasks at runtime.
• Task Groups: Perform atomic operations on related tasks (start/stop/restart groups together).
• Group Health Monitoring: Aggregate health status across task groups for easy monitoring.
• Task Information: Query detailed task status, metrics, and health at any time.

Scheduling & Timing

• Cron Scheduling: Schedule tasks using cron expressions (integrates with foxtive-cron).
• Delayed Starts: Configure initial delays before task execution begins.
• Random Jitter: Add randomness to prevent thundering herd in distributed deployments.
• Rate Limiting: Enforce minimum intervals between restarts to prevent resource exhaustion.
• Time Windows: Restrict task execution to specific time periods (e.g., business hours only).

Advanced Reliability

• Circuit Breaker: Fail fast and prevent cascading failures when external services are down.
• Persistence Layer: Persist task state (attempts, failures, status) across application restarts.
• Multiple Storage Backends: In-memory and filesystem persistence out of the box.
• Graceful Shutdown: Configurable timeouts with forced termination and cleanup hooks.
• Cascading Shutdown: Hierarchical shutdown propagation for nested supervisors.

Observability & Monitoring

• Event System: Comprehensive lifecycle events for monitoring and alerting.
• Distributed Tracing: Built-in tracing integration with correlation IDs.
• Structured Logging: Rich, contextual logs throughout the supervision lifecycle.
• Health Checks: Per-task and group-level health status reporting.
• Custom Metrics: Expose task-specific metrics for monitoring dashboards.

Concurrency & Performance

• Concurrency Control: Global and per-task limits to prevent resource exhaustion.
• Priority Scheduling: Control the order in which tasks are started and restarted.
• Task Pools: Load-balanced worker pools with multiple strategies (RoundRobin, Random, LeastLoaded).
• Supervisor Hierarchies: Create nested supervisor trees for organized task management.

Testing & Development

• Testing Utilities: Built-in mock tasks and test harness for reliable testing.
• Conditional Dependencies: Environment-based dependency activation for flexible testing.
• Hot Reload Ready: Architecture designed for future configuration reload support.

Installation

[dependencies]
foxtive-supervisor = "0.3"
tokio = { version = "1", features = ["full"] }
anyhow = "1"
async-trait = "0.1"

Feature Flags

Foxtive Supervisor uses feature flags to enable optional functionality:

[dependencies]
foxtive-supervisor = { version = "0.3", features = ["cron"] }

Available Features:

• cron - Enable cron scheduling support (requires foxtive-cron and chrono-tz)
• All other features are enabled by default

Optional Dependencies

Some features require additional dependencies:

# For distributed tracing
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

# For persistence with filesystem
serde_json = "1"

# For cron scheduling (enabled with 'cron' feature)
foxtive-cron = "0.1"
chrono-tz = "0.8"
rand = "0.8"

Quick Start

use foxtive_supervisor::{Supervisor, SupervisedTask};

struct MyWorker;

#[async_trait::async_trait]
impl SupervisedTask for MyWorker {
    fn id(&self) -> &'static str {
        "my-worker"
    }

    async fn run(&self) -> anyhow::Result<()> {
        process_messages().await?;
        Ok(())
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    Supervisor::new()
        .add(MyWorker)
        .start_and_wait_any()
        .await?;

    Ok(())
}

Advanced Examples

Cron-Scheduled Tasks

Run tasks on a schedule using cron expressions:

struct DailyReport;

#[async_trait::async_trait]
impl SupervisedTask for DailyReport {
    fn id(&self) -> &'static str { "daily-report" }
    
    fn cron_schedule(&self) -> Option<&'static str> {
        // Run every day at 9:00 AM UTC
        Some("0 0 9 * * * *")
    }
    
    async fn run(&self) -> anyhow::Result<()> {
        generate_report().await?;
        Ok(())
    }
}

Task Groups with Health Monitoring

Group related tasks and monitor them together:

struct DatabaseService;
struct CacheService;
struct ApiService;

#[async_trait::async_trait]
impl SupervisedTask for DatabaseService {
    fn id(&self) -> &'static str { "database" }
    fn group_id(&self) -> Option<&'static str> { Some("infrastructure") }
    // ... implementation
}

#[async_trait::async_trait]
impl SupervisedTask for CacheService {
    fn id(&self) -> &'static str { "cache" }
    fn group_id(&self) -> Option<&'static str> { Some("infrastructure") }
    fn dependencies(&self) -> &'static [&'static str] { &["database"] }
    // ... implementation
}

// Start all infrastructure tasks atomically
let mut runtime = supervisor.start().await?;
runtime.start_group("infrastructure");

// Check group health
let health = runtime.get_group_health("infrastructure").await;
match health {
    HealthStatus::Healthy => println!("All systems operational"),
    HealthStatus::Degraded { reason } => eprintln!("Degraded: {}", reason),
    HealthStatus::Unhealthy { reason } => eprintln!("Unhealthy: {}", reason),
    _ => {}
}

Rate Limiting & Backoff

Prevent resource exhaustion with rate limiting:

struct ExternalApiConsumer;

#[async_trait::async_trait]
impl SupervisedTask for ExternalApiConsumer {
    fn id(&self) -> &'static str { "api-consumer" }
    
    fn backoff_strategy(&self) -> BackoffStrategy {
        BackoffStrategy::Exponential {
            initial: Duration::from_secs(1),
            max: Duration::from_secs(60),
            multiplier: 2.0,
        }
    }
    
    fn min_restart_interval(&self) -> Option<Duration> {
        // Never restart more than once per 5 seconds
        Some(Duration::from_secs(5))
    }
    
    async fn run(&self) -> anyhow::Result<()> {
        fetch_from_api().await?;
        Ok(())
    }
}

Supervisor Hierarchies

Create nested supervisor trees for complex architectures:

use foxtive_supervisor::hierarchy::{SupervisorHierarchy, SupervisorNode};

let mut hierarchy = SupervisorHierarchy::new("root");

// Add child supervisors
let mut api_node = SupervisorNode::root("api-services");
api_node.add_child_supervisor("auth-service", auth_supervisor);
api_node.add_child_supervisor("user-service", user_supervisor);

let mut worker_node = SupervisorNode::root("background-workers");
worker_node.add_child_supervisor("email-worker", email_supervisor);
worker_node.add_child_supervisor("report-worker", report_supervisor);

hierarchy.root_mut().add_child(api_node);
hierarchy.root_mut().add_child(worker_node);

// Cascading shutdown - shuts down all children first, then parent
hierarchy.root().shutdown_all().await;

Task Pools with Load Balancing

Distribute work across multiple workers:

use foxtive_supervisor::task_pool::{TaskPool, LoadBalancingStrategy};

// Create a pool of 4 workers with round-robin distribution
let pool = TaskPool::new(
    "message-processors",
    4,
    LoadBalancingStrategy::RoundRobin
);

// Build supervisor with pooled workers
let supervisor = pool.build_pool(|worker_id| {
    MessageProcessor::new(worker_id)
});

Conditional Dependencies

Activate dependencies based on environment or runtime conditions:

struct Microservice;

#[async_trait::async_trait]
impl SupervisedTask for Microservice {
    fn id(&self) -> &'static str { "my-service" }
    
    fn dependencies(&self) -> &'static [&'static str] {
        &["database"] // Always required
    }
    
    fn conditional_dependencies(&self) -> Vec<(&'static str, Box<dyn Fn() -> bool + Send + Sync>)> {
        vec![
            // Only depend on cache if USE_CACHE is enabled
            ("cache", Box::new(|| {
                std::env::var("USE_CACHE").is_ok()
            })),
            // Only depend on message queue if ENABLE_QUEUE is set
            ("message-queue", Box::new(|| {
                std::env::var("ENABLE_QUEUE").is_ok()
            })),
        ]
    }
}

State Persistence

Persist task state across application restarts:

use foxtive_supervisor::persistence::{FsStateStore, TaskStateStore};
use std::path::PathBuf;

struct MessageProcessor;

#[async_trait::async_trait]
impl SupervisedTask for MessageProcessor {
    fn id(&self) -> &'static str { "message-processor" }
    
    async fn run(&self) -> anyhow::Result<()> {
        process_messages().await?;
        Ok(())
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create filesystem-based state store
    let store = FsStateStore::new(PathBuf::from("./task-state"));
    
    // Supervisor will automatically restore task state on startup
    let supervisor = Supervisor::new()
        .with_state_store(Box::new(store))
        .add(MessageProcessor);
    
    // Task state (attempts, failures, last success) persists across restarts
    supervisor.start_and_wait_any().await?;
    
    Ok(())
}

The persistence layer supports:

• InMemoryStateStore: Default, no configuration needed
• FsStateStore: Filesystem-based persistence with JSON serialization
• Automatic state recovery when tasks restart
• Custom storage backends via TaskStateStore trait

Documentation

Guides

• Testing Guide - Comprehensive testing strategies and utilities

Examples

Real-world examples demonstrating various use cases:

• Microservice Orchestration (examples/microservice-orchestration)

  • Database to Cache to API Server architecture
  • Dependency management and health monitoring

• Circuit Breaker Pattern (examples/circuit-breaker)

  • Automatic failure detection and recovery
  • Half-open state testing

• Graceful Shutdown (examples/graceful-shutdown)

  • Configurable shutdown timeouts
  • Cleanup hooks and forced termination

• Database Message Consumer (examples/db-message-consumer)

  • Persistent message processing with retry logic
  • State persistence across restarts

• Panic Recovery (examples/panic-catcher)

  • Handling task panics gracefully
  • Maintaining supervision despite crashes

• Axum Integration (examples/axum-cron)

  • Web framework integration
  • Scheduled task management

• Tracing & Observability (examples/tracing)

  • Distributed tracing with correlation IDs
  • Structured logging throughout task lifecycle

Troubleshooting Guide

1. Task not starting

• Check Dependencies: Ensure all IDs in dependencies() exist and their setup() succeeds.
• Check Prerequisites: If any require() gate fails, no tasks will start.
• Concurrency Limits: If the global_concurrency_limit is 0 or very low, tasks might be queued indefinitely.

2. Immediate Restart Loops

• Setup Failure: If setup() fails, the task will not enter the run() loop. Check logs for TaskSetupFailed.
• Backoff Strategy: Ensure your backoff_strategy() provides enough time for external resources to recover.
• Should Restart Hook: Check if should_restart() is returning false unexpectedly.

3. State Not Persisting

• Storage Permissions: Ensure the directory passed to FsStateStore is writable.
• Task ID Consistency: Persistence is tied to the task's id(). If you change the ID, previous state is lost.

4. High Resource Usage

• Limit Concurrency: Use with_global_concurrency_limit to cap the number of active tasks.
• Check Leaks: Ensure your run() loop or setup() doesn't leak memory or file handles. The supervisor restarts tasks, but it can't fix underlying leaks.

Test Coverage & Quality

Foxtive Supervisor is thoroughly tested with 89 comprehensive tests covering:

Test Categories

• Core Functionality (16 tests) - Basic supervision, restarts, lifecycle
• Dynamic Management (7 tests) - Add/remove/pause/resume at runtime
• Dependency Resolution (6 tests) - Complex dependency graphs, diamond patterns
• Persistence (2 tests) - State storage and recovery
• Circuit Breaker (3 tests) - All state transitions and recovery
• Concurrency Control (2 tests) - Global and per-task limits
• Graceful Shutdown (4 tests) - Timeouts, forced termination, cascading shutdown
• Event System (3 tests) - Lifecycle event emission and listeners
• Prerequisites (7 tests) - Async gates and validation
• Error Handling (3 tests) - Panic recovery, error propagation
• Edge Cases (17 tests) - Boundary conditions, error scenarios
• Task Groups (4 tests) - Atomic operations, health aggregation
• Task Pools (4 tests) - Load balancing strategies
• Conditional Dependencies (3 tests) - Environment-based activation
• Cron Scheduling (7 tests) - Scheduled execution, rate limiting, time windows
• Real-World Scenarios (2 tests) - Microservice architecture, message pipelines
• Complex Dependency Graphs (6 tests) - Fan-out/fan-in, linear chains, mixed deps

Quality Metrics

• Test Pass Rate: 98.9% (88/89 tests passing)
• Clippy Compliance: Zero warnings
• Documentation: Every public API documented with examples
• Integration Tests: Real-world scenarios with multiple features combined
• Performance: Benchmarks for persistence overhead and concurrency

License

MIT

---

Built with 🦊 by the Foxtive team. Made for humans who write Rust.