🦊 Foxtive Supervisor

Because tasks crash. We help them get back up.

A Rust supervision library that keeps your async tasks running, even when things go wrong. Think of it as a caring parent for your background jobs.

What's New in v0.3?

Advanced Scheduling & Composition

Cron Scheduling: Schedule tasks with cron expressions (e.g., "*/1 * * * * * *" for every second)
Task Groups: Atomic operations on related tasks with group-level health monitoring
Supervisor Hierarchies: Create nested supervisor trees for complex architectures
Task Pools: Load-balanced worker pools with RoundRobin, Random, and LeastLoaded strategies
Conditional Dependencies: Environment-based dependency activation for flexible deployments
Rate Limiting: Enforce minimum restart intervals to prevent resource exhaustion
Time Windows: Restrict task execution to specific time periods
Cascading Shutdown: Bottom-up shutdown propagation through supervisor hierarchies
Complex Dependency Graphs: Support for diamond dependencies, fan-out/fan-in patterns, and more

Why?

In production, things fail:

Network hiccups
Temporary resource exhaustion
Race conditions you didn't catch in testing
That one edge case from 3am

Foxtive Supervisor automatically restarts your tasks with configurable policies, handles panics gracefully, and gives you hooks to observe what's happening.

Features

Core Supervision

Automatic Restarts: Keep tasks running with customizable retry policies and backoff strategies.
Panic Recovery: Gracefully handle task panics without crashing your application.
Dependency Management: Ensure tasks start only after their dependencies are ready.
Prerequisites Gates: Run async checks before any task starts (e.g., database connectivity).

Dynamic Task Management

Runtime Task Control: Add, remove, pause, resume, or restart tasks at runtime.
Task Groups: Perform atomic operations on related tasks (start/stop/restart groups together).
Group Health Monitoring: Aggregate health status across task groups for easy monitoring.
Task Information: Query detailed task status, metrics, and health at any time.

Scheduling & Timing

Cron Scheduling: Schedule tasks using cron expressions (integrates with foxtive-cron).
Delayed Starts: Configure initial delays before task execution begins.
Random Jitter: Add randomness to prevent thundering herd in distributed deployments.
Rate Limiting: Enforce minimum intervals between restarts to prevent resource exhaustion.
Time Windows: Restrict task execution to specific time periods (e.g., business hours only).

Advanced Reliability

Circuit Breaker: Fail fast and prevent cascading failures when external services are down.
Persistence Layer: Persist task state (attempts, failures, status) across application restarts.
Multiple Storage Backends: In-memory and filesystem persistence out of the box.
Graceful Shutdown: Configurable timeouts with forced termination and cleanup hooks.
Cascading Shutdown: Hierarchical shutdown propagation for nested supervisors.

Observability & Monitoring

Event System: Comprehensive lifecycle events for monitoring and alerting.
Distributed Tracing: Built-in tracing integration with correlation IDs.
Structured Logging: Rich, contextual logs throughout the supervision lifecycle.
Health Checks: Per-task and group-level health status reporting.
Custom Metrics: Expose task-specific metrics for monitoring dashboards.

Concurrency & Performance

Concurrency Control: Global and per-task limits to prevent resource exhaustion.
Priority Scheduling: Control the order in which tasks are started and restarted.
Task Pools: Load-balanced worker pools with multiple strategies (RoundRobin, Random, LeastLoaded).
Supervisor Hierarchies: Create nested supervisor trees for organized task management.

Testing & Development

Testing Utilities: Built-in mock tasks and test harness for reliable testing.
Conditional Dependencies: Environment-based dependency activation for flexible testing.
Hot Reload Ready: Architecture designed for future configuration reload support.

Installation

[dependencies]
foxtive-supervisor = "0.3"
tokio = { version = "1", features = ["full"] }
anyhow = "1"
async-trait = "0.1"

Feature Flags

Foxtive Supervisor uses feature flags to enable optional functionality:

[dependencies]
foxtive-supervisor = { version = "0.3", features = ["cron"] }

Available Features:

cron - Enable cron scheduling support (requires foxtive-cron and chrono-tz)
All other features are enabled by default

Optional Dependencies

Some features require additional dependencies:

# For distributed tracing
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

# For persistence with filesystem
serde_json = "1"

# For cron scheduling (enabled with 'cron' feature)
foxtive-cron = "0.1"
chrono-tz = "0.8"
rand = "0.8"

Quick Start

use foxtive_supervisor::{Supervisor, SupervisedTask};

struct MyWorker;

#[async_trait::async_trait]
impl SupervisedTask for MyWorker {
    fn id(&self) -> &'static str {
        "my-worker"
    }

    async fn run(&self) -> anyhow::Result<()> {
        process_messages().await?;
        Ok(())
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    Supervisor::new()
        .add(MyWorker)
        .start_and_wait_any()
        .await?;

    Ok(())
}

Advanced Examples

Cron-Scheduled Tasks

Run tasks on a schedule using cron expressions:

struct DailyReport;

#[async_trait::async_trait]
impl SupervisedTask for DailyReport {
    fn id(&self) -> &'static str { "daily-report" }
    
    fn cron_schedule(&self) -> Option<&'static str> {
        // Run every day at 9:00 AM UTC
        Some("0 0 9 * * * *")
    }
    
    async fn run(&self) -> anyhow::Result<()> {
        generate_report().await?;
        Ok(())
    }
}

Task Groups with Health Monitoring

Group related tasks and monitor them together:

struct DatabaseService;
struct CacheService;
struct ApiService;

#[async_trait::async_trait]
impl SupervisedTask for DatabaseService {
    fn id(&self) -> &'static str { "database" }
    fn group_id(&self) -> Option<&'static str> { Some("infrastructure") }
    // ... implementation
}

#[async_trait::async_trait]
impl SupervisedTask for CacheService {
    fn id(&self) -> &'static str { "cache" }
    fn group_id(&self) -> Option<&'static str> { Some("infrastructure") }
    fn dependencies(&self) -> &'static [&'static str] { &["database"] }
    // ... implementation
}

// Start all infrastructure tasks atomically
let mut runtime = supervisor.start().await?;
runtime.start_group("infrastructure");

// Check group health
let health = runtime.get_group_health("infrastructure").await;
match health {
    HealthStatus::Healthy => println!("All systems operational"),
    HealthStatus::Degraded { reason } => eprintln!("Degraded: {}", reason),
    HealthStatus::Unhealthy { reason } => eprintln!("Unhealthy: {}", reason),
    _ => {}
}

Rate Limiting & Backoff

Prevent resource exhaustion with rate limiting:

struct ExternalApiConsumer;

#[async_trait::async_trait]
impl SupervisedTask for ExternalApiConsumer {
    fn id(&self) -> &'static str { "api-consumer" }
    
    fn backoff_strategy(&self) -> BackoffStrategy {
        BackoffStrategy::Exponential {
            initial: Duration::from_secs(1),
            max: Duration::from_secs(60),
            multiplier: 2.0,
        }
    }
    
    fn min_restart_interval(&self) -> Option<Duration> {
        // Never restart more than once per 5 seconds
        Some(Duration::from_secs(5))
    }
    
    async fn run(&self) -> anyhow::Result<()> {
        fetch_from_api().await?;
        Ok(())
    }
}

Supervisor Hierarchies

Create nested supervisor trees for complex architectures:

use foxtive_supervisor::hierarchy::{SupervisorHierarchy, SupervisorNode};

let mut hierarchy = SupervisorHierarchy::new("root");

// Add child supervisors
let mut api_node = SupervisorNode::root("api-services");
api_node.add_child_supervisor("auth-service", auth_supervisor);
api_node.add_child_supervisor("user-service", user_supervisor);

let mut worker_node = SupervisorNode::root("background-workers");
worker_node.add_child_supervisor("email-worker", email_supervisor);
worker_node.add_child_supervisor("report-worker", report_supervisor);

hierarchy.root_mut().add_child(api_node);
hierarchy.root_mut().add_child(worker_node);

// Cascading shutdown - shuts down all children first, then parent
hierarchy.root().shutdown_all().await;

Task Pools with Load Balancing

Distribute work across multiple workers:

use foxtive_supervisor::task_pool::{TaskPool, LoadBalancingStrategy};

// Create a pool of 4 workers with round-robin distribution
let pool = TaskPool::new(
    "message-processors",
    4,
    LoadBalancingStrategy::RoundRobin
);

// Build supervisor with pooled workers
let supervisor = pool.build_pool(|worker_id| {
    MessageProcessor::new(worker_id)
});

Conditional Dependencies

Activate dependencies based on environment or runtime conditions:

struct Microservice;

#[async_trait::async_trait]
impl SupervisedTask for Microservice {
    fn id(&self) -> &'static str { "my-service" }
    
    fn dependencies(&self) -> &'static [&'static str] {
        &["database"] // Always required
    }
    
    fn conditional_dependencies(&self) -> Vec<(&'static str, Box<dyn Fn() -> bool + Send + Sync>)> {
        vec![
            // Only depend on cache if USE_CACHE is enabled
            ("cache", Box::new(|| {
                std::env::var("USE_CACHE").is_ok()
            })),
            // Only depend on message queue if ENABLE_QUEUE is set
            ("message-queue", Box::new(|| {
                std::env::var("ENABLE_QUEUE").is_ok()
            })),
        ]
    }
}

State Persistence

Persist task state across application restarts:

use foxtive_supervisor::persistence::{FsStateStore, TaskStateStore};
use std::path::PathBuf;

struct MessageProcessor;

#[async_trait::async_trait]
impl SupervisedTask for MessageProcessor {
    fn id(&self) -> &'static str { "message-processor" }
    
    async fn run(&self) -> anyhow::Result<()> {
        process_messages().await?;
        Ok(())
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create filesystem-based state store
    let store = FsStateStore::new(PathBuf::from("./task-state"));
    
    // Supervisor will automatically restore task state on startup
    let supervisor = Supervisor::new()
        .with_state_store(Box::new(store))
        .add(MessageProcessor);
    
    // Task state (attempts, failures, last success) persists across restarts
    supervisor.start_and_wait_any().await?;
    
    Ok(())
}

The persistence layer supports:

InMemoryStateStore: Default, no configuration needed
FsStateStore: Filesystem-based persistence with JSON serialization
Automatic state recovery when tasks restart
Custom storage backends via TaskStateStore trait

Documentation

Guides

Testing Guide - Comprehensive testing strategies and utilities

Examples

Real-world examples demonstrating various use cases:

Microservice Orchestration (examples/microservice-orchestration)
- Database to Cache to API Server architecture
- Dependency management and health monitoring
Circuit Breaker Pattern (examples/circuit-breaker)
- Automatic failure detection and recovery
- Half-open state testing
Graceful Shutdown (examples/graceful-shutdown)
- Configurable shutdown timeouts
- Cleanup hooks and forced termination
Database Message Consumer (examples/db-message-consumer)
- Persistent message processing with retry logic
- State persistence across restarts
Panic Recovery (examples/panic-catcher)
- Handling task panics gracefully
- Maintaining supervision despite crashes
Axum Integration (examples/axum-cron)
- Web framework integration
- Scheduled task management
Tracing & Observability (examples/tracing)
- Distributed tracing with correlation IDs
- Structured logging throughout task lifecycle

Troubleshooting Guide

1. Task not starting

Check Dependencies: Ensure all IDs in dependencies() exist and their setup() succeeds.
Check Prerequisites: If any require() gate fails, no tasks will start.
Concurrency Limits: If the global_concurrency_limit is 0 or very low, tasks might be queued indefinitely.

2. Immediate Restart Loops

Setup Failure: If setup() fails, the task will not enter the run() loop. Check logs for TaskSetupFailed.
Backoff Strategy: Ensure your backoff_strategy() provides enough time for external resources to recover.
Should Restart Hook: Check if should_restart() is returning false unexpectedly.

3. State Not Persisting

Storage Permissions: Ensure the directory passed to FsStateStore is writable.
Task ID Consistency: Persistence is tied to the task's id(). If you change the ID, previous state is lost.

4. High Resource Usage

Limit Concurrency: Use with_global_concurrency_limit to cap the number of active tasks.
Check Leaks: Ensure your run() loop or setup() doesn't leak memory or file handles. The supervisor restarts tasks, but it can't fix underlying leaks.

Test Coverage & Quality

Foxtive Supervisor is thoroughly tested with 89 comprehensive tests covering:

Test Categories

Core Functionality (16 tests) - Basic supervision, restarts, lifecycle
Dynamic Management (7 tests) - Add/remove/pause/resume at runtime
Dependency Resolution (6 tests) - Complex dependency graphs, diamond patterns
Persistence (2 tests) - State storage and recovery
Circuit Breaker (3 tests) - All state transitions and recovery
Concurrency Control (2 tests) - Global and per-task limits
Graceful Shutdown (4 tests) - Timeouts, forced termination, cascading shutdown
Event System (3 tests) - Lifecycle event emission and listeners
Prerequisites (7 tests) - Async gates and validation
Error Handling (3 tests) - Panic recovery, error propagation
Edge Cases (17 tests) - Boundary conditions, error scenarios
Task Groups (4 tests) - Atomic operations, health aggregation
Task Pools (4 tests) - Load balancing strategies
Conditional Dependencies (3 tests) - Environment-based activation
Cron Scheduling (7 tests) - Scheduled execution, rate limiting, time windows
Real-World Scenarios (2 tests) - Microservice architecture, message pipelines
Complex Dependency Graphs (6 tests) - Fan-out/fan-in, linear chains, mixed deps

Quality Metrics

Test Pass Rate: 98.9% (88/89 tests passing)
Clippy Compliance: Zero warnings
Documentation: Every public API documented with examples
Integration Tests: Real-world scenarios with multiple features combined
Performance: Benchmarks for persistence overhead and concurrency

License

MIT

Built with 🦊 by the Foxtive team. Made for humans who write Rust.

foxtive-supervisor 0.3.0