RustFS Workers - Background Job Processing

📖 Overview

RustFS Workers provides a distributed background job processing system for the RustFS distributed object storage system. It handles asynchronous tasks such as data replication, cleanup, healing, indexing, and other maintenance operations across the cluster.

Note: This is a core submodule of RustFS that provides essential background processing capabilities for the distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

🔄 Job Processing

Distributed Execution: Jobs run across multiple cluster nodes
Priority Queues: Multiple priority levels for job scheduling
Retry Logic: Automatic retry with exponential backoff
Dead Letter Queue: Failed job isolation and analysis

🛠️ Built-in Workers

Replication Worker: Data replication across nodes
Cleanup Worker: Garbage collection and cleanup
Healing Worker: Data integrity repair
Indexing Worker: Metadata indexing and search
Metrics Worker: Performance metrics collection

🚀 Scalability Features

Horizontal Scaling: Add worker nodes dynamically
Load Balancing: Intelligent job distribution
Circuit Breaker: Prevent cascading failures
Rate Limiting: Control resource consumption

🔧 Management & Monitoring

Job Tracking: Real-time job status monitoring
Health Checks: Worker health and availability
Metrics Collection: Performance and throughput metrics
Administrative Interface: Job management and control

📦 Installation

Add this to your Cargo.toml:

[dependencies]
rustfs-workers = "0.0.3"

🔧 Usage

📋 Requirements

Rust: 1.70.0 or later
Platforms: Linux, macOS, Windows
Network: Cluster connectivity required
Storage: Persistent queue storage recommended

🌍 Related Projects

This module is part of the RustFS ecosystem:

RustFS Main - Core distributed storage system
RustFS Common - Common types and utilities
RustFS Lock - Distributed locking

📚 Documentation

For comprehensive documentation, visit:

🔗 Links

Documentation - Complete RustFS manual
Changelog - Release notes and updates
GitHub Discussions - Community support

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

rustfs-workers 0.0.3