Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
Backbone Queue Module
A high-performance, asynchronous queue implementation for the Backbone Framework with support for Redis, AWS SQS, and RabbitMQ backends.
Overview
The Backbone Queue Module provides a unified, high-level interface for message queuing across multiple backend technologies. Built with async/await support and comprehensive error handling, it abstracts the complexity of different queue systems while maintaining performance and flexibility.
Key Features
- ✅ Multiple Backends: Redis, AWS SQS, and RabbitMQ support
- ✅ Priority Queues: Four priority levels (Low, Normal, High, Critical)
- ✅ Batch Operations: Efficient bulk processing of messages
- ✅ Dead Letter Queues: Automatic handling of failed messages
- ✅ Message Expiration: Time-to-live (TTL) support
- ✅ Visibility Timeouts: Prevent concurrent processing
- ✅ Message Attributes: Custom metadata and headers support
- ✅ FIFO Support: First-in, first-out queue semantics (SQS)
- ✅ Exchange Types: RabbitMQ routing with Direct, Fanout, Topic, and Headers exchanges
- ✅ Compression: Automatic compression for large messages
- ✅ Async/Await: Full tokio async support
- ✅ Health Monitoring: Built-in health checks and statistics
- ✅ Type Safety: Strong typing with comprehensive error handling
Architecture
┌─────────────────────────────────────┐
│ Application Layer │
│ (Your Business Logic) │
└─────────────┬───────────────────────┘
│
┌─────────────▼───────────────────────┐
│ Backbone Queue Module │
│ ┌─────────────────────────────┐ │
│ │ QueueService Trait │ │
│ └─────────────┬───────────────┘ │
│ │ │
│ ┌─────────────▼───────────────┐ │
│ │ Backend Implementations │ │
│ │ ┌─────────┬─────────────┐ │ │
│ │ │ Redis │ SQS │ │ │
│ │ │ │ │ │ │
│ │ │ RabbitMQ│ │ │ │
│ │ └─────────┴─────────────┘ │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────────┘
│
┌─────────────▼───────────────────────┐
│ Infrastructure Layer │
│ ┌─────────┬─────────────┐ │
│ │ Redis │ SQS │ │
│ │ Server │ AWS │ │
│ │ │ │ │
│ │ RabbitMQ│ │ │
│ │ Server │ │ │
│ └─────────┴─────────────┘ │
└─────────────────────────────────────┘
Supported Backends
| Backend | Use Case | Performance | Features |
|---|---|---|---|
| Redis | High-speed in-memory queuing | Very High | Simple pub/sub, priority queues |
| AWS SQS | Cloud-native, scalable | High | Durability, at-least-once delivery |
| RabbitMQ | Complex routing, enterprise | High | AMQP, exchange routing, acknowledgments |
Usage
Quick Start Examples
Redis Queue
use ;
async
AWS SQS Queue
use ;
async
RabbitMQ Queue
use ;
async
Backend Selection Guide
Choose the right backend based on your requirements:
Redis
- Best for: High-speed in-memory queuing, simple use cases
- Pros: Fastest performance, simple setup, low latency
- Cons: Limited durability, memory-based, limited routing
// Use Redis when you need:
// - Maximum speed
// - Simple pub/sub patterns
// - Temporary queues
// - Low-latency processing
AWS SQS
- Best for: Cloud-native applications, durability at scale
- Pros: Highly durable, managed service, auto-scaling
- Cons: Higher latency, AWS-specific, limited routing
// Use SQS when you need:
// - Cloud-native solution
// - High durability guarantees
// - Automatic scaling
// - Serverless architecture
RabbitMQ
- Best for: Complex routing, enterprise messaging, microservices
- Pros: Rich routing, AMQP protocol, acknowledgments, exchanges
- Cons: More complex, requires infrastructure management
// Use RabbitMQ when you need:
// - Complex message routing
// - Exchange patterns (direct, fanout, topic)
// - Publisher confirms
// - Microservices communication
Advanced Configuration
Redis Configuration
use RedisQueueBuilder;
let queue = new
.url
.queue_name
.key_prefix
.pool_size
.build
.await?;
SQS Configuration
use SqsQueueBuilder;
let queue = new
.queue_url
.region
.credentials
.visibility_timeout
.wait_time_seconds
.build
.await?;
RabbitMQ Configuration
use ;
// Development configuration
let dev_config = dev_config;
// Production configuration with TLS
let prod_config = prod_config;
// Manual configuration
let custom_config = RabbitMQConfig ;
let queue = new.await?;
RabbitMQ Exchange Types
use ;
// Direct Exchange - Exact routing key matches
let direct_config = RabbitMQConfig ;
// Fanout Exchange - Broadcast to all bound queues
let fanout_config = RabbitMQConfig ;
// Topic Exchange - Pattern-based routing with wildcards
let topic_config = RabbitMQConfig ;
Message Creation
Basic Message
use ;
let message = builder
.payload
.priority
.build;
Message with Headers
use HashMap;
use ;
let mut headers = new;
headers.insert;
headers.insert;
let message = builder
.payload
.priority
.headers
.build;
RabbitMQ Message with Routing
use ;
let message = builder
.payload
.priority
.routing_key
.header
.header
.build;
Message with Expiration
use QueueMessage;
let message = builder
.payload
.visibility_timeout // seconds
.build;
FIFO Message (SQS)
use QueueMessage;
let message = builder
.payload
.message_group_id
.message_deduplication_id
.build;
JSON Payload
use QueueMessage;
use json;
let message = builder
.payload
.expect
.routing_key
.priority
.build;
Priority Levels
Messages are processed in priority order:
use QueuePriority;
// Priority levels (higher number = higher priority)
Low // 1
Normal // 5 (default)
High // 10
Critical // 20
Queue Operations
Basic Operations
// Enqueue single message
let message_id = queue.enqueue.await?;
// Dequeue single message
if let Some = queue.dequeue.await?
// Get message by ID
if let Some = queue.get_message.await?
Batch Operations
// Enqueue multiple messages
let messages = vec!;
let message_ids = queue.enqueue_batch.await?;
println!;
// Dequeue batch
let batch_result = queue.dequeue_batch.await?;
println!;
// Batch acknowledge
let message_ids: = batch_result.messages
.into_iter
.map
.collect;
let ack_count = queue.ack_batch.await?;
println!;
Queue Management
// Get queue statistics
let stats = queue.get_stats.await?;
println!;
println!;
println!;
// Get queue size
let size = queue.size.await?;
println!;
// Check if empty
if queue.is_empty.await?
// Purge all messages
let purged_count = queue.purge.await?;
println!;
Health Monitoring
// Health check
let health = queue.health_check.await?;
println!;
println!;
println!;
// Test connection
let is_connected = queue.test_connection.await?;
println!;
Error Handling
use QueueError;
match queue.enqueue.await
Advanced Usage
Dead Letter Queue Handling
use QueueMessage;
let message = builder
.payload
.max_receive_count // Send to DLQ after 3 failed attempts
.build;
queue.enqueue.await?;
// Process message
if let Some = queue.dequeue.await?
// Check dead letter queue
let stats = queue.get_stats.await?;
if stats.dead_letter_messages > 0
Message Compression
use QueueMessage;
let large_payload = "x".repeat; // 1MB payload
let message = builder
.payload
.compress // Enable compression
.build;
let message_id = queue.enqueue.await?;
Visibility Timeouts
use QueueMessage;
let message = builder
.payload
.visibility_timeout // 5 minutes
.build;
queue.enqueue.await?;
if let Some = queue.dequeue.await?
Monitoring and Metrics
Queue Statistics
let stats = queue.get_stats.await?;
println!;
println!;
println!;
println!;
println!;
println!;
println!;
Health Check
let health = queue.health_check.await?;
println!;
println!;
println!;
println!;
println!;
println!;
println!;
Testing
Running Tests
# Run all tests
# Run specific backend tests
# Run integration tests (requires Redis/SQS setup)
# Run performance tests
Test Configuration
For integration tests, set these environment variables:
Performance
Redis Performance
- Enqueue: ~10,000 messages/second
- Dequeue: ~8,000 messages/second
- Batch Operations: ~50,000 messages/second
SQS Performance
- Enqueue: ~1,000 messages/second
- Dequeue: ~800 messages/second
- Batch Operations: ~5,000 messages/second
Performance varies based on message size, network latency, and AWS region.
Best Practices
Message Design
// ✅ Good: Small, focused messages
let message = builder
.payload
.priority
.build;
// ❌ Avoid: Large payloads in message
let message = builder
.payload // Use S3 instead
.build;
Error Handling
// ✅ Good: Proper error handling and retries
if let Some = queue.dequeue.await?
Resource Management
// ✅ Good: Use connection pooling
let queue = new
.url
.pool_size // Match to your concurrency needs
.build
.await?;
// ✅ Good: Batch operations when possible
let messages = create_messages;
queue.enqueue_batch.await?;
Monitoring
// ✅ Good: Regular health checks
let health = queue.health_check.await?;
match health.status
Troubleshooting
Common Issues
Redis Connection Errors
// Check connection
if !queue.test_connection.await?
SQS Permission Errors
Ensure your AWS credentials have these permissions:
sqs:SendMessagesqs:ReceiveMessagesqs:DeleteMessagesqs:GetQueueAttributes
Large Message Errors
// Check message size
let message = builder.payload.build;
if let Ok = message.size_bytes
Technical Details
Core Architecture
The Backbone Queue Module is built around a unified trait-based architecture that provides consistent behavior across all backends:
use QueueService;
// All backends implement the same QueueService trait
async
QueueService Trait
Message Flow
graph TD
A[Producer creates QueueMessage] --> B[QueueService.enqueue]
B --> C[Backend Processing]
C --> D[Queue Storage]
D --> E[Consumer calls QueueService.dequeue]
E --> F[Backend retrieves message]
F --> G[Message processing]
G --> H{Processing success?}
H -->|Yes| I[QueueService.ack]
H -->|No| J[QueueService.nack]
I --> K[Message removed]
J --> L[Message requeued/DLQ]
Backend Implementations
Redis Backend
- Protocol: Redis RESP protocol
- Storage: Sorted sets for priority queues
- Connection: Connection pooling with bb8
- Durability: In-memory with optional persistence
- Scalability: Cluster support with Redis Cluster
// Redis data structure
ZADD queue:pending <priority_score> <message_id>
HSET queue:messages <message_id> <serialized_message>
AWS SQS Backend
- Protocol: AWS SDK HTTP API
- Storage: Managed AWS infrastructure
- Durability: 99.999999999% durability
- Scalability: Auto-scaling to any throughput
- Compliance: SOC, ISO, HIPAA compliant
// SQS message flow
SendMessage API
RabbitMQ Backend
- Protocol: AMQP 0.9.1
- Storage: Managed RabbitMQ server
- Routing: Exchanges and binding patterns
- Reliability: Publisher confirms and consumer acknowledgments
- Cluster: High availability with clustering
// AMQP flow
Publisher
Error Handling Strategy
use QueueError;
Performance Characteristics
Throughput Benchmarks
| Backend | Enqueue | Dequeue | Batch Ops |
|---|---|---|---|
| Redis | 10,000 msg/s | 8,000 msg/s | 50,000 msg/s |
| SQS | 1,000 msg/s | 800 msg/s | 5,000 msg/s |
| RabbitMQ | 8,000 msg/s | 6,000 msg/s | 20,000 msg/s |
Latency (P99)
| Backend | Enqueue | Dequeue |
|---|---|---|
| Redis | <1ms | <1ms |
| SQS | 10-50ms | 10-50ms |
| RabbitMQ | 2-5ms | 2-5ms |
Configuration Reference
Redis Configuration
| Option | Type | Default | Description |
|---|---|---|---|
url |
String |
redis://localhost:6379 |
Redis connection URL |
queue_name |
String |
default_queue |
Queue name |
key_prefix |
String |
backbone:queue |
Redis key prefix |
pool_size |
u32 |
10 |
Connection pool size |
SQS Configuration
| Option | Type | Default | Description |
|---|---|---|---|
queue_url |
String |
Required | SQS queue URL |
region |
String |
us-east-1 |
AWS region |
visibility_timeout |
i32 |
30 |
Message visibility timeout (seconds) |
wait_time_seconds |
i32 |
5 |
Long polling wait time |
max_number_of_messages |
i32 |
10 |
Max messages per receive |
RabbitMQ Configuration
| Option | Type | Default | Description |
|---|---|---|---|
connection_url |
String |
amqp://guest:guest@localhost:5672/%2f |
AMQP connection URL |
queue_name |
String |
default_queue |
Queue name |
exchange_name |
String |
default_exchange |
Exchange name |
exchange_type |
ExchangeType |
Direct |
Exchange type (Direct, Fanout, Topic) |
routing_key |
Option<String> |
None |
Default routing key |
Testing
Unit Tests
# Run all tests
# Run backend-specific tests
Integration Tests
# Run integration tests (requires infrastructure)
# Run performance tests
Test Coverage
- ✅ All QueueService trait methods
- ✅ Configuration validation
- ✅ Error handling scenarios
- ✅ Priority processing
- ✅ Batch operations
- ✅ Health checks
- ✅ Connection failure scenarios
Dependencies
[]
# Core
= { = "1.0", = ["full"] }
= { = "1.0", = ["derive"] }
= "1.0"
= "0.1"
# Backends
= { = "0.24", = ["tokio-comp"] }
= { = "1.0" }
= "1.0"
= "2.5" # RabbitMQ
= "2.0"
# Utilities
= { = "1.0", = ["v4"] }
= { = "0.4", = ["serde"] }
= "0.1"
= "1.0"
Real-World Examples
Microservices Communication
// User service publishes events
let event_queue = new.await?;
let user_created = builder
.payload
.expect
.routing_key
.priority
.build;
event_queue.enqueue.await?;
Webhook Processing
// Process incoming webhooks asynchronously
let webhook_queue = builder
.url
.queue_name
.build
.await?;
// Enqueue webhook for processing
let webhook = builder
.payload
.expect
.priority
.build;
webhook_queue.enqueue.await?;
Background Job Processing
// Queue background tasks
let job_queue = builder
.queue_url
.build
.await?;
let job = builder
.payload
.expect
.build;
job_queue.enqueue.await?;
Migration Guide
Between Backends
use ;
// Factory function for backend creation
async
// Use with any backend
let queue = create_queue.await;
process_messages.await?;
License
This project is part of the Backbone Framework and follows the same licensing terms.
Contributing
Please read the contribution guidelines before submitting pull requests.
Support
For questions and support:
- Create an issue on GitHub
- Check the documentation
- Review the test examples
- See RabbitMQ Integration Guide for detailed RabbitMQ setup