Module advanced_routing 

Advanced Routing and Load Balancing Module

This module provides intelligent request routing, load balancing, and provider health monitoring for the Ultrafast Gateway. It includes circuit breakers, health checks, and adaptive routing strategies.

Overview

The advanced routing system provides:

• Intelligent Provider Selection: Route requests to the best available provider
• Health Monitoring: Real-time provider health tracking
• Circuit Breakers: Automatic failover and recovery
• Load Balancing: Distribute requests across healthy providers
• Adaptive Routing: Dynamic routing based on performance metrics
• Failover Strategies: Automatic fallback to backup providers

Routing Strategies

The system supports multiple routing strategies:

• Single Provider: Route all requests to a single provider
• Load Balancing: Distribute requests across multiple providers
• Failover: Primary provider with automatic fallback
• A/B Testing: Route requests to different providers for testing
• Geographic Routing: Route based on geographic location
• Cost-Based Routing: Route to the most cost-effective provider

Health Monitoring

Real-time provider health tracking includes:

• Latency Monitoring: Track response times for each provider
• Success Rate Tracking: Monitor success/failure ratios
• Consecutive Failures: Track consecutive failure counts
• Automatic Health Checks: Periodic provider health verification
• Degraded State Detection: Identify underperforming providers

Circuit Breaker Pattern

The system implements circuit breakers for each provider:

• Closed State: Normal operation, requests allowed
• Open State: Provider failing, requests blocked
• Half-Open State: Testing if provider has recovered
• Automatic Recovery: Automatic state transitions based on health

Usage

use ultrafast_gateway::advanced_routing::{AdvancedRouter, RoutingConfig};
use ultrafast_models_sdk::routing::RoutingStrategy;

// Create advanced router
let config = RoutingConfig {
    strategy: RoutingStrategy::LoadBalancing,
    health_check_interval: Duration::from_secs(30),
    failover_threshold: 0.8,
};

let router = AdvancedRouter::new(RoutingStrategy::LoadBalancing, config);

// Select provider for request
let selection = router.select_provider(&providers, &context).await;

// Update provider health after request
router.update_provider_health("openai", true, 150).await;

Health Check Configuration

Health checks can be configured via RoutingConfig:

[routing]
strategy = "load_balancing"
health_check_interval = "30s"
failover_threshold = 0.8

Performance Metrics

The routing system tracks comprehensive metrics:

• Provider Latency: Average response times per provider
• Success Rates: Request success percentages
• Failure Counts: Consecutive failure tracking
• Health Status: Current health state of each provider
• Routing Decisions: Provider selection statistics

Failover Strategies

The system supports multiple failover strategies:

• Immediate Failover: Switch to backup on first failure
• Threshold-Based: Switch after N consecutive failures
• Latency-Based: Switch when latency exceeds threshold
• Cost-Based: Switch to cheaper provider when possible

Monitoring and Alerts

The routing system provides monitoring capabilities:

• Health Status Endpoints: Real-time provider health
• Performance Metrics: Detailed routing statistics
• Circuit Breaker Status: Current state of each provider
• Alert Integration: Notifications for unhealthy providers

Structs

AdvancedRouter

CircuitBreaker

HealthChecker

Health checker for monitoring provider health and performance.

ProviderCircuitBreaker

ProviderHealthStatus

Health status for a specific provider.

RoutingStats

Enums

CircuitState