Struct LlmBasedRouter 

pub struct LlmBasedRouter { /* private fields */ }

LLM-powered router that uses a model to make routing decisions

Uses the configured tier to analyze requests and choose optimal target. Provides intelligent fallback when rule-based routing is ambiguous.

Construction-Time Validation

Uses TierSelector to validate that the specified tier has available endpoints. The tier is chosen via config.routing.router_tier at construction time.

Implementations

impl LlmBasedRouter

pub fn new( selector: Arc<ModelSelector>, tier: TargetModel, router_timeout_secs: u64, metrics: Arc<Metrics>, ) -> AppResult<Self>

Create a new LLM-based router using the specified tier

Returns an error if no endpoints are configured for the specified tier.

Arguments

• selector - The underlying ModelSelector
• tier - Which tier (Fast, Balanced, Deep) to use for routing decisions
• router_timeout_secs - Timeout for router queries in seconds
• metrics - Metrics collector for observability

Tier Selection

• Fast: Lowest latency (~50-200ms) but may misroute complex requests
• Balanced: Recommended default (~100-500ms) with good accuracy
• Deep: Highest accuracy (~2-5s) but rarely worth the latency overhead

Construction-Time Validation

The TierSelector validates tier availability at construction, ensuring at least one endpoint exists for the specified tier.

pub fn tier(&self) -> TargetModel

Returns the configured router tier

pub async fn route( &self, user_prompt: &str, meta: &RouteMetadata, ) -> AppResult<RoutingDecision>

Route request using LLM analysis

Async Behavior

This method is async because it:

• Waits for LLM inference: ~100-500ms for 30B model routing decision (dominant latency)
• Makes HTTP requests to LLM endpoints (network I/O, ~10-100ms connection overhead)
• Awaits endpoint selection from ModelSelector (async lock acquisition, <1ms)
• Performs health tracking mark_success/mark_failure (async lock, <1ms)

Total typical latency: ~110-600ms (dominated by LLM inference)

Retry Logic & Failure Tracking (Dual-Level)

Implements sophisticated retry with TWO failure tracking mechanisms:

1. Request-Scoped Exclusion (failed_endpoints): Prevents retrying the same endpoint within THIS request. Clears when function returns.
2. Global Health Tracking: Marks endpoints unhealthy after 3 consecutive failures across ALL requests. Persists via ModelSelector’s health_checker.

Cancellation Safety

If the returned Future is dropped (cancelled), in-flight LLM queries will be aborted but endpoint health state remains consistent (mark_success/mark_failure only called after query completes).

Trait Implementations

impl LlmRouter for LlmBasedRouter

Implementation of LlmRouter trait for LlmBasedRouter

This allows LlmBasedRouter to be used as a trait object for dependency injection in tests.

fn route<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, user_prompt: &'life1 str, meta: &'life2 RouteMetadata, ) -> Pin<Box<dyn Future<Output = AppResult<RoutingDecision>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Route a request based on LLM analysis