Struct RouterProvider 

pub struct RouterProvider { /* private fields */ }

Multi-provider LLM router implementing LlmProvider.

Construct with RouterProvider::new and configure a routing strategy via the builder methods. All configuration is immutable after construction except for runtime state (EMA statistics, Thompson distribution, bandit weights) which is stored behind Arc<Mutex<_>> and updated on every successful call.

Cloning is cheap: RouterState and all per-strategy state are Arc-wrapped and shared between the original and all clones — clone cost is proportional to the number of Arc fields, not to provider count or strategy complexity.

Implementations

impl RouterProvider

pub fn new(providers: Vec<AnyProvider>) -> Self

Create a new router over providers.

Use the builder methods (e.g., with_thompson, with_cascade) to configure a routing strategy. The default strategy is RouterStrategy::Ema.

pub fn with_embed_timeout(self, timeout_ms: u64) -> Self

Set the per-call timeout for embed across all non-bandit providers.

A stalled provider is skipped and the next candidate is tried. Default is 5000 ms. Pass 0 to disable the timeout (not recommended for production).

Examples

let router = RouterProvider::new(vec![]).with_embed_timeout(3000);

pub fn with_embed_concurrency(self, limit: usize) -> Self

Set the maximum number of concurrent embed_batch calls.

A value of 0 disables the semaphore (unlimited). Default is no semaphore.

pub fn set_memory_confidence(&self, confidence: Option<f32>)

Set the MAR (Memory-Augmented Routing) signal for the current turn.

Must be called before chat / chat_stream to influence bandit provider selection. Pass None to disable MAR for this turn.

pub fn with_ema(self, alpha: f64, reorder_interval: u64) -> Self

Enable EMA-based adaptive provider ordering.

pub fn with_coe( self, config: CoeConfig, secondary: AnyProvider, embed: AnyProvider, ) -> Self

Enable Collaborative Entropy (CoE) for Ema/Thompson strategies.

CoE detects uncertain responses via intra-entropy and inter-divergence signals, escalating to secondary when either threshold is exceeded.

No-op (with a warn!) when the active strategy is Cascade or Bandit.

pub fn coe_metrics(&self) -> Option<(u64, u64, u64, u64)>

Return session-level CoE metrics snapshot, or None if CoE is disabled.

pub fn with_asi(self, config: AsiRouterConfig) -> Self

Enable Agent Stability Index (ASI) coherence tracking.

When enabled, each successful response is embedded in a background task and added to a per-provider sliding window. The coherence score (cosine similarity of the latest embedding vs. window mean) penalizes Thompson/EMA routing priors for providers whose responses drift.

pub fn with_quality_gate(self, threshold: f32) -> Self

Enable embedding-based quality gate for Thompson/EMA routing.

After provider selection, computes cosine similarity between the query embedding and the response embedding. If below threshold, tries the next provider in the ordered list. On full exhaustion, returns the best response seen (highest similarity). Fail-open: embedding errors disable the gate for that request.

pub fn with_thompson(self, state_path: Option<&Path>) -> Self

Enable Thompson Sampling strategy.

Loads existing state from state_path if present; falls back to uniform prior. Prunes stale entries for providers not in the current chain.

pub fn with_bandit( self, config: BanditRouterConfig, state_path: Option<&Path>, embedding_provider: Option<AnyProvider>, ) -> Self

Enable PILOT bandit routing strategy (LinUCB contextual bandit).

Loads existing state from state_path (or the default path) using tokio::task::block_in_place to avoid blocking the async executor. Applies session-level decay if config.decay_factor < 1.0, and prunes arms for removed providers.

embedding_provider is used to obtain feature vectors for each query. When None, the bandit falls back to Thompson/uniform selection whenever an embedding cannot be obtained within config.embedding_timeout_ms.

The warmup_queries default of 0 in BanditRouterConfig is overridden here to 10 * num_providers to ensure sufficient initial exploration.

pub async fn save_bandit_state(&self)

Persist current bandit state to disk. No-op if bandit strategy is not active.

Uses tokio::task::spawn_blocking so it is safe to call from any async context.

pub fn bandit_stats(&self) -> Vec<(String, u64, f32)>

Return bandit diagnostic stats: (provider_name, pulls, mean_reward).

Returns an empty vec if bandit strategy is not active.

pub fn with_reputation( self, decay_factor: f64, weight: f64, min_observations: u64, state_path: Option<&Path>, ) -> Self

Enable Bayesian reputation scoring (RAPS).

Loads existing state from state_path (or the default path) using tokio::task::block_in_place to avoid blocking the async executor. Applies session-level decay and prunes stale provider entries.

No-op for Cascade routing (reputation is not used for cost-tier ordering).

pub fn record_quality_outcome(&self, _provider_name: &str, success: bool)

Record a quality outcome for the last active sub-provider (tool execution result).

Call only for semantic failures (invalid tool args, parse errors). Do NOT call for network errors, rate limits, or transient I/O failures. No-op when reputation scoring is disabled, strategy is Cascade, or no tool call has been made yet in this session.

The _provider_name parameter is ignored — quality is attributed to the sub-provider that served the most recent chat_with_tools call, tracked via last_active_provider.

pub fn last_selected_provider_kind(&self) -> &'static str

Returns the provider_kind_str of the last provider selected by the router.

Used by crate::any::AnyProvider::provider_kind_str to attribute cost to the actual child provider rather than returning the generic "local" sentinel for all router-dispatched calls. Falls back to "local" when no call has been made yet.

pub async fn save_reputation_state(&self)

Persist current reputation state to disk. No-op if reputation is disabled. Uses tokio::task::spawn_blocking so it is safe to call from any async context.

pub fn reputation_stats(&self) -> Vec<(String, f64, f64, f64, u64)>

Return reputation stats for all tracked providers: (name, alpha, beta, mean, observations).

pub fn with_cascade(self, config: CascadeRouterConfig) -> Self

Enable Cascade routing strategy.

Providers are tried in chain order (cheapest first). Each response is evaluated by the quality classifier; if it falls below quality_threshold, the next provider is tried. At most max_escalations quality-based escalations occur.

Network/API errors do not count against the escalation budget. The best response seen so far is returned if all escalations are exhausted.

When config.cost_tiers is set, providers are reordered once at construction time (no per-request cost). Providers absent from cost_tiers are appended after listed ones in original chain order. Unknown names in cost_tiers emit a warning and are otherwise ignored.

pub async fn save_thompson_state(&self)

Persist current Thompson state to disk.

No-op if Thompson strategy is not active.

Uses tokio::task::spawn_blocking so it is safe to call from any async context, including mid-request paths.

pub fn thompson_stats(&self) -> Vec<(String, f64, f64)>

Return a snapshot of Thompson distribution parameters for all tracked providers.

Returns an empty vec if Thompson strategy is not active.

pub fn set_status_tx(&mut self, tx: StatusTx)

pub async fn list_models_remote(&self) -> Result<Vec<RemoteModelInfo>, LlmError>

Aggregate model lists from all sub-providers, deduplicating by id.

Individual sub-provider errors are logged as warnings and skipped.

Errors

Always succeeds (errors per-provider are swallowed).

impl RouterProvider

pub fn embed_cache_metrics(&self) -> (u64, u64)

Return session-level embedding cache metrics: (total_calls, cache_hits).

Trait Implementations

impl Clone for RouterProvider

fn clone(&self) -> RouterProvider

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RouterProvider

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LlmProvider for RouterProvider

fn context_window(&self) -> Option<usize>

Report the model’s context window size in tokens.

fn chat( &self, messages: &[Message], ) -> impl Future<Output = Result<String, LlmError>> + Send

Send messages to the LLM and return the assistant response.

fn chat_stream( &self, messages: &[Message], ) -> impl Future<Output = Result<ChatStream, LlmError>> + Send

Send messages and return a stream of response chunks.

fn supports_streaming(&self) -> bool

Whether this provider supports native streaming.

fn embed( &self, text: &str, ) -> impl Future<Output = Result<Vec<f32>, LlmError>> + Send

Generate an embedding vector from text.

fn embed_batch( &self, texts: &[&str], ) -> impl Future<Output = Result<Vec<Vec<f32>>, LlmError>> + Send

Embed multiple texts in a single API call.

fn supports_embeddings(&self) -> bool

Whether this provider supports embedding generation.

fn name(&self) -> &str

Provider name for logging and identification.

fn model_identifier(&self) -> &str

Model identifier string (e.g. gpt-4o-mini, claude-sonnet-4-6). Used by cost-estimation heuristics. Returns "" when not applicable.

fn supports_tool_use(&self) -> bool

Whether this provider supports native tool_use / function calling.

fn list_models(&self) -> Vec<String>

Return the list of model identifiers this provider can serve. Default: empty (provider does not advertise models).

fn chat_with_tools( &self, messages: &[Message], tools: &[ToolDefinition], ) -> impl Future<Output = Result<ChatResponse, LlmError>> + Send

Send messages with tool definitions, returning a structured response.

fn debug_request_json( &self, messages: &[Message], tools: &[ToolDefinition], stream: bool, ) -> Value

Return the request payload that will be sent to the provider, for debug dumps.

fn last_cache_usage(&self) -> Option<(u64, u64)>

Return the cache usage from the last API call, if available. Returns (cache_creation_tokens, cache_read_tokens).

fn supports_vision(&self) -> bool

Whether this provider supports image input (vision).

fn last_usage(&self) -> Option<(u64, u64)>

Return token counts from the last API call, if available. Returns (input_tokens, output_tokens).

fn last_reasoning_tokens(&self) -> Option<u64>

Return reasoning tokens from the last API call, if the provider reports them.

fn take_compaction_summary(&self) -> Option<String>

Return the compaction summary from the most recent API call, if a server-side compaction occurred (Claude compact-2026-01-12 beta). Clears the stored value.

fn chat_with_extras( &self, messages: &[Message], ) -> impl Future<Output = Result<(String, ChatExtras), LlmError>> + Send

Send messages and return the assistant response together with per-call extras.

fn supports_structured_output(&self) -> bool

Whether this provider supports native structured output.

async fn chat_typed<T>(&self, messages: &[Message]) -> Result<T, LlmError>

where T: DeserializeOwned + JsonSchema + 'static, Self: Sized,

Send messages and parse the response into a typed value T.

impl RouterAware for RouterProvider

fn set_memory_confidence(&self, confidence: Option<f32>)

Set the MAR (Memory-Augmented Routing) confidence signal for the current turn.

fn record_quality_outcome(&self, provider_name: &str, success: bool)

Record a semantic quality outcome for the last active sub-provider (RAPS).