Struct LatencyRouter 

pub struct LatencyRouter { /* private fields */ }

Router that selects providers based on observed latency.

Implementations

impl LatencyRouter

pub fn new(providers: Vec<Arc<dyn Provider>>) -> Result<Self>

Create a latency router with default configuration.

Errors

Returns a routing error if no providers are supplied.

pub fn with_config( providers: Vec<Arc<dyn Provider>>, config: LatencyRouterConfig, ) -> Result<Self>

Create a latency router with explicit configuration.

Errors

Returns a routing error if no providers are supplied.

pub fn provider_count(&self) -> usize

Return the number of configured providers.

pub async fn complete( &self, request: &CompletionRequest, ) -> Result<CompletionResponse>

Execute a completion request using latency-based selection.

pub async fn stream( &self, request: &CompletionRequest, ) -> Result<Box<dyn Stream<Item = Result<CompletionChunk>> + Send + Unpin>>

Execute a streaming request using latency-based selection.