Struct Client 

pub struct Client<CTXEXT, FENSLLM, CUSG> {
    pub ensemble_llm_fetcher: Arc<CachingFetcher<CTXEXT, FENSLLM>>,
    pub usage_handler: Arc<CUSG>,
    pub upstream_client: Client,
    pub backoff_current_interval: Duration,
    pub backoff_initial_interval: Duration,
    pub backoff_randomization_factor: f64,
    pub backoff_multiplier: f64,
    pub backoff_max_interval: Duration,
    pub backoff_max_elapsed_time: Duration,
}

Client for creating chat completions.

Handles Ensemble LLM fetching, upstream provider selection with fallbacks, retry logic with exponential backoff, and usage tracking.

Fields

ensemble_llm_fetcher: Arc<CachingFetcher<CTXEXT, FENSLLM>>

Caching fetcher for Ensemble LLM definitions.

usage_handler: Arc<CUSG>

Handler for tracking usage after completion.

upstream_client: Client

Client for communicating with upstream providers.

backoff_current_interval: Duration

Current backoff interval for retry logic.

backoff_initial_interval: Duration

Initial backoff interval for retry logic.

backoff_randomization_factor: f64

Randomization factor for backoff jitter.

backoff_multiplier: f64

Multiplier for exponential backoff growth.

backoff_max_interval: Duration

Maximum backoff interval.

backoff_max_elapsed_time: Duration

Maximum total time to spend on retries.

Implementations

impl<CTXEXT, FENSLLM, CUSG> Client<CTXEXT, FENSLLM, CUSG>

pub fn new( ensemble_llm_fetcher: Arc<CachingFetcher<CTXEXT, FENSLLM>>, usage_handler: Arc<CUSG>, upstream_client: Client, backoff_current_interval: Duration, backoff_initial_interval: Duration, backoff_randomization_factor: f64, backoff_multiplier: f64, backoff_max_interval: Duration, backoff_max_elapsed_time: Duration, ) -> Self

Creates a new chat completions client.

impl<CTXEXT, FENSLLM, CUSG> Client<CTXEXT, FENSLLM, CUSG>

where CTXEXT: ContextExt + Send + Sync + 'static, FENSLLM: Fetcher<CTXEXT> + Send + Sync + 'static, CUSG: UsageHandler<CTXEXT> + Send + Sync + 'static,

pub async fn create_unary_for_chat_handle_usage( self: Arc<Self>, ctx: Context<CTXEXT>, request: Arc<ChatCompletionCreateParams>, ) -> Result<ChatCompletion, Error>

Creates a unary chat completion, tracking usage after completion.

Internally streams the response and aggregates chunks into a single response.

pub async fn create_streaming_for_chat_handle_usage( self: Arc<Self>, ctx: Context<CTXEXT>, request: Arc<ChatCompletionCreateParams>, ) -> Result<impl Stream<Item = Result<ChatCompletionChunk, Error>> + Send + Unpin + 'static, Error>

Creates a streaming chat completion, tracking usage after the stream ends.

pub async fn create_streaming_for_vector_handle_usage( self: Arc<Self>, ctx: Context<CTXEXT>, request: Arc<VectorCompletionCreateParams>, vector_pfx_indices: Vec<Arc<Vec<(String, usize)>>>, ensemble_llm: EnsembleLlmWithFallbacksAndCount, ) -> Result<impl Stream<Item = Result<ChatCompletionChunk, Error>> + Send + Unpin + 'static, Error>

Creates a streaming completion for vector voting, tracking usage after the stream ends.

Used internally by vector completions to generate LLM votes.

impl<CTXEXT, FENSLLM, CUSG> Client<CTXEXT, FENSLLM, CUSG>

where CTXEXT: ContextExt + Send + Sync + 'static, FENSLLM: Fetcher<CTXEXT> + Send + Sync + 'static,

pub async fn create_streaming_for_chat( &self, ctx: Context<CTXEXT>, request: Arc<ChatCompletionCreateParams>, ) -> Result<impl Stream<Item = Result<ChatCompletionChunk, Error>> + Send + Unpin + 'static, Error>

Creates a streaming chat completion without usage tracking.

Handles model validation, Ensemble LLM fetching, fallback logic, and retry with exponential backoff.

pub async fn create_streaming_for_vector( &self, ctx: Context<CTXEXT>, request: Arc<VectorCompletionCreateParams>, vector_pfx_indices: Vec<Arc<Vec<(String, usize)>>>, ensemble_llm: EnsembleLlmWithFallbacksAndCount, ) -> Result<impl Stream<Item = Result<ChatCompletionChunk, Error>> + Send + Unpin + 'static, Error>

Creates a streaming completion for vector voting without usage tracking.

Used internally by vector completions. Handles fallback logic and retry with exponential backoff.

Trait Implementations

impl<CTXEXT: Clone, FENSLLM: Clone, CUSG: Clone> Clone for Client<CTXEXT, FENSLLM, CUSG>

fn clone(&self) -> Client<CTXEXT, FENSLLM, CUSG>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<CTXEXT: Debug, FENSLLM: Debug, CUSG: Debug> Debug for Client<CTXEXT, FENSLLM, CUSG>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.