pub struct OpenAiCompatibleClient { /* private fields */ }Expand description
A client for any OpenAI-compatible /chat/completions endpoint (OpenAI,
Azure-OpenAI-compatible gateways, vLLM, Ollama, LM Studio, …).
Uses reqwest with rustls; no default TLS features.
Transient failures are retried with capped, jittered exponential backoff
(same classification policy as crate::remote::RemoteNode): connect
errors, timeouts, and HTTP 5xx / 408 / 429 are retryable; a Retry-After
header (integer seconds) floors the delay. Other 4xx statuses and
request/response decode errors are fatal. Configure with
OpenAiCompatibleClient::with_retries /
OpenAiCompatibleClient::with_backoff.
Implementations§
Source§impl OpenAiCompatibleClient
impl OpenAiCompatibleClient
Sourcepub fn new(
base_url: impl Into<String>,
api_key: Option<String>,
model: impl Into<String>,
) -> Self
pub fn new( base_url: impl Into<String>, api_key: Option<String>, model: impl Into<String>, ) -> Self
A client for base_url (e.g. https://api.openai.com/v1) serving
model. Trailing slashes on base_url are trimmed.
Sourcepub fn from_env(
base_url: impl Into<String>,
api_key_env: &str,
model: impl Into<String>,
) -> Self
pub fn from_env( base_url: impl Into<String>, api_key_env: &str, model: impl Into<String>, ) -> Self
Read the API key from an environment variable.
A missing variable maps to “no key” (requests go out unauthenticated and typically fail 401 far from the cause), so a warning is logged at construction time to keep the cause close to the effect.
Sourcepub fn with_http_client(self, client: Client) -> Self
pub fn with_http_client(self, client: Client) -> Self
Override the underlying reqwest::Client (timeouts, proxies, …).
Sourcepub fn with_retries(self, max_retries: u32) -> Self
pub fn with_retries(self, max_retries: u32) -> Self
Override the number of retries after the initial attempt
(0 = single attempt, no retries).
Sourcepub fn with_backoff(self, base_backoff: Duration) -> Self
pub fn with_backoff(self, base_backoff: Duration) -> Self
Override the base backoff delay (attempt n waits roughly
base * 2^n, exponent capped and jittered).
Trait Implementations§
Source§impl ChatModel for OpenAiCompatibleClient
impl ChatModel for OpenAiCompatibleClient
Source§fn chat<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
messages: &'life1 [ChatMessage],
tools: &'life2 [Value],
) -> Pin<Box<dyn Future<Output = Result<ChatResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn chat<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
messages: &'life1 [ChatMessage],
tools: &'life2 [Value],
) -> Pin<Box<dyn Future<Output = Result<ChatResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Source§fn chat_stream<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
messages: &'life1 [ChatMessage],
tools: &'life2 [Value],
on_token: &'life3 mut (dyn FnMut(TokenChunk) + Send),
) -> Pin<Box<dyn Future<Output = Result<ChatResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
fn chat_stream<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
messages: &'life1 [ChatMessage],
tools: &'life2 [Value],
on_token: &'life3 mut (dyn FnMut(TokenChunk) + Send),
) -> Pin<Box<dyn Future<Output = Result<ChatResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
on_token as they arrive. Read moreSource§impl Clone for OpenAiCompatibleClient
impl Clone for OpenAiCompatibleClient
Source§fn clone(&self) -> OpenAiCompatibleClient
fn clone(&self) -> OpenAiCompatibleClient
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more