Trait Provider 

pub trait Provider: Send + Sync {
    // Required methods
    fn chat_completion(
        &self,
        request: &ChatCompletionRequest,
    ) -> impl Future<Output = Result<ChatCompletionResponse, Error>> + Send;
    fn chat_completion_stream(
        &self,
        request: &ChatCompletionRequest,
    ) -> impl Future<Output = Result<BoxStream<'static, Result<ChatCompletionChunk, Error>>, Error>> + Send;
    fn anthropic_messages(
        &self,
        request: &AnthropicRequest,
    ) -> impl Future<Output = Result<AnthropicResponse, Error>> + Send;
    fn anthropic_messages_stream(
        &self,
        request: &AnthropicRequest,
    ) -> impl Future<Output = Result<BoxStream<'static, Result<AnthropicStreamEvent, Error>>, Error>> + Send;
    fn gemini_generate_content_stream(
        &self,
        model: &str,
        request: &GeminiRequest,
    ) -> impl Future<Output = Result<BoxStream<'static, Result<GeminiResponse, Error>>, Error>> + Send;

    // Provided methods
    fn gemini_generate_content(
        &self,
        model: &str,
        request: &GeminiRequest,
    ) -> impl Future<Output = Result<GeminiResponse, Error>> + Send { ... }
    fn complete(
        &self,
        _request: &Request,
    ) -> impl Future<Output = Result<Response, Error>> + Send { ... }
    fn complete_stream(
        &self,
        _request: &Request,
    ) -> impl Future<Output = Result<BoxStream<'static, Result<StreamEvent, Error>>, Error>> + Send { ... }
    fn embedding(
        &self,
        _request: &EmbeddingRequest,
    ) -> impl Future<Output = Result<EmbeddingResponse, Error>> + Send { ... }
    fn image_generation(
        &self,
        _request: &ImageRequest,
    ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send { ... }
    fn audio_speech(
        &self,
        _request: &AudioSpeechRequest,
    ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send { ... }
    fn audio_transcription(
        &self,
        _model: &str,
        _fields: &[MultipartField],
    ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send { ... }
    fn is_openai_compat(&self) -> bool { ... }
    fn is_anthropic_compat(&self) -> bool { ... }
    fn is_gemini_compat(&self) -> bool { ... }
    fn chat_completion_stream_passthrough(
        &self,
        _model: &str,
        _body_stream: ByteStream,
    ) -> impl Future<Output = Result<ByteStream, Error>> + Send { ... }
    fn chat_completion_stream_raw(
        &self,
        _model: &str,
        _raw_body: Bytes,
    ) -> impl Future<Output = Result<ByteStream, Error>> + Send { ... }
    fn chat_completion_raw(
        &self,
        _model: &str,
        raw_body: Bytes,
    ) -> impl Future<Output = Result<Bytes, Error>> + Send { ... }
    fn anthropic_messages_raw(
        &self,
        _raw_body: Bytes,
    ) -> impl Future<Output = Result<Bytes, Error>> + Send { ... }
    fn anthropic_messages_stream_raw(
        &self,
        _raw_body: Bytes,
    ) -> impl Future<Output = Result<ByteStream, Error>> + Send { ... }
    fn gemini_generate_content_raw(
        &self,
        model: &str,
        raw_body: Bytes,
    ) -> impl Future<Output = Result<Bytes, Error>> + Send { ... }
    fn gemini_generate_content_stream_raw(
        &self,
        _model: &str,
        _raw_body: Bytes,
    ) -> impl Future<Output = Result<ByteStream, Error>> + Send { ... }
}

The dispatch surface every provider implementation satisfies.

Returns futures via RPITIT (return-position impl Trait in trait) so the type system can monomorphize through the trait without dyn dispatch or per-call boxing. The proxy crate is generic over P: Provider; the binary crate picks the concrete type by defining a workspace-level union enum that delegates each method.

Streaming responses use BoxStream because returning an opaque stream from an async-returning trait method requires a fixed type at the trait boundary; the boxing is one allocation per stream creation, not per item. Implementors must clone any borrowed data from the request before returning the stream — the returned BoxStream is 'static and cannot borrow from the request reference.

The optional methods (embedding, image_generation, audio_speech, audio_transcription) default to returning Error::not_implemented, so concrete providers only override the methods they actually support. Overrides are free to capture self or the request reference — only the default impl bodies happen to capture nothing, and that’s an implementation detail of the defaults, not a constraint on the trait.

Required Methods

fn chat_completion( &self, request: &ChatCompletionRequest, ) -> impl Future<Output = Result<ChatCompletionResponse, Error>> + Send

fn chat_completion_stream( &self, request: &ChatCompletionRequest, ) -> impl Future<Output = Result<BoxStream<'static, Result<ChatCompletionChunk, Error>>, Error>> + Send

fn anthropic_messages( &self, request: &AnthropicRequest, ) -> impl Future<Output = Result<AnthropicResponse, Error>> + Send

fn anthropic_messages_stream( &self, request: &AnthropicRequest, ) -> impl Future<Output = Result<BoxStream<'static, Result<AnthropicStreamEvent, Error>>, Error>> + Send

fn gemini_generate_content_stream( &self, model: &str, request: &GeminiRequest, ) -> impl Future<Output = Result<BoxStream<'static, Result<GeminiResponse, Error>>, Error>> + Send

Gemini Generative Language API: :streamGenerateContent.

Returns native GeminiResponse items — each SSE chunk from Google is a full response object with a single candidate.

Provided Methods

fn gemini_generate_content( &self, model: &str, request: &GeminiRequest, ) -> impl Future<Output = Result<GeminiResponse, Error>> + Send

Gemini Generative Language API: :generateContent.

Default impl converts GeminiRequest to canonical Anthropic IR (filling model from the path), dispatches to anthropic_messages, and converts the response back. Providers that natively speak Gemini (e.g. GoogleProvider) override for direct dispatch.

fn complete( &self, _request: &Request, ) -> impl Future<Output = Result<Response, Error>> + Send

fn complete_stream( &self, _request: &Request, ) -> impl Future<Output = Result<BoxStream<'static, Result<StreamEvent, Error>>, Error>> + Send

fn embedding( &self, _request: &EmbeddingRequest, ) -> impl Future<Output = Result<EmbeddingResponse, Error>> + Send

fn image_generation( &self, _request: &ImageRequest, ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send

fn audio_speech( &self, _request: &AudioSpeechRequest, ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send

fn audio_transcription( &self, _model: &str, _fields: &[MultipartField], ) -> impl Future<Output = Result<(Bytes, String), Error>> + Send

fn is_openai_compat(&self) -> bool

Whether this provider speaks the OpenAI wire format and can forward raw JSON bytes without deserialization.

fn is_anthropic_compat(&self) -> bool

Whether this provider speaks the Anthropic wire format and can forward raw /v1/messages bytes without translation.

fn is_gemini_compat(&self) -> bool

Whether this provider speaks the Gemini wire format and can forward raw :generateContent bytes without translation.

fn chat_completion_stream_passthrough( &self, _model: &str, _body_stream: ByteStream, ) -> impl Future<Output = Result<ByteStream, Error>> + Send

Stream a request body directly to an OpenAI-compatible endpoint and return the raw SSE response stream. The body is consumed — no retries.

fn chat_completion_stream_raw( &self, _model: &str, _raw_body: Bytes, ) -> impl Future<Output = Result<ByteStream, Error>> + Send

Stream raw OpenAI SSE bytes from an OpenAI-compatible endpoint. The default returns not_implemented. OpenAI-compatible providers override to forward bytes without deserialization.

fn chat_completion_raw( &self, _model: &str, raw_body: Bytes, ) -> impl Future<Output = Result<Bytes, Error>> + Send

Forward raw OpenAI-format JSON body and return raw response bytes. The default deserializes, calls chat_completion, and re-serializes. OpenAI-compatible providers override to skip serde.

fn anthropic_messages_raw( &self, _raw_body: Bytes, ) -> impl Future<Output = Result<Bytes, Error>> + Send

Forward raw Anthropic-format JSON body and return raw response bytes. The default translates Anthropic → OpenAI, calls [chat_completion], and translates back. The Anthropic provider overrides to skip serde.

fn anthropic_messages_stream_raw( &self, _raw_body: Bytes, ) -> impl Future<Output = Result<ByteStream, Error>> + Send

Stream raw Anthropic SSE bytes from an Anthropic-compatible endpoint.

fn gemini_generate_content_raw( &self, model: &str, raw_body: Bytes, ) -> impl Future<Output = Result<Bytes, Error>> + Send

Forward raw Gemini-format JSON body and return raw response bytes. Defaults to deserialize → gemini_generate_content → re-serialize. Gemini-compatible providers override to skip serde.

fn gemini_generate_content_stream_raw( &self, _model: &str, _raw_body: Bytes, ) -> impl Future<Output = Result<ByteStream, Error>> + Send

Stream raw Gemini SSE bytes from a Gemini-compatible endpoint.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl<P: Provider> Provider for Retrying<P>