Skip to main content

ChatProvider

recursive::llm

Trait ChatProvider 

Source 
pub trait ChatProvider: Send + Sync {
    // Required method
    fn complete<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        messages: &'life1 [Message],
        tools: &'life2 [ToolSpec],
    ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;

    // Provided methods
    fn supports_deferred_tools(&self) -> bool { ... }
    fn complete_with_search<'life0, 'life1, 'life2, 'life3, 'async_trait>(
        &'life0 self,
        messages: &'life1 [Message],
        eager_tools: &'life2 [(ToolSpec, Option<String>)],
        deferred_tools: &'life3 [(ToolSpec, Option<String>)],
    ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait,
             'life3: 'async_trait { ... }
    fn complete_structured<'life0, 'async_trait>(
        &'life0 self,
        _req: StructuredRequest,
    ) -> Pin<Box<dyn Future<Output = Result<Value>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait { ... }
    fn stream<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        messages: &'life1 [Message],
        tools: &'life2 [ToolSpec],
        stream_tx: Option<StreamSender>,
        cancel_token: Option<CancellationToken>,
    ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait { ... }
    fn stream_with_search<'life0, 'life1, 'life2, 'life3, 'async_trait>(
        &'life0 self,
        messages: &'life1 [Message],
        eager_tools: &'life2 [(ToolSpec, Option<String>)],
        deferred_tools: &'life3 [(ToolSpec, Option<String>)],
        stream_tx: Option<StreamSender>,
        cancel_token: Option<CancellationToken>,
    ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait,
             'life3: 'async_trait { ... }
    fn complete_simple<'life0, 'life1, 'async_trait>(
        &'life0 self,
        prompt: &'life1 str,
        _temperature: f32,
    ) -> Pin<Box<dyn Future<Output = Result<String>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
}
Expand description

Core trait for LLM chat-completion providers.

Implementors must provide ChatProvider::complete. All other methods have default implementations that delegate to complete or return appropriate defaults, so adding a new provider only requires implementing one async fn.

Required Methods§

Source

fn complete<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, messages: &'life1 [Message], tools: &'life2 [ToolSpec], ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Provided Methods§

Source

fn supports_deferred_tools(&self) -> bool

Whether this provider supports deferred tool loading via tool_reference content blocks (Anthropic API feature).

When true, run_core will:

  • Strip deferred tools from the tools array sent to the LLM
  • Inject <available-deferred-tools> into the messages
  • Rely on serialize_messages_anthropic to convert ToolSearchTool results into tool_reference blocks for schema expansion.

When false (default), all tools are sent eagerly and ToolSearchTool is NOT registered in the registry.

Variant that accepts a partition between eager and deferred tools. Providers that support native deferred loading (e.g. Anthropic via defer_loading: true + tool_reference content blocks) override this. The default implementation concatenates the two lists (dropping the hints) and calls complete() — i.e., it ignores the partition and behaves identically to the legacy interface.

Providers that do NOT support deferred tool loading (e.g. the OpenAI provider) inherit this default and see every tool as eager. No code change is required in those providers.

Source

fn complete_structured<'life0, 'async_trait>( &'life0 self, _req: StructuredRequest, ) -> Pin<Box<dyn Future<Output = Result<Value>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Request a JSON response conforming to a caller-supplied schema. Default impl returns an error. Providers that support structured output (e.g. OpenAI-compatible) override this.

Source

fn stream<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, messages: &'life1 [Message], tools: &'life2 [ToolSpec], stream_tx: Option<StreamSender>, cancel_token: Option<CancellationToken>, ) -> Pin<Box<dyn Future<Output = Result<Completion>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Stream a completion token-by-token.

The default implementation delegates to ChatProvider::complete and emits the entire content as a single delta via the channel (if configured). Providers that support native SSE streaming should override this.

The stream_tx channel receives partial-token deltas as they are parsed. The returned Completion is the fully accumulated result.

The optional cancel_token, when set, enables cooperative cancellation of the underlying SSE read loop via tokio::select!. When the token fires, the provider drops the HTTP response (closing the TCP connection) and returns Err(Error::Cancelled).

Search-aware streaming: model sees eager tools + ToolSearchTool; deferred tools become available only after the model requests them.

The default implementation merges both lists and delegates to ChatProvider::stream — i.e., behaves like the legacy eager-only path. Providers that support deferred tool loading (e.g. Anthropic) override this to inject ToolSearchTool into the eager set and handle the search loop across multiple streaming rounds.

Source

fn complete_simple<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, _temperature: f32, ) -> Pin<Box<dyn Future<Output = Result<String>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Simple completion with a single user prompt.

Wraps the prompt in a user Message and calls complete with no tools. Providers that support temperature or other controls should override this method. The default implementation ignores temperature.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§