LLM Provider Clients and Abstractions

This module provides a unified interface for interacting with various Large Language Model (LLM) providers. It abstracts away provider-specific implementations behind common traits, allowing the rest of the application to work with any supported LLM.

Architecture

The module follows a factory pattern:

[LLMClient] - The core trait that all providers implement
[LLMClientFactory] - Factory trait for creating provider clients
[ProviderRegistry] - Registry for managing multiple providers
[ConfigBasedLLMFactory] - Creates clients based on ares.toml configuration
ToolCoordinator - Generic multi-turn tool calling coordinator
ClientPool - Connection pooling for efficient client reuse (DIR-44)

Supported Providers

Enable providers via Cargo features:

openai - OpenAI API (GPT-4, GPT-3.5, etc.)
azure - Azure AI Foundry OpenAI-compatible chat completions
bedrock - AWS Bedrock Claude via Anthropic Messages JSON
anthropic - Anthropic API (Claude 3, Claude 3.5, etc.)
ollama - Local Ollama server
llamacpp - llama.cpp server

Example

use ares::llm::{ConfigBasedLLMFactory, LLMClientFactory, Provider};

let factory = ConfigBasedLLMFactory::new(&config);
let client = factory.create_client(Provider::OpenAI)?;

let response = client.generate("What is 2+2?", None).await?;
println!("{}", response.content);

Connection Pooling (DIR-44)

Use the ClientPool for efficient connection reuse:

use ares::llm::pool::{ClientPool, PoolConfig};

let pool = ClientPool::new(PoolConfig::default());
pool.register_provider("openai", provider);

// Get a pooled client - automatically returned when guard is dropped
let guard = pool.get("openai").await?;
let response = guard.generate("Hello!").await?;

Tool Calling

Use the ToolCoordinator for multi-turn tool calling with any provider:

use ares::llm::coordinator::{ToolCoordinator, ToolCallingConfig};

let tools = std::sync::Arc::new(ares_tools::Tools::from_static(
    Vec::<std::sync::Arc<dyn ares_tools::Tool>>::new(),
));
let coordinator = ToolCoordinator::new(client, tools, ToolCallingConfig::default());
let ctx = cordis::Context::new_root();
let result = coordinator.execute(Some("System prompt"), "User query", &ctx).await?;

Streaming