Module llm

Expand description

§LLM Integration Layer

This module provides a unified, modular interface for integrating multiple LLM providers with VT Code, supporting Gemini, OpenAI, Anthropic, Meta AI, xAI, and DeepSeek.

§Architecture Overview

The LLM layer is designed with several key principles:

Unified Interface: Single AnyClient trait for all providers
Provider Agnostic: Easy switching between providers
Configuration Driven: TOML-based provider configuration
Error Handling: Comprehensive error types and recovery
Async Support: Full async/await support for all operations

§Supported Providers

Provider	Status	Models
Gemini	✓	gemini-3.1-pro-preview, gemini-3-flash-preview
OpenAI	✓	gpt-5, o3, o4-mini, gpt-5-mini, gpt-5-nano
Anthropic	✓	claude-4.1-opus, claude-4-sonnet
xAI	✓	grok-4.6, grok-4.5, grok-build-0.1, grok-4.3
DeepSeek	✓	deepseek-chat, deepseek-reasoner
Meta AI	✓	muse-spark-1.1, muse-spark-1.2
Z.AI	✓	glm-5
Ollama	✓	gpt-oss:20b (local)

§Basic Usage

use vtcode_core::llm::{AnyClient, make_client};
use vtcode_core::utils::dot_config::ProviderConfigs;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Configure providers
    let providers = ProviderConfigs {
        gemini: Some(vtcode_core::utils::dot_config::ProviderConfig {
            api_key: std::env::var("GEMINI_API_KEY")?,
            model: "gemini-3-flash-preview".to_string(),
            ..Default::default()
        }),
        ..Default::default()
    };

    // Create client
    let client = make_client(&providers, "gemini")?;

    // Make a request
    let messages = vec![
        vtcode_core::llm::types::Message {
            role: "user".to_string(),
            content: "Hello, how can you help me with coding?".to_string(),
        }
    ];

    let response = client.chat(&messages, None).await?;
    println!("Response: {}", response.content);

    Ok(())
}

§Provider Configuration

use vtcode_core::utils::dot_config::{ProviderConfigs, ProviderConfig};

let config = ProviderConfigs {
    gemini: Some(ProviderConfig {
        api_key: "your-api-key".to_string(),
        model: "gemini-3-flash-preview".to_string(),
        temperature: Some(0.7),
        max_tokens: Some(4096),
        ..Default::default()
    }),
    openai: Some(ProviderConfig {
        api_key: "your-openai-key".to_string(),
        model: "gpt-5".to_string(),
        temperature: Some(0.3),
        max_tokens: Some(8192),
        ..Default::default()
    }),
    ..Default::default()
};

§Advanced Features

§Streaming Responses

use vtcode_core::llm::AnyClient;
use futures::StreamExt;

let client = make_client(&providers, "gemini")?;

let mut stream = client.chat_stream(&messages, None).await?;
while let Some(chunk) = stream.next().await {
    match chunk {
        Ok(response) => print!("{}", response.content),
        Err(e) => eprintln!("Error: {}", e),
    }
}

§Function Calling

use vtcode_core::llm::types::{FunctionDeclaration, FunctionCall};

let functions = vec![
    FunctionDeclaration {
        name: "read_file".to_string(),
        description: "Read a file from the filesystem".to_string(),
        parameters: serde_json::json!({
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "File path to read"}
            },
            "required": ["path"]
        }),
    }
];

let response = client.chat_with_functions(&messages, &functions, None).await?;

if let Some(function_call) = response.function_call {
    match function_call.name.as_str() {
        "read_file" => {
            // Handle function call
        }
        _ => {}
    }
}

§Error Handling

The LLM layer provides comprehensive error handling:

use vtcode_core::llm::LLMError;

match client.chat(&messages, None).await {
    Ok(response) => println!("Success: {}", response.content),
    Err(LLMError::Authentication) => eprintln!("Authentication failed"),
    Err(LLMError::RateLimit { metadata: None }) => eprintln!("Rate limit exceeded"),
    Err(LLMError::Network { message: e, metadata: None }) => eprintln!("Network error: {}", e),
    Err(LLMError::Provider { message: e, metadata: None }) => eprintln!("Provider error: {}", e),
    Err(e) => eprintln!("Other error: {}", e),
}

§Performance Considerations

Connection Pooling: Efficient connection reuse
Request Batching: Where supported by providers
Caching: Built-in prompt caching for repeated requests
Timeout Handling: Configurable timeouts and retries
Rate Limiting: Automatic rate limit handling

§LLM abstraction layer with modular architecture

This module provides a unified interface for different LLM providers with provider-specific implementations.

Re-exports§

pub use client::AnyClient;
pub use client::ProviderClientAdapter;
pub use client::make_client;
pub use factory::create_provider_with_config;
pub use factory::get_factory;
pub use factory::get_models_manager;
pub use lightweight_routing::LightweightFeature;
pub use lightweight_routing::LightweightRouteResolution;
pub use lightweight_routing::LightweightRouteSource;
pub use lightweight_routing::ModelRoute;
pub use lightweight_routing::auto_lightweight_model;
pub use lightweight_routing::create_provider_for_model_route;
pub use lightweight_routing::lightweight_model_choices;
pub use lightweight_routing::main_model_route;
pub use lightweight_routing::resolve_api_key_for_model_route;
pub use lightweight_routing::resolve_lightweight_route;
pub use mock_client::StaticResponseClient;

Modules§

capabilities: Provider capability declarations and feature detection.
cgp: Context-Generic Provider (CGP) wiring for the LLM factory. Context-generic provider wiring for VT Code’s LLM factory.
client: Simplified LLM client trait and adapter.
config_adapter: Adapter between config-level and factory-level provider configurations. Re-exported from vtcode_llm::config_adapter to eliminate duplication.
error_display: Human-readable error formatting for LLM errors. Re-exported from vtcode_llm::error_display to eliminate duplication.
factory: LLM provider factory and global registry.
http_client: Shared HTTP client utilities for provider implementations. Centralized HTTP client factory for LLM providers.
lightweight_routing: Lightweight (cheap/fast) model routing for auxiliary features.
mock_client: Mock LLM client for testing. Utilities for deterministic tests that need an LLMClient implementation without performing network calls.
model_resolver: Model resolution, availability checks, and dynamic metadata. Re-exported from vtcode_llm::model_resolver to eliminate duplication.
provider: Core LLM provider trait and error types. Universal LLM provider abstraction with API-specific role handling.
provider_base: Shared provider utilities to eliminate duplicate code. Re-exported from vtcode_llm::provider_base to eliminate duplication.
provider_builder: Generic provider builder with builder-pattern construction.
provider_config: Per-provider configuration types and the unified creation shim.
providers: Re-exported provider implementations. LLM provider implementations.
reasoning_effort: Capability-driven reasoning validation before a provider request is sent.
request_gap: Shared idle-gap tracker for detecting when the provider prompt cache has likely expired between dispatched LLM requests. Shared tracker for detecting idle gaps between dispatched LLM requests.
rig_adapter: Adapter for the Rig agent framework. Re-exported from vtcode_llm::rig_adapter to eliminate duplication.
tool_bridge: Tool-call correlation and intent extraction for LLM responses. Re-exported from vtcode_llm::tool_bridge to eliminate duplication.
types: LLM request/response types, errors, and backend kind.
usage_cost: Provider-normalized usage accumulation and cache-aware session cost estimation. Provider-normalized usage accumulation and cache-aware session cost estimation.
utils: Shared utilities for request/response processing. Re-exported from vtcode_llm::utils to eliminate duplication.

Structs§

AdapterHooks: Shared adapter that enriches ProviderConfig conversions with WorkspacePaths, telemetry, and error-reporting hooks from vtcode-commons.
AnthropicProvider
CorrelationStats
DynamicModelMeta
DynamicModelRef
GeminiProvider
HuggingFaceProvider
LLMResponse: Universal LLM response structure
MergeGatewayProvider
MessageCorrelationTracker: Track correlations across a session
MessageToolCorrelation: Correlation between message intent and tool execution
MetaProvider
ModelResolver
OllamaProvider
OpenAIProvider
OwnedProviderConfig: Simple builder-friendly provider configuration backed by owned values.
ProviderCapabilities: Cached provider capabilities to reduce repeated trait method calls
ResolvedModel
ToolExecution: Tool execution record tied to message
ToolIntentExtractor: Extractor for tool intents from messages
Usage
ZAIProvider

Enums§

AdapterEvent: Telemetry event emitted when adapter hooks adjust provider configuration.
BackendKind
FinishReason
IntentFulfillment: Tracks intent fulfillment
LLMError: LLM error types with optional provider metadata
LLMStreamEvent
ModelAvailability
ToolIntent: Stated intent extracted from message

Traits§

AdapterHooksProvider: Trait that bundles the adapter hook dependencies into a single interface. This replaces the previous four-parameter generic bound (Paths, Telemetry, Reporter, Formatter) with a single trait, following the Config Trait pattern from the Rust Patterns guide (Ch 3 — Config Trait Pattern).
ProviderConfig: Trait describing the configuration required to instantiate an LLM provider.

Functions§

as_factory_config: Convert an implementor of ProviderConfig into the configuration used by the vtcode_core provider factory.
as_factory_config_with_hooks: Convert a ProviderConfig into the factory configuration using the supplied adapter hooks for workspace, telemetry, and error integration.
collect_single_response
infer_provider_from_model: Infer provider from model slug.

Type Aliases§

LLMStream

Module llm

Module llm Copy item path

§LLM Integration Layer

§Architecture Overview

§Supported Providers

§Basic Usage

§Provider Configuration

§Advanced Features

§Streaming Responses

§Function Calling

§Error Handling

§Performance Considerations

§LLM abstraction layer with modular architecture

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Module llm