Skip to main content

Module llm

vtcode_core

Module llm 

Source
Expand description

§LLM Integration Layer

This module provides a unified, modular interface for integrating multiple LLM providers with VT Code, supporting Gemini, OpenAI, Anthropic, xAI, and DeepSeek.

§Architecture Overview

The LLM layer is designed with several key principles:

  • Unified Interface: Single AnyClient trait for all providers
  • Provider Agnostic: Easy switching between providers
  • Configuration Driven: TOML-based provider configuration
  • Error Handling: Comprehensive error types and recovery
  • Async Support: Full async/await support for all operations

§Supported Providers

ProviderStatusModels
Gemini✓gemini-3.1-pro-preview, gemini-3-flash-preview
OpenAI✓gpt-5, o3, o4-mini, gpt-5-mini, gpt-5-nano
Anthropic✓claude-4.1-opus, claude-4-sonnet
xAI✓grok-2-latest, grok-2-mini
DeepSeek✓deepseek-chat, deepseek-reasoner
Z.AI✓glm-5
Ollama✓gpt-oss:20b (local)

§Basic Usage

use vtcode_core::llm::{AnyClient, make_client};
use vtcode_core::utils::dot_config::ProviderConfigs;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Configure providers
    let providers = ProviderConfigs {
        gemini: Some(vtcode_core::utils::dot_config::ProviderConfig {
            api_key: std::env::var("GEMINI_API_KEY")?,
            model: "gemini-3-flash-preview".to_string(),
            ..Default::default()
        }),
        ..Default::default()
    };

    // Create client
    let client = make_client(&providers, "gemini")?;

    // Make a request
    let messages = vec![
        vtcode_core::llm::types::Message {
            role: "user".to_string(),
            content: "Hello, how can you help me with coding?".to_string(),
        }
    ];

    let response = client.chat(&messages, None).await?;
    println!("Response: {}", response.content);

    Ok(())
}

§Provider Configuration

use vtcode_core::utils::dot_config::{ProviderConfigs, ProviderConfig};

let config = ProviderConfigs {
    gemini: Some(ProviderConfig {
        api_key: "your-api-key".to_string(),
        model: "gemini-3-flash-preview".to_string(),
        temperature: Some(0.7),
        max_tokens: Some(4096),
        ..Default::default()
    }),
    openai: Some(ProviderConfig {
        api_key: "your-openai-key".to_string(),
        model: "gpt-5".to_string(),
        temperature: Some(0.3),
        max_tokens: Some(8192),
        ..Default::default()
    }),
    ..Default::default()
};

§Advanced Features

§Streaming Responses

use vtcode_core::llm::AnyClient;
use futures::StreamExt;

let client = make_client(&providers, "gemini")?;

let mut stream = client.chat_stream(&messages, None).await?;
while let Some(chunk) = stream.next().await {
    match chunk {
        Ok(response) => print!("{}", response.content),
        Err(e) => eprintln!("Error: {}", e),
    }
}

§Function Calling

use vtcode_core::llm::types::{FunctionDeclaration, FunctionCall};

let functions = vec![
    FunctionDeclaration {
        name: "read_file".to_string(),
        description: "Read a file from the filesystem".to_string(),
        parameters: serde_json::json!({
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "File path to read"}
            },
            "required": ["path"]
        }),
    }
];

let response = client.chat_with_functions(&messages, &functions, None).await?;

if let Some(function_call) = response.function_call {
    match function_call.name.as_str() {
        "read_file" => {
            // Handle function call
        }
        _ => {}
    }
}

§Error Handling

The LLM layer provides comprehensive error handling:

use vtcode_core::llm::LLMError;

match client.chat(&messages, None).await {
    Ok(response) => println!("Success: {}", response.content),
    Err(LLMError::Authentication) => eprintln!("Authentication failed"),
    Err(LLMError::RateLimit { metadata: None }) => eprintln!("Rate limit exceeded"),
    Err(LLMError::Network { message: e, metadata: None }) => eprintln!("Network error: {}", e),
    Err(LLMError::Provider { message: e, metadata: None }) => eprintln!("Provider error: {}", e),
    Err(e) => eprintln!("Other error: {}", e),
}

§Performance Considerations

  • Connection Pooling: Efficient connection reuse
  • Request Batching: Where supported by providers
  • Caching: Built-in prompt caching for repeated requests
  • Timeout Handling: Configurable timeouts and retries
  • Rate Limiting: Automatic rate limit handling

§LLM abstraction layer with modular architecture

This module provides a unified interface for different LLM providers with provider-specific implementations.

Re-exports§

pub use capabilities::ProviderCapabilities;
pub use client::AnyClient;
pub use client::ProviderClientAdapter;
pub use client::make_client;
pub use factory::create_provider_with_config;
pub use factory::get_factory;
pub use factory::get_models_manager;
pub use factory::infer_provider_from_model;
pub use lightweight_routing::LightweightFeature;
pub use lightweight_routing::LightweightRouteResolution;
pub use lightweight_routing::LightweightRouteSource;
pub use lightweight_routing::ModelRoute;
pub use lightweight_routing::auto_lightweight_model;
pub use lightweight_routing::create_provider_for_model_route;
pub use lightweight_routing::lightweight_model_choices;
pub use lightweight_routing::main_model_route;
pub use lightweight_routing::resolve_api_key_for_model_route;
pub use lightweight_routing::resolve_lightweight_route;
pub use model_resolver::DynamicModelMeta;
pub use model_resolver::DynamicModelRef;
pub use model_resolver::ModelAvailability;
pub use model_resolver::ModelResolver;
pub use model_resolver::ResolvedModel;
pub use optimized_client::OptimizedLLMClient;
pub use optimized_client::OptimizedRequest;
pub use optimized_client::OptimizedResponse;
pub use provider::LLMStream;
pub use provider::LLMStreamEvent;
pub use providers::AnthropicProvider;
pub use providers::GeminiProvider;
pub use providers::HuggingFaceProvider;
pub use providers::OllamaProvider;
pub use providers::OpenAIProvider;
pub use providers::ZAIProvider;
pub use tool_bridge::CorrelationStats;
pub use tool_bridge::IntentFulfillment;
pub use tool_bridge::MessageCorrelationTracker;
pub use tool_bridge::MessageToolCorrelation;
pub use tool_bridge::ToolExecution;
pub use tool_bridge::ToolIntent;
pub use tool_bridge::ToolIntentExtractor;

Modules§

capabilities
Compatibility re-export for provider capability metadata.
cgp
Context-generic provider wiring for VT Code’s LLM factory.
client
error_display
LLM error display utilities with enhanced ANSI color support
factory
http_client
Centralized HTTP client factory for LLM providers.
lightweight_routing
model_resolver
optimized_client
Optimized LLM client with connection pooling and request batching
provider
Universal LLM provider abstraction with API-specific role handling
provider_base
Base traits and utilities for LLM providers
provider_builder
provider_config
providers
rig_adapter
tool_bridge
Bridge between messages and tool executions
types
utils
Shared utilities for LLM request/response processing

Structs§

LLMResponse
Universal LLM response structure
Usage

Enums§

BackendKind
FinishReason
LLMError
LLM error types with optional provider metadata

Functions§

collect_single_response