llm-connector 0.1.0

A lightweight Rust library for protocol adaptation across multiple LLM providers. Focuses solely on converting between different LLM provider APIs and providing a unified OpenAI-compatible interface.
Documentation

llm-connector

A lightweight Rust library for protocol adaptation across multiple LLM providers. This library focuses solely on converting between different LLM provider APIs and providing a unified OpenAI-compatible interface.

🎯 What llm-connector Does

Core Purpose: Protocol adaptation and API normalization

  • βœ… Protocol Conversion: Convert different LLM provider APIs to OpenAI-compatible format
  • βœ… Request/Response Standardization: Unified data structures across providers
  • βœ… Streaming Adaptation: Normalize SSE streams from different providers
  • βœ… Provider Abstraction: Extensible trait system for adding new providers
  • βœ… Simple Configuration: Basic API key and endpoint management

🚫 What llm-connector Does NOT Do

This library intentionally does not include:

  • ❌ Load balancing (use a reverse proxy or service mesh)
  • ❌ Health checking (use external monitoring)
  • ❌ Circuit breaking (use infrastructure-level solutions)
  • ❌ Complex routing strategies (keep it simple)
  • ❌ Built-in metrics collection (use tracing/metrics crates)
  • ❌ Request queuing or rate limiting

Features

  • Provider-agnostic types: Unified request/response structures
  • Simple provider trait: Easy to implement new providers
  • Streaming support: Unified SSE handling across providers
  • Minimal dependencies: Focused on core functionality
  • OpenAI compatibility: Drop-in replacement for OpenAI client

Supported Providers

  • OpenAI - GPT models (gpt-4, gpt-3.5-turbo, etc.)
  • Anthropic - Claude models (claude-3-5-sonnet, claude-3-haiku, etc.)
  • DeepSeek - DeepSeek models (deepseek-chat, deepseek-coder, etc.)
  • Zhipu GLM - GLM models (glm-4, glm-3-turbo, etc.)
  • Alibaba Qwen - Qwen models (qwen-turbo, qwen-plus, etc.)
  • Moonshot Kimi - Kimi models (moonshot-v1-8k, moonshot-v1-32k, etc.)

Quick Start

Add to your Cargo.toml:

[dependencies]
llm-connector = "0.1.0"
tokio = { version = "1.0", features = ["full"] }

Basic Usage

use llm_connector::{Client, ChatRequest, Message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize client with environment variables
    let client = Client::from_env();

    // Create a chat request
    let request = ChatRequest {
        model: "deepseek/deepseek-chat".to_string(),
        messages: vec![
            Message {
                role: "user".to_string(),
                content: "Hello, how are you?".to_string(),
                ..Default::default()
            }
        ],
        ..Default::default()
    };

    // Send request
    let response = client.chat(request).await?;
    println!("Response: {}", response.choices[0].message.content);

    Ok(())
}

Configuration

Environment Variables (Recommended)

Set environment variables for the providers you want to use:

# OpenAI
export OPENAI_API_KEY="your-openai-key"
export OPENAI_BASE_URL="https://api.openai.com/v1"  # optional

# DeepSeek
export DEEPSEEK_API_KEY="your-deepseek-key"
export DEEPSEEK_BASE_URL="https://api.deepseek.com/v1"  # optional

# Anthropic
export ANTHROPIC_API_KEY="your-anthropic-key"
export ANTHROPIC_BASE_URL="https://api.anthropic.com"  # optional

# Add other providers as needed...

Explicit Configuration

use llm_connector::{Client, Config, ProviderConfig};

let config = Config {
    openai: Some(ProviderConfig {
        api_key: "your-openai-key".to_string(),
        base_url: Some("https://api.openai.com/v1".to_string()),
    }),
    deepseek: Some(ProviderConfig {
        api_key: "your-deepseek-key".to_string(),
        base_url: Some("https://api.deepseek.com/v1".to_string()),
    }),
    // ... other providers
    ..Default::default()
};

let client = Client::with_config(config);

Streaming

use futures_util::StreamExt;
use llm_connector::{Client, ChatRequest, Message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::from_env();

    let request = ChatRequest {
        model: "openai/gpt-4".to_string(),
        messages: vec![
            Message {
                role: "user".to_string(),
                content: "Tell me a story".to_string(),
                ..Default::default()
            }
        ],
        stream: Some(true),
        ..Default::default()
    };

    let mut stream = client.chat_stream(request).await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        if let Some(choice) = chunk.choices.first() {
            if let Some(content) = &choice.delta.content {
                print!("{}", content);
            }
        }
    }

    Ok(())
}

Model Naming Convention

Use the format provider/model for explicit provider selection:

// Explicit provider selection (recommended)
"openai/gpt-4"
"anthropic/claude-3-5-sonnet-20241022"
"deepseek/deepseek-chat"
"glm/glm-4"
"qwen/qwen-turbo"
"kimi/moonshot-v1-8k"

// Direct model names (auto-detected)
"gpt-4"           // -> openai/gpt-4
"claude-3-haiku"  // -> anthropic/claude-3-haiku
"deepseek-chat"   // -> deepseek/deepseek-chat

Error Handling

The library provides structured error types:

use llm_connector::{Client, LlmConnectorError};

match client.chat(request).await {
    Ok(response) => println!("Success: {}", response.choices[0].message.content),
    Err(LlmConnectorError::AuthenticationError(msg)) => {
        eprintln!("Auth error: {}", msg);
    },
    Err(LlmConnectorError::RateLimitError(msg)) => {
        eprintln!("Rate limited: {}", msg);
    },
    Err(LlmConnectorError::NetworkError(msg)) => {
        eprintln!("Network error: {}", msg);
    },
    Err(e) => eprintln!("Other error: {}", e),
}

Extending with New Providers

Implement the Provider trait to add support for new LLM providers:

use llm_connector::{Provider, ProviderConfig, ChatRequest, ChatResponse};
use async_trait::async_trait;

struct MyCustomProvider {
    config: ProviderConfig,
}

#[async_trait]
impl Provider for MyCustomProvider {
    async fn chat(&self, request: &ChatRequest) -> Result<ChatResponse, LlmConnectorError> {
        // Implement your provider's API call here
        todo!()
    }

    async fn chat_stream(&self, request: &ChatRequest) -> Result<ChatStream, LlmConnectorError> {
        // Implement streaming support
        todo!()
    }
}

Design Philosophy

llm-connector follows the Unix philosophy: "Do one thing and do it well."

  • Single Responsibility: Only handles protocol adaptation between LLM providers
  • Minimal Dependencies: Keeps the dependency tree small and focused
  • Composable: Designed to be used as a building block in larger systems
  • No Magic: Explicit configuration and clear error messages
  • Provider Agnostic: Treats all providers equally, no special cases

What's NOT Included (By Design)

If you need these features, consider these alternatives:

  • Load Balancing: Use nginx, HAProxy, or a service mesh
  • Rate Limiting: Use Redis-based rate limiters or API gateways
  • Caching: Use Redis, Memcached, or HTTP caching proxies
  • Monitoring: Use Prometheus, Grafana, or APM solutions
  • Circuit Breaking: Use Hystrix-style libraries or service mesh features
  • Request Queuing: Use message queues like RabbitMQ or Apache Kafka

Contributing

We welcome contributions! Please focus on:

  1. Adding new providers - Implement the Provider trait
  2. Improving protocol compatibility - Better OpenAI API compliance
  3. Bug fixes - Especially around streaming and error handling
  4. Documentation - Examples and provider-specific notes

Please avoid adding features outside the core scope (load balancing, complex routing, etc.).

License

MIT