Crate flyllm 

FlyLLM is a Rust library that provides a load-balanced, multi-provider client for Large Language Models.

It enables developers to seamlessly work with multiple LLM providers through a unified API with request routing, load balancing, and failure handling.

Features

• Multi-provider support: Integrate with OpenAI, Anthropic, Google, Mistral, Ollama, LM Studio, Groq, Cohere, Together AI, and Perplexity
• Load balancing: Distribute requests across multiple providers
• Automatic retries: Handle provider failures with configurable retry policies
• Task routing: Route specific tasks to the most suitable providers
• Metrics tracking: Monitor response times, error rates, and token usage

Example

use flyllm::{LlmManager, ProviderType, GenerationRequest, TaskDefinition};

async fn example() {
    // Create a manager using the builder pattern
    let manager = LlmManager::builder()
        .define_task(TaskDefinition::new("chat"))
        .add_instance(ProviderType::OpenAI, "gpt-4-turbo", "api-key")
        .supports("chat")
        .build()
        .await
        .expect("Failed to build manager");

    // Generate a response
    let request = GenerationRequest {
        prompt: "Explain Rust in one paragraph".to_string(),
        task: None,
        params: None,
    };

    let responses = manager.generate_sequentially(vec![request]).await;
    println!("{}", responses[0].content);
}

Re-exports

pub use providers::ProviderType;

pub use providers::LlmRequest;

pub use providers::LlmResponse;

pub use providers::LlmInstance;

pub use providers::create_instance;

pub use providers::AnthropicInstance;

pub use providers::OpenAIInstance;

pub use providers::ModelInfo;

pub use providers::ModelDiscovery;

pub use providers::StreamChunk;

pub use providers::LlmStream;

pub use errors::LlmError;

pub use errors::LlmResult;

pub use load_balancer::LlmManager;

pub use load_balancer::GenerationRequest;

pub use load_balancer::LlmManagerResponse;

pub use load_balancer::TaskDefinition;

Modules

config

Configuration module for TOML-based LlmManager configuration.

constants

errors

load_balancer

providers

Functions

use_logging

Initialize the logging system