CloudLLM

CloudLLM is a batteries-included Rust toolkit for working with remote Large Language Models. It provides:

• ergonomic provider clients built on a shared ClientWrapper trait,
• a stateful LLMSession that automates context trimming and token accounting,
• a multi-agent council orchestration engine, and
• a protocol-agnostic tool interface with adapters for MCP and OpenAI function calling.

The entire public API is documented with compilable examples—run cargo doc --open to browse the crate-level manual.

Installation

Add CloudLLM to your project:

[dependencies]
cloudllm = "0.4.0"

The crate targets tokio 1.x and Rust 1.70+.

Quick start

1. Initialising a session

use std::sync::Arc;

use cloudllm::{init_logger, LLMSession, Role};
use cloudllm::clients::openai::{Model, OpenAIClient};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    init_logger();

    let api_key = std::env::var("OPEN_AI_SECRET")?;
    let client = OpenAIClient::new_with_model_enum(&api_key, Model::GPT41Nano);

    let mut session = LLMSession::new(
        Arc::new(client),
        "You write product update haikus.".to_owned(),
        8_192,
    );

    let reply = session
        .send_message(Role::User, "Announce the logging feature.".to_owned(), None)
        .await?;

    println!("Assistant: {}", reply.content);
    println!("Usage (tokens): {:?}", session.token_usage());
    Ok(())
}

2. Streaming tokens in real time

use cloudllm::{LLMSession, Role};
use cloudllm::clients::openai::{Model, OpenAIClient};
use futures_util::StreamExt;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = std::env::var("OPEN_AI_SECRET")?;
    let client = Arc::new(OpenAIClient::new_with_model_enum(&api_key, Model::GPT41Mini));
    let mut session = LLMSession::new(client, "You think out loud.".into(), 16_000);

    if let Some(mut stream) = session
        .send_message_stream(Role::User, "Explain type erasure.".into(), None)
        .await? {
        while let Some(chunk) = stream.next().await {
            let chunk = chunk?;
            print!("{}", chunk.content);
            if let Some(reason) = chunk.finish_reason {
                println!("\n<terminated: {reason}>");
            }
        }
    }

    Ok(())
}

Provider wrappers

CloudLLM ships wrappers for popular OpenAI-compatible services:

Providers share the ClientWrapper contract, so you can swap them without changing downstream code.

use cloudllm::ClientWrapper;
use cloudllm::clients::claude::{ClaudeClient, Model};
use cloudllm::client_wrapper::{Message, Role};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("ANTHROPIC_KEY")?;
    let claude = ClaudeClient::new_with_model_enum(&key, Model::ClaudeSonnet4);

    let response = claude
        .send_message(
            &[Message { role: Role::User, content: "Summarise rice fermentation.".into() }],
            None,
        )
        .await?;

    println!("{}", response.content);
    Ok(())
}

Every wrapper exposes token accounting via ClientWrapper::get_last_usage.

Tooling

Describe tools once, reuse them everywhere:

use std::sync::Arc;

use cloudllm::tool_adapters::CustomToolAdapter;
use cloudllm::tool_protocol::{ToolMetadata, ToolParameter, ToolParameterType, ToolRegistry, ToolResult};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let adapter = Arc::new(CustomToolAdapter::new());

    adapter
        .register_tool(
            ToolMetadata::new("area", "Compute the area of a rectangle")
                .with_parameter(ToolParameter::new("width", ToolParameterType::Number).required())
                .with_parameter(ToolParameter::new("height", ToolParameterType::Number).required()),
            Arc::new(|params| {
                let width = params["width"].as_f64().unwrap();
                let height = params["height"].as_f64().unwrap();
                Ok(ToolResult::success(serde_json::json!({"area": width * height})))
            }),
        )
        .await;

    let registry = ToolRegistry::new(adapter.clone());
    assert_eq!(registry.list_tools()[0].name, "area");
    Ok(())
}

Adapters for the Model Context Protocol (McpAdapter) and OpenAI function calling (OpenAIFunctionAdapter) are ready to use. Implementors can create additional adapters by implementing ToolProtocol.

Councils: multi-agent orchestration

The council module orchestrates conversations between agents built on any ClientWrapper. Choose from parallel, round-robin, moderated, hierarchical, or debate modes.

use std::sync::Arc;

use cloudllm::council::{Agent, Council, CouncilMode};
use cloudllm::clients::openai::{Model, OpenAIClient};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("OPEN_AI_SECRET")?;

    let architect = Agent::new(
        "architect",
        "System Architect",
        Arc::new(OpenAIClient::new_with_model_enum(&key, Model::GPT4o)),
    )
    .with_expertise("Distributed systems")
    .with_personality("Pragmatic, direct");

    let tester = Agent::new(
        "qa",
        "QA Lead",
        Arc::new(OpenAIClient::new_with_model_enum(&key, Model::GPT41Mini)),
    )
    .with_expertise("Test automation")
    .with_personality("Sceptical, detail-oriented");

    let mut council = Council::new("design-review", "Deployment Review")
        .with_mode(CouncilMode::RoundRobin)
        .with_system_context("Collaboratively review the proposed architecture.");

    council.add_agent(architect)?;
    council.add_agent(tester)?;

    let outcome = council
        .discuss("Evaluate whether the blue/green rollout plan is sufficient.", 2)
        .await?;

    for msg in outcome.messages {
        if let Some(name) = msg.agent_name {
            println!("{name}: {}", msg.content);
        }
    }

    Ok(())
}

For a deep dive, read COUNCIL_TUTORIAL.md which walks through each collaboration mode with progressively sophisticated examples.

Examples

Clone the repository and run the provided examples:

export OPEN_AI_SECRET=...
export ANTHROPIC_KEY=...
export GEMINI_KEY=...
export XAI_KEY=...

cargo run --example interactive_session
cargo run --example streaming_session
cargo run --example council_demo

Each example corresponds to a module in the documentation so you can cross-reference the code with explanations.

Support & contributing

Issues and pull requests are welcome via GitHub. Please open focused pull requests against main and include tests or doc updates where relevant.

CloudLLM is released under the MIT License.

Happy orchestration! 🤖🤝🤖