EdgeQuake LLM

edgequake-llm is a Rust AI runtime with a single abstraction over cloud APIs, local gateways, enterprise deployments, and testing backends. It ships first-class support for chat, streaming, tool calling, embeddings, image generation, caching, retries, rate limiting, cost tracking, and release-grade CI/CD.

Python users should use edgequake-litellm, the LiteLLM-compatible package backed by this crate.

What It Covers

• One trait-based surface for LLMs, embeddings, and Rust image generation.
• Production backends: OpenAI, Azure OpenAI, Anthropic, Gemini, Vertex AI, xAI, OpenRouter, Mistral, AWS Bedrock.
• Local and gateway backends: Ollama, LM Studio, VSCode Copilot proxy, generic OpenAI-compatible APIs.
• Additional embedding backend: Jina.
• Image generation backends in the Rust crate: Gemini image generation, Vertex Imagen, FAL, mock image generation.
• Operational layers: caching, retry, rate limiting, cost tracking, tracing, reranking, mock providers.

Install

[dependencies]
edgequake-llm = "0.5.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

bedrock is feature-gated:

[dependencies]
edgequake-llm = { version = "0.5.1", features = ["bedrock"] }

Note: the base crate declares rust-version = 1.83.0, but AWS Bedrock dependencies currently require a newer toolchain when the bedrock feature is enabled. Use stable Rust for release builds.

Quick Start

use edgequake_llm::{ChatMessage, LLMProvider, OpenAIProvider};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = OpenAIProvider::from_env()?;
    let messages = vec![ChatMessage::user("Explain Rust ownership in one sentence.")];
    let response = provider.chat(&messages, None).await?;

    println!("{}", response.content);
    Ok(())
}

Environment:

export OPENAI_API_KEY=sk-...

Provider Matrix

Image Generation Providers

Rust-only image generation support is exposed through ImageGenProvider and ImageGenFactory:

Common Setup

Image generation environment:

Factory Usage

ProviderFactory is the fastest way to wire environments or provider/model routing:

use edgequake_llm::{ProviderFactory, ProviderType};

let (llm, embedding) = ProviderFactory::from_env()?;
println!("llm={} embedding={}", llm.name(), embedding.name());

let (vertex_llm, _) = ProviderFactory::create_with_model(
    ProviderType::VertexAI,
    Some("gemini-2.5-flash"),
)?;

let custom = ProviderFactory::create_llm_provider(
    "openai-compatible",
    "deepseek-chat",
)?;

For generic OpenAI-compatible routing, set:

export OPENAI_COMPATIBLE_BASE_URL=https://api.groq.com/openai/v1
export OPENAI_COMPATIBLE_API_KEY=...

For Rust image generation, use:

use edgequake_llm::{ImageGenFactory, ImageGenRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = ImageGenFactory::from_env()?;
    let response = provider
        .generate(&ImageGenRequest::new("Editorial product photo on a concrete desk"))
        .await?;

    println!("generated {} image(s)", response.images.len());
    Ok(())
}

Python Package

edgequake-litellm is the Python package in this repo. It is a drop-in LiteLLM replacement backed by the Rust runtime:

import edgequake_litellm as litellm

resp = litellm.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Install:

pip install edgequake-litellm

See edgequake-litellm/README.md for provider routing, migration notes, wheel coverage, and release instructions. The Python package does not expose the Rust image-generation APIs yet.

Development

Local validation:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --locked
cargo doc --workspace --no-deps --all-features

Python package validation:

cd edgequake-litellm
pip install . -v
pytest -q -k "not e2e"

Release

Release guides:

• docs/providers.md: provider-by-provider setup
• docs/releasing.md: release checklist, tags, registry setup
• docs/release-cycle.md: end-to-end CI/CD flow
• CHANGELOG.md: release notes for the Rust crate
• edgequake-litellm/CHANGELOG.md: release notes for the Python package

Tag conventions:

• Rust crate: vX.Y.Z
• Python package: py-vX.Y.Z

Both publish workflows validate versions before publishing and can attach release artifacts to GitHub Releases.

License

Apache-2.0. See LICENSE-APACHE.