EdgeQuake LLM

edgequake-llm is a Rust AI runtime with a single abstraction over cloud APIs, local gateways, enterprise deployments, and testing backends. It ships first-class support for chat, streaming, tool calling, embeddings, image generation, caching, retries, rate limiting, cost tracking, and release-grade CI/CD.

Python users should use edgequake-litellm, the LiteLLM-compatible package backed by this crate.

What It Covers

• One trait-based surface for LLMs, embeddings, and Rust image generation.
• Production backends: OpenAI, Azure OpenAI, Anthropic, Gemini, Vertex AI, xAI, OpenRouter, NVIDIA NIM, Mistral, AWS Bedrock.
• Local and gateway backends: Ollama, LM Studio, GitHub Copilot direct mode (proxy optional), generic OpenAI-compatible APIs.
• Additional embedding backend: Jina.
• Image generation backends in the Rust crate: Gemini image generation, Vertex Imagen, FAL, mock image generation.
• Model Discovery — programmatic capability discovery across 11 providers with zero heuristics.
• Operational layers: caching, retry, rate limiting, cost tracking, tracing, reranking, mock providers.

Install

[dependencies]
edgequake-llm = "0.8.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

bedrock is feature-gated:

[dependencies]
edgequake-llm = { version = "0.8.0", features = ["bedrock"] }

Note: the repository is now pinned to Rust 1.95.0, and the Bedrock integration is verified against the latest published AWS SDK crate set, including the current Bedrock runtime release.

Provider compatibility highlights in this release:

• Anthropic-compatible adapters now preserve final streamed tool-call deltas even when the upstream SSE stream ends without a trailing newline.
• Provider-side schema normalization now aligns Anthropic, Bedrock, and Gemini tool declarations with the stricter subsets those APIs actually accept.
• Normalized usage reporting distinguishes cache writes from cache hits where the upstream provider reports both.
• Gemini provider now preserves function-call IDs across assistant tool calls, streamed deltas, and tool-result follow-ups.
• Mistral provider now includes native audio (speech, transcriptions, voices) and OCR endpoint wrappers.

New capabilities in this release:

• Structured Outputs — CompletionOptions::json_schema(schema) for JSON Schema-constrained responses
• OpenRouter fallback models — with_fallback_models() for sequential model failover
• Azure Managed Identity — AzureCredential::BearerToken for Entra ID auth
• Vertex AI ADC — automatic GCE metadata token refresh
• Mistral variable embedding dimensions — with_embed_dimensions(dim)
• Ollama truncate parameter in embedding requests
• Bedrock Nova 2 multimodal embeddings
• NVIDIA vision model detection improvements

Latest model IDs validated on 2026-07-04 from official provider documentation:

• OpenAI (docs): gpt-5.5, gpt-5.4, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4o, o3, o4-mini
• Anthropic (docs): claude-fable-5, claude-opus-4-8, claude-sonnet-5, claude-opus-4-7, claude-sonnet-4-6
• Gemini (docs): gemini-3.5-flash, gemini-2.5-flash (default), gemini-2.5-pro, gemini-3.1-pro-preview
• Mistral (docs): mistral-small-latest (default), mistral-large-latest, codestral-latest, magistral-medium-latest
• xAI (docs): grok-4, grok-3, grok-3-mini

Quick Start

use edgequake_llm::{ChatMessage, LLMProvider, OpenAIProvider};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = OpenAIProvider::from_env()?;
    let messages = vec![ChatMessage::user("Explain Rust ownership in one sentence.")];
    let response = provider.chat(&messages, None).await?;

    println!("{}", response.content);
    Ok(())
}

Environment:

export OPENAI_API_KEY=sk-...

Provider Matrix

Image Generation Providers

Rust-only image generation support is exposed through ImageGenProvider and ImageGenFactory:

Common Setup

GitHub Copilot direct mode

Use vscode-copilot/auto unless you have a strong reason to pin a specific model.

Why this is now the default:

• GitHub's live Auto routing knows which chat-capable model family is actually available for the current account and session.
• Some Copilot catalog entries are responses-only or temporarily throttled; Auto avoids hard-coding a brittle premium path.
• Reusing the real VS Code auth cache keeps parity with the official extension instead of depending on stale local token copies.

Legacy proxy setups still work through VSCODE_COPILOT_PROXY_URL, but no proxy is required for the normal path anymore.

Image generation environment:

Factory Usage

ProviderFactory is the fastest way to wire environments or provider/model routing:

use edgequake_llm::{ProviderFactory, ProviderType};

let (llm, embedding) = ProviderFactory::from_env()?;
println!("llm={} embedding={}", llm.name(), embedding.name());

let (vertex_llm, _) = ProviderFactory::create_with_model(
    ProviderType::VertexAI,
    Some("gemini-2.5-flash"),
)?;

let custom = ProviderFactory::create_llm_provider(
    "openai-compatible",
    "deepseek-chat",
)?;

For generic OpenAI-compatible routing, set:

export OPENAI_COMPATIBLE_BASE_URL=https://api.groq.com/openai/v1
export OPENAI_COMPATIBLE_API_KEY=...

For Rust image generation, use:

use edgequake_llm::{ImageGenFactory, ImageGenRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = ImageGenFactory::from_env()?;
    let response = provider
        .generate(&ImageGenRequest::new("Editorial product photo on a concrete desk"))
        .await?;

    println!("generated {} image(s)", response.images.len());
    Ok(())
}

Header Propagation

All five major providers support caller-supplied HTTP headers for B2B/multi-tenant deployments (closes edgequake#132):

Use with_extra_headers() to inject custom headers — trace IDs, tenant identifiers, HMAC tokens, or traceparent — into every outgoing LLM API call:

use edgequake_llm::{AnthropicProvider, GeminiProvider, MistralProvider};

// Anthropic
let provider = AnthropicProvider::from_env()?
    .with_extra_headers([
        ("x-request-id".to_string(), "req-abc-123".to_string()),
        ("x-tenant-id".to_string(), "tenant-42".to_string()),
    ]);

// Gemini (Google AI or VertexAI)
let provider = GeminiProvider::from_env()?
    .with_extra_headers([
        ("x-correlation-id".to_string(), "corr-xyz".to_string()),
    ]);

// Mistral
let provider = MistralProvider::from_env()?
    .with_extra_headers([
        ("traceparent".to_string(), "00-abc123-def456-01".to_string()),
    ]);

Reserved headers (authorization, x-api-key, anthropic-version, content-type, content-length, host, user-agent) are silently dropped to prevent accidental credential overrides. All other headers pass through to every request made by that provider instance.

Python Package

edgequake-litellm is the Python package in this repo. It is a drop-in LiteLLM replacement backed by the Rust runtime:

import edgequake_litellm as litellm

resp = litellm.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Install:

pip install edgequake-litellm

See edgequake-litellm/README.md for provider routing, migration notes, wheel coverage, and release instructions. The Python package does not expose the Rust image-generation APIs yet.

Model Discovery

Programmatic model discovery across all providers — no hardcoded model IDs, no name-pattern heuristics:

use edgequake_llm::{ModelDiscoveryService, CapabilityFilter};

let service = ModelDiscoveryService::new();

// Find all reasoning models with 100K+ context
let filter = CapabilityFilter {
    requires_thinking: Some(true),
    min_context_length: Some(100_000),
    ..Default::default()
};
let models = service.find_models(&filter).await?;

Python:

import edgequake_litellm as litellm

# Find vision-capable models
models = litellm.discovery.find_models(requires_vision=True, requires_tools=True)
for m in models:
    print(f"  {m.provider}/{m.id}: {m.context_length} tokens")

Features:

• 11 providers with dynamic, hybrid, or static discovery strategies
• Per-provider caching with configurable TTL and stale-on-error fallback
• Capability filtering — vision, tools, thinking, context length, cost, deprecation
• Zero heuristics — capabilities from API responses or cited documentation only
• Static registry — 30+ models with verified data for offline/fallback use

See docs/discovery.md for full documentation.

Development

Local validation:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --locked
cargo doc --workspace --no-deps --all-features

Python package validation:

cd edgequake-litellm
pip install . -v
pytest -q -k "not e2e"

Documentation

Specifications

Design specifications are tracked in specs/001-edgequake-llm/:

Release

Release guides:

• docs/releasing.md: release checklist, tags, registry setup
• docs/release-cycle.md: end-to-end CI/CD flow
• CHANGELOG.md: release notes for the Rust crate
• edgequake-litellm/CHANGELOG.md: release notes for the Python package

Tag conventions:

• Rust crate: vX.Y.Z → crates.io
• Python package: py-vX.Y.Z → PyPI

Both publish workflows validate versions before publishing and attach release artifacts to GitHub Releases.

License

Apache-2.0. See LICENSE-APACHE.