Crate edgequake_llm 

EdgeQuake LLM - LLM and Embedding Provider Abstraction

Implements

• FEAT0017: Multi-Provider LLM Support
• FEAT0018: Embedding Provider Abstraction
• FEAT0019: LLM Response Caching
• FEAT0020: API Rate Limiting
• FEAT0005: Embedding Generation (via providers)

Enforces

• BR0301: LLM API rate limits (configurable per provider)
• BR0302: Document size limits (context window awareness)
• BR0303: Cost tracking per request
• BR0010: Embedding dimension validated (1536 default)

This crate provides traits and implementations for:

• Text completion (LLM providers)
• Text embedding (embedding providers)
• Token counting and management
• Rate limiting for API calls
• Response caching for cost reduction

Providers

Provider	FEAT0017	Chat	Embeddings	Notes
OpenAI	✓	✓	✓	Primary production provider
Azure OpenAI	✓	✓	✓	Enterprise deployments
Ollama	✓	✓	✓	Local/on-prem models
LM Studio	✓	✓	✓	Local OpenAI-compatible API
Gemini	✓	✓	✓	Google AI
Mock	✓	✓	✓	Testing (no API calls)

Architecture

The crate uses trait-based abstraction to support multiple LLM backends:

• OpenAI (GPT-4, GPT-3.5)
• OpenAI-compatible APIs (Ollama, LM Studio, etc.)
• Anthropic (Claude 3.5, Claude 3)
• Future: Mistral, local models

Example

use edgequake_llm::{LLMProvider, OpenAIProvider};

let provider = OpenAIProvider::new("your-api-key");
let response = provider.complete("Hello, world!").await?;

See Also

• crate::traits for provider trait definitions
• crate::providers for concrete implementations
• crate::cache for response caching

Re-exports

pub use api_format::ApiFormat;

pub use api_format::ENV_API_FORMAT;

pub use responses_map::build_responses_request;

pub use responses_map::extract_output_text;

pub use responses_map::responses_url;

pub use application_context::ApplicationContext;

pub use application_context::ApplicationContextBuilder;

pub use application_context::AttributionPolicy;

pub use application_context::AttributionProviderKind;

pub use application_context::AttributionWarning;

pub use omit_env::apply_omit_reasoning_effort;

pub use omit_env::env_truthy;

pub use omit_env::omit_reasoning_effort_from_env;

pub use omit_env::omit_temperature_from_env;

pub use omit_env::reasoning_effort_for_wire;

pub use omit_env::temperature_for_wire;

pub use omit_env::ENV_OMIT_REASONING_EFFORT;

pub use omit_env::ENV_OMIT_TEMPERATURE;

pub use temperature::effective_temperature_for_model;

pub use temperature::model_requires_default_temperature;

pub use temperature::resolve_effective_temperature;

pub use cache::CacheConfig;

pub use cache::CacheStats;

pub use cache::CachedProvider;

pub use cache::LLMCache;

pub use cache_prompt::apply_cache_control;

pub use cache_prompt::parse_cache_stats;

pub use cache_prompt::prepare_chat;

pub use cache_prompt::prompt_cache_key;

pub use cache_prompt::provider_prompt_cache_enabled;

pub use cache_prompt::CachePromptConfig;

pub use cache_prompt::CacheStats as PromptCacheStats;

pub use cache_prompt::OpenAiChatApi;

pub use cache_prompt::PromptCachePolicy;

pub use cost_tracker::format_cost;

pub use cost_tracker::format_tokens;

pub use cost_tracker::CostEntry;

pub use cost_tracker::CostSummary;

pub use cost_tracker::ModelPricing;

pub use cost_tracker::SessionCostTracker;

pub use discovery::find_static_models;

pub use discovery::search_models;

pub use discovery::search_static_models;

pub use discovery::static_lookup_by_name;

pub use discovery::CapabilityFilter;

pub use discovery::DiscoveredModel;

pub use discovery::DiscoveryError;

pub use discovery::DiscoverySource;

pub use discovery::DiscoveryStrategy;

pub use discovery::ModelCapability;

pub use discovery::ModelDiscoveryProvider;

pub use discovery::ModelDiscoveryService;

pub use discovery::ModelMatchKind;

pub use discovery::ModelSearchMatch;

pub use discovery::ModelSearchQuery;

pub use error::LlmError;

pub use error::Result;

pub use error::RetryStrategy;

pub use factory::ProviderFactory;

pub use factory::ProviderType;

pub use imagegen::AspectRatio;

pub use imagegen::AzureImageGen;

pub use imagegen::FalImageGen;

pub use imagegen::GeminiImageGenProvider;

pub use imagegen::GeneratedImage;

pub use imagegen::ImageFormat;

pub use imagegen::ImageGenData;

pub use imagegen::ImageGenError;

pub use imagegen::ImageGenFactory;

pub use imagegen::ImageGenOptions;

pub use imagegen::ImageGenProvider;

pub use imagegen::ImageGenRequest;

pub use imagegen::ImageGenResponse;

pub use imagegen::ImageResolution;

pub use imagegen::MockImageGenProvider;

pub use imagegen::NvidiaImageGen;

pub use imagegen::OpenAIImageGen;

pub use imagegen::SafetyLevel;

pub use imagegen::ThinkingLevel;

pub use imagegen::VertexAIImageGen;

pub use imagegen::XAIImageGen;

pub use inference_metrics::InferenceMetrics;

pub use middleware::LLMMiddleware;

pub use middleware::LLMMiddlewareStack;

pub use middleware::LLMRequest;

pub use middleware::LogLevel;

pub use middleware::LoggingLLMMiddleware;

pub use middleware::MetricsLLMMiddleware;

pub use middleware::MetricsSummary;

pub use model_config::DefaultsConfig;

pub use model_config::ModelCapabilities;

pub use model_config::ModelCard;

pub use model_config::ModelConfigError;

pub use model_config::ModelCost;

pub use model_config::ModelType;

pub use model_config::ModelsConfig;

pub use model_config::ProviderConfig;

pub use model_config::ProviderType as ConfigProviderType;

pub use provider_catalog::AttributionSupport;

pub use provider_catalog::ProviderCatalog;

pub use provider_catalog::ProviderDescriptor;

pub use provider_catalog::ProviderFeatures;

pub use providers::azure_openai::AzureCredential;

pub use providers::azure_openai::AzureOpenAIProvider;

pub use providers::gemini::GeminiProvider;

pub use providers::jina::JinaProvider;

pub use providers::llamacpp::api_key_from_env as llamacpp_api_key_from_env;

pub use providers::llamacpp::builder as llamacpp_builder;

pub use providers::llamacpp::from_env as llamacpp_from_env;

pub use providers::llamacpp::from_env_with_model as llamacpp_from_env_with_model;

pub use providers::llamacpp::host_from_env as llamacpp_host_from_env;

pub use providers::llamacpp::normalize_llamacpp_host;

pub use providers::llamacpp::resolve_llamacpp_runtime_config;

pub use providers::llamacpp::LlamaCppProvider;

pub use providers::llamacpp::DEFAULT_LLAMACPP_HOST;

pub use providers::llamacpp::DEFAULT_LLAMACPP_PORT;

pub use providers::llamacpp::LLAMACPP_IDENTITY;

pub use providers::lmstudio::LMStudioProvider;

pub use providers::lmstudio::LmStudioModelMetadata;

pub use providers::local_openai_common::normalize_local_openai_host;

pub use providers::local_openai_common::parse_openai_models_list;

pub use providers::local_openai_common::LocalOpenAiIdentity;

pub use providers::local_openai_common::LocalOpenAiProvider;

pub use providers::local_openai_common::LocalOpenAiRuntimeConfig;

pub use providers::mlx_lm::api_key_from_env as mlx_lm_api_key_from_env;

pub use providers::mlx_lm::builder as mlx_lm_builder;

pub use providers::mlx_lm::from_env as mlx_lm_from_env;

pub use providers::mlx_lm::from_env_with_model as mlx_lm_from_env_with_model;

pub use providers::mlx_lm::host_from_env as mlx_lm_host_from_env;

pub use providers::mlx_lm::normalize_mlx_lm_host;

pub use providers::mlx_lm::resolve_mlx_lm_runtime_config;

pub use providers::mlx_lm::MlxLmProvider;

pub use providers::mlx_lm::DEFAULT_MLX_LM_HOST;

pub use providers::mlx_lm::DEFAULT_MLX_LM_PORT;

pub use providers::mlx_lm::MLX_LM_IDENTITY;

pub use providers::mock::MockProvider;

pub use providers::mtplx::list_cached_model_ids;

pub use providers::mtplx::load_mtplx_settings_file;

pub use providers::mtplx::normalize_mtplx_host;

pub use providers::mtplx::resolve_mtplx_runtime_config;

pub use providers::mtplx::MtplxProvider;

pub use providers::mtplx::MtplxRuntimeConfig;

pub use providers::mtplx::DEFAULT_MTPLX_HOST;

pub use providers::mtplx::DEFAULT_MTPLX_PORT;

pub use providers::ollama::OllamaModelDetails;

pub use providers::ollama::OllamaModelInfo;

pub use providers::ollama::OllamaModelsResponse;

pub use providers::ollama::OllamaProvider;

pub use providers::omlx::api_key_from_env;

pub use providers::omlx::host_from_env;

pub use providers::omlx::load_omlx_settings_file;

pub use providers::omlx::normalize_omlx_host;

pub use providers::omlx::resolve_omlx_runtime_config;

pub use providers::omlx::OmlxProvider;

pub use providers::omlx::OmlxRuntimeConfig;

pub use providers::omlx::DEFAULT_OMLX_HOST;

pub use providers::omlx::DEFAULT_OMLX_PORT;

pub use providers::openai::OpenAIProvider;

pub use providers::vllm_mlx::api_key_from_env as vllm_mlx_api_key_from_env;

pub use providers::vllm_mlx::builder as vllm_mlx_builder;

pub use providers::vllm_mlx::from_env as vllm_mlx_from_env;

pub use providers::vllm_mlx::from_env_with_model as vllm_mlx_from_env_with_model;

pub use providers::vllm_mlx::host_from_env as vllm_mlx_host_from_env;

pub use providers::vllm_mlx::normalize_vllm_mlx_host;

pub use providers::vllm_mlx::resolve_vllm_mlx_runtime_config;

pub use providers::vllm_mlx::VllmMlxProvider;

pub use providers::vllm_mlx::DEFAULT_VLLM_MLX_HOST;

pub use providers::vllm_mlx::DEFAULT_VLLM_MLX_PORT;

pub use providers::vllm_mlx::VLLM_MLX_IDENTITY;

pub use providers::mistral::MistralProvider;

pub use providers::anthropic::AnthropicProvider;

pub use providers::openrouter::ModelArchitecture as OpenRouterModelArchitecture;

pub use providers::openrouter::ModelInfo as OpenRouterModelInfo;

pub use providers::openrouter::ModelPricing as OpenRouterModelPricing;

pub use providers::openrouter::ModelsResponse as OpenRouterModelsResponse;

pub use providers::openrouter::OpenRouterProvider;

pub use providers::openai_compatible::OpenAICompatibleProvider;

pub use providers::vscode::Model as CopilotModel;

pub use providers::vscode::ModelsResponse as CopilotModelsResponse;

pub use providers::vscode::VsCodeCopilotProvider;

pub use providers::xai::XAIProvider;

pub use providers::trace_llm_arc;

pub use providers::TracingProvider;

pub use stream_tool_calls::StreamToolCallAccumulator;

pub use stream_tools::finalize_streamed_tool_calls_with_repair;

pub use stream_tools::FinalizeStreamToolCallsOptions;

pub use stream_tools::PartialStreamToolCall;

pub use trace_context::with_trace_context;

pub use providers::nvidia::NvidiaModelInfo;

pub use providers::nvidia::NvidiaModelsResponse;

pub use providers::nvidia::NvidiaProvider;

pub use providers::cohere::CohereProvider;

pub use providers::ollama_capabilities::capabilities_include_thinking;

pub use providers::ollama_capabilities::is_thinking_model_legacy;

pub use providers::ollama_capabilities::map_think;

pub use providers::ollama_capabilities::ThinkCapabilityMode;

pub use providers::ollama_capabilities::ThinkingSupport;

pub use rate_limiter::RateLimitedProvider;

pub use rate_limiter::RateLimiter;

pub use rate_limiter::RateLimiterConfig;

pub use reasoning_capabilities::capabilities as reasoning_capabilities_for;

pub use reasoning_capabilities::clamp_options_reasoning_effort;

pub use reasoning_capabilities::clamp_reasoning_effort;

pub use reasoning_capabilities::lowest_for_structured_output;

pub use reasoning_capabilities::ollama_thinking_effort_vocab;

pub use reasoning_capabilities::parse_openai_reasoning_effort;

pub use reasoning_capabilities::ReasoningCapabilities;

pub use reasoning_capabilities::EFFORT_SCALE;

pub use registry::ProviderRegistry;

pub use reranker::create_bm25_reranker;

pub use reranker::create_cross_encoder_reranker;

pub use reranker::create_production_reranker;

pub use reranker::try_http_cross_encoder_reranker;

pub use reranker::BM25Reranker;

pub use reranker::BiEncoderReranker;

pub use reranker::HttpReranker;

pub use reranker::HybridReranker;

pub use reranker::MockReranker;

pub use reranker::RRFReranker;

pub use reranker::RerankConfig;

pub use reranker::RerankResult;

pub use reranker::Reranker;

pub use reranker::ScoreAggregation;

pub use reranker::TermOverlapReranker;

pub use retry::RetryExecutor;

pub use tokenizer::Tokenizer;

pub use traits::CacheControl;

pub use traits::ChatMessage;

pub use traits::ChatRole;

pub use traits::CompletionOptions;

pub use traits::EmbeddingProvider;

pub use traits::FunctionCall;

pub use traits::FunctionDefinition;

pub use traits::ImageData;

pub use traits::LLMProvider;

pub use traits::LLMResponse;

pub use traits::ToolCall;

pub use traits::ToolChoice;

pub use traits::ToolDefinition;

pub use traits::ToolResult;

Modules

api_format

SPEC-131 — upstream LLM HTTP API format selection.

application_context

Application identity and attribution context for LLM provider calls.

cache

LLM response caching for reducing API costs and latency.

cache_prompt

Provider prompt / KV-prefix cache policy (August 2026 SOTA).

cost_tracker

Session Cost Tracker

discovery

Model Discovery System

error

LLM error types with retry strategies.

factory

LLM provider factory for environment-based selection.

http

HTTP utilities for provider request construction.

imagegen

Image generation provider abstraction.

inference_metrics

Inference Metrics for Real-Time Streaming Display

middleware

LLM Provider Middleware System (OODA-125)

model_config

Model Configuration Module

omit_env

SPEC-131 — fleet env knobs for omitting illegal LLM wire fields.

provider_catalog

Unified provider catalog — single source of truth for provider metadata.

providers

LLM provider implementations.

rate_limiter

Async-aware rate limiting for LLM API calls.

reasoning_capabilities

Model reasoning-effort capability registry (SPEC-109).

registry

LLM Provider Registry - Pluggable Provider Management

reranker

Reranking functionality for improved retrieval quality.

responses_http

SPEC-131 — shared HTTP client for POST …/responses.

responses_map

SPEC-131 — Chat Completions ↔ Responses API mapper (SRP / DRY).

retry

Retry executor for LLM operations with exponential backoff.

stream_tool_calls

Incremental streamed tool-call accumulator (EdgeCrab proxy / clients).

stream_tools

Streaming tool-call assembly helpers used by EdgeCrab’s provider call path.

temperature

SPEC-131 — effective temperature resolution (product + wire policy).

tokenizer

Token counting utilities.

trace_context

Lightweight task-local / span context for LLM calls (EdgeCrab integration).

traits

LLM provider traits for text completion and embedding.