zeph-llm

LLM provider abstraction with Ollama, Claude, OpenAI, Gemini, Gonka, and Candle backends.

Overview

Defines the LlmProvider trait and ships concrete backends for Ollama, Claude, OpenAI, Google Gemini, Gonka (signed native transport), and OpenAI-compatible endpoints. Includes a RouterProvider that selects among backends via four strategies (EMA, Thompson sampling, cascade, LinUCB bandit), a typed structured-extraction layer (Extractor / chat_typed), an optional Candle backend for local inference, and a disk-backed cache for remote model listings.

Key modules

Re-exports: LlmProvider, LlmProviderDyn, LlmError, Extractor, ChatStream, StreamChunk, CompatibleConfig, OpenAiConfig, MaskedProvider, SpeechToText

Router strategies

The router supports four strategies — EMA and Thompson sampling reorder the fallback chain (covered below); cascade and LinUCB bandit have their own sections further down. Set the strategy in [llm.router]:

EMA (default)

Exponential moving average latency tracking. After each response, EmaTracker records provider latency and periodically reorders the chain so the fastest reliable provider is tried first.

[llm]
router_ema_enabled      = true
router_ema_alpha        = 0.1   # smoothing factor; lower = slower to adapt
router_reorder_interval = 60    # seconds between reordering

[llm.router]
strategy = "ema"

Thompson Sampling

Adaptive model selection using Beta distributions. Each provider maintains a Beta(alpha, beta) distribution initialized with a uniform prior (1, 1). On each request the router samples all distributions and picks the provider with the highest sample; after the response it updates alpha (success) or beta (failure). This naturally balances exploration of less-tested providers with exploitation of known-good ones.

State persists across restarts to ~/.zeph/router_thompson_state.json (configurable). Stale entries for removed providers are pruned automatically on startup.

[llm.router]
chain    = ["claude", "openai", "ollama"]
strategy = "thompson"
# thompson_state_path = "~/.zeph/router_thompson_state.json"  # optional

CLI commands for inspecting and managing Thompson state:

zeph router stats   # show per-provider alpha/beta and success rate
zeph router reset   # reset all distributions to uniform prior

TUI: /router stats displays the same information in the dashboard.

[!NOTE] Thompson Sampling is most useful when you have multiple providers with varying reliability and want the router to automatically converge on the best one while still occasionally probing alternatives.

Cascade routing

The cascade strategy tries providers in order and escalates to the next when a quality threshold is not met. Configure via [llm.router.cascade]:

[llm.router]
strategy = "cascade"
chain = ["ollama", "claude", "openai"]

[llm.router.cascade]
quality_threshold = 0.7
max_escalations = 2
cost_tiers = ["ollama", "claude", "openai"]  # optional: explicit cheapest-first ordering

cost_tiers reorders providers once at construction time (zero per-request cost). Providers absent from the list are appended after listed ones in original chain order. Unknown names are silently ignored.

Complexity triage routing

The triage strategy classifies each request into a complexity tier before inference and routes it to the provider pool configured for that tier. This avoids sending simple queries to expensive models and reserves high-capability models for genuinely complex tasks.

[llm.router]
strategy = "triage"

[llm.complexity_routing]
simple_providers  = ["ollama"]
medium_providers  = ["ollama", "openai"]
complex_providers = ["claude", "openai"]
expert_providers  = ["claude"]

Tier assignment uses a lightweight classifier (TriageClassifier) that runs before the primary LLM call. The classifier dispatches to LlmRoutingStrategy::Triage on the RouterProvider.

[!TIP] Use ClassifierMode::Judge to route classification through a separate LLM call when heuristic scoring is insufficient for your workload.

PILOT LinUCB bandit routing

The bandit strategy applies a contextual LinUCB bandit to provider selection. On each request, context features (query complexity score, recent per-provider latency, time-of-day bucket) are assembled into a feature vector; the bandit computes an upper confidence bound per provider and selects the highest. After each response, the reward signal (success × inverse latency) updates the ridge regression weights.

State is persisted to ~/.zeph/router_bandit_state.json (configurable) and restored on restart.

[llm.router]
strategy = "bandit"
chain    = ["ollama", "claude", "openai"]

[llm.router.bandit]
alpha            = 1.0     # exploration parameter; higher = more exploration
state_path       = "~/.zeph/router_bandit_state.json"
feature_dim      = 8       # dimensionality of the context feature vector

[!NOTE] PILOT (Provider Intelligent Linucb Online Tracking) is most effective when providers have meaningfully different latency/quality profiles and the workload has varied query complexity. For uniform workloads, Thompson Sampling may converge faster.

[!TIP] Inspect learned weights and UCB scores with zeph router stats (same command as Thompson Sampling) or /router stats in the TUI.

Gonka native provider

GonkaProvider connects to Gonka inference nodes using a signed transport. Requests are signed per-call via RequestSigner using an HMAC key stored in the age vault under ZEPH_GONKA_API_KEY. EndpointPool distributes load across nodes by weight and falls back automatically when a node is unreachable.

Supported operations: chat, chat_stream, embed, chat_with_tools, and chat_typed (structured output via JSON Schema).

[[llm.providers]]
name = "gonka"
type = "gonka"
model = "qwen3-235b"
default = true

[[llm.gonka_nodes]]
url    = "https://node.example.gonka.ai"
weight = 1

Store the key in the vault:

zeph vault set ZEPH_GONKA_API_KEY <your-key>

Configure via the --init wizard by selecting the GonkaGate / Gonka Native option.

[!NOTE] The GonkaGate path (type = "compatible") is still available for access via the OpenAI-compatible gateway. Use type = "gonka" for the native signed-transport path with full multi-node pool support.

SLM provider recommendations

For cost-sensitive or resource-constrained deployments, the following Small Language Models are verified to work well with Zeph:

Pair SLMs with a cloud provider for complex/expert tasks using triage routing:

[llm.router]
strategy = "triage"

[llm.complexity_routing]
simple_providers  = ["ollama"]   # qwen3:8b handles simple queries
medium_providers  = ["ollama"]
complex_providers = ["claude"]
expert_providers  = ["claude"]

Claude extended thinking

ClaudeProvider supports two thinking modes via ThinkingConfig:

[llm.claude]
thinking = { mode = "extended", budget_tokens = 16000 }

CLI: --thinking extended:16000 or --thinking adaptive. When thinking is enabled and max_tokens is below 16000, it is raised automatically. Thinking deltas are parsed from the SSE stream and suppressed from the user-facing output; MessagePart::ThinkingBlock variants preserve thinking blocks verbatim across tool-use turns.

The thinking budget and OpenAI/Compatible/Gemini reasoning_effort can also be changed mid-session via AnyProvider::set_thinking_budget() / apply_reasoning_effort() — surfaced as the /think-tokens [N|Nk|NM|off] and /reasoning-effort [low|medium|high] slash commands (and a matching --reasoning-effort CLI flag / --init wizard prompt). Overrides are session-only: never persisted across restarts or /provider switches, and unsupported providers return an explicit "not supported" message instead of a silent no-op.

Prompt cache TTL

ClaudeProvider supports a configurable prompt cache TTL via the CacheTtl enum:

[[llm.providers]]
type = "claude"
model = "claude-sonnet-5"
prompt_cache_ttl = "1h"   # "ephemeral" (default) or "1h"

[!NOTE] The 1-hour TTL costs approximately 2× more per cache write but dramatically reduces repeated-prefix costs for long sessions. Default "ephemeral" is byte-identical to the previous wire format — no rollout risk for existing deployments.

Gemini configuration

[llm]
provider = "gemini"

[llm.gemini]
model = "gemini-2.0-flash"   # or "gemini-2.5-pro" for extended thinking
max_tokens = 8192
# base_url = "https://generativelanguage.googleapis.com/v1beta"

Store the API key in the vault: zeph vault set ZEPH_GEMINI_API_KEY AIza...

[!NOTE] Gemini does not expose an embeddings endpoint. For semantic memory and skill matching, pair Gemini with an Ollama embedding model via [llm.orchestrator].

Features

All backends (Ollama, Claude, OpenAI, Gemini, compatible) and the OpenAI Whisper STT path are always compiled — no default features are required. schemars, chat_typed, and Extractor are always available. The optional features below add local inference and specialized backends.

Installation

cargo add zeph-llm

# With local inference via Candle
cargo add zeph-llm --features candle

# With the Gonka native backend
cargo add zeph-llm --features gonka

Documentation

Full documentation: https://bug-ops.github.io/zeph/

License

Licensed under either of MIT or Apache License, Version 2.0 at your option.