cortiq-gateway 0.2.2

Universal LLM gateway with intelligent routing and an embedded multilingual admin console
cortiq-gateway-0.2.2 is not a library.

Cortiq Gateway

English · Русский

Crates.io CI License: Apache 2.0 Rust

A universal LLM gateway with intelligent routing. One OpenAI-compatible endpoint → automatic model selection from your pool (cheap local + expensive hosted) based on the task type and complexity decided by allaigate / cortiq-router.

In your agent/SDK you only change base_url — and you get "smart" routing across models. No model-selection logic on the client side.

┌─────────────┐   OpenAI / Anthropic / MCP    ┌──────────────────┐
│   Agent /   │ ─────────────────────────────▶│  Cortiq Gateway  │
│  developer  │   (standard protocol)          │  (this project)  │
└─────────────┘ ◀─────────────────────────────└──────────────────┘
                       response + metadata        │          │
                                                  │ /v1/route│ call LLM
                                                  ▼          ▼
                                         ┌──────────────┐  ┌─────────────────────┐
                                         │ cortiq-router│  │  Model pool         │
                                         │ (task type,  │  │  • local llama.cpp  │
                                         │  complexity) │  │  • ollama / vLLM    │
                                         └──────────────┘  │  • OpenAI / Claude  │
                                                           └─────────────────────┘

✨ Highlights

  • Fast. Rust/Tokio: sub-millisecond p99 gateway overhead — ~10× the throughput of Portkey and ~47× of LiteLLM in a like-for-like proxy benchmark (details).
  • Drop-in OpenAI API + streaming. Point any OpenAI client at the gateway and send model: "cortiq-auto"; stream: true is fully supported (SSE).
  • Multi-protocol. OpenAI Chat & Embeddings, Anthropic Messages (in/out, streaming), and an MCP server (POST /mcp) exposing routing as tools for agent orchestrators.
  • Intelligent routing. complexity.tier → ordered model pool (low → local, high → cloud), with fallback and graceful degradation when the router is down.
  • Semantic cache. Optional embedding-based cache returns a stored answer for prompts semantically near a previous one — skipping the model call (hit-rate & savings on the dashboard).
  • Built-in multilingual admin console. Manage models, routing, protocols, keys and secrets from a web UI — no hand-editing TOML, no restart (changes are hot-reloaded). Available in 7 languages (en, ru, de, fr, es, zh, tr) with light/dark themes.
  • Live analytics. Per-request stats (tokens, cost, latency, success rate, failovers), time-series charts, breakdowns by model/tier/task, and a Prometheus GET /metrics endpoint.
  • Playground. Send a prompt through the live pipeline and inspect the routing decision.
  • Single self-contained binary. The SPA is embedded into the Rust binary — nothing extra to deploy.
  • Secrets stay in the gateway. Agents hold only a virtual gateway key; provider keys live in the gateway and never leak to clients.

📦 Install

# Docker (GitHub Container Registry)
docker run -p 9000:9000 -e CORTIQ_ADMIN_TOKEN=secret \
  -v "$PWD/config:/app/config" ghcr.io/infosave2007/cortiq-gateway:latest
# → admin console at http://localhost:9000/admin

# from crates.io
cargo install cortiq-gateway

# or build from source
git clone https://github.com/infosave2007/cortiq-gateway
cd cortiq-gateway
cargo build --release   # ./target/release/cortiq-gateway

The image ships a default config and runs out of the box; mount /app/config (with a gateway.toml) to use your own. Tags: :latest, :X.Y.Z, :edge (linux/amd64 + arm64).

🖥️ Admin console

Dashboard

The gateway ships with an embedded web console at /admin:

Models Routing
Models Routing
add/edit/probe models, manage provider keys visual tier editor (ordered, reorderable)
Playground
Playground
test the live pipeline and see the routing decision 
cargo run --release -- --config config/gateway.toml --admin-token <YOUR_TOKEN>
# open  http://localhost:9000/admin?token=<YOUR_TOKEN>

If --admin-token / [admin].token_env are not set, a token is generated at startup and printed to the log. All /admin/api/* endpoints require a Bearer admin token; secret values are never returned by the API (only their presence: store / env / missing).

🚀 Quick start

# 1. start a router (the hosted allaigate router, or a local cortiq-router)
#    it listens e.g. on http://localhost:8080 (or https://api.allaigate.com)

# 2. describe your model pool
cp config/gateway.example.toml config/gateway.toml
$EDITOR config/gateway.toml

# 3. run the gateway
cortiq-gateway --config config/gateway.toml
# Gateway listens on 0.0.0.0:9000 and serves the admin console at /admin

Now any OpenAI client works through the gateway:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:9000/v1", api_key="sk-gw-...")

resp = client.chat.completions.create(
    model="cortiq-auto",                       # ← magic model = "choose for me"
    messages=[{"role": "user", "content": "Solve x^2 - 5x + 6 = 0"}],
)
print(resp.choices[0].message.content)
# response headers show what the gateway picked:
#   X-Cortiq-Task-Label: math
#   X-Cortiq-Complexity-Tier: low
#   X-Cortiq-Selected-Model: local-qwen

model: "cortiq-auto" enables routing. Any real model name from the config ("gpt-4o-mini", "local-qwen") is a direct passthrough, without routing.

Using the hosted allaigate router? Set url = "https://138.226.222.209", verify_tls = false, taxonomy_id = "data-assistant", and a cortiq_… key in CORTIQ_ROUTER_KEY. On hard prompts the router escalates to an oracle (~10 s), so set timeout_ms = 12000+ — otherwise the gateway gracefully degrades to the default model.

🧭 The router (allaigate)

cortiq-auto routing is powered by a semantic router — the classifier that reads a prompt and returns its task type + complexity. Use the hosted allaigate router (~94% accuracy on close task types, patent-pending, privacy-first) or self-host cortiq-router. Pinned models need no router.

🎉 Launch promo: grab a $1 Starter key with code LAUNCH1 at api.allaigate.com (limited activations).

Setup, pricing, API contract and how to verify accuracy: docs/ROUTER.md.

Request flow

  1. Client → POST /v1/chat/completions (or Anthropic/MCP) with model: "cortiq-auto".
  2. The gateway extracts the text to route on (strategy is configurable, see docs/ROUTING.md).
  3. Gateway → cortiq-router /v1/route → gets task_label + complexity.tier.
  4. Gateway selects a model from the pool via the routing table (with a fallback order).
  5. Gateway → the selected model's provider (translating the protocol if needed), returns the response.
  6. Gateway attaches routing metadata in headers/usage and records cost & statistics.

Details — docs/ARCHITECTURE.md.

Supported protocols (inbound)

Each adapter is toggled in the config — take only what you need.

Protocol Endpoint Status
OpenAI Chat Completions POST /v1/chat/completions ✅ implemented (+ streaming)
OpenAI Embeddings POST /v1/embeddings ✅ implemented
Anthropic Messages POST /v1/messages ✅ implemented (+ streaming)
MCP (Model Context Protocol) POST /mcp (JSON-RPC) ✅ implemented
OpenAI Completions (legacy) POST /v1/completions ✅ implemented (+ streaming)
OpenAI Models GET /v1/models ✅ implemented
Native passthrough POST /route ✅ implemented

Supported providers (outbound)

Provider Covers
openai (OpenAI-compatible) OpenAI, OpenRouter, Together, Groq, vLLM, llama.cpp server, LM Studio, Ollama (/v1)
anthropic Claude Messages API (+ streaming)
ollama native Ollama API
http arbitrary HTTP endpoint (custom adapter)

Most local servers already expose an OpenAI-compatible API — so the openai adapter makes the gateway universal with almost no effort.

Configuration

listen = "0.0.0.0:9000"

[router]
url = "http://localhost:8080"
api_key_env = "CORTIQ_ROUTER_KEY"

[[models]]
id        = "local-qwen"
provider  = "openai"
base_url  = "http://localhost:8000/v1"
model     = "qwen2.5-7b-instruct"
cost_tier = "cheap"

[[models]]
id          = "claude-opus"
provider    = "anthropic"
base_url    = "https://api.anthropic.com"
model       = "claude-opus-4"
api_key_env = "ANTHROPIC_API_KEY"
cost_tier   = "expensive"

[routing]
low     = ["local-qwen"]
high    = ["claude-opus", "local-qwen"]
default = "local-qwen"

Full example — config/gateway.example.toml. Routing & cost-aware selection — docs/ROUTING.md. Everything in the config is also editable from the admin console at runtime.

Cross-standards (like grown-up APIs)

  • Auth: Authorization: Bearer sk-gw-... (virtual gateway keys).
  • Errors: OpenAI-compatible {"error": {...}} envelope — parsed natively by SDKs.
  • Routing metadata: X-Cortiq-* headers on every response.
  • Cost accounting: token usage + USD cost and the selected model.
  • Observability: GET /metrics (Prometheus), GET /healthz, GET /readyz.

Details — docs/PROTOCOLS.md.

⚡ Performance & accuracy

Latency — gateway overhead proxying the same instant backend, ab -k -r -c 20 -n 5000:

Gateway req/s p50 p99
Cortiq Gateway (Rust) ~57,300 0 ms 1 ms
Portkey (Node) ~5,800 3 ms 9 ms
LiteLLM (Python, 4 workers) ~1,200 9 ms 59 ms

Accuracy — task-type routing on natural-language prompts (7 task types). LiteLLM/Portkey have no semantic task router; a keyword heuristic is the DIY stand-in:

Classifier Accuracy
allaigate semantic router 100% (37/37)
keyword heuristic (no classifier) 32% (12/37)

Methodology, caveats and a reproducible harness: BENCHMARKS.md (bash bench/run.sh, python3 bench/accuracy.py). Numbers vary; the gaps are the point.

Build & test

cargo build --release      # single self-contained binary (SPA embedded)
cargo test                 # config round-trip, routing validation

✅ Tested

Area How it's verified Status
Config load/save round-trip, routing validation cargo test (unit) ✅
Build · format · lint CI: cargo build / fmt --check / clippy -D warnings on Linux · macOS · Windows ✅
OpenAI Chat + SSE streaming integration (mock upstream) ✅
Anthropic Messages — in/out, streaming + non-streaming integration (mock) ✅
Embeddings (POST /v1/embeddings) integration (mock) ✅
Semantic cache — hit / miss / savings integration (mock) ✅
MCP — initialize / tools/list / tools/call integration (mock) ✅
Models · Completions · native POST /route integration (mock) ✅
Routing — auto / pinned / failover / degrade integration (mock and live allaigate router) ✅
Hot config reload (no restart) integration ✅
Latency benchmark vs LiteLLM / Portkey bench/run.sh (Apache Bench) — see BENCHMARKS.md ✅
Task-type routing accuracy bench/accuracy.py — live router 100% vs keyword heuristic 32% ✅

Status

Implemented: OpenAI Chat Completions with SSE streaming, Anthropic provider + inbound /v1/messages (streaming), embeddings, a semantic cache, MCP server, routing with fallback & graceful degradation, cost/token accounting, the embedded multilingual admin console with hot config reload, statistics and Prometheus GET /metrics. Planned: per-account routing tables, router feedback loop. See the roadmap in docs/ARCHITECTURE.md. Contributions welcome — see CONTRIBUTING.md.

License

Apache-2.0 — see LICENSE.