polyglotmesh

A fast, queue-based Rust LLM router. You register multiple OpenAI-compatible and Anthropic-compatible upstreams; the router exposes a single base URL for each, with priority-based load balancing, per-key limits, health checks, and live queue stats. Speaks every LLM dialect in the room — hence polyglot — and weaves them into a single mesh — hence mesh.

A fast, queue-based Rust LLM router. You register multiple OpenAI-compatible and Anthropic-compatible upstreams; the router exposes a single base URL for each, with priority-based load balancing, per-key limits, health checks, and live queue stats.

        ┌──────────┐    ┌───────────────────┐    ┌──────────┐
client ─┤ OpenAI   │───▶│   polyglotmesh   │───▶│ upstream │
        │ Anthropic│    │  (single baseurl, │    │   A      │
        └──────────┘    │   one self-issued │    │   B      │
                        │   API key)        │    │   C      │
                        └───────────────────┘    └──────────┘

Documentation map:

• docs/ARCHITECTURE.md — module layout, request lifecycle, queue internals, auth/limits pipeline, state machine.
• docs/FEATURES.md — every CLI subcommand, every HTTP endpoint, every config field, with examples.
• examples/config.sample.toml — the annotated reference config.
• scripts/install.sh — build + install the binary, copy the sample config into ~/.polyglotmesh/.

Quick start

# Build + install
./scripts/install.sh
# → drops ~/.local/bin/polyglotmesh AND ~/.polyglotmesh/config.sample.toml

# Initialize (generates a fresh API key)
polyglotmesh init --bind 0.0.0.0:8080
# → prints the OpenAI + Anthropic base URLs and your Bearer token

# Add 3-4 OpenAI upstreams (one CLI call each)
polyglotmesh upstream add --id openai-1 --kind openai \
  --base-url https://api.openai.com/v1 --api-key "$OPENAI_KEY" \
  --models gpt-4o-mini,gpt-4o --priority 30 --max-concurrency 50

polyglotmesh upstream add --id openai-2 --kind openai \
  --base-url https://api.openrouter.ai/v1 --api-key "$OR_KEY" \
  --models gpt-4o-mini --priority 10

polyglotmesh upstream add --id openai-3 --kind openai \
  --base-url http://gpu.local:8000/v1 --api-key EMPTY \
  --models gpt-4o-mini --priority 5 --tags local

# Add 1-4 Anthropic upstreams
polyglotmesh upstream add --id anthropic-1 --kind anthropic \
  --base-url https://api.anthropic.com --api-key "$ANTHROPIC_KEY" \
  --models claude-3-5-sonnet-20241022 --priority 30

# Generate an admin token
polyglotmesh key --role admin

# Run
polyglotmesh serve

The config file to edit

After init, the active config lives at:

$ polyglotmesh where
config: /home/you/.polyglotmesh/config.toml

That single file controls everything: server bind address, every upstream's base URL and key, every virtual key with its rpm_limit / tpm_limit / max_budget / models / expires, model aliases, queue tuning, health-check tuning. Run polyglotmesh show to print the active (merged) version with defaults filled in.

A fully-commented reference is at examples/config.sample.toml. It is also installed alongside the live config as config.sample.toml so you can diff and copy fields.

For the complete field list, see docs/FEATURES.md.

Calling the router

# OpenAI-compatible (chat completions)
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $AILR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

# OpenAI-compatible (streaming)
curl -N http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $AILR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"hi"}]}'

# Anthropic-compatible
curl http://localhost:8080/v1/messages \
  -H "Authorization: Bearer $AILR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":256,"messages":[{"role":"user","content":"hi"}]}'

# OpenAI-compatible (model listing)
curl -H "Authorization: Bearer $AILR_KEY" http://localhost:8080/v1/models

# Health (no auth)
curl http://localhost:8080/healthz

Endpoints at a glance

See docs/FEATURES.md for the complete table.

Operational behavior

• Queue — per-provider kind (openai, anthropic). Greedy in priority order; if all eligible upstreams are saturated, the request waits on a Notify (exponential backoff up to 200ms) up to queue.queue_wait_timeout_ms. See ARCHITECTURE.md.
• Per-key limits — rpm_limit, tpm_limit, max_parallel_requests, max_budget, soft_budget, expires, models allow-list, allowed_providers. Returning 429 / 402 / 401 as appropriate. See FEATURES.md.
• Per-upstream limits — max_concurrency, rate_limit_rpm, rate_limit_tpm, max_budget, timeout_ms.
• Health checks — background task probes every healthcheck_interval_ms; consecutive failures flip the upstream Unhealthy; the next success flips it back.
• Model aliases — three layers (model_list, model_aliases, per-upstream models), all evaluated at request time.

Scripts

• scripts/install.sh — build + install + copy the sample config.
• examples/bootstrap.sh — end-to-end installer: registers N upstreams from arrays, prints the final URLs and the active config path.
• scripts/smoke-test.sh — self-contained end-to-end test using mock upstreams.
• scripts/mock_slow.py — mock upstream for the smoke test.

License

MIT

Persistence & hot reload

Every router run opens a SQLite database at $POLYGLOTMESH_HOME/state.db (WAL mode, bundled via rusqlite). It persists per-key counters, rolling RPM/TPM windows, in-flight slot counts, and an audit trail of every successful request.

Edit config.toml to add an upstream or change a limit, then have the router pick it up without a restart:

curl -sS -X POST http://127.0.0.1:8080/v1/admin/reload \
  -H "Authorization: Bearer $AILR_ADMIN_TOKEN"
# -> {"status":"reloaded","upstreams":3,"keys":2}

On reload the router rebuilds the in-memory upstream registry, atomically swaps the auth store, and re-hydrates per-key counters from SQLite — so request volumes, token totals, and rolling windows survive the reload.

Pricing & cost

The router ships with a built-in price table (USD per token) for popular models: OpenAI gpt-4o*, gpt-3.5-turbo, o1*, o3-mini; Anthropic claude-3-5-*, claude-3-*. On every successful response, real cost_usd is computed and persisted to usage_events and key_usage.

Override per-upstream with model_info:

[[upstreams]]
id = "openai-prod"
kind = "openai"
base_url = "https://api.openai.com/v1"
api_key  = "sk-…"
models   = ["gpt-4o-mini"]

[upstreams.model_info]
"gpt-4o-mini" = { input_cost_per_token = 0.0000001, output_cost_per_token = 0.0000004 }

Auto-reload (no restart)

The router watches $POLYGLOTMESH_HOME/config.toml for changes every 2s and auto-applies them — rebuilds the upstream registry, atomically swaps the auth store, re-hydrates per-key counters from SQLite. No POST /v1/admin/reload needed; the manual endpoint still works for scripted deploys.

Usage analytics

curl -sS "http://127.0.0.1:8080/v1/admin/usage?group_by=model" \
  -H "Authorization: Bearer $AILR_ADMIN_TOKEN"
# -> { "group_by":"model", "totals":{...}, "buckets":[{ key:"gpt-4o-mini", requests:8, cost_usd:4.2e-05 }] }

curl -sS "http://127.0.0.1:8080/v1/admin/usage/recent?limit=10" \
  -H "Authorization: Bearer $AILR_ADMIN_TOKEN"

group_by may be alias, upstream, model, or all. Add since/until as Unix seconds to bound the window.

Retention policy

usage_events rows can grow without bound. Set a retention window in config.toml and the router prunes rows older than N days once per day:

[server]
usage_retention_days = 30   # 0 = keep forever

Or change it live:

curl -X POST http://127.0.0.1:8080/v1/admin/usage/retention \
  -H "Authorization: Bearer $AILR_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"days": 30}'

Budget reset

Keys with a budget_duration (e.g. "1h", "7d", "30d") have their total_spend_usd automatically reset to 0 when the window expires — even if the key has zero traffic. A background task fires every 60s and the reset is persisted to SQLite so it survives restarts.

Config auto-reload (inotify/FSEvents)

The router uses kernel-level file events (inotify on Linux, FSEvents on macOS) to detect config changes within ~150 ms — no polling. It falls back to a 2 s stat() poll automatically on platforms where inotify is unavailable (e.g. inside some sandboxes).

Per-upstream pricing overrides

curl -X POST "http://127.0.0.1:8080/v1/admin/upstreams/openai-prod/prices" \
  -H "Authorization: Bearer $AILR_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{
        "merge": true,
        "prices": {
          "gpt-4o-mini": { "input_cost_per_token": 1e-6, "output_cost_per_token": 2e-6 }
        }
      }'

merge: true keeps existing overrides; false replaces the whole map. Persisted to config.toml AND SQLite so it survives reload + restart.

Observability

Lightweight in-process metrics — zero external dependencies (no Prometheus client crate). Counters, fixed-bucket latency histograms, and gauges, all backed by atomics. Persisted to SQLite every 10s and rehydrated on startup.

# JSON snapshot (counts + P50/P95/P99 + active gauges)
curl -sS http://127.0.0.1:8080/v1/admin/metrics -H "Authorization: Bearer $TOKEN"

# Prometheus text format (compatible with any scraper)
curl -sS http://127.0.0.1:8080/v1/admin/metrics/prom -H "Authorization: Bearer $TOKEN"

# Built-in HTML dashboard with live refresh
open http://localhost:8080/dashboard

The dashboard is a single-file HTML page served from static/dashboard.html (Chart.js via CDN, no build step). It includes P50/P95/P99 latency cards, per-upstream RPS sparkline, sliding-window rates, live event SSE tail, and a dark/light theme toggle. The SQLite store backs the persisted metrics that survive restarts.

Metric set (Bifrost-parity, no Prom crate)

Counters are labeled by method, model, upstream_id, reason (errors). Histograms have 14 log-scale buckets from 1 ms to 30 s — the P50/P95/P99 quantile is computed on demand from the cumulative bucket counts.

Sliding-window rates, traces, live events, reset, audit

# 1m/5m/1h RPS, TPS, cost-per-second
curl -sS http://127.0.0.1:8080/v1/admin/rates -H "Authorization: Bearer $TOKEN"

# OTLP-shaped recent spans
curl -sS 'http://127.0.0.1:8080/v1/admin/traces/recent?limit=20' -H "Authorization: Bearer $TOKEN"

# Server-Sent Events of every completed request (dashboard live-tail)
curl -N  http://127.0.0.1:8080/v1/admin/events/stream -H "Authorization: Bearer $TOKEN"

# Operational reset — zeros in-memory state; long-term tables untouched
curl -X POST http://127.0.0.1:8080/v1/admin/metrics/reset -H "Authorization: Bearer $TOKEN"

# Audit log (key ops, config reload, metrics reset, …)
curl -sS 'http://127.0.0.1:8080/v1/admin/audit?limit=50' -H "Authorization: Bearer $TOKEN"

Docs

• docs/ARCHITECTURE.md — internals, storage, observability layer
• docs/FEATURES.md — full feature catalog
• docs/FUTURE_IMPROVEMENTS.md — working list of planned enhancements